I absolutely love the quantecon project and it has inspired me to use jupyter-notebooks + sphinx to try to build a couple courses (albeit on a far more modest scale) on applied micro theory and trade/development topics for a couple courses aimed at undergrad and MA students.
Could you help me understand how you are using Sphinx to generate your site? I understand the basics of how Sphinx is used to generate static sites and I clearly see how, say, the notebook gallery was generated from the notebooks.rst file in the github repo but I’m failing to find the similar restructured text files on the repo(s) to build the main textbook/course site page http://quant-econ.net/py/index.html (or the PDF). I see some of the content files (e.g. solution notebooks) on the repo but I don’t see the main text which must have involved a fair bit of latex. You appear to be using Sphinx to tie together content from ipython/ijulia notebook, latex, py, and perhaps other formats but I’m failing to see how all the pieces are tied together? Am I failing to see files in the repo, or is some of the content not there?
Could you explain briefly how you generate the main textbook/course site? Are you using any special scripts or extensions? I’m not looking for a lengthy explanation just a few pointers to help save me (and perhaps others) some time in trying to recreate something similar.
You are providing a huge public service to the Economics profession with this material. It’s not just that you are making a lot of really interesting economic material much easier to understand and access, you’re also introducing new tools: python, julia, git, and Sphinx which are way underused in our field. This is fun. Thanks!
Jonathan
Hi Jonathan,
Many thanks for your kind words. It’s great to hear back from people who are as excited as we are about working with all these great tools. As you say, they are way under-utilized.
In terms of the site, it’s true that we don’t make the lecture source files and all the code that generates the site public. One reason is that we’d like to eventually publish a hard copy version of the site through somewhere like MIT or Princeton UP. (I think that will help with adoption in courses.) Publishing would be a bit more problematic if all of the source was open.
The lecture site started as a very basic static site, built, as you say, with Sphinx. The theme we used is called f6:
https://github.com/vimalkvn/sphinx-themes/tree/master/f6
We’re very lucky to be supported by the Sloan Foundation and we have great postdocs and research assistants (particularly Matt McKay, Chase Coleman and Spencer Lyon), as well as some input from a web developer (my brother Andrij Stachurski). That’s allowed us to get a nice looking front page, the two versions of the sites, nicely compiled PDFs and so on. (The two versions of the site also involve some Jinja templating.)
But even without all those add-ons, you can go a long way with just a straight up Sphinx site and a nice Sphinx theme. That will already produce a static site and lecture pages that look similar to ours, including highlighted code, latex for equations, etc. If you know a web developer they can probably customize the theme to fit your tastes (colors, margins, etc) very quickly by tweaking the CSS file that comes with the theme. Hopefully that alone will get you to a site that’s suitable for your course and that you’re happy with the look of.
Please feel free to ask further questions.
All the best, John.
Hi Jon,
Thanks for your explanation. It’s helpful to understand your publishing plan (which makes sense!) and how that relates to the repository and documents structure.
With some of your tips and after much experimentation I think I’ve stumbled upon a nice set of tools and relatively efficient workflow for build a website similar to what you’ve created, all built from jupyter notebook content and without the need for a web developer. I went through my fair share of frustrating experiences before stumbling upon the right tools. In case it’s useful to anybody else wanting to do something similar here is what works for me:
- Post your notebooks to a github repository (lets call it XYZ).
- Create a readthedocs account.
- Get the nbsphinx sphinx extension (this should also install sphinx if you don’t have it already).
- Follow/adapt nbsphinx’s usage instructions to your needs, connecting readthedocs to your repo, setting up a github webhook, and adding a requirements.txt file to readthedocs.
- Create an index.rst file to give structure to your site, listing the names of any notebooks you want in the toctree.
If your setup went well you are now practically done. Push your index.rst file and any jupyter notebooks it refers to to your XYZ repo on github, wait a minute or two (longest on first build) and now browse to XYZ.readthedocs.org and you should see a beautifully Sphinx rendered site. Readthedocs is a free hosting service that runs a virtual linux machine that in turn runs sphinx and nbsphinx on the files in your github repository automatically every time you push new files, converting your notebooks on the fly and even executing them if it finds non-executed cells. The nbsphinx developer, Matthias Geier is super-responsive to any issues you might have. It’s really amazing that it works so well and is free.
I do have one last question for you. You have a lot of math in your site/document. I’m assuming you wrote this up first as Latex and then to restructured text (the hidden files). Did you integrate such documents into the Sphinx workflow using pandoc or similar latex --> rst converter? It’s the missing piece for understanding how all the pieces come together.
Thanks again for your help and for creating this site which I hope inspires others to also create new economics jupyter notebook content and sites/documents of this sort. With tools like these improving all the time the barriers to doing so keep falling.
best Jonathan
Hi Jonathan
that’s a very interesting work flow. I had no idea that readthedocs could generate and publish the documents via nbsphinx.
We are closer to a more traditional Sphinx model. Our process is this:
- Write in rst (restructured text)
- Run the files through the Jinja templating engine to produce a Julia and Python version of the rst files
- Run Sphinx over each one to produce a static html version of the site and a PDF version (both Python and Julia)
- Do some postprocessing to tidy up the PDF
- Copy the compiled files (static version of site and PDFs) up to an S3 bucket at AWS, which acts as our web server
So to answer your question more directly, we write the LaTeX directly into the rst files.
We like directly working with rst files rather than notebooks because of features like cross-referencing that restructured text provides.
Anyway, thanks for sharing your work flow. It looks very neat.
Regards, John.