Hi,
I was looking at your site to get some inspiration for creating new course material for my mechanics courses. I’m wondering how exactly the lectures site work? The individual pages look a lot like Jupyter notebooks, but from what I understand they are actually generated from rst files. Is there a way to directly run the course lectures as a notebook, and why is the lecture site built this way rather than just writing notebooks directly?
Cheers,
Bart
Hi Bart,
Thanks for your questions.
First up, we maintain these as rst (restructured text) source files because maintaining a large set of notebooks is problematic. Think of LaTeX vs Microsoft Word. Do you want to maintain a large Word document or a large LaTeX document? Of course the latter, and the main reason is that everything is text files expressing what you want to do. Same story with rst files.
It’s even more crucial to work with raw text files here because it’s a large distributed project with many RAs. Everything runs through version control (GitHub) and pull requests, as happens with distributed software projects. For this text files are far better.
In our case, the rst files are compiled into a website and PDFs using Sphinx. Because we have Python and Julia versions, we have an extra layer, where we use Jinja templating to produce two Sphinx projects from our source files, one for each language. Those who don’t need to support two languages can ignore this step.
(Credits: @mamckay and @spencer.lyon built most of the back end of our site. Recently @mamckay has put in a lot of hours refactoring it. The front end was put together by @drdrij )
All that said, we love Jupyter notebooks and we’re currently working on a tool that can convert all the lectures (in fact any rst file) into notebooks. When that’s done we’ll provide notebook versions of every lecture. We hope to have that feature within a month. Happy to provide more info if you need it.
Hope that helps!
OK, thanks for the clarification, I’ll play around a bit with Sphinx, it looks very nice.
A quick follow-up here. I also have visited quantecon lectures many times for ideas and inspiration in creating my own econ lectures using jupyter, sphinx and github as tools. It seems that the quantecon repo with the .rst files for the lectures appears to be private – for good reasons I’m sure – maybe a book project requires it be so – but that makes it’s hard to reverse engineer things to figure out how all the pieces come together. Can you reveal a bit more about your workflow. Maybe show us the pieces for an example lecture?
Take the Linear Algebra lecture as an example. Did you just write the math straight into a restructured text file? Did you in parallel create a jupyter notebook with the examples and plots and then you pasted relevant code pieces into the .rst file? Di you then use a sphinx directive to run thins with a python interpreter to add the output, or did you just paste in the plots and other output as well yourselves? If so, at the end of the day do you have two files (.rst and .ipynb) associated with that particular lecture? Or are there also .png and other graphic files in the folder. Is there one folder per lecture?
Sorry to press for such details but I keep struggling to find the right workflow for building content . It’s easy to to learn best practice python or Julia using web resources but very hard to learn best practices on things like this.
As I’ve mentioned on this forum before I’ve created some lecture content using jupyter notebooks and nbsphinx to convert notebooks (via .rst) to html content then hosted on readthedocs (all semi-automated via github webhook). Using only jupyter to create content has many advantages (just one file with text, code and output together) but also drawbacks: (a) hard to keep notebooks under version control and (b) too much code embedded distracts from the text/lecture.
So I’m looking for something like your method. So I would love to know more details or see just one example.
Thanks!
Jonathan
Hey Jonathan,
Sorry I missed this! Discourse sometimes notifies me of new posts and sometimes decides not to. If I’d seen it I would have got back to you quicker.
Yeah, it’s hard to get the right work flow! Text (rst) files are much easier to work with when you have a big batch of them. But notebooks are executable, so you need them to test your code.
For that reason, we’ve been working on a new tool, which we’ve called Jupinx, that takes a Sphinx project with a bunch of rst files and builds a set of corresponding notebooks. It does it in an intelligent way, so that only rst files that have been edited since the last build are recreated. As a result it’s pretty fast.
The work flow is then to work with the rst files, re-build notebooks, test code, work more with rst files, repeat.
It’s kind of like the edit tex file -> latex builder -> view pdf -> edit tex file workflow.
We’re about to release Jupinx as an open source project. @mamckay Would you mind to update Jonathan on the status of the project? Jonathan, we’d appreciate any feedback if you do give it a go.
Hi @jonathan.conning.
It looks like we will migrate the Jupyter Sphinx extension to be an installable open source project next week.
We are currently working on documenting the code, setting up test cases etc.
Once the extension has been published I will make a note to notify you through this channel.
You can follow progress of the extension at this newly created repository:
We are now just waiting to hear back from the sphinx project before releasing 0.1 on PyPI.
This looks very useful and I see it has now also been nicely described in this post: Introducing Jupinx post. Thank you for open sourcing this as a Sphinx extension.
The option to include or suppress output in the .. code-block:: python via the :class: output or :class: no-execute' options is nice.
It may take awhile to adapt my own workflow to this (which flowed the other way around from jupyter to RST/sphinx via nbsphinx) but your method offers much better control over final output and version control. I will give it a try.
Quick updated (CC @jonathan.conning) is that we have now shifted all lectures to Jupyter Book. @mamckay is now one of the lead developers on JB and can provide more information.
Yes, thanks @john.stachurski, @mamckay and all others who helped to conceptualize, build, and extend jupyterbook, a great open source project built upon sphinx and other great tools. I have been watching it evolve and already love how easy it has made the task of combining content from markdown files, jupyter notebooks and other resources to build fully functional and beautiful sites and books. It really takes ‘document as you build’ concept to a new level.
I look forward to seeing how the project continues to evolve, and to support and sing its praises however I can.