10 years of PyPy
From a software engineering perspective, 10 years is indistinguishable from infinity, so I don't care what happens 10 years from now -- as long as you don't blame me. :-)
- Guido van Rossum, Python creator.10 years is indeed a long time. PyPy was created approximately 10 years ago, with the exact date being lost in the annals of the version control system. We've come a long way during those 10 years, from a "minimal Python" that was supposed to serve mostly as an educational tool, through to a vehicle for academic research to a high performance VM for Python and beyond.
Some facts from the PyPy timeline:
- In 2007, at the end of the EU funding period, we promised the JIT was just around the corner. It turned out we misjudged it pretty badly -- the first usable PyPy was released in 2010.
- At some point we decided to have a JavaScript backend so one could compile RPython programs to JavaScript and run them in a browser. Turned out it was a horrible idea.
- Another option we tried was using RPython to write CPython C extensions. Again, it turned out RPython is a bad language and instead we made a fast JIT, so you don't have to write C extensions.
- We made N attempts to use LLVM. Seriously, N is 4 or 5. But we haven't fully given up yet :-) They all run into issues one way or another.
- We were huge fans of ctypes at the beginning. Up to the point where we tried to make a restricted subset with static types, called rctypes for RPython. Turned out to be horrible. Twice.
- We were very hopeful about creating a JIT generator from the beginning. But the first one failed miserably, generating too much assembler. The second failed too. The third first burned down and then failed. However, we managed to release a working JIT in 2010, against all odds.
- Martijn Faassen used to ask us "how fast is PyPy" so we decided to name an option enabling all optimizations "--faassen". Then "--no-faassen" was naturally added too. Later we decided to grow up and renamed it to "-O2", and now "-Ojit".
- The first time the Python interpreter successfully compiled to C, it segfaulted because the code generator used signed chars instead of unsigned chars...
- To make it more likely to be accepted, the proposal for the EU project contained basically every feature under the sun a language could have. This proved to be annoying, because we had to actually implement all that stuff. Then we had to do a cleanup sprint where we deleted 30% of codebase and 70% of features.
- At one sprint someone proposed a new software development methodology: 'Terminology-Driven Programming' means to pick a fancy name, then discuss what it could mean, then implement it. Examples: timeshifter, rainbow interpreter, meta-space bubble, hint annotations (all but one of these really existed).
- There is a conspiracy theory that the reason why translation is so slow is because time is stored away during it, which is later retrieved when an actual program runs to make them appear faster
Overall, it was a really long road. However, 10 years later we are in good shape. A quick look on the immediate future: we are approaching PyPy 2.0 with stackless+JIT and cffi support, the support for Python 3 is taking shape, non-standard extensions like STM are slowly getting ready (more soon), and there are several non-Python interpreters around the corner (Hippy, Topaz and more).
Cheers,
fijal, arigo, hodgestar, cfbolz and the entire pypy team.
Comments
My best wishes to whole PyPy team! And thanks for all the hard work!
You guys rock!
Best blog posting - ever! Heres to another 10 pypy years and N llvm endeavours. -- rxe
You've made a great work so far, please continue with it!!
Only those who dare to fail greatly can ever achieve greatly. --RFK
Congrats, guys!
Congratulations and thank you for the great work, looking forward to the next 10 years!
Great work!
How will PyPy impact Python future and it's adoption as preferred language?
indeed: congratulations and much respect for the perseverance and hard work you have put into this project over the years!
First, congratulations for keeping at it for 10 years! PyPy is one of the most interesting project I know of.
This blog post is also very interesting but by reading it I can't help but think: are all those "failures" documented somewhere in one place? It could be a very interesting read.
Or more specifically:
* Why was the JavaScript backend a horrible idea?
* Why is RPython a bad language (for writing CPython extensions)?
* What went wrong in the different attempts at using LLVM?
* What were those "70% of features" that were dropped after the EU project?
Congratulations! Here's to another 10 years!
And the JavaScript backend was a great idea - bring it back! It's certainly better than the other Python-to-JS translators out there, at least in terms of actually parsing some Python. I want Python in my browser!
I was and always will be impressed by PyPy. And the self-critic of this post only furthers it. You are cool people, looking forward to meet you again.
I remember 10 years ago, when I decided to learn to program... I didn't know what language to choose, and someone suggested python. It was someone I approached through a mailing list, and he was passionate explaining why python is so special.
I remember reading about it being cool but with a "performance problem". However, there were some nerds out there talking about a minimal python, that would eventually become a fast python, so I said "cool, perhaps in a few months there will be a fast python...".
I spent ten years following silently this story, and I'm happy to say "Happy birthday Pypy!".
I've never met any of you, but I feel I know you.
You showed me the value of perseverance, that every failure is one step closer to success.
Congratulations and a big THANK YOU!
Luis Gonzalez, from Buenos Aires.
PyPy is my favorite open-source project. Best of wishes for the future development.
May you find all the funding you need, become the leading STM Implementation and become the defacto Python standard.
+1 on Gaëtan de Menten's comment.
One more +1 on Gaëtan de Menten's comment. :)
You are incredible people and you do such cool stuff! Best of luck to you and keep up the great work!
Thank you for the great post - and thank you for sticking to it and finding ways to get time to make it work - including to add everything under the sun into that EU project to be able to go full-time!
You’re a great example how to really do stuff right - by actually doing it and keeping at it through every stumbling block on the way.
Happy birthday - and thank you for pypy!
+1 on Gaëtan de Menten's comment.
I'd also like to see the failures documented. Trying and failing is a great way to learn - but even better is to learn from other's failures.
Great work guys! Happy birthday PyPy!
Thanks for making fast Python possible and creating a masterpiece in process!
About Terminology-Driven Programming: let me guess, the only nonexistent thing is a timeshifter? Three other names make a lot of sense in context of PyPy.
Электроник: no :-) Try again.