Python is one of the most used programming languages in the world, and the Python Package Index contains nearly half a million projects with over 14 terrabytes of published code.
Exploring the contents of PyPi is currently impractical due to the total size, API, and complexity of Python packages. This means it is harder than it should be to:
Detect malicious packages
Quantitatively understand how the Python ecosystem is evolving
Detect credentials that have been accidentally published.
I will develop a system that will make it possible to explore the total set of all Python code published on PyPi with consumer-grade hardware + internet in a matter of minutes rather than weeks. This will be achieved by leveraging Git Packfiles to massively compress huge quantities of PyPi code into manageable chunks (~14tb to ~100gb), distributing them via Github or another service and then providing tooling to explore/analyse the contained code. I will leverage third party secret scanning/detection services to scan the contents for leaked credentials, and notify the owners automatically.
This won't change the world, cure malaria or save a life. But it will have a positive impact on the security of the internet (specifically companies that publish code to PyPi) and hopefully on the evolution of one of the worlds most popular programming language.
I've built a system that trawls PyPi releases for AWS keys in real time, and I've been working on a proof of concept for this project. It's more than possible and I've already found a worrying amount of credentials leaked on PyPi.
I need to rent some beefy server hardware to bootstrap the project and create the initial packfiles, and I need some compensation for the time it will take to work on this project.
Tom Forbes
almost 2 years ago
Python code is mirrored to GitHub here: https://github.com/pypi-data
Auto updates website is here: https://py-code.org/
secrets are being reported to the PSF.
Domain name: 10$
GitHub subscription for increased CI capacity: 50$
Server costs for initial development: 300$
Tom Forbes
almost 2 years ago
The project is now complete! All PyPI code is mirrored to GitHub, and new uploads are automatically added as well.
Tom Forbes
over 2 years ago
Update: This project was pretty delayed due to unrelated personal issues, but I completed the first stage last week: https://github.com/pypi-data
All the Python code up until the 8th of July is within this organisation. It's not currently live updating but the infrastructure is all there.
I'm working with https://www.gitguardian.com/ to scan the contents for credentials, then produce some analysis on the contents, documentation and finally publicity.
Thank you for your funding Austin! This helped pay for some infrastructure to set up the project and begin scoping out a way to do this as well as give me motivation to continue.