The PyPI Universe Field Guide

Orientation

Ten repositories, and everything that isn't

Ten of them are the actual index: the application, its satellites, the infrastructure underneath, and the libraries it depends on. This guide is a map of those ten.

Nothing warns you about this. The pypi organization's front page, which lives in the pypi/.github repository, links exactly one codebase by name:

“This organization hosts Warehouse, the codebase that powers PyPI, along with the supporting tools, infrastructure, and services that keep the index running for millions of developers worldwide.”

pypi/.github → profile/README.md

“Supporting tools, infrastructure, and services” is doing a lot of work in that sentence. This guide unpacks it into five buckets: core, satellite, library, infrastructure, and process.

This is entry 00 in a thirteen-part series. It is the map. Every later entry zooms into one square of it.

The map

Three rings and two libraries

Ten repositories live under github.com/pypi and see regular commits. They sort cleanly by what breaks when they break. The exception is the two libraries, which don't run anywhere and so can't break in that sense at all.

The rings are about blast radius, not importance. Break warehouse and uploads and installs stop. Break camo and some README images go missing. Break infra and you may not notice for a week, then everything breaks at once.

The core

warehouse, in one paragraph

pypi/warehouse is the application that serves pypi.org. Its README opens with the whole story: “Warehouse is the software that powers PyPI.” It is a Pyramid application on PostgreSQL, with Celery and celery-redbeat for background work, and it carries 9,008 commits on main. It renders the web site, serves the Simple API, accepts every upload, sends every email, runs the admin interface, and holds the Trusted Publishing implementation. If you only ever read one PyPI repository, read this one.

That is all this guide will say about it. Warehouse gets four entries of its own later in the series: what it is, how its roughly forty Pyramid “plugins” wire together through includeme, how it swaps storage and email backends through zope.interface, and how its API surface accumulated. Treat the box in Fig. 1 as a placeholder for four guides.

The satellites

Four small services with one job each

Each satellite runs on its own, deploys on its own, and can fail without taking pypi.org with it. Three deploy through Cabotage, an open-source, MIT-licensed platform-as-a-service built on Kubernetes, Consul, and Vault. It isn't PSF-exclusive tooling: anyone can run it, and there's a public preview at preview.cabotage.io to try it. The PSF deploys its own instance for PyPI's satellites, which is as good a production reference as the project has. The fourth satellite runs on Google Cloud instead.

camo: the image proxy

PyPI renders project READMEs, and READMEs contain images hosted anywhere. Serving those images directly would leak visitor IP addresses to arbitrary third parties and would break the padlock on any image served over plain HTTP. So warehouse rewrites every image URL to point at a proxy, and pypi/camo is that proxy.

The code is not PyPI's. go.mod declares module github.com/cactus/go-camo/v2: this is a fork of Cactus's go-camo, itself a Go rewrite of GitHub's original Ruby camo. Of 896 commits, most are dependency bumps and merges from upstream, but a handful genuinely change behaviour, including a Cabotage-specific Dockerfile, an Access-Control-Allow-Origin header, and a --max-size-redirect flag PyPI wrote and later upstreamed. The Camo field guide audits the full list. The Procfile shows the whole PyPI-specific deployment in one line. The binary listens on a Unix socket rather than a port, which is how Cabotage expects a web process to behave:

web: /bin/go-camo --socket-listen=/var/run/cabotage/cabotage.sock \
     --server-name=https://github.com/pypi/camo \
     --max-size=$GOCAMO_MAXSIZE --max-size-redirect=$GOCAMO_MAXSIZEREDIRECT \
     --header="Access-Control-Allow-Origin: $ALLOW_ORIGIN"

The last commit landed in July 2024. That is not neglect. A dependency-bump fork of a stable proxy is supposed to be quiet.

conveyor: the redirector

PyPI is old, and old URLs still get requested. pypi/conveyor exists to answer them. Its README is two sentences: it “translates the old URL scheme for packages into redirects and serves legacy user documentation,” and “canonical data for redirects is pulled from the JSON documents that PyPI serves.”

It is an aiohttp application, 69 commits, run under gunicorn with the aiohttp worker class, deployed by Cabotage for production and test. It reads two environment variables: CONVEYOR_ENDPOINT, the PyPI host it asks for JSON, and DOCS_BUCKET, the S3 bucket pypi-docs holding user documentation uploaded before PyPI stopped accepting it.

The interesting part is where conveyor sits. It has no end-user-facing hostname; its publicly resolvable Cabotage ingress is used as a Fastly backend, named Conveyor, wired into both the file-hosting and docs-hosting Terraform modules through a conveyor_address variable. A real client reaches it only when the edge decides to. You will meet it again in the infra section.

inspector: the file browser

pypi/inspector is a small Flask application at inspector.pypi.io. Its README is blunt about scope: it “provides a mechanism for security researchers to quickly share specific contents of a project hosted on PyPI when submitting security reports. Inspector is not currently supported for any other purpose.”

It unpacks a distribution and gives you a stable, linkable URL for one file inside it, so a report can say “look at this line” instead of “download this wheel and look around.” The routes follow the shape you would guess: /project/<name>/, then /project/<name>/<version>/, then paths into the archive.

Inspector is the one satellite with a full Terraform module of its own, including a Next-Gen WAF configuration at 100 percent enablement. It has no entry in the thirteen-topic series. It is real, it is small, and it is deliberately out of scope for the fundamentals.

linehaul-cloud-function: the telemetry pipeline

The only satellite that does not run on PSF infrastructure. main.py holds two Google Cloud Functions: process_fastly_log, triggered when Fastly drops a log file into a Cloud Storage bucket, and load_processed_files_into_bigquery, triggered by a Pub/Sub message on a schedule. Together they turn CDN log lines into rows in the public BigQuery download dataset. Cloud Build deploys them on merge to main.

The repository also ships a second, unrelated product: the linehaul user-agent parser library, which warehouse imports. That split is the first thing that confuses readers, and it is covered in full in the Linehaul field guide, already written.

The layer underneath

infra: three lines of README, everything else

pypi/infra has the shortest README in the organization, and it undersells the repository badly. Here it is, complete:

# PyPI Infrastructure

Primarily used for terraform configuration.

Under that sits terraform/, 234 commits, and every piece of PyPI that is not application code. There are eight modules. The top-level main.tf instantiates them thirteen times, because almost everything exists twice: once for production and once for TestPyPI.

Two details are worth carrying around. First, both zones share one Route 53 delegation set, so pypi.org and pythonhosted.org answer from the same nameservers. Second, only pypi.org is signed: the module creates a KMS key, an aws_route53_key_signing_key, and an aws_route53_hosted_zone_dnssec resource for the primary zone only. pythonhosted.org gets a CAA record and no DNSSEC. That asymmetry is deliberate. Signing the zone that serves files means every resolver failure becomes a download failure. It still surprises people who expect both zones to match.

The third detail: the repository does not hold values. Bucket names, memory limits, and credentials come from variables supplied at apply time. Reading the Terraform tells you the shape of production, not its contents.

Process and paper

Two repositories that ship no code

support: a queue with a README

pypi/support has 70 commits and 11,770 issues. That ratio is the whole point. The repository is not a codebase; it is the intake desk for anything that needs a human with administrator rights. Its README draws the line precisely:

“Any issue which relates to an individual user or project, or requires administrator or moderator intervention, should be opened here. … Any issue that affects how PyPI works for all users, or require code or configuration changes, should not be opened here.”

pypi/support → README.md

Two policies live here and nowhere else. The first is upload limits. The README states, in plain lists, what earns a limit increase and what does not: small pre-trained models and platform binaries generally yes, nightly releases and large machine-learning models generally no, and always after a best effort to shrink the release. Those lists are what a support volunteer quotes back to a requester.

The second is name retention. name-retention/README.md holds the PEP 541 runbook as a Mermaid flowchart: does the project have uploads, does it have functionality, is this a transfer or a replacement, is it name squatting. Each branch ends in a specific action and a specific canned notice. The document opens by admitting it is incomplete and instructs the reader to escalate whenever the process feels wrong, which is a better piece of policy design than most of what it replaces.

Support is topic 11 in this series, and the full guide walks the runbook branch by branch. It is where policy becomes the product.

.github: four commits of front door

pypi/.github holds two files that matter: profile/README.md, the org landing page quoted at the top of this guide, and FUNDING.yml, which points the sponsor button at github: [python] and the PSF donation page. Four commits, one of them titled “get rid of weird ai text.” It ships nothing. It is the first thing a new contributor reads, which makes it worth knowing exists.

The libraries

Two repositories that install, not deploy

Everything above either runs somewhere or governs something that does. These two just ship to PyPI as ordinary packages, the same as any project they support.

pypi-attestations: the PEP 740 glue

pypi/pypi-attestations converts between Sigstore bundles and PEP 740 attestation objects. Its own README calls it “an implementation detail within the reference implementation of PEP 740” and tells most users they shouldn't need to import it directly. Warehouse depends on it for Trusted Publishing's attestation handling; nothing deploys it, because there is nothing to deploy.

stdlib-list: the answer for older Pythons

pypi/stdlib-list ships static lists of every standard-library module name, for Python 2.6 through 3.14, for tools that need to answer “is this import a stdlib module” without launching the matching interpreter. Its README recommends against using it at all on 3.10+, where sys.stdlib_module_names now answers the same question built in. A library that documents its own obsolescence is still worth knowing exists, because plenty of packaging tools still target older Pythons.

The other org

pypi is not pypa

Two GitHub organizations sit next to each other with names four characters long and three characters shared. pypi is the subject of this guide: it runs the index. pypa, the Python Packaging Authority, is a separate organization with a separate job. It maintains the standards and the client tools that talk to the index. Different owners, different release cadence, and neither one is the other's parent.

The dependency runs in a direction people find surprising. Warehouse imports from pypa, not the reverse. requirements/main.txt pins trove-classifiers==2026.6.1.19, and warehouse/packaging/models.py carries a trove_classifiers foreign key, so every classifier on every project is validated against a list published by pypa/trove-classifiers. PyPI enforces that list. PyPI does not own it. pypa/packaging is the version-parsing and metadata library that most of this ecosystem imports, Warehouse included. And pypa/pip is the client that all ten repositories in Fig. 1 ultimately exist to serve, sitting in an org none of them control.

There's a corollary worth carrying forward. An origin of github.com/pypa/… is not a stray directory or a mislabelled clone. It is a real, load-bearing project, filed under a different authority, and a bug you find there gets reported somewhere else. Entry 12 of this series, pip Meets PyPI, covers the client side of that boundary in depth.

The map

Where everything is

The rest of the series

Thirteen entries, all written. In order:

1. 00 · The PyPI Universe. This guide.
2. 01 · Warehouse Overview.
3. 02 · Warehouse's Plugin Architecture.
4. 03 · Swappable Backends.
5. 04 · PyPI's API History.
6. 05 · Uploads, Then and Now.
7. 06 · Wheels & What's (Wheel)Next.
8. 07 · Infra.
9. 08 · Camo.
10. 09 · Conveyor.
11. 10 · Linehaul.
12. 11 · Support.
13. 12 · pip Meets PyPI.

Inspector and the wilder corners are not in it. They are real; they are not fundamentals.

If you're going to work on this

Four habits, learned the hard way.

1. Clone the ten active pypi/* repositories into one directory and nothing else into that directory. Keep experiments elsewhere. The confusion this guide describes is self-inflicted and avoidable.
2. When you change behaviour at a hostname, open the application repository and its Terraform module. Ask which layer owns the behaviour. Half the time it is the edge.
3. Read pypi/support's two READMEs early even though they contain no code. They explain more about how PyPI makes decisions than any module will.
4. Before filing a client-behaviour bug (how pip resolves, which wheel it picks) against pypi/warehouse, check whether it actually belongs in pypa/pip or pypa/packaging instead. Different org, different maintainers.