Warehouse Overview

Orientation

One codebase, six process types

“Warehouse” and “PyPI” name the same thing at two layers. PyPI is the service the world talks to. Warehouse is the Python code that answers. There is one repository, one WSGI application object, and one database. Everything that looks like a separate service is the same application wearing a different hostname or a different gunicorn config.

The Procfile declares the whole production fleet in seven lines:

release:      bin/release
web:          gunicorn -c gunicorn-prod.conf.py    warehouse.wsgi:application
web-api:      gunicorn -c gunicorn-prod.conf.py    warehouse.wsgi:application
web-uploads:  gunicorn -c gunicorn-uploads.conf.py warehouse.wsgi:application
worker:       celery -A warehouse worker -Q default --max-tasks-per-child 1024
worker-beat:  celery -A warehouse beat -S redbeat.RedBeatScheduler
worker-traced: celery worker, same thing, with tracing and a smaller child budget

Read the first four lines again. web and web-api are byte-identical commands. They exist as separate process types so the deployment can scale browser traffic and API traffic apart, and so a flood of one does not starve the other. Nothing in the Python knows which one it is running as.

web-uploads is the only web process with its own config, and the diff is short and revealing:

Each number is a scar. The four-minute timeout is there because a single upload runs validation, hashing, a YARA scan of every archive member, and an object-storage PUT inside the request. The whole gauntlet is in No. 05, the PEP 694 guide. max_requests = 32 recycles an upload worker after 32 requests instead of 2048, because uploads leak memory faster than they leak anything else. And the quadrupled header limit exists for one reason: a PyPI API token is a macaroon, and macaroons with several caveats attached do not fit in gunicorn's default 8 KiB header field.

The admin console is not a fourth web process. It is /admin/ on the ordinary Warehouse domain, registered by warehouse/admin/routes.py against the same warehouse.domain setting as the public site, and guarded by an ACL on the root factory rather than by a network boundary. There is no admin deploy to fail over to.

The read path

Cache first, origin second

Warehouse's own docs are blunt about the traffic mix. Of the four classes of user it names (anonymous visitors, project maintainers, moderators, administrators) the first accounts for “the majority of browser traffic and all API download traffic,” and the last numbers “fewer than ten.” Almost everything is a read by a stranger. The conclusion the docs draw from that is one sentence long:

“Since reads are much more common than writes (much more goes out than goes in), we try to cache as much as possible.”

warehouse → docs/dev/application.md

So Fastly sits in front of the origin, and the origin's job is partly to tell Fastly when it is wrong. Warehouse registers cache keys per model class; a SQLAlchemy after_flush listener collects purge keys for every new, dirty, or deleted object, and an after_commit listener sends the purges. Edit a project, and the purge goes out when the transaction commits, not when the view returns.

Package files never come from Warehouse at all. Responses point clients at files.pythonhosted.org, which Fastly serves from a Backblaze B2 bucket, falling back to S3 when B2 is down or missing the file. The reason is money: B2 is primary “for cost savings over S3 for egress, as Backblaze has an agreement with Fastly to waive egress fees.” Nothing about that choice is visible in the application code. It is a storage backend behind an interface, which is No. 03's subject, not this guide's.

Writes go the other way. Uploads bypass Fastly entirely and hit the origin ingress directly, because there is nothing to cache in a 100 MiB POST and a CDN in the path only adds buffering and its own size limits.

Anatomy

What happens between the socket and the view

Follow one request: GET https://pypi.org/project/requests/2.32.3/, a cache miss.

1. 01

   Fastly misses, and asks the origin

   pypi/infra → terraform/warehouse · gunicorn-prod.conf.py

   The edge forwards to the Cabotage ingress, which hands the request to gunicorn over a Unix domain socket at /var/run/cabotage/cabotage.sock. There is no TCP port to point at, which is worth knowing the first time you try to reproduce a production networking problem. preload_app = True means the application was built once before forking, so every worker shares one already-configured Pyramid registry.

   Cabotage is not a PyPI-specific tool, and it is not PSF-exclusive either: it's an open-source, MIT-licensed platform-as-a-service, and anyone can run it or try the public preview at preview.cabotage.io. Its README calls it “deployment tooling built on top of Kubernetes, Hashicorp Consul, Hashicorp Vault, Docker, and Python to power automated rollout and management of software and configuration changes,” providing end-to-end verified TLS, secure secret management, and continuous delivery, while its tagline settles for the “We're doing our best” Platform-as-a-Service. The PSF deploys its own instance to run Warehouse, camo, conveyor, and everything else PSF infrastructure puts through it, which is as close to a production reference as the project has. The socket above is the convention it imposes: every app under Cabotage exposes a Unix domain socket at a well-known path instead of a TCP port, and Cabotage's ingress is what terminates TLS and proxies into that socket.

2. 02

   ProxyFixer decides whether to believe the headers

   warehouse/utils/wsgi.py → ProxyFixer.__call__

   This runs as WSGI middleware, outside Pyramid, and it is the security boundary for every piece of client identity in the system. It reads Warehouse-Token and compares it against the configured WAREHOUSE_TOKEN with hmac.compare_digest. On a match it trusts the sibling Warehouse-* headers: protocol, client IP, pre-hashed IP, host, and eight GeoIP fields the edge resolved. On a miss it falls back to X-Forwarded-*, counting in from the right by WAREHOUSE_NUM_PROXIES so a client-supplied prefix cannot win, and hashes the IP itself with WAREHOUSE_IP_SALT.

   Then it deletes every one of those headers from the environ. An application-level view cannot see them, cannot accidentally trust them, and cannot leak them. A token that is present but wrong is not silently ignored either. It raises a Sentry warning called Invalid Proxy Token, because that means either a misconfiguration or somebody guessing.

3. 03

   VhmRootRemover deletes one header, for one reason

   warehouse/utils/wsgi.py → VhmRootRemover

   Four lines that drop X-Vhm-Root. It exists because that header changes how a URL generator writes absolute links, so a client that could set it could poison the cache with URLs pointing somewhere else. It is an ugly piece of code, and removing it would be a security bug.

4. 04

   The tween stack, with sanity bolted on either end

   warehouse/config.py · warehouse/sanity.py · warehouse/cache/http.py

   Seven modules add tweens: structlog request context, require_https (403 on any :action URL over plain HTTP, with the Fastly error code attached), gzip compression, conditional-HTTP handling for ETag and If-Modified-Since, rate-limit response headers, Content-Security-Policy, and Referrer-Policy. pyramid_tm wraps the request in a transaction (skipped for /static/ and the debug toolbar), and pyramid_retry will replay the whole request up to three times if the transaction hits a retryable conflict.

   Around all of that sits warehouse/sanity.py, which does not add a tween so much as rebuild the chain: it takes the ingress tween, then re-adds every implicit tween in order, then the egress tween. That guarantees it is genuinely first in and last out. The ingress half turns undecodable query strings and unparseable form bodies into 400s instead of 500s; the egress half percent-encodes non-ASCII redirect targets. The module comment is the honest one: “I'm doing bad things, I'm sorry. - dstufft”. It is included last in configure() for exactly this reason.

5. 05

   The router matches a route, and a domain

   warehouse/routes.py · warehouse/predicates.py → DomainPredicate

   Warehouse registers around 130 routes, and nearly every public one carries domain=warehouse. The predicate is four lines: compare request.domain against the configured value with Pyramid's is_same_domain, and, in the branch that matters, match everything if the setting is None. That is how the whole site, the uploader, and the auth endpoint can share one process in local development and on Test PyPI, and how they separate cleanly in production where all three settings are filled in.

6. 06

   A factory turns the URL into a model object

   warehouse/routes.py · warehouse/packaging/models.py → ProjectFactory, Project.__getitem__

   The route is /project/{name}/{version}/ with factory=ProjectFactory and traverse="/{name}/{version}". Pyramid matches the pattern by dispatch, then traverses the matched segments: ProjectFactory["requests"] queries for a project whose normalized_name equals normalize_pep426_name('requests'), and the resulting Project["2.32.3"] canonicalizes the version and looks up the release. A miss raises KeyError, which Pyramid renders as a 404. The view never sees a string; it receives a Release that already carries its own __acl__.

   This hybrid of dispatch and traversal is the pattern that surprises people arriving from Django or Flask, and it is the subject of No. 02. Here, one sentence is enough: the URL is resolved to an object before authorization runs, and permissions live on the object.

7. 07

   The view runs, and the response unwinds

   warehouse/packaging/views.py · warehouse/cache/origin/__init__.py

   The view renders a Jinja2 template. Any task it queued is still sitting on the transaction manager's after-commit hook. Any object it modified has already contributed purge keys to the session. pyramid_tm commits; the purges fire; the tasks are sent; the egress tween tidies the response; gzip and CSP headers go on; Fastly stores the result and every subsequent visitor gets it from the edge.

The data model

Project, Release, File, and the scars

Three tables carry the packaging world. A Project is a name. A Release is a version of that name. A File is one artifact of that version: an sdist, a wheel. Everything else in warehouse/packaging/models.py hangs off those three.

They are also older than Warehouse. The docs say so plainly:

“Since Warehouse was built on top of an existing database (for legacy PyPI) and developers had to fit our ORM to the existing tables, some of the code in the ORM may not look like code from the SQLAlchemy documentation. There are some places where joins are done using name-based logic instead of a foreign key.”

warehouse → docs/dev/application.md

That warning is not decorative. Here is where it bites.

normalized_name is written by Postgres, not by Python

Project.normalized_name is a real, unique column, but the model declares it with server_default=FetchedValue() and server_onupdate=FetchedValue(): SQLAlchemy is being told somebody else fills this in, go read it back. That somebody is a trigger. Migration 8bee9c119e41 (June 2022) added the column, backfilled it, dropped the old functional unique index, and installed projects_update_normalized_name, a BEFORE INSERT OR UPDATE OF name trigger calling normalize_pep426_name(NEW.name). The model file carries a matching comment: TODO: Cannot update columns that are used in triggers.

The practical consequence: you cannot normalize a name in Python and trust it to match. Traversal itself calls the SQL function, Project.normalized_name == func.normalize_pep426_name(project), so that lookups use exactly the same normalization the trigger used at write time.

Uniqueness on File is aggressive, and then apologetic

File declares unique=True on five columns at once: filename, path, md5_digest, sha256_digest, and blake2_256_digest. The last two are CITEXT with check constraints matching ^[A-F0-9]{64}$ case-insensitively. A filename is globally unique across all of PyPI forever, which is why a deleted release does not free its filenames.

Then there is the partial index release_files_single_sdist: unique on (release_id, packagetype), but only WHERE packagetype = 'sdist' AND allow_multiple_sdist = false. One sdist per release, unless the row is one of the pre-existing double sdists, which get an escape hatch column. The comment is candid: “Eventually we should try to get rid of all of them and then remove this column.” That is 2018 data still shaping the 2026 schema.

The journal is a serial number the database cannot supply

JournalEntry records every mutating action, and downstream consumers (mirrors, the XML-RPC changelog, last_serial) depend on its integer id being monotonic in commit order. Postgres SERIAL does not promise that: it hands out the next integer when the INSERT runs, not when the transaction commits, so two overlapping transactions can commit out of ID order. Not even SERIALIZABLE fixes it.

So Warehouse serializes journal writes itself. A before_flush listener takes a Postgres advisory transaction lock keyed on the journals table's OID before any journal insert. And there is a second layer of care underneath: if the same flush also contains non-journal changes, holding the advisory lock while those rows do their own constraint checks can deadlock, so the listener expunges the journal entries, stashes them in session.info, and an after_flush listener adds them back for a flush of their own. Two listeners and a stash, to buy one guarantee no SQL type could give.

Small things worth knowing before you read further

Six more details in the same file, none of them big enough for a heading, all of them capable of costing you an afternoon.

• JournalEntry.submitted_by is a CITEXT foreign key to users.username, the string rather than the UUID, with ON UPDATE CASCADE so a rename propagates. That is the name-based join the docs warned about, in the flesh.
• Classifiers are a foreign key, not free text. warehouse/packaging/models.py points at ForeignKey("trove_classifiers.id"), and that table is populated from pypa/trove-classifiers, pinned in requirements/main.txt at trove-classifiers==2026.6.1.19. Note the org: the canonical list of classifiers lives under pypa, the Python Packaging Authority, not under pypi. Warehouse validates every project's classifiers against a vocabulary it does not own.
• LifecycleStatus has four values: quarantine-enter, quarantine-exit, archived, archived-noindex, and it applies to both Project and Release. Do not confuse it with ProjectStatusMarker (active, archived, quarantined, deprecated), which is the public PEP 792 vocabulary. One is internal state; the other is what the API says.
• Size limits are a three-way maximum, not a single number. Defaults live in warehouse/constants.py: 100 MiB per file, 10 GiB per project. A project or its organization can carry a larger upload_limit, and upload_limit_size picks the largest applicable value. But a check constraint, projects_upload_limit_max_value, caps any per-project override at exactly 1 GiB in the database. Nobody can raise it past that through the admin UI, because the table refuses.
• Release.version is unique per project, and canonical_version exists alongside it because two different version strings can canonicalize the same. Project.__getitem__ looks up by canonical version, catches MultipleResultsFound, and retries on the exact string. Another pre-uniqueness-check inheritance.
• File.requires_python has a @validates hook whose entire body raises RuntimeError("Cannot set File.requires_python"). The value is derived; assigning to it is a bug, so the model makes it impossible rather than surprising.

The map of the source

Forty-odd subpackages, two of them called “legacy”

warehouse/ is flat and wide. Most directories are a self-contained Pyramid plugin with an includeme(), and config.py pulls in 54 of them by name. The naming is mostly honest; the exceptions are worth learning before they cost you an hour.

Configuration

Environment variables, all the way down

Warehouse has no configuration file. configure() in warehouse/config.py builds a settings dict from os.environ, hands it to a Pyramid Configurator, includes 54 subpackages, adds tweens, scans for views and tasks, and commits. Just under a thousand lines, read top to bottom, and it is the best single file to read first.

Three helpers do nearly all the work:

maybe_set(settings, name, envvar, coercer, default)

Copy one environment variable into one setting, optionally coerced, optionally defaulted. Uses setdefault, so a value passed into configure() (from a test, say) always wins over the environment.

maybe_set_compound(settings, base, name, envvar)

The interesting one. It shlex.splits the variable: the first token becomes base.name, and every remaining key=value becomes base.key. So FILES_BACKEND="warehouse.packaging.services.LocalFileStorage path=/var/opt/warehouse/packages/ url=http://files:9001/packages/{path}" selects a class and configures it, in one string. Fourteen subsystems are wired this way: file storage, archive storage, the simple index, docs, mail, metrics, billing, the breach-check services, forge OAuth, the origin cache.

maybe_set_redis(settings, name, envvar, db=N)

Takes one REDIS_URL and rewrites its path to pick a numbered database, so eight subsystems share one Redis without sharing a keyspace.

The docstring on that last one states the convention: general caches in databases 1–9, Celery in 10–15, nothing above 15. In practice sessions take 2, rate limiting 3, the XML-RPC cache 4, query results 5, the OIDC key cache 1, the Celery broker 10, and results 12. The scheduler takes 0, which is neither a cache nor in 10–15. Conventions drift; the code is the record.

That maybe_set_compound string reaches further than configuration. Every one of those backends is selected by dotted path at boot and registered as a service against a zope.interface, which is how local disk stands in for S3 and a console printer stands in for SES without a single if environment == "development" anywhere in the application. Naming the mechanism is enough here; No. 03 is entirely about it.

Two small things that confuse newcomers. SOURCE_COMMIT becomes the warehouse.commit setting, which is how a running instance reports the code it is on. And development.ini in the repository root is two lines long. It exists purely so Pyramid's ptweens and proutes scripts, which insist on a PasteDeploy file, can reach a factory that ignores the file and calls configure() anyway. If you want the real tween order rather than the one inferred from reading config.py, that file is how you ask.

Background work

Celery, and the commit that gates it

The workers are the same application again. celery -A warehouse boots the identical Pyramid configuration, and config.scan() picks up task functions the same way it picks up views. A @tasks.task() decorator attaches a venusian callback that registers the function on the Celery app during the scan. One queue, default. Redis as broker, result backend, and schedule store. Redbeat holds the schedule so that worker-beat can be restarted without losing time.

Two mechanics in warehouse/tasks.py matter more than the rest.

Tasks are sent after the commit, not at the call. WarehouseTask.apply_async looks for the current request, and if it finds one with a transaction manager it does not send anything. It registers an after-commit hook and returns None. The task goes to Redis only if the transaction succeeds. This closes the oldest race in web development: queueing work about a row, then rolling back the row. The price is stated in the docstring: you get None instead of an AsyncResult, so you cannot wait on the task, which inside a request you should not be doing anyway.

Tasks get a fake request. So much of the codebase assumes request exists that get_request() manufactures one with pyramid.scripting.prepare, gives it an explicit transaction manager, and sets remote_addr = "127.0.0.1". Every task body runs inside with request.tm, wrapped in metrics, and a pyramid_retry-retryable exception is turned into a Celery retry. If you have ever wondered why a task signature starts with self, request, that is where the request comes from, and why its IP is always localhost.

Twenty-three periodic tasks are registered across the codebase via config.add_periodic_task. They range from every minute (checking whether the file-cache sync is falling behind, notifying users of terms updates) to daily at a fixed hour (search reindex, expiring OIDC macaroons, SES cleanup). They are declared next to the code they serve, not in one schedule file, so the way to find them all is to grep.

The dev environment fakes the cloud with an HTTP server

docker-compose.yml brings up nineteen services. Some are the real thing at a smaller size: Postgres, Redis, OpenSearch, the Camo image proxy, the RSTUF pair. Some are the real application: web, worker, tests, the static asset watcher, three documentation sites.

The rest are stand-ins, and the naming is not subtle. notdatadog is a Python script that listens on the statsd port and throws metrics away. notgithub impersonates GitHub's token-scanning callback. stripe-mock answers billing calls. mailpit catches every outgoing email and shows it in a browser at port 1080.

And object storage (S3, B2, the whole file-hosting stack) is a container running python -m http.server 9001 over a shared volume. The worker service sets FILES_BACKEND, ARCHIVE_FILES_BACKEND, and SIMPLE_BACKEND to the Local* storage classes pointed at that volume, and the application is none the wiser. That is the swappable-backend pattern paying for itself: the most expensive external dependency in production is a standard-library HTTP server on a laptop.

History

What Warehouse deliberately did not bring with it

The old PyPI codebase ran the index from the early 2000s until April 2018. Warehouse replaced it, and the replacement was not feature-complete on purpose. The developer docs keep the list of dropped features, and it reads as a statement of what an index is for:

• Hidden releases: gone.
• Documentation hosting on pythonhosted.com: no longer accepts uploads.
• Download counts in the API: replaced by a public BigQuery dataset, which is where No. 10, the Linehaul guide picks up the thread.
• GPG and SSH key management, and GPG signatures on releases: no UI, no index links, though an existing .asc is still fetchable by appending the suffix.
• Uploading a release through the web UI, and editing a description after the fact: use a tool, publish a new version.
• Uploading before verifying an email address.
• Plain HTTP for the APIs; OpenID and Google login.

Every one of those removals shifted work from the index to the client, or from a human to a token. Read the list and the modern design follows: files are immutable, metadata comes from the artifact, identity is an API token or a Trusted Publisher, and analytics live outside the request path.

The map

Where everything is

Everything this guide asserted, and the file it came from.

If you're going to work on this

In order, and the order matters more than any single item on the list.

1. Read config.py start to finish before anything else. It is the table of contents for the entire application, and every unfamiliar subpackage appears in it as one line you can go read next.
2. Bring the dev stack up and use ptweens and proutes through development.ini. The real tween order and the real route table beat any diagram, including the one above.
3. When a model surprises you, check the migration that created the column before you assume the ORM is the whole story. Triggers, partial indexes, and check constraints are doing real work that models.py only hints at.
4. Treat every “temporary” and “eventually” comment as a permanent feature with a plan attached. forklift/, allow_multiple_sdist, and the name-based journal foreign key have all outlived several roadmaps.
5. Then read No. 02 on the plugin architecture and hybrid dispatch, and No. 03 on swappable backends. This guide named both mechanisms and explained neither; that was the deal.