Warehouse Architecture Field Guide

Orientation

Fifty plugins, one function

Open warehouse/config.py and scroll to configure(). Past four hundred lines of pulling settings out of the environment, it builds a Configurator and then does this, fifty-four times:

config.include("pyramid_services")
config.include(".metrics")
config.include(".csrf")
…
config.include(".sanity")
config.commit()

That is the whole architecture. Pyramid's config.include(x) imports x, finds a function named includeme, and calls it with the configurator. Whatever that function registers, the application gets. Warehouse takes the convention seriously: grep the tree and you find 50 modules that define includeme. Forty-six of them are named directly in configure(); the other four hang off two of those (admin pulls in admin.routes, admin.flags and admin.bans; forklift pulls in forklift.action_routing). Eight more include calls reach third-party packages: pyramid_services, pyramid_jinja2, pyramid_tm, pyramid_retry, pyramid_mailer, pyramid_rpc.xmlrpc, pyramid_redirect, and the debug toolbar in development only.

So there is no registry to discover, no entry-point scan, no dependency resolver. The order of the plugins is the order of the lines. If you want to know why something is available on a request, you read one function top to bottom and stop at the line that put it there.

The cost of this design is that ordering is a real constraint held only by comments and by whoever wrote the line. The payoff is that the whole app is legible. Any subpackage's __init__.py tells you exactly what that subpackage contributes to the running process, and nothing contributes anything else.

Anatomy

Four plugins, dissected

Plugins vary from four lines to a hundred. Read these four and you have seen every move the rest make.

1. 01

   warehouse/db.py: an engine, a request attribute, a directive

   warehouse/db.py → includeme

   Twenty lines at the bottom of a two-hundred-line module. It adds an alembic_config directive so migration commands can build an Alembic config from live settings; creates the SQLAlchemy engine (pool_size=35, max_overflow=65, READ COMMITTED) and parks it in config.registry["sqlalchemy.engine"]; attaches request.db as a reified request method; and sets a JSON serializer on psycopg.

   The session factory is overridable from settings (warehouse.db_create_session) purely so tests can swap it. Note what _create_session does on every request: it asks for the metrics service, registers the session with zope.sqlalchemy against request.tm, adds a finished-callback to close the connection, and reads the READ_ONLY admin flag, dooming the transaction if it is set. Four other plugins are already required for that one attribute to work.

2. 02

   warehouse/rate_limiting/__init__.py: three lines that fifteen callers depend on

   warehouse/rate_limiting/__init__.py → includeme

   The entire body:

   def includeme(config):
       config.add_directive("register_rate_limiter", _register_rate_limiter)
       config.registry["ratelimiter.storage"] = storage_from_string(
           config.registry.settings["ratelimit.url"]
       )
       config.add_tween("warehouse.rate_limiting.headers.rate_limit_headers_tween_factory")

   A directive, a Redis-backed limits store, and a tween that stamps rate-limit headers on responses. The directive is the point. _register_rate_limiter registers a named IRateLimiter service, and fifteen calls in five later plugins use it: nine in accounts (login, 2FA, email add, password reset, account search), two in packaging (project creation per user and per IP), two in manage, one in search, one in legacy.api.xmlrpc. Every one of those plugins is included after this one. That is not a coincidence, it is the ordering constraint.

3. 03

   warehouse/metrics/__init__.py: a service, six subscribers, a deriver

   warehouse/metrics/__init__.py → includeme

   The widest surface for its size. It registers the metrics backend as an IMetricsService factory, chosen by dotted path from settings. It subscribes handlers to five Pyramid events (NewRequest, BeforeTraversal, ContextFound, BeforeRender, NewResponse) plus IBeforeRetry, which is how route-match time, traversal time, view time and render time all get timed separately. It adds a view deriver under viewderivers.INGRESS to wrap and time every view. And it attaches request.metrics.

   This is why .metrics is the second include in the whole function, right after pyramid_services. It must be in place before anything it measures.

4. 04

   warehouse/email/__init__.py: nine lines at the end of 1,193

   warehouse/email/__init__.py → includeme (line 1185 of 1193)

   The email module holds every transactional message Warehouse sends. Its includeme registers one service factory for IEmailSender and one nightly periodic task to trim the EmailMessage table. That is all.

   Take the general lesson: plugin surface has nothing to do with plugin size. A plugin's includeme tells you what it hands the application. Everything else in the module is reached by import, by config.scan(), or not at all.

The centerpiece

The wiring order, annotated

Below is the include sequence from configure(), with the settings and Jinja plumbing elided. The comments quoted here are the load-bearing ones. They document constraints, not intentions.

Why sanity has to cheat

Tweens are normally positioned with over= and under= hints, and Pyramid sorts them. The sanity plugin does not want a position in that sort; it wants to be the outermost and the innermost wrapper, so that a request with undecodable bytes in the query string returns a 400 instead of a 500, and a redirect to a non-ASCII location gets quoted on the way out. So it converts the whole implicit tween chain into an explicit one at the last possible moment:

def includeme(config):
    # I'm doing bad things, I'm sorry. - dstufft
    config.action(
        ("tween", "warehouse.sanity.sanity_tween_factory", True),
        _add_tween,
        args=(config,),
        order=PHASE3_CONFIG,
    )

_add_tween adds the ingress tween explicitly, copies every implicit tween in order, then adds the egress tween. Deferring it to PHASE3_CONFIG means it runs after every other plugin's tween registration, at commit() time. The apology has survived in the source for over a decade. It is still the clearest single example of what "the ordering comments are load-bearing" means in this codebase.

The toolbox

Six ways a plugin extends the app

Every one of the fifty plugins uses some subset of these. There is nothing else.

register_service_factory

Bind an implementation to an interface, optionally under a name. packaging registers IFileStorage twice, as "cache" and "archive", plus ISimpleStorage, IDocsStorage and IProjectService. Views then ask for what they need by interface: request.find_service(IFileStorage, name="archive"). 41 registrations across 21 plugins. The interface pattern that makes local disk, S3, GCS and B2 interchangeable is its own topic; see Swappable Backends, next in this series.

add_request_method

Hang an attribute on every request, usually reify=True so it is computed once. Twenty-three calls, which together are the request object you actually use: db, metrics, user, oidc_publisher, oidc_claims, task, log, id, locale, _, flags, banned, http, opensearch, sentry, sponsors, camo_url, canonical_url, ip_address, remote_addr_hashed, and forklift's own help_url and user_docs_url.

add_directive

Extend the configurator itself, for later plugins. Fourteen calls: register_rate_limiter, register_origin_cache_keys, alembic_config, add_periodic_task, make_celery_app, task, add_redirect, add_template_view, add_pypi_action_route, add_pypi_action_redirect, add_legacy_action_route, and three whitenoise_* directives. Every one creates an ordering constraint.

add_tween

Middleware inside Pyramid, so it sees the request object rather than the raw WSGI environ. Seven ordinary ones: structlog request context, conditional HTTP, rate-limit headers, HTTPS enforcement on legacy action POSTs, response compression, CSP headers, referrer policy. Plus the sanity pair described above.

add_view_deriver

Wrap every view that opts in. Ten of them, and their relative order is spelled out in the arguments: csrf_view goes under=INGRESS, over="secured_view" because, in the source's words, "we do not want to access the ambient authority provided by the session cookie without first checking to ensure that this is not a cross-site request." The method check goes over CSRF so that CSRF never fires on a view that was never going to accept the method anyway.

add_subscriber, predicates, periodic tasks

Six event subscribers, all in metrics. Two route predicates (domain, auth_methods) and one more from action routing (pypi_action); two view predicates (require_headers, require_active_organization). Twenty-three add_periodic_task calls declaring the Celery beat schedule inline, next to the code they schedule. packaging alone contributes seven, including a per-minute file-cache check and a quarter-hourly storage reconcile.

Worth noticing what is not on this list: there is no lifecycle hook, no plugin-to-plugin messaging, no configuration schema. Plugins talk to each other in exactly two ways: by calling a directive an earlier plugin added, or by asking for a service at request time.

Dispatch

The router matches; the ORM supplies the context

Most Python web frameworks pick a view by matching the URL, then hand the view some string captures and let it query. Pyramid can do that, and it can also do traversal, where the URL path is walked segment by segment through a graph of objects. Warehouse does both at once. The dev docs call it by name: hybrid URL traversal and dispatch.

Of 127 routes in warehouse/routes.py, 54 use traverse=. They name a factory and a traversal path built from the matched pattern:

config.add_route(
    "packaging.release",
    "/project/{name}/{version}/",
    factory="warehouse.packaging.models:ProjectFactory",
    traverse="/{name}/{version}",
    domain=warehouse,
)

Seven factories cover all of them: ProjectFactory on 26 routes, OrganizationFactory on 17, TeamFactory and UserFactory on 5 each, plus OrganizationApplicationFactory and two JSON-API release factories. The traversal itself is nothing but __getitem__ on ordinary SQLAlchemy models.

The three __getitem__ methods are four to twenty lines each and live in warehouse/packaging/models.py. All three end the same way:

    except NoResultFound:
        raise KeyError from None

That translation is the whole error-handling story. Pyramid's traverser catches KeyError, stops walking, and hands the view lookup whatever object it had reached plus the leftover segments. The views are registered against a type (@view_config(route_name="packaging.release", context=Release, ...)), so a walk that stopped early produces a context of the wrong type, no view matches, and Pyramid returns 404. The view function never runs. Its first argument is guaranteed to be a real, loaded ORM object:

@view_config(
    route_name="packaging.release",
    context=Release,
    renderer="warehouse:templates/packaging/detail.html",
    decorator=[origin_cache(1 * 24 * 60 * 60, stale_if_error=5 * 24 * 60 * 60)],
    has_translations=True,
)
def release_detail(release, request):
    project = release.project
    ...

Two consequences worth carrying around. First, authorization gets easier: the context object is loaded before the permission check, so a route can declare a permission and Pyramid evaluates it against that object's __acl__. Project.__acl__ builds its list from the project's roles and its organization. Second, "not found" and "no permission" are decided by the same machinery, in the same place, for every one of those 54 routes, and you never write the lookup twice.

The other 73 routes are plain dispatch: static pages, JSON endpoints that do their own querying, health checks, form handlers. Traversal is a tool here, not a religion.

Domains

One process, more than one website

Warehouse serves pypi.org and upload.pypi.org from the same Python process, the same registry, the same database session. The only thing separating them is a route predicate, seventeen lines of it in warehouse/predicates.py:

class DomainPredicate:
    def __call__(self, info, request):
        # Support running under the same instance for local development and for
        # test.pypi.io which will continue to host it's own uploader.
        if self.val is None:
            return True
        return is_same_domain(request.domain, self.val)

Every route in routes.py passes domain=warehouse, where warehouse is the warehouse.domain setting. Every route in forklift/__init__.py passes domain=forklift, from FORKLIFT_DOMAIN. The predicate's None branch is the escape hatch that lets one dev container answer to both, and lets TestPyPI host its own uploader on its own hostname.

Four routes deliberately carry no domain= at all, so they answer on any hostname the process is reachable at: health, force-status, packaging.file, and legacy.docs. The first two are for the load balancer and for exercising error templates; the other two are URL shapes that predate the domain split.

The /pypi pile-up

The same predicate mechanism resolves a much older mess. The single path /pypi on pypi.org is claimed by three different protocols, and the router picks between them on request properties:

• XML-RPC, via three add_xmlrpc_endpoint calls (/pypi, /pypi/, /RPC2). Each one matches on the request's Content-Type: header="Content-Type:text/xml".
• Ten legacy action routes, all on the path /pypi, distinguished by a pypi_action predicate that reads the :action query parameter: file_upload, submit, submit_pkg_info, doc_upload, doap, list_classifiers, search, browse, files, display.
• Redirects for everything else, /pypi/{name}/ to /project/{name}/, plus the modern JSON API hanging off /pypi/{name}/json.

The predicate is a dozen lines and does the obvious thing:

class PyPIActionPredicate:
    def __call__(self, context, request) -> bool:
        return self.action_name == request.params.get(":action", None)

Four of those ten action routes, the upload-shaped ones, are served by a single view, forklifted, in warehouse/legacy/api/pypi.py. It returns 410 Gone with a message pointing at the forklift domain. The upload path exists on the main site only so that a client using a decade-old URL gets told where to go.

The tenant

Forklift, which never moved out

Forklift is the upload API. It has been about to become its own project for roughly ten years, and the file says so at the top, before any code:

"NOTE: warehouse.forklift is a temporary package, and it is assumed that it will go away eventually, once we split forklift out into it's own project."

warehouse/forklift/__init__.py, lines 3–5

Read as architecture rather than as a promise, that comment is a design constraint that has held: forklift is coupled to the rest of Warehouse only through the shared toolbox, so it stays extractable. Its includeme is 58 lines and contributes exactly six things.

1. 01

   Its own copy of the action-routing directive

   warehouse/forklift/action_routing.py

   Eleven lines, near-identical to warehouse/legacy/action_routing.py, with one difference that is the whole point: the legacy version's add_pypi_action_route registers the path /pypi, forklift's add_legacy_action_route registers /legacy/. Both reuse the same pypi_action route predicate, which the legacy module registered earlier in configure(). Duplicating eleven lines rather than sharing a parameterized helper is what keeps the eventual split a directory move.

2. 02

   Five routes, all domain-scoped

   warehouse/forklift/__init__.py → includeme

   file_upload, submit, submit_pkg_info and doc_upload on /legacy/, plus /legacy without the slash so a missing slash gets a 308 rather than a 404. Only file_upload still does anything; the other three return 410 Gone. Note the one that carries auth_methods={"basic-auth", "macaroon"} , a storage-only predicate that always matches and exists purely so the security policy can read it back off request.matched_route.predicates.

3. 03

   Three template views and two request methods

   warehouse/forklift/__init__.py → includeme

   The upload host serves an index page, a robots.txt, and a catch-all for /legacy/ requests with no valid :action. It attaches request.help_url and request.user_docs_url, which build absolute URLs back to the main site and to docs.pypi.org. They exist because every error message the upload host returns has to point somewhere else.

4. 04

   Three service lookups, and that is the whole coupling

   warehouse/forklift/legacy.py (1,899 lines)

   file_upload is one of the largest view functions in the codebase, and it reaches into the shared machinery through three find_service calls: IProjectService to create a project, IIntegrityService for PEP 740 attestations, and IFileStorage named "archive" for the object-storage PUT. Everything else it needs it gets from the request attributes other plugins attached (request.db, request.metrics, request.user, request.help_url) and from view-config options other plugins registered, such as require_csrf=False, require_methods=["POST"] and permit_duplicate_post_keys=True.

   What that view actually validates (the metadata forms, hashing, size limits, YARA scan, filename rules, attestations) is the subject of the PEP 694 field guide, which walks the gauntlet in order. This guide is only about how it is plugged in.

Add it up and forklift is a second website in the same process: its own hostname, its own routes, its own request methods, its own directive, and 1,899 lines of view code. It is joined to everything else by five service and request-method names.

The map

Where everything is

If you're going to work on this

Six things worth carrying into your first patch, roughly in the order they bite.

1. Read configure() once, end to end, before you touch anything. It is long but it is linear, and it is the only place the whole application exists at once.
2. Before moving an include line, grep for the directives that plugin adds. A directive moved down the file breaks startup; a service moved down usually does not. Know which one you have.
3. When adding a resource-shaped route, copy the factory= plus traverse= plus context= pattern rather than querying in the view. You inherit the 404 and the ACL for free.
4. When adding a plugin, put its includeme at the bottom of the module and keep it to registrations only. If it contains logic, that logic is untestable without booting a configurator.
5. Anything you add to a subpackage that is not in its includeme and not decorated for config.scan() is dead code. Check which category yours is in.
6. Next in this series, Swappable Backends: the zope.interface and pyramid_services pattern that lets the same find_service(IFileStorage) return local disk in development and S3 in production. See the series page for status.