PEP 694 Field Guide

Orientation

One API, two eras

There has never been a standardized upload API for Python package indexes. Everyone reverse-engineers what PyPI does. What PyPI does is accept a multipart/form-data POST. The original upload API was not an API at all: it was a browser form on the first PyPI, and the tools learned to submit that form programmatically. Twenty-odd years later, the whole ecosystem still holds one HTTP request open while the index validates, scans, stores, and journals a file.

PEP 694 defines Upload API 2.0: a JSON protocol where a release is a publishing session, each file goes up in its own file upload session through a negotiated mechanism, the whole set can be installed and tested from an unguessable stage URL before anyone else can see it, and a single publish makes every file public at the same instant, or none of them.

The PEP does not deprecate the legacy API. Both will run side by side, at different endpoints, for years. That is why its atomicity rules keep talking about conflicts arriving "through the legacy API." The two doors open into the same room.

How it works today

The legacy path, end to end

Six stages, from the client's first byte to the moment strangers can install the thing:

1. 01

   The client POSTs a form

   https://upload.pypi.org/legacy/?:action=file_upload · docs.pypi.org/api/upload

   One request per file: :action=file_upload, protocol_version=1, the artifact as a content part, digests (sha256 or blake2_256 required), and every core-metadata field flattened into ~40 form keys whose names only roughly match METADATA. Auth is HTTP Basic with username __token__, or a Trusted Publishing OIDC exchange that mints the same kind of short-lived token on a third hostname (auth.domain).

2. 02

   DNS, not Fastly

   pypi/infra → terraform/main.tf (absence) · upload.pypi.org A records

   upload.pypi.org appears nowhere in the Fastly Terraform. It is a bare A record pointing at warehouse.ingress.us-east-2.pypi.io, the Cabotage ingress, and it answers as gunicorn with no x-cache headers. The main pypi.org service even has VCL to un-shield POSTs, but a CDN still buffers and caps large request bodies, so uploads get their own hostname that skips the edge entirely. TestPyPI is asymmetric: test.pypi.org/legacy/ does go through Fastly.

   On the warehouse side, the split is a Pyramid route domain: every forklift route binds to FORKLIFT_DOMAIN (warehouse/forklift/__init__.py), and the same paths on pypi.org return 410 Gone with a pointer to the upload host.

3. 03

   The synchronous gauntlet

   warehouse/forklift/legacy.py → file_upload (~lines 613–1850)

   One view function runs everything while the client waits: form and metadata validation, project auto-creation (rate limited: 20/hour per user, 40/hour per IP), permission, verified-email and 2FA checks, README rendering, release creation (which row-locks every release of the project to re-sort them), filename rules, streaming the body to a temp file while computing md5+sha256+blake2b and enforcing size limits (100 MiB/file and 10 GiB/project by default, overrides capped at 1 GiB), archive inspection with a decompression-bomb check and a YARA scan of every member, PEP 625/639 filename and license checks, wheel RECORD and platform-tag validation, PEP 658 METADATA extraction into a sidecar file, PEP 740 attestation verification, and finally the object-storage PUT to the archive bucket and the journal writes, all before HTTPOk. The DB connection sets a 600-second statement timeout because this can genuinely take that long.

4. 04

   Two quirks every tool depends on

   warehouse/forklift/legacy.py → _is_duplicate_file, _exc_with_message

   Re-uploading a byte-identical file is a silent no-op: the transaction is doomed and the server answers 200 OK anyway. A same-name-different-bytes upload gets a 400 whose message must begin with the literal string File already exists, because twine --skip-existing matches on that prefix. And every error travels in the HTTP reason phrase, a field deprecated since HTTP/2, encoded down to latin-1 so gunicorn doesn't choke.

5. 05

   After the response

   warehouse/packaging/tasks.py → sync_file_to_cache, update_bigquery_release_files

   Only two things defer to Celery: copying the file from the archive bucket to the B2 cache bucket that files.pythonhosted.org serves (Fastly → B2 primary, restart-failover to an S3 Glacier archive), and inserting the metadata row into the public BigQuery distribution_metadata table. The file is durable and DB-visible on response but not yet in the CDN-served bucket.

6. 06

   And it is public. Now. Per file.

   pypi.org/simple/<project>/

   Each file appears in the simple index the moment its own POST commits. A release with an sdist and thirty wheels is public in thirty-one increments. If a consumer resolves the project after the sdist lands but before their platform's wheel does, they build from source. Or fail. This is the race that downstream CI feels as "the release exists but is broken for an hour."

The new mechanism

Sessions, stages, and one publish

Upload 2.0 is a small number of JSON resources linked together. Everything below is from the current draft (post–Round 4, July 2026). Every request carries Content-Type: application/vnd.pypi.upload.v2+json, except executing an upload mechanism itself, which uses whatever that mechanism specifies (application/octet-stream for http-post-bytes). The proposed PyPI root is https://upload.pypi.org/2.0.

The requests, concretely

POST /2.0  HTTP/1.1
Content-Type: application/vnd.pypi.upload.v2+json

{"meta": {"api-version": "2.0"}, "name": "foo", "version": "1.0"}

→ 201 Created
{"meta": {"api-version": "2.0"},
 "links": {"session": "…", "upload": "…", "publish": "…",
           "extend": "…", "stage": "…"},
 "mechanisms": ["http-post-bytes"],
 "session-token": "<unguessable>",
 "expires-at": "2026-08-20T12:00:00Z", "status": "open", "files": {}}

POST <links.upload>
{"meta": {"api-version": "2.0"}, "filename": "foo-1.0-py3-none-any.whl",
 "size": 1048576, "hashes": {"sha256": "…"}, "mechanism": "http-post-bytes"}

→ 202 Accepted   (then POST the raw bytes to mechanism.file_url,
                    then POST links.complete, poll until "completed")

Details that matter, each of which exists because someone argued for it:

• One session per name+version. A second create for the same pair while a session is live returns 409 Conflict with a Location header to the existing session. That is also how several Trusted Publishing jobs join one session. twine upload foo-1.1* foo-2.0* bar-2.0* means three sessions; the client manages that invisibly.
• Authorization is checked per request, not per session. Any principal currently allowed to upload to the project can contribute to, extend, cancel, or publish the session. Revoke a token mid-session and its next request 403s; rotate a Trusted Publishing token and the new one continues where the old one stopped.
• The fail-fast is real. filename, size, and hashes are required up front so the index can reject a quota overrun or a filename conflict before a gigabyte crosses the wire. The legacy API can only fail after.
• No metadata field. Round 3 deleted it: the uploaded file is the only source of metadata, extracted server-side. Twenty years of "the form said one thing, the wheel said another" ends here.
• Publish is an allow-list. Every file must be completed or the publish is a 409 naming the offenders. Nothing is silently included, nothing silently dropped.
• Publish is atomic against both doors. At publish time the index reserves every filename in the session; a concurrent upload of the same filename, even through the legacy API, gets a 409 until the publish succeeds (reservation becomes permanent) or fails (reservation released).
• First release of a new name reserves the name for the life of the stage, invisible to regular index protocols, so two clients cannot race to claim it at publish time. Cancel the stage and the name is released as if it never existed.
• Name reservation without files is a feature: create a session with a placeholder version, publish it empty, and the project name is yours. No more 0.0.1 stub uploads.

The state machines

Round 3 made both lifecycles normative. The asymmetry between them is the part worth memorizing:

A session can be edited back to health because its files are replaceable; a half-uploaded file cannot be edited in place, only discarded. Cancellation is always available from any editable state (DELETE links.session) but is refused with 409 during processing, because by then the outcome is already being decided.

Stages are the killer feature

Staged previews are optional for an index. One that doesn't support them omits links.stage and session-token entirely, and still gets atomic publish out of the deal. For PyPI, though, staging is the whole point. The stage URL and session-token are cryptographically unguessable, server-minted capabilities. Hand the URL to CI and it can pip install --extra-index-url $STAGE_URL my-package against the real index (real filenames, real metadata, real resolver behavior) before the public can see anything. Deliberately, the preview capability is not gated by upload permission, so a test job needs no project credentials. This is the feature Barry Warsaw thinks "would solve many of the problems with test.pypi.org, and maybe even allow us to decommission that service."

History

Four years, five threads, one PEP

PEP 694 is a 2022 design that died, was exhumed in 2024, and was then argued into its current shape across four numbered rounds. Reading the threads in order is watching scope get traded for shippability.

2022: born as tus2, died of scope

Stufft's original was, in his words, an "application specific protocol that happens to look almost exactly like tus2" (the IETF resumable-upload draft), with chunked uploads, offsets, and upload tokens in the core spec. The killer use case arrived in the thread immediately, from downstream CI:

"One of the biggest pain points for our projects is that many of our dependencies have significant delays (hours or even days) between uploading some parts of their releases."

fungi (Jeremy Stanley), original thread #8

But implementers pushed back on weight: "the proposed API requires much more from service implementations than the current legacy upload API" (mmarston, #32). Chunk-size semantics tangled with S3's multipart limits, and after Stufft's last post in August 2022 the thread simply stopped. The companion warehouse PR went untouched for two years. Procedurally the PEP was stuck too: Stufft is the natural PEP-Delegate for PyPI PEPs and can't delegate his own PEP, and nobody assigned anyone else.

2024–25: the revival, and the great scope trade

Barry Warsaw picked the PEP up in mid-2024, Alpha-Omega funded implementation work, and two decisions made the 2025 draft shippable. First, Ee Durbin cut resumable uploads out of the core: file transfer became a negotiated mechanism, with dumb-simple http-post-bytes required and anything fancier (S3 multipart via presigned URLs, resumable protocols, whatever) addable as vnd-<operator>-<impl> identifiers without a new PEP:

"I would very much like to see us offload the multi-part upload work to an S3 compatible multi-part upload interface rather than implement it as part of warehouse."

EWDurbin (Ee Durbin), original thread #76

Second, Round 2's defining argument: the January 2025 draft let clients supply a nonce from which the stage URL could be computed out-of-band, via a specified gentoken() hash. William Woodruff found the hash construction lacked domain separation (("foo", 11) and ("foo1", 1) both concatenate to foo11), then asked the question that deleted the whole design: "what if the server was entirely responsible for generating the session token?" Barry conceded the out-of-band property, "as cool as it might be," for a simpler protocol where the index mints an opaque, unguessable token by any algorithm it likes. The same round pinned timestamps to RFC 3339 UTC (after a long localization debate that fungi ended with daylight-saving time) and made duplicate session creation an explicit 409 instead of a get-or-create.

2026: the delegate's round, then the races

Round 3 is the PEP responding to off-thread review by Dustin Ingram, the PEP-Delegate (Stufft and Durbin being conflicted out as co-authors, the Steering Council's fallback). The changes read like a reviewer's checklist of everything that was previously vibes: dedicated links.publish/links.extend/links.complete endpoints instead of magic action strings in request bodies; the optional metadata upload key deleted (files are the source of truth); both state machines written down with transition tables and diagrams; session access relaxed from creator-bound to contemporaneously-authorized; a security-implications section covering name squatting and staged-malware hosting; and a client-recommendations section sketching twine upload --stage, uv publish --test-install, and a stage→test→publish GitHub Actions pipeline.

Round 4 then closed the concurrency gaps Barry had spotted himself: publication became atomic with filename reservation across both APIs, first-release name reservation went from MAY to MUST (otherwise two clients could race to first-publish a new name), publish-with-uploads-in-flight became a defined 409, and cancellation was guaranteed available regardless of file states.

Where it stands (August 2026)

Pre-acceptance, converging. The Round 4 thread is nearly silent; the newest actionable feedback is four implementation-level defects from someone prototyping a client in a uv fork against a server that does not exist: blake2b without a pinned digest length (hashlib defaults to 512-bit, PyPI uses Blake2b-256), file_url/attestations_url in snake_case amid a kebab-case protocol, the error example saying details where RFC 9457 says detail, and an unspecified Location resolution. The one design-level loose end is authentication: konstin (Astral) argued you still cannot write a universal client from the spec text, and the resolution so far is RFC 7235 WWW-Authenticate negotiation rather than a single mandated scheme. Ee Durbin: "I don't much like masquerading basic auth under the Bearer path."

Warehouse contains zero PEP 694 code today. No session models, no stage storage, nothing. The closest thing is the July 2026 change rejecting new files on releases older than 14 days, whose own blog post defers the real semantics: "stay tuned for the future where Upload 2.0 API provides semantics for releases that are 'closed' instead of 'open'."

The review lens

Reading the PEP like an operator

If you review this PEP from the PyPI side (as reviews on the draft PR, python/peps#4794, did), the questions are less about JSON shapes and more about what the protocol does to the index. It is a useful exercise to trace how each concern fared:

The map

Where everything is

If you're going to work on this

Five things to do first, roughly in order:

1. Read the PEP's Change History bottom-up. It is the argument in compressed form.
2. Read file_upload in forklift/legacy.py start to end, and for each validation ask: does this belong at file-session creation (fail fast), mechanism execution, file completion, or publish? That mapping is the warehouse implementation design.
3. Decide early where staged artifacts live. They must be downloadable via the stage URL but invisible to /simple/ and files.pythonhosted.org. The current archive/cache bucket split is suggestive but not sufficient.
4. Watch the Round 4 thread for the auth-specification question and the four client-reported defects; both are likely to produce the next revision.
5. Prototype against the uv fork's client: there is a client and no server, a rare gift for server implementers.