PyPI's API History Field Guide

Orientation

An index older than the tools that talk to it

Richard Jones filed PEP 301 on 24 October 2002. It asked for “a central package index server, tools for submitting package information to the index and extensions to the package metadata to include Trove information,” and it explicitly left out dependency handling and file storage. The PyPA's own history of Python packaging records the result in five words: “2003: PyPI was up and running.” Hosting files came two years later, at the PyCon US 2005 sprints.

Line that up against the rest of the toolchain and the ordering is strange. WSGI (PEP 333) was created in December 2003, after the index was live. Django's first public commit landed in July 2005. JSON did not get a media type until RFC 4627 in July 2006. setuptools and easy_install arrived in 2004; pip in 2008; wheels with PEP 427 in 2012, accepted 2013. npm's public repository history starts in January 2010, and crates.io went online in November 2014.

So PyPI had to choose an API style before most of the reference points existed. It chose the one that was fashionable in 2003, then added a second when JSON became the obvious answer, then discovered that the thing it had built by accident for easy_install was the one worth standardizing. All three survive. You can see them in the tree right now:

warehouse/legacy/api/xmlrpc/views.py     489 lines   POST, method name in the body
warehouse/legacy/api/json.py             377 lines   GET, unspecified, load-bearing
warehouse/api/simple.py                  124 lines   GET, PEP 503, standardized

Read the paths, not the line counts. Two of the three sit under legacy/. The third does not, and it used to. On 26 June 2022 a commit titled “The simple API is no longer considered a legacy api” moved warehouse/legacy/api/simple.py to warehouse/api/simple.py and its template with it. That git mv is the whole argument of this guide in one commit.

Stratum 1

XML-RPC: the API a cache cannot see

Open warehouse/legacy/api/xmlrpc/views.py and the first thing worth reading is not a view. It is the decorator that every view wears:

def xmlrpc_method(**kwargs):
    """
    Support multiple endpoints serving the same views by chaining calls to
    xmlrpc_method
    """
    # Add some default arguments
    kwargs.update(
        require_csrf=False,
        require_methods=["POST"],
        decorator=(submit_xmlrpc_metrics(method=kwargs["method"]), ratelimit()),
        mapper=TypedMapplyViewMapper,
    )

require_methods=["POST"] is the sentence the rest of this guide reacts to. Every XML-RPC call is a POST. The thing being called, changelog_since_serial or package_roles or browse, is not in the URL. It is an XML element inside the request body:

POST /pypi HTTP/1.1
Content-Type: text/xml

<?xml version="1.0"?>
<methodCall>
  <methodName>package_roles</methodName>
  <params><param><value><string>django</string></value></param></params>
</methodCall>

HTTP caching keys on the method and the URL. Both are constant here. A cache looking at ten thousand XML-RPC requests sees ten thousand identical POST /pypi lines and cannot tell a cheap call from an expensive one. Nothing in the HTTP spec forbids caching a POST outright, but there's no reusable representation keyed on a request body an ordinary shared cache ever looks at, so in practice nothing gets reused. PyPI's own Fastly VCL settles the question for its own traffic: it passes every non-GET method straight through the shield to the origin. Three add_xmlrpc_endpoint calls in routes.py register /pypi, /pypi/ and /RPC2, each matching on header="Content-Type:text/xml", which is how one path carries an RPC endpoint and a redirect at the same time. No. 02 draws that overloaded path in full.

PyPI's own documentation states the consequence without ceremony. From docs.pypi.org/api:

“The XML-RPC API does not have the ability to indicate cached responses.”

docs.pypi.org → APIs

And, on the same page: “No new integrations should use the XML-RPC APIs as they are planned for deprecation. Existing consumers should migrate to JSON/RSS/Index APIs.”

Six live methods, ten dead ones

Of the sixteen methods the file still registers, six do work: changelog_last_serial, changelog_since_serial, list_packages_with_serial, package_roles, user_packages, and browse. The other ten are live routes that exist only to raise. They are worth reading as a record of what the interface used to promise:

Every one of those messages ends by pointing at XMLRPC_DEPRECATION_URL, a constant at the top of the file holding https://warehouse.pypa.io/api-reference/xml-rpc.html#deprecated-methods. That page now returns 404, and so does its successor at docs.pypi.org/api/xml-rpc/. The documentation for the deprecated methods was deprecated harder than the methods were.

Four of the ten died on a published schedule. Ee Durbin announced it on discuss.python.org on 5 September 2024:

“These methods have seen continued excess traffic that requires a full transit to our backends to service. Given that we have had them marked as deprecated for years, and they are continuing to cause availability issues for the service, it is time for them to go.”

Deprecation of PyPI XMLRPC Methods, 2024-09-05

The removal ran as three weeks of widening hourly brownouts (ten minutes an hour, then twenty-five, then forty) so that clients not subscribed to pypi-announce would find out the way they actually find things out, by failing. On 26 September 2024 the follow-up post reads, in full: “This deprecation period is complete. These endpoints are now permanently decommissioned.”

Note the phrase “requires a full transit to our backends.” That is the cost of a body-dispatched protocol stated as an operations problem. Nothing in the path can answer on PyPI's behalf.

Which is why the edge reads request bodies

Almost nothing else, but XML-RPC gets a special case. In pypi/infra, at terraform/warehouse/vcl/main.vcl, the Fastly configuration matches on the body:

# Deprecated XMLRPC Methods
if ((req.url.path ~ "^/pypi$" || req.url.path ~ "^/pypi/$") &&
    req.http.Content-Type ~ "text/xml" &&
    req.body ~ "<methodName>search</methodName>") {
        error 666 "Disable XMLRPC search method";
}

Five blocks like that, one per dead method, differing only in the string and the error number. vcl_error then turns each internal code back into a well-formed XML-RPC fault. The status line is 200, because an XML-RPC client reads faults out of the body and would treat a 4xx as a transport failure:

} else if (obj.status == 666) {
    set obj.status = 200;
    set obj.http.Content-Type = "text/xml; charset=UTF-8";
    synthetic "<?xml version='1.0'?><methodResponse><fault>…
        <int>-32500</int>…RuntimeError: PyPI no longer supports
        'pip search' (or XML-RPC search)…";
    return (deliver);
}

The faults are byte-compatible with the ones the Python views raise, down to the -32500 code that XMLRPCWrappedError sets. The edge is impersonating the application so that the application never has to wake up. To block one method on a URL-dispatched API you write a rule about a URL. To block one method here, a CDN has to parse XML.

Two more details from the same file. POSTs skip the shield tier and go straight to a backend (if (req.request == "POST") { set req.backend = F_Application; }), and every non-GET, non-HEAD, non-purge request returns pass, which is Fastly for “do not even consider caching this.” And there is a carve-out explaining why /pypi cannot be force-redirected to HTTPS the way everything else is:

# If we're on the /pypi page and we've received something other than a
# GET or HEAD request, then we have no way to determine if a particular
# request is an API call or not because it'll be in the request body
# and that isn't available to us here. So in those cases, we won't
# do a redirect.

The same sentence keeps coming back in different accents. The edge cannot see what the request is for.

Stratum 1, continued

A Redis cache built because HTTP caching was unavailable

If the CDN cannot cache XML-RPC, and the calls are expensive, the caching has to happen behind the door. Warehouse carries a whole subsystem for it: warehouse/legacy/api/xmlrpc/cache/, five files, 382 lines, serving exactly two of the sixteen methods.

The shape is a Redis-backed LRU (fncache.py, class RedisLru) plus a Pyramid view deriver that wraps any view marked xmlrpc_cache=True, plus SQLAlchemy after_flush and after_commit listeners that turn model changes into purge tags, plus a Celery task per tag to do the purging. It reuses the vocabulary of the CDN layer (keys, tags, purge on commit) because it is solving the same problem one layer lower down. Two call sites use it:

xmlrpc_cache_by_project = functools.partial(
    xmlrpc_method,
    xmlrpc_cache=True,
    xmlrpc_cache_expires=48 * 60 * 60,  # 48 hours
    xmlrpc_cache_tag="project/%s",
    xmlrpc_cache_arg_index=0,
    xmlrpc_cache_tag_processor=canonicalize_name,
)

Above that functools.partial sits a 39-line comment, the most useful thing in the directory. It exists because the cache stores return values as JSON, and a JSON round trip is not the identity function. Item 1 is the one that bites hardest:

“Types the stdlib encoder rejects raise TypeError from inside the view deriver, and nothing catches it: the RPC call fails outright instead of degrading to an uncached response, after the view has already done its work.”

warehouse/legacy/api/xmlrpc/views.py

Which is why changelog_since_serial converts timestamps by hand before returning them (int(e.submitted_date.replace(tzinfo=datetime.UTC).timestamp())). A datetime in a cached view is a runtime error waiting for the first cache miss.

The list continues, and every entry is a way for a cached answer to differ from an uncached one. Tuples come back as lists, so a hit and a miss hand different Python types to the same caller. Non-string dict keys are coerced silently. And item 5 is the canonicalization mismatch, which is the interesting one:

“The cache key is the JSON-encoded arguments verbatim while the purge tag is canonicalized, so package_roles("Django") and package_roles("django") occupy separate entries holding identical data. Both are purged together.”

warehouse/legacy/api/xmlrpc/views.py

Look back at the functools.partial and you can see the seam. xmlrpc_cache_tag_processor=canonicalize_name normalizes the tag. The key comes from json.dumps(request.rpc_args[slice_obj]) in derivers.py, untouched. So the store fragments by spelling while invalidation is correct, which is the right way round to get it wrong, and still means Django and django each pay for their own miss.

Item 4 warns that falsy results are never hits, because fetch was get(...) or add(...). That one has since been fixed in the file below it, and the fix carries its own comment:

def fetch(self, func, args, kwargs, key, tag, expires):
    # `get` returns None for both a miss and a Redis error, so compare against
    # None rather than testing truthiness: an empty list or dict is a real hit.
    # Treating it as a miss counts the request as both a hit and a miss and
    # re-runs the query on every call.
    value = self.get(func.__name__, str(key), str(tag))
    if value is not None:
        return value

The comment above the wrapper and the code below it now disagree, which is worth knowing before you trust either one. It is also a fair summary of the whole subsystem: a careful, documented, actively maintained cache, hand-built to recover a property that a plain GET would have handed over for free.

One more line of defence sits in front of all of it. Every XML-RPC view, cached or not, passes through a rate limiter keyed on request.remote_addr, registered as xmlrpc.client and defaulting to 3600 per hour. That's not unique to XML-RPC (the /search/ view carries its own origin rate limit too, on top of edge caching), but among the three package-data APIs this guide covers, it's the only one that needs an application-level limiter to survive at all: the legacy JSON and Simple APIs lean on edge and origin caching instead, and only XML-RPC's every-request-looks-identical shape forces a second, independent defence.

Stratum 2

The legacy JSON API: useful, unspecified, load-bearing

warehouse/legacy/api/json.py serves four routes that are really two: /pypi/{name}/json and /pypi/{name}/{version}/json, each with a trailing-slash twin. Nothing about it is standardized. No PEP describes it. Its own documentation at docs.pypi.org/api/json opens by steering you elsewhere: “This page documents the PyPI-specific JSON API. If all you need is a JSON index API … you can use the Index API.”

It is also, in the shape it has, unchangeable. The reason is a three-line comment in json_project:

# Build our json data, including all releases because this is the root url
# and changing this breaks bandersnatch
# TODO: Eventually it would be nice to drop all_releases.
return _json_data(request, project, release, all_releases=True)

That is the only mention of bandersnatch in Warehouse's Python source. The all_releases=True flag is what makes the project-level response include a releases object holding every file of every version, so a request for one project can return megabytes. The versioned endpoint sets all_releases=False and stays small. PyPI's docs say why the split exists in the first place: “Due to stability concerns, this had to be removed from the release specific page.” Removed there, kept here, because the recommended way to mirror PyPI reads it.

The rest of the file is an archaeology site with the labels still attached. "has_sig": False and "downloads": -1 are constants, each carrying the same comment: “TODO: Remove this once we've had a long enough time with it here to consider it no longer in use.” "bugtrack_url" is hard-coded None. The docs list all three, plus releases, as deprecated, and warn that they “may be removed entirely from this API response.” Above the serialization block sits the honest version of the file's mission statement: “Serialize our database objects to match the way that PyPI legacy presented this data.”

The part that matters: it is a GET

Unlike its neighbour in legacy/, this API is cached properly, because it was built the ordinary way. The identity of the thing is in the URL, the method is GET, and the view decorator says so:

_PROJECT_CACHE_DECORATOR = [
    cache_control(15 * 60),  # 15 minutes
    origin_cache(
        1 * 24 * 60 * 60,  # 1 day
        stale_while_revalidate=5 * 60,  # 5 minutes
        stale_if_error=1 * 24 * 60 * 60,  # 1 day
        keys=["all-legacy-json", "project-legacy-json"],
    ),
]

Fifteen minutes at the client, a day at the edge, and a day of serving stale data if the origin is unreachable. The keys are extra surrogate keys, so every legacy-JSON response can be purged as a class without touching anything else.

The edge helps further. Before the request reaches Warehouse, the VCL rewrites the name so that spelling variants collapse onto one cache object:

# Normalize project name in JSON API and collapse requests, redirect if
# trailing slash present.
if (req.url.path ~ "^/pypi/([^/]+)(/?[^/]+)?/json/?$") {
    set var.json_url = "/pypi/" +
        std.tolower(regsuball(re.group.1, "(\.|_|-)+", "-")) + "/json";

That is PEP 503's normalization rule, implemented in VCL, applied to an API that PEP 503 has nothing to do with. Flask-SQLAlchemy, flask_sqlalchemy and flask.sqlalchemy become one cached object instead of three. Warehouse would have redirected all of them anyway, via HTTPMovedPermanently in the view, but a redirect costs a round trip and a second cache lookup. The rewrite costs nothing.

So: same directory, same legacy label, same lack of a specification. One of them the edge can absorb entirely and one of it cannot, and the difference is not the data or the format. It is that one put the project name in the URL.

Stratum 3

The Simple API, which was never designed

The third stratum began as a convenience. easy_install, shipped with setuptools in 2004, needed somewhere to look for a project's files, and what it got was a web page with links on it. Not a data format. A page. That is still exactly what /simple/django/ is.

PEP 503, created 4 September 2015 by Donald Stufft, did not invent this. It wrote down what was already being served, eleven years after the fact. Its normative text is short enough to read as a whole:

“Within a repository, the root URL … MUST be a valid HTML5 page with a single anchor element per project in the repository. The text of the anchor tag MUST be the name of the project and the href attribute MUST link to the URL for that particular project.”

PEP 503 → Specification

And for a project page:

“This URL must respond with a valid HTML5 page with a single anchor element per file for the project. The href attribute MUST be a URL that links to the location of the file for download, and the text of the anchor tag MUST match the final path component (the filename) of the URL. The URL SHOULD include a hash in the form of a URL fragment…”

PEP 503 → Specification

The PEP also fixes the normalization rule that the VCL above reimplements, as four lines of Python:

import re

def normalize(name):
    return re.sub(r"[-_.]+", "-", name).lower()

Warehouse's implementation is a Jinja template. This is the whole of warehouse/templates/api/simple/detail.html, minus the file loop, which is one long line:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta name="pypi:repository-version" content="{{ meta['api-version'] }}">
    {% if project_status %}<meta name="pypi:project-status" content="{{ project_status.status }}">{% endif %}
    <title>Links for {{ name }}</title>
  </head>
  <body>
    <h1>Links for {{ name }}</h1>
    {% for file in files -%}
    <a href="{{ file.url }}#sha256={{ file.hashes.sha256 }}" …>{{ file.filename }}</a><br />
    {% endfor -%}
  </body>
</html>
<!--SERIAL {{ meta['_last-serial'] }}-->

An h1, a list of anchors, and a serial number in an HTML comment. Every capability added since 2020 rides on that page as an attribute or a meta tag: data-requires-python, data-yanked, data-core-metadata, data-provenance. A client written in 2004 that reads anchors and ignores attributes it does not recognise still works against it, which is not a courtesy. It is what HTML's parsing rules require, and it is the reason the format could keep growing.

The view is 124 lines including imports, and half of it is content negotiation. The core is _select_content_type, which offers three types and takes the client's first preference:

offers = request.accept.acceptable_offers(
    [
        MIME_TEXT_HTML,                 # "text/html"
        MIME_PYPI_SIMPLE_V1_HTML,       # "application/vnd.pypi.simple.v1+html"
        MIME_PYPI_SIMPLE_V1_JSON,       # "application/vnd.pypi.simple.v1+json"
    ]
)
if not offers:
    return MIME_TEXT_HTML

When the answer is JSON the view swaps its own renderer at runtime, request.override_renderer = "orjson", and returns the same context dictionary the template would have received. One view, one query, one context, two serializations. That is the machinery of PEP 691, and the next section explains why it was built that way rather than as a second URL.

The ladder

Standardizing in place, six PEPs deep

Once the format was written down it could be versioned, and once it could be versioned it could grow without a flag day. PEP 629 (16 July 2020) supplied the version number, and its choice of mechanism is the tell. It is not a header. It is a tag in the document:

<meta name="pypi:repository-version" content="1.0">

The PEP considered a header and rejected it, because that “would make mirrors have to start modifying headers instead of being able to operate as a 'dumb' HTTP server of files.” Read that as a design rule and the rest of the ladder follows from it: the Simple API stays a thing you can serve by copying files onto a static host, and every extension has to survive that constraint.

PEP 691 (2022) added the JSON serialization and, crucially, refused to add a URL for it. From its Goals:

“Enable zero configuration discovery. Clients of the simple API MUST be able to gracefully determine whether a target repository supports this PEP without relying on any form of out of band communication.”

PEP 691 → Goals

Hence content negotiation: “To enable zero configuration discovery and to minimize the amount of additional HTTP requests, this PEP extends PEP 503 such that all of the API endpoints … will utilize HTTP content negotiation.” The URL stays /simple/django/ for both formats. PEP 691 also deliberately did not bump the version, arguing that a new serialization of existing features is not a new feature.

PEP 658 (2021) is the one that reveals PyPI's data model. Its problem: to learn a distribution's dependencies you had to download the distribution, and for an sdist that historically meant executing its setup.py. Its rationale rejects the clever fix and takes the boring one:

“Tools have been exploring methods to reduce the download size by partially downloading wheels with HTTP range requests. This, however, adds additional run-time requirements to the repository server … It is therefore desired to make the server extract the metadata file in advance, and serve it as an independent file.”

PEP 658 → Rationale

An independent file, per artifact, at a URL. It is per artifact rather than per release because it has to be: two wheels of the same version can declare different dependencies, and the PEP's rejected-ideas section notes that inlining every distribution's metadata into the project page would bloat a page whose readers mostly want one file from it. PyPI extracts that sidecar during upload, which is No. 05's territory.

PEP 714 (2023) is the ladder's cautionary tale. PyPI shipped PEP 658 with the JSON key misnamed. Fixing the name turned out to be impossible:

“when attempting to fix that bug, it was discovered that pip also had a bug, where any use of dist-info-metadata in the JSON representation would cause pip to hard fail with an exception.”

PEP 714 → Motivation

The pip bug had shipped in v22.3, roughly eight months earlier, long enough to be baked into distro packages and container images. And a pip broken this way “cannot install anything from PyPI once it fixes its bug, including installing a new, fixed version of pip.” The escape was to rename the field in the specification: dist-info-metadata became core-metadata, data-dist-info-metadata became data-core-metadata. The PEP is candid that this is not normal practice, since “the spec is fine and these are just genuine bugs in pip and PyPI.”

You can still see the settlement in warehouse/packaging/utils.py, where one hash is emitted under two names:

"data-dist-info-metadata": (
    {"sha256": file.metadata_file_sha256_digest}
    if file.metadata_file_sha256_digest
    else False
),
"core-metadata": (
    {"sha256": file.metadata_file_sha256_digest}
    if file.metadata_file_sha256_digest
    else False
),

The version numbers, and what each one cost

packaging.python.org's simple repository API spec keeps the ledger. Warehouse keeps the current value, one line, in warehouse/packaging/utils.py: API_VERSION = "1.4".

Two rows are worth pausing on. PEP 708 spent three years provisionally accepted, claimed version 1.2, and was then rejected because its acceptance conditions (an implementation in Warehouse, an implementation in another repository, opt-in support in pip) were never met. The version history still lists 1.2 as declared by it. A minor version number is cheap enough that burning one on a proposal that did not land is not worth undoing.

And PEP 792's status markers are visible in the code you have already read. The {% if project_status %} line in the template, the "project-status" key in _simple_detail, and _valid_simple_detail_context renaming that key because Jinja cannot address a hyphenated name, are all one PEP arriving in 2025 in a document format from 2004.

The payoff

Why the Simple API is the cheapest surface PyPI has

Everything above is history. This section is the mechanism, and it is short, because a cacheable API needs very little machinery.

Start with the view decorator in warehouse/api/simple.py, identical on the index and the detail page:

decorator=[
    add_vary("Accept"),
    cache_control(10 * 60),  # 10 minutes
    origin_cache(
        1 * 24 * 60 * 60,  # 1 day
        stale_while_revalidate=5 * 60,  # 5 minutes
        stale_if_error=1 * 24 * 60 * 60,  # 1 day
    ),
]

Note what is missing: no keys=[...]. The Simple API does not need to name its surrogate keys, because the object it is about supplies them. simple_detail is registered with context=Project, and warehouse/packaging/__init__.py already declares what a Project is worth:

config.register_origin_cache_keys(
    Project,
    cache_keys=["project/{obj.normalized_name}"],
    purge_keys=[
        key_factory("project/{obj.normalized_name}"),
        key_factory("user/{itr.username}", iterate_on="users"),
        key_factory("all-projects"),
        key_factory("org/{attr.normalized_name}", if_attr_exists="organization"),
    ],
)

So /simple/django/ goes out with Surrogate-Key: project/django, and any commit that touches that project purges exactly that key. One project, one key, one URL. The general mechanism (the after_flush collector, the after_commit sender) is No. 01's subject; what matters here is that the Simple API is the case the mechanism fits perfectly, with nothing left over.

The edge collapses the variants

Content negotiation is the one thing that could have ruined this. Vary: Accept means every distinct Accept string is a separate cache object, and browsers send long, idiosyncratic Accept headers. So the VCL normalizes it before it becomes a cache key, at the edge only:

# We want to normalize our Accept header for our /simple/ index pages to
# support the content negotiation feature added in PEP 691.
if (req.url ~ "^/simple/" && fastly.ff.visits_this_service == 0) {

What follows is roughly sixty lines of careful work. A ?format= query parameter wins outright if it names a known type. Otherwise accept.media_lookup maps whatever arrived onto one of the three known types, with a fallback dance through a sentinel value that the comment apologizes for at length. The latest aliases are resolved to their concrete versions. Then the real answer is stashed aside and text/html is folded into the versioned type, so that both spellings share one cached object:

# We want to stash what our "real" content type is, prior to doing any sort of
# aliasing for cache efficiency, so that we can serve it correctly.
set req.http.Warehouse-Real-Content-Type = req.http.Accept;

# text/html is really just application/vnd.pypi.simple.v1+html, so we'll turn it into that
if (req.http.Accept == "text/html") {
    set req.http.Accept = "application/vnd.pypi.simple.v1+html";
}

On the way out, vcl_deliver puts the client's spelling back on the response. Five acceptable ways to ask for the HTML page, one object in cache, and the client still sees the content type it asked for.

Rendered once, stored as a file

The last piece is the one that proves the point. render_simple_detail in warehouse/packaging/utils.py takes a project, renders the Simple page through the same Jinja template the view uses, hashes the result with BLAKE2b, and writes it to object storage:

content = template.render(**context, request=request)
content_hasher = hashlib.blake2b(digest_size=256 // 8)
content_hasher.update(content.encode("utf-8"))
content_hash = content_hasher.hexdigest().lower()

simple_detail_path = (
    f"{project.normalized_name}/{content_hash}.{project.normalized_name}.html"
)

With store=True it writes twice: once at the content-addressed path, once at {project}/index.html, both tagged with the project name, its last serial, and the hash. The destination is ISimpleStorage, a swappable backend covered in No. 03: a directory in development, an object store in production.

A Simple page is a static file. Not metaphorically. Warehouse can render one, hash it, and put it in a bucket, and the result is byte-identical to what the view would have produced, because there is no per-request state in it. Try that with POST /pypi and a method name in the body.

The other side of the wire

Macaroons, and caveats that only subtract

Everything so far is anonymous. Reads need no identity, which is most of why they cache. When PyPI does need to know who you are, it uses macaroons, and the design rhymes with the cacheability argument in a way worth ending on.

The primitive comes from “Macaroons: Cookies with Contextual Caveats for Decentralized Authorization in the Cloud”, presented at NDSS 2014 by Birgisson, Politz, Erlingsson, Taly, Vrable and Lentczner. The paper's own summary:

“Although macaroons are bearer credentials, like Web cookies, macaroons embed caveats that attenuate and contextually confine when, where, by who, and for what purpose a target service should authorize requests.”

Macaroons, NDSS 2014 → Abstract

Attenuate is the operative verb. A caveat is a predicate that must hold for the token to be accepted, and adding one produces a new token that can do strictly less. There is no operation that adds authority, because the construction is a chain of HMACs where each caveat is folded into the signature: anyone holding the token can extend the chain, nobody without the root key can shorten it.

Warehouse's caveats are in warehouse/macaroons/caveats/, and there are only five things a PyPI token can say:

Serialization is deliberately austere. Each caveat is a JSON array of [TAG, ...fields], field order fixed by declaration order in the class, and the file states the consequence as a rule:

“This means that fields cannot be reordered or deleted, and that new fields must be added at the end of the class. New fields must also include a default value. If a change can't be made under these constraints, then a new Caveat class (and tag) should be created instead.”

warehouse/macaroons/caveats/__init__.py

Verification runs through pymacaroons: build a Verifier, register a general satisfier that deserializes each first-party caveat and evaluates it against the request, then check the HMAC chain against the stored root key. Failure returns WarehouseDenied with a reason; success returns Allowed("signature and caveats OK").

The caveats PyPI keeps in the database

Now the detail that makes the design more than textbook. DatabaseMacaroonService.verify appends caveats to your token before checking it, out of a JSONB column on the macaroon row:

for caveat in dm.caveats:
    m.add_first_party_caveat(caveats.serialize(caveat))

verified = caveats.verify(m, dm.key, request, context, permission)

The comment above it is 24 lines and explains the whole trade. Every embedded caveat lengthens the token, and “in the common case, end users often don't add their own caveats to the macaroons we give them,” so tokens were paying for self-description they rarely used. Keeping the system-generated caveats server-side buys three things, in the file's own numbering: the caveats no longer need to be embedded, PyPI can add new restrictions to an already-issued token by inserting a row, and appending them when they are also embedded is harmless because it just verifies the same restriction twice. The note at the end is the honest part:

“NOTE: We choose to mutate the macaroon that was given to us by appending the stored caveats to the end of it. … However, it does mean that the macaroon we actually end up verifying is a 'sub' macaroon of what the user provided us.”

warehouse/macaroons/services.py

That is only safe because caveats subtract. The server is not granting itself anything by appending them; it is choosing to check more than the bearer asked it to check. The database column that holds them carries a comment saying the same thing from the other direction: users may add caveats without telling PyPI, so this list “MUST be used as additional caveats along with whatever caveats are attached to the Macaroon,” never as the authoritative set.

Token mechanics on the upload path (how the token rides in a header, and what that costs) belong to No. 05.

The map

Where everything is

If you're going to work on this

Five habits worth carrying.

1. Before adding an endpoint, ask what a cache would key on. If the answer involves the request body, you are building the XML-RPC cache subsystem again, and you will be writing your own purge tags by Thursday.
2. Treat the legacy/ directory as a description of an API's shape, not its traffic. The legacy JSON API is heavily used and heavily cached; the Simple API left the directory because its design stopped being legacy, not because it got busier.
3. When you change the Simple API, check three places, not one: the view, the Jinja template, and the VCL ^/simple/ block. A new content type that the edge does not know about will fragment the cache silently.
4. Read the packaging.python.org spec rather than the PEPs when you want current behaviour. The PEPs are the argument; the spec is the state, and it is the only place that records PEP 708 claiming version 1.2 before it was rejected.
5. Do not add a caveat type to macaroons without reading the serialization note. The format is positional, tags are permanent, and a new tag is cheaper than any change to an old one.