pip Meets PyPI

Scope

Two orgs, one wire

Entry 00 of this series drew the line between the two organizations: pypi is not pypa. The pypi org runs the index. The pypa org maintains the standards and the client tools. That guide made the point from the server's side. This guide takes the other side, and reads only client code: pypa/pip, plus the library it leans on hardest, pypa/packaging.

Nothing here is about Warehouse. When a fact about the server matters, this guide points at the entry that owns it and moves on.

One property of pip shapes everything below. pip does not import your installed libraries. It carries copies of them. src/pip/_vendor/vendor.txt lists what ships inside the wheel you install pip from, and at pip 26.3.dev0 that list pins packaging==26.3, resolvelib==1.2.1, requests==2.34.2, CacheControl==0.14.4, urllib3==2.7.0, certifi, rich, truststore, and a handful more. Every import inside pip reads from pip._vendor import requests, never import requests.

That has a practical consequence people trip over, in a standard install. Upgrading packaging in your virtual environment changes nothing about how pip parses a version or computes a tag. Two pip releases can disagree about which wheel fits your machine because they vendor two different versions of packaging, and no amount of reinstalling the library will move either of them. To change pip's behaviour, upgrade pip. The one exception is a pip built by a Linux distribution: pip's own vendoring bootstrap has a DEBUNDLED path, meant for redistributors like Debian and Fedora, that aliases pip._vendor.packaging to the system's own packaging package instead of the bundled copy. If you're on a distro-patched pip, this paragraph may not hold.

The first request

One GET, three content types, one preference order

Ask pip for a project and it asks each configured index for one URL. src/pip/_internal/models/search_scope.py builds it in get_index_urls_locations(), a list comprehension over every index you've configured, one Simple-detail URL apiece; with the default configuration that's just PyPI, but add --extra-index-url and it's one request per index, a point this guide comes back to in the last section. Each URL joins the index URL to canonicalize_name(project_name), quotes it, and appends a trailing slash. The comment there explains the slash: PyPI copes without it, but the code adds it “for maximum compatibility with easy_install” and for other index implementations that copied easy_install's behaviour. So pip install Requests and pip install requests both request https://pypi.org/simple/requests/.

The request itself is built in _get_simple_response(), in src/pip/_internal/index/collector.py. The interesting part is one header:

headers = {
    "Accept": ", ".join(
        [
            "application/vnd.pypi.simple.v1+json",
            "application/vnd.pypi.simple.v1+html; q=0.1",
            "text/html; q=0.01",
        ]
    ),
}

Read the quality values. JSON has no q, so it is 1.0. The versioned HTML type is worth a tenth of that. Plain text/html, the only thing any index spoke before PEP 691, is worth a hundredth. pip is not negotiating a preference so much as declaring one and leaving two fire escapes open. Against PyPI the JSON always wins. Against a nine-year-old internal index that has never heard of a vendor media type, the last line is what keeps the install working.

The response is checked before it is trusted, but only the media type. _ensure_api_header() lowercases Content-Type and requires it to start with text/html, application/vnd.pypi.simple.v1+html, or application/vnd.pypi.simple.v1+json. Anything else raises _NotAPIContent, and the message pip prints names all three types back at you. What this guards against is narrower than it sounds: a response with an unexpected media type gets caught (a JSON API error body served as application/json, say), but a captive portal or corporate proxy that serves its login page as ordinary text/html passes the check and goes on to the HTML parser, which will usually just find no distribution links worth following. The guard stops the wrong content type, not a wrong page wearing the right one.

One more guard runs before the GET. If the URL's filename looks like an archive, _ensure_api_response() sends a HEAD first and checks the content type there, so a mistyped --index-url pointing at a 200MB tarball costs one HEAD instead of the tarball. The comment in the code is candid that this only catches URLs that look like archives, which is why the same check runs again on the real response.

Parsing, and where the two shapes converge

parse_links() branches once on the content type. For JSON it loads the document and calls Link.from_json() on each entry in files. For HTML it runs HTMLLinkParser, honours any <base href>, and calls Link.from_element() per anchor. Both paths produce the same Link objects, and every stage after this point cannot tell which wire format it came from. The client's JSON support is a parser swap, not a second code path.

The result is cached in two independent places, and confusing them costs an afternoon. The parse is memoised in process by with_cached_index_content, keyed on the page URL, and deliberately disabled for index pages (cache_link_parsing=False in collect_sources()) while enabled for --find-links pages. The HTTP response is cached on disk by CacheControl, honouring the index's own Cache-Control headers. When you need to defeat the second one, pip has a flag for it: --refresh-package NAME (or --refresh-package :all:) sets Cache-Control: max-age=0 on the simple request. The comment cites pypa/pip#5670 and explains the choice of max-age=0 over no-cache: it still allows a conditional request, so an unchanged page costs a 304 rather than a full body.

The payload

What pip reads, and what it ignores

The Simple API's JSON document is richer than pip's use of it. Entry 04 covers the server side of that document and the PEPs that grew it, so this section only answers one question: which keys reach pip's data model?

Link.from_json(), in src/pip/_internal/models/link.py, reads six things per file entry.

Now the absences. pip reads no filename key; it derives the filename from the URL path, which is why a file entry whose filename disagrees with its url is decided by the URL. It reads no size. It reads no per-file provenance. At the document level it reads no meta block and no api-version, no name, no PEP 700 versions array, and no PEP 708 alternate-locations or tracks. A grep for those strings across src/pip/_internal returns hits only in unrelated commands. The index may publish them; this client walks past them.

The exception, and it is recent, is upload-time. --uploaded-prior-to takes either an ISO 8601 datetime or a duration in whole days written P7D, and package_finder.py rejects every link uploaded at or after that instant. The behaviour when the data is missing is the part worth remembering. The evaluator returns LinkType.upload_time_missing, and pip fails immediately with a message saying the index does not provide upload-time metadata. It does not quietly install the newest thing it found. A time-travel flag that silently stopped travelling would be worse than no flag.

The resolver

The resolver you are actually running

pip has two resolvers on disk and one of them is a museum piece. src/pip/_internal/resolution/ holds legacy/ and resolvelib/, and the choice happens in five lines in cli/req_command.py:

@staticmethod
def determine_resolver_variant(options: Values) -> str:
    """Determines which resolver should be used, based on the given options."""
    if "legacy-resolver" in options.deprecated_features_enabled:
        return "legacy"

    return "resolvelib"

The default is resolvelib. Reaching the old one takes --use-deprecated legacy-resolver, and the flag's own option group is named for what it does: --use-deprecated enables “deprecated functionality, that will be removed in the future.” install also prints a warning when the legacy variant is active, saying the legacy resolver does not consider dependency conflicts. If you are reading a blog post about pip “backtracking” and it is older than 2020, it describes a code path you are almost certainly not running.

The current resolver is a thin layer over the vendored resolvelib. resolvelib knows nothing about Python packaging. It asks four questions through a provider interface, and pip's resolvelib/provider.py answers them: identify a requirement, find matches for it, test whether a candidate satisfies it, and rank which requirement to work on next. Everything specific to wheels, extras, and interpreters lives on pip's side of that interface.

Two things that are not packages, modelled as packages

The first is extras. ExtrasCandidate in resolvelib/candidates.py wraps a base candidate and gives it a distinct name of the form foo[extra], so the graph gets a separate node. It reports the base's dependencies plus the extra's, and then adds a dependency on the base candidate itself. The docstring says why: without that edge, the resolver could pick foo[extra1] at 1.0 and foo[extra2] at 2.0 and never notice it had chosen one package twice.

The second is the interpreter. RequiresPythonCandidate holds a single Version representing the target Python version. By default that's built from sys.version_info[:3], the interpreter actually running pip, but a target-interpreter option overrides it with whatever py_version_info the resolver factory was given instead. Its project name is the literal string <Python from Requires-Python>, chosen with angle brackets because no real project can be named that. Every Requires-Python field in every distribution becomes a RequiresPythonRequirement pointed at that one candidate. The consequence is that a Python version conflict is not a special case in the error reporting code. It is an ordinary unsatisfiable requirement against an ordinary pinned candidate, which is why pip's conflict messages can say “the requested Python version” in the same breath as they list package versions.

Which requirement to work on first

Order matters enormously for how long resolution takes, and it happens in two passes, not one. Before pip ever ranks requirements against each other, narrow_requirement_selection() gives absolute priority to whichever of three things is present, in order: the synthetic Requires-Python identifier always goes first if it's in play, then any requirement currently listed as a cause of the active backtrack, then requirements that have been promoted after repeatedly causing conflicts. Only once none of those apply does resolvelib fall through to asking pip's get_preference() for a sort key per requirement, lower being more urgent, and pip returns a seven-element tuple. In order:

1. 1

   conflict-promoted

   not conflict_promoted

   This is the narrowing pass's third case, not every past backtrack: a requirement earns this flag only after it has caused conflicts often enough to cross a threshold and get promoted, so pip stops re-deriving the same dead end.

2. 2

   direct

   not direct

   An explicit URL or local path. There is exactly one answer, so resolve it and be done.

3. 3

   pinned

   not pinned

   == without a wildcard, or ===. Also one answer.

4. 4

   upper-bounded

   not upper_bounded

   <, <=, ~=, or == with a wildcard. The docstring's reasoning: pip tries the newest version first, so a known ceiling rules out infeasible candidates sooner. It adds, unprompted, that this does not make upper bounds good practice.

5. 5

   requested order

   requested_order

   The position on your command line, or math.inf for anything you did not ask for by name. Your requirements come before their dependencies.

6. 6

   constrained at all

   not unfree

   Any specifier beats no specifier.

7. 7

   alphabetical

   identifier

   The tiebreaker exists for one reason, stated in the docstring: consistency, which aids debuggability. A resolver that reorders itself run to run cannot be debugged by rerunning it.

None of this changes which answer is correct. It changes how many wrong turns pip takes before it finds one, and on a large requirements file that is the difference between four seconds and four minutes.

Selection

Picking one wheel among equals

Suppose the page listed forty files for one version: sdist, pure-Python wheel, and wheels for several platforms and interpreters. Several of them will run on your machine. pip picks one, and the rule is a single tuple in CandidateEvaluator._sort_key(), in src/pip/_internal/index/package_finder.py. Candidates are sorted with it, and the winner is max(), so higher is better:

return (
    has_allowed_hash,
    yank_value,
    binary_preference,
    candidate.version,
    pri,
    build_tag,
)

The order surprises people who guess it. Version is fourth, not first.

has_allowed_hash

int(link.is_hash_allowed(self._hashes)). If you pinned hashes in a requirements file, a file matching one of them outranks everything, including newer versions and unyanked files. The docstring gives the case that forced this: if the only candidate with an allowed hash has been yanked, pip still wants it, because you said that exact file and no other.

yank_value

-1 * int(link.is_yanked), so 0 for a normal file and -1 for a yanked one. This is PEP 592 working as designed, but by the time a candidate reaches this tuple, most of the filtering is already done: the resolver drops yanked candidates before sorting, unless the requirement is pinned exactly with == or === and every remaining candidate for that pin is yanked. This field is what breaks the tie in that narrow surviving case, not what lets a yanked file back in generally.

binary_preference

Zero unless --prefer-binary is set, in which case every wheel gets a 1 and every sdist keeps 0. Because this sits above version, the flag does what its name says and nothing subtler: it will take an older wheel over a newer sdist.

candidate.version

A packaging.version.Version, compared by PEP 440 rules. Prereleases were already filtered out upstream by the specifier unless they were asked for.

pri

Tag specificity, negated. Covered below.

build_tag

The optional build field in a wheel filename, as in foo-1.0-2-py3-none-any.whl. It isn't purely numeric: the grammar requires a leading digit but allows an alphanumeric remainder after it, and packaging parses it as a (leading integer, remainder string) pair, comparing the whole pair. Two wheels identical in every other respect are separated by that comparison. It is last because almost nothing on PyPI uses it; it exists for redistributors who rebuild the same version.

What tag specificity means

Your interpreter supports a long, ordered list of tags. packaging's sys_tags() generates it, best first, and its docstring gives the ranking as interpreter, then platform, then ABI, with the reasoning that an ABI is tied to a platform while platform-specific code is not necessarily tied to an ABI. On a CPython 3.13 macOS machine the list opens with cp313-cp313-macosx_14_0_arm64, works down through older macOS deployment targets and the stable ABI, and ends, hundreds of entries later, with py3-none-any.

pip precomputes a dict from tag to index once, in CandidateEvaluator.__init__(), then Wheel.find_most_preferred_tag() returns the min() index across the wheel's own file tags. Lower index means better match, and _sort_key negates it into pri so that higher still wins. An sdist gets pri = -len(valid_tags), one step worse than the worst possible wheel, which is what puts source distributions last without a special case.

Two details are worth carrying. The precomputed dict replaced a linear scan of the tag list per wheel, and the docstring says so; on a project with hundreds of files the scan was measurable. And the ordering is not frozen: packaging 26.3 changed it so native linux_* platform tags now sort before manylinux and musllinux tags. That is a client-side behaviour change, shipped in a vendored library, that will alter which wheel some Linux users get, with no change on PyPI at all.

Metadata

Learning dependencies without the wheel

To know what a package depends on, you once had to download the package. For a 300MB machine-learning wheel that pip may reject two seconds later, that is an absurd price for a few hundred bytes of Requires-Dist lines. PEP 658 fixed it by having the index publish each distribution's METADATA file next to it, and PEP 714 fixed PEP 658's key name after the original one turned out to be unusable in the wild.

pip's client half is small. Link.metadata_link() in models/link.py is the whole URL rule:

def metadata_link(self) -> Link | None:
    """Return a link to the associated core metadata file (if any)."""
    if self.metadata_file_data is None:
        return None
    metadata_url = f"{self.url_without_fragment}.metadata"
    if self.metadata_file_data.hashes is None:
        return Link(metadata_url)
    return Link(metadata_url, hashes=self.metadata_file_data.hashes)

The URL is the distribution's URL with .metadata stuck on the end, fragment removed. pip does not invent that URL on spec; it only builds it when the index said the file exists, and the index says so through the keys read back in section 3. Both parsers implement the PEP 714 fallback the same way, and both carry a comment stating the rule: indexes must use the new name, clients should accept the old one for compatibility. In JSON that is core-metadata falling back to dist-info-metadata. In HTML it is data-core-metadata falling back to data-dist-info-metadata. The HTML form also has to parse a string: "true" means the file exists with no hashes, and anything else is split on = into a hash name and digest, with a debug log and a graceful downgrade to “exists, no hashes” if the split fails.

The fetch itself lives in RequirementPreparer._fetch_metadata_using_link_data_attr(), in src/pip/_internal/operations/prepare.py, and its four numbered comments are the whole algorithm: get the link, download the file with its declared hashes, build a distribution object from the bytes, then check that the Name: field inside matches the requirement's name. That last step raises MetadataInconsistent if it does not, which closes an obvious attack: a file served from a URL under one project's name whose metadata claims to be a different package.

Two conditions turn the whole thing off, both in _fetch_metadata_only(), both logged at debug level. The legacy resolver never uses it. And --require-hashes disables it, because in that mode pip must hash the artifact it installs, so there is nothing to save by not downloading it.

The fallback, and its status

When the index publishes no metadata file, pip has a second trick, and it is still in the tree: _fetch_metadata_using_lazy_wheel() uses HTTP range requests to pull just the central directory and the METADATA member out of a remote wheel, through network/lazy_wheel.py. It is not on by default. It requires --use-feature=fast-deps, and enabling it prints a warning that says the feature is experimental and “not ready for production.” If the server does not support range requests, HTTPRangeRequestUnsupported is caught and pip falls through to downloading the wheel. Against PyPI it matters less than it used to, but not never: current uploads normally carry a .metadata sidecar, but plenty of older files on PyPI don't. A same-day check of two popular projects' Simple pages found roughly half their listed files missing a core-metadata value. Pin one of those older wheels with fast-deps enabled and the range-request fallback is exactly what runs. Against an index that never publishes sidecars, it is the difference between reading two kilobytes and reading the whole artifact, every time.

The trace

The whole exchange, in order

Here is pip install requests from an empty HTTP cache into an empty environment, as requests on the wire. Nothing below is a simplification of several requests into one; it is the shape of the conversation.

1. 1

   One page fetch

   index/collector.py → _get_simple_response()

   A single GET for /simple/requests/ with the three-way Accept. PyPI answers JSON. pip parses every file entry into a Link, filters by Requires-Python and platform tags, and sorts the survivors with the tuple from section 5.

2. 2

   One small metadata fetch

   operations/prepare.py → _fetch_metadata_using_link_data_attr()

   Only for the winning candidate, and only because the page said core-metadata was published. A few kilobytes. pip now knows the dependencies without holding the wheel.

3. 3

   One page fetch per new project

   the same code, once per name

   Four more GETs for requests' dependencies. The order in which pip works through them is get_preference() from section 4, not the order they appear in the metadata.

4. 4

   Metadata again, per candidate

   prepare.py

   Steps 3 and 4 repeat until the graph closes. On a conflict, resolvelib backtracks and pip evaluates a different candidate, which usually costs one more .metadata GET and no page fetch, because the page is already parsed in memory.

5. 5

   Then, and only then, the wheels

   network/download.py

   Five wheels, requested with Accept-Encoding: identity (the constant HEADERS in network/utils.py: asking a server to gzip an already-compressed zip wastes both ends' CPU) and streamed to disk in 256KB chunks. An interrupted download resumes with a Range header and, when the server offered an ETag or Last-Modified, an If-Range guard so a file that changed mid-download restarts rather than corrupts.

What the server learns about you

Every one of those requests carries a User-Agent that pip builds in user_agent(), in src/pip/_internal/network/session.py. The format is a short prefix and then a JSON blob:

return "{data[installer][name]}/{data[installer][version]} {json}".format(
    data=data,
    json=json.dumps(data, separators=(",", ":"), sort_keys=True),
)

The dict holds the installer name and version, the Python version, the implementation and its version, the OS and release, the CPU architecture, the OpenSSL version, the installed setuptools version, the distro name and libc on Linux or the macOS version on darwin, the output of rustc --version if a Rust compiler is on PATH, and a ci field that is true or null, never false. The comment explains that last choice: a null says the check ran and was inconclusive, rather than claiming pip knows it is not in CI. There is also an escape hatch, PIP_USER_AGENT_USER_DATA, whose contents pip copies into the blob verbatim.

Those strings are the raw material for PyPI's download statistics. The parser that turns them into BigQuery rows, and the reason the pipeline is a separate repository on a different cloud, is entry 10. Reading the producing code and the consuming code together is the fastest way to understand why a field like rustc_version exists at all.

Anonymous down, authenticated up

Not one request in Fig. 1 carries a credential. pip has full authentication machinery, MultiDomainBasicAuth in network/auth.py, with keyring support, URL-embedded credentials, and interactive prompting, and against PyPI none of it fires, because no read endpoint asks for it. That is why the responses are cacheable at the edge, why the same bytes serve every user, and why an install works from a machine that has never held a token.

Uploads are the mirror image. Every write is authenticated, and the modern credential is a macaroon with caveats rather than a password, issued to a person or minted for a CI run by Trusted Publishing. Entry 05 covers the upload protocol and where it is going. Hold the asymmetry in mind: the read path is anonymous, cacheable, and enormous; the write path is authenticated, uncacheable, and rare. Almost every design decision on both sides falls out of that split.

Configuration

Steering pip at the index

Five options decide where the conversation in Fig. 1 happens. None of them change what it is.

--index-url / -i / --pypi-url

The base URL. Its default is PyPI.simple_url from models/index.py, which is https://pypi.org/ plus simple. The help text asks for “a repository compliant with PEP 503 (the simple repository API) or a local directory laid out in the same format.” Every pip option has an environment twin: configuration.py reads any variable starting with PIP_, so PIP_INDEX_URL works everywhere the flag does, as does a line in pip.conf.

--extra-index-url

Appends. The construction in LinkCollector.create() is one line: index_urls = [options.index_url] + options.extra_index_urls. Read the rest of the code before assuming that list is a priority order, because it is not.

--no-index

Empties the list entirely, with a debug log naming the indexes being ignored. Combined with --find-links pointing at a directory, this is how an air-gapped install works.

--find-links / -f

A URL or a path. Given a directory, pip lists it; given an HTML page, pip parses it for archive links. Note one difference from index URLs: collect_sources() passes expand_dir=True and cache_link_parsing=True for find-links sources, and the opposite for index sources.

--trusted-host

Mounts an insecure adapter for a host, over both http and https, so TLS failures stop being fatal for that host. pip still caches those responses, through InsecureCacheControlAdapter, but the comment above the adapter setup is worth reading: pip caches only securely fetched or explicitly trusted origins, because it cannot validate anything else and does not want a poisoned cache entry that needs manual eviction.

The flat namespace

pip's own documentation states the consequence plainly. From docs/html/cli/pip_install.rst: pip looks on PyPI or the index given as --index-url, in the local filesystem, and in anything added by --find-links or --extra-index-url, and “there is no priority in the locations that are searched. Rather they are all checked, and the ‘best’ match for the requirements… is selected.” The architecture notes in docs/html/development/architecture/package-finding.rst describe the same collection step.

So the candidates from your private index and the candidates from PyPI go into one pool and are ranked by the tuple in section 5. If a name exists in both places, the higher version wins, wherever it came from. Publish internal-billing 99.0 to PyPI and every developer whose --extra-index-url points at the real one will fetch the impostor. This is the dependency-confusion shape, it has been documented behaviour for years, and it is still current in the source read for this guide.

The fix is not a pip flag. It is to make the private index the only index, with --index-url pointing at a repository that proxies PyPI, so that one server decides what a name resolves to. Some indexes and some other installers offer per-package pinning; pip does not.

The map

Where everything is

All paths are relative to the root of pypa/pip.

Closing the loop

This is the last of thirteen entries, and it is deliberately the one that faces outward. Entry 00 mapped the repositories. Entries 01, 02 and 03 went through Warehouse: what it is, how its plugins wire together, how its backends swap. 04 explained why three read APIs still answer and which one grew. 05 took the upload path. 06 looked at where wheels are going. 07 read the Terraform under all of it, and 08, 09 and 10 took the satellites one at a time. 11 covered the part with no code in it, where policy is the product.

Every one of those describes a system that exists to answer the five requests in Fig. 1. The Simple API's cacheability, the immutable file URLs, the CDN, the log pipeline, the yank semantics, the upload protocol: all of it is shaped by one client's expectations, and that client is maintained by people in a different organization who did not have to agree to any of it. They did agree, through PEPs, slowly, in public. That is the loop this guide closes.

If you're going to work on this

Five habits for the boundary between the two orgs.

1. Run pip install -vvv once, on purpose, and read the whole log. The GET for the simple page, the content type it came back as, the “Obtaining dependency information for X from Y” lines, and the eventual download are all there in order. Every claim in this guide is visible in that output.
2. Before filing a client bug against pypi/warehouse, decide which side of the wire it is on. Resolution, wheel choice, tag computation, and caching behaviour are pypa/pip. Version and specifier parsing is pypa/packaging. The page contents are Warehouse. Wrong-repo issues cost a volunteer's time twice.
3. When pip's behaviour changes and no code you own changed, check what pip vendors. The diff between two releases of src/pip/_vendor/vendor.txt answers this in seconds and is the least obvious place to look.
4. Treat --extra-index-url as a security decision, not a convenience. Every name you install becomes a name anyone can claim on the other index in the pool.
5. Read a PEP and its client implementation together. PEP 714 is three pages; pip's implementation of it is two if statements and a comment in each parser. The pair teaches more about how packaging standards actually land than either does alone.