Conveyor: The Redirector With No Hostname

Why it exists

Two hostnames, no database

PyPI changed how it addresses files. The old scheme put the metadata in the path:

/packages/source/r/requests/requests-2.7.0.tar.gz
/packages/2.7/s/six/six-1.9.0-py2.py3-none-any.whl

Four segments: the Python version the file was built for, the first letter of the project name, the project name, the filename. Warehouse replaced that with a content-addressed path built from the file's BLAKE2b-256 digest. The Fastly config in front of files.pythonhosted.org spells the modern shape out as a regex, and you will meet it half a dozen times reading terraform/file-hosting/vcl/files.vcl:

req.url ~ "^/packages/[a-f0-9]{2}/[a-f0-9]{2}/[a-f0-9]{60}/"

Two hex bytes, two more, then the remaining sixty characters of the digest, then the filename. Nothing about the project name or the Python version survives in the path. The two schemes share a prefix and share nothing else.

Links in the old shape are still out there: in requirements files, in blog posts, in pinned build scripts, in vendored lockfiles nobody has touched in a decade. Something has to answer them. That something is pypi/conveyor, and its README is two sentences long:

“This project translates the old URL scheme for packages into redirects and serves legacy user documentation.”

“Canonical data for redirects is pulled from the JSON documents that PyPI serves.”

pypi/conveyor → README.md

The second sentence is the design. Conveyor holds no table of files, no mirror of the index, no cache of what exists. When an old URL arrives, it asks PyPI's own public JSON API what that project has, finds the matching file, and returns a 302 pointing at the current URL. Authority stays in one place. Conveyor is a translator standing next to it.

The other half of the job is documentation. PyPI used to accept documentation uploads and serve them under pythonhosted.org/<project>/. It stopped accepting them. The uploaded HTML did not go anywhere, and neither did the links to it. Conveyor serves that frozen archive out of an S3 bucket, plus a redirect map for projects that have since moved their docs elsewhere.

Four dependencies do all of it. requirements.in lists aiohttp, aiobotocore, gunicorn, and packaging. No database driver, because there is no database.

Anatomy

The whole app is one configure()

Five Python files, and one of them is __init__.py. conveyor/app.py is a license header, one import, and application = configure(). Everything structural lives in conveyor/config.py.

Settings, in full

Two environment variables, both read with os.environ[...], so a missing one kills the process at import time rather than at the first request:

app["settings"] = {
    "endpoint": os.environ["CONVEYOR_ENDPOINT"],
    "docs_bucket": os.environ["DOCS_BUCKET"],
}

CONVEYOR_ENDPOINT is the host to ask for JSON documents. DOCS_BUCKET is the S3 bucket holding the uploaded documentation, pypi-docs. The README's own run instructions supply https://pypi.python.org for the first. That is the entire configuration surface: no secrets, no credentials, no connection strings.

Two shared clients

configure() builds one aiohttp.ClientSession for the process, with a User-Agent of conveyor, so every JSON lookup reuses the same connection pool back to PyPI. A shutdown hook closes it.

The S3 side is stranger. Instead of one session there is a factory that caches an aiobotocore session in a threading.local(), with a comment pointing at a botocore issue as the reason. Views call app["boto.session"]() to get the thread's session, then open a client from it per request. Read that as scar tissue, not as a pattern to copy.

The route table

Ten add_route calls, and the ordering matters more than the count.

The two not_found routes exist to stop the catch-all documentation route from swallowing malformed package URLs. Without them, /packages/garbage reads as a request for a project named packages and goes looking in the docs bucket. The last two routes match nearly everything, which is why anything that must not reach them has to be declared above them.

One route table serves both hostnames. Conveyor never looks at Host. The path shape decides which half of the app answers, and the edge is responsible for only sending it paths it can handle.

The redirect half

How an old download URL gets answered

redirect() in conveyor/views.py is about eighty lines. Walked in order:

1. 01

   Check the letter bucket, cheaply

   conveyor/views.py → redirect()

   The old scheme sharded projects by first letter. If the letter in the URL does not match the first character of the project name, no file can exist at that path, and there is no reason to make a network call to find that out. One if compares project_l against both project_name[0].lower() and project_name[0], accepting either, because old PyPI was not consistent about case. A mismatch returns 404 with a Reason: Incorrect project bucket header.

   That Reason header is conveyor's own invention. It is not a standard header, and it is the fastest way to tell which branch produced a 404 when you are staring at a log line.

2. 02

   Split off a .asc suffix

   conveyor/views.py → redirect()

   PyPI once served a detached GPG signature next to each file, at the file's URL plus .asc. If the requested filename ends in .asc, conveyor strips the four characters, sets a signature flag, and carries on looking for the underlying file. Where that leads is its own section below.

3. 03

   Ask PyPI what the project has

   conveyor/views.py → redirect()

   It joins the configured endpoint with /pypi/<project>/json and fetches it over the shared session. The status handling is three lines and worth copying: a 4xx passes straight through to the client, a 5xx becomes a 503, and anything else is asserted to be 200. A project that does not exist therefore produces a 404 from PyPI that reaches the client as a 404 from conveyor, without conveyor knowing anything about which projects exist.

4. 04

   Scan every file of every release

   conveyor/views.py → redirect()

   Two nested loops over data["releases"]. A file matches when its python_version equals the one in the URL and its filename matches, either normalized or exactly. Normalization is _normalize_filename(), built out of packaging.utils: parse_wheel_filename or parse_sdist_filename pull the pieces apart, then canonicalize_name and canonicalize_version rebuild them, for .whl, .tar.gz and .zip. Anything else comes back unchanged.

5. 05

   302, cached for a week

   conveyor/views.py → redirect()

   On a match the response is a bare 302 carrying the file's current URL out of the JSON document, plus one cache header. If no file matched, the fall-through is 404 with Reason: no file found.

That last response is worth seeing whole, because it is the entire product of the service:

return web.Response(
    status=302,
    headers={
        "Location": file_["url"],
        "Cache-Control": "max-age=604800, public",
    },
)

604800 seconds is seven days. The mapping from an old URL to a current one does not change, so the edge can hold the answer for a long time and conveyor can go back to doing nothing.

if (
    # Prefer that the normalized filename has been specified
    _normalize_filename(file_["filename"]) == filename
    # But also allow specifying the exact filename, to maintain
    # backwards compatiblity
    or file_["filename"] == filename
) and file_["python_version"] == python_version:

The signature branch is a fossil

Follow the signature flag to its end and it checks file_.get("has_sig") in the JSON document. If that is truthy it redirects to the file's URL plus .asc; if not, it returns 404 with Reason: missing signature file.

Now look at what PyPI publishes in that field. pypi/warehouse's JSON view hard-codes it:

# TODO: Remove this once we've had a long enough time with it
#       here to consider it no longer in use.
"has_sig": False,

The column behind it is gone as well. An Alembic migration titled “Remove PGP signature support” does one thing, op.drop_column("release_files", "has_signature"), and its downgrade() raises RuntimeError("Cannot undelete data!"). The recorded creation date in that file is 2023-05-21. The upload path agrees: forklift writes a literal "has_signature": False into the metadata it hands downstream.

So conveyor's signature branch is reachable, correct, and permanently dead. Every .asc request under an old-style URL walks the letter check, splits the suffix, fetches the JSON, matches the file, and then 404s because the field it needs is a constant. Worth knowing before you spend an afternoon on why one specific old URL fails.

The docs half

An S3 bucket with a web server bolted on

Everything under pythonhosted.org that is not a package URL lands in documentation(). It does three things in order, and the order is the design.

1. Check the redirect map

Before touching storage, the view looks the project up in app["redirects"], an in-memory dict. On a hit it returns a 302 to that project's base_uri, appending the requested path only when the entry's include_path flag is set:

if project_name in request.app["redirects"]:
    location = request.app["redirects"][project_name]["base_uri"]
    if request.app["redirects"][project_name]["include_path"]:
        location = f"{location}/{path}"
    return web.Response(status=302, headers={"Location": location})

include_path is the difference between a project that moved its docs to a site with matching page structure (send the visitor to the matching page) and one that rebuilt from scratch (send everyone to the front door). This redirect carries no Cache-Control at all, unlike the package redirect. It inherits whatever the edge decides, which on the docs service is a one-day default TTL.

2. Build a key and fetch it anonymously

No redirect entry means the archived HTML is still the answer. The view builds an S3 key from the project name and the rest of the path, appends index.html if the path ends in a slash, and opens an S3 client with signing turned off:

ANON_CONFIG = BotoCoreConfig(signature_version=botocore.UNSIGNED)

Unsigned means conveyor holds no AWS credentials for this. The bucket is anonymously readable and conveyor reads it the way anyone else could. That is why the deployment needs nothing but two environment variables.

3. Fall back, or redirect to the directory form

If the key is missing, the view tries the same key with /index.html appended. If that exists, it does not serve it. It returns a 301 to the path with a trailing slash added, and lets the client come back around:

except botocore.exceptions.ClientError:
    try:
        key = await fetch_key(s3, request, bucket, path + "/index.html")
    except botocore.exceptions.ClientError:
        return web.Response(status=404)
    else:
        return web.HTTPMovedPermanently(location="/" + path + "/")

That is directory-index behaviour rebuilt on top of flat object storage. The redirect rather than a direct serve is deliberate: relative links inside the served page only resolve correctly when the browser's URL ends in a slash. documentation_top() is the same idea one level up, a 301 from /<project> to /<project>/, and it is why that route is declared separately from the catch-all.

Serving is a StreamResponse read from the S3 body in 4096-byte chunks, with the content type guessed by mimetypes.guess_type from the key. Nothing is buffered whole and nothing is cached in the process.

One more route completes the picture. GET / serves index.html from the bucket root, so the apex of pythonhosted.org is a stored object rather than a rendered page.

The control plane

redirects.txt, polled by ETag

The map that documentation() consults is loaded by redirects_refresh_task in conveyor/tasks.py, started once by configure() and left running for the life of the process. The loop is about twenty lines:

while True:
    bucket = app["settings"]["docs_bucket"]
    session = app["boto.session"]()
    etag = app["redirects"].get('_ETag')
    async with session.create_client('s3', config=ANON_CONFIG) as s3:
        try:
            key = await fetch_key(s3, bucket, 'redirects.txt')
        except botocore.exceptions.ClientError:
            await asyncio.sleep(60)
            continue
    if etag != key['ETag']:
        redirects = await key['Body'].read()
        ...
    await asyncio.sleep(600)

Four things to take from it. The object lives in the same bucket as the docs, read with the same anonymous client. The previous ETag is stashed under the key _ETag inside the redirect dict itself, so an unchanged file skips both the download and the parse. A failed fetch backs off for 60 seconds; a successful one sleeps for 600. And a broad except Exception: pass wraps the body, so a malformed file leaves the last good map in place and logs nothing, which is either resilience or a silent failure depending on how your afternoon is going.

The format is JSON Lines. One object per line, split on b'\n', blank lines dropped:

{"project_name": "somepkg", "base_uri": "https://somepkg.readthedocs.io/en/latest", "include_path": true}
{"project_name": "otherpkg", "base_uri": "https://otherpkg.example/docs", "include_path": false}

Now the part worth sitting with. A maintainer who has moved their documentation to Read the Docs and wants pythonhosted.org/<project>/ to follow needs one line added to one text file in one bucket. No migration, no admin screen, no row anywhere, no deploy. Within ten minutes the ETag differs, the map is rebuilt, and every conveyor worker serves the new redirect. Grep both pypi/warehouse and pypi/infra for redirects.txt and you get nothing. The file is managed by neither repository. It is an operational artifact, and the only code that knows its shape is the eight-line dict comprehension that parses it.

What is being preserved

“Legacy documentation” means the uploads stopped

PyPI once accepted a zip of built HTML alongside a release and hosted it. That door is shut, and the tombstone is legible in pypi/warehouse at warehouse/forklift/legacy.py, lines 1871 to 1881:

@view_config(
    route_name="forklift.legacy.doc_upload",
    require_csrf=False,
    require_methods=["POST"],
)
def doc_upload(request):
    return _exc_with_message(
        HTTPGone,
        "Uploading documentation is no longer supported, we recommend using "
        "https://readthedocs.org/.",
    )

A view whose only purpose is to return 410 Gone, with the replacement named in the body. There is a second one: warehouse/legacy/api/pypi.py maps legacy.api.pypi.doc_upload onto a shared forklifted() view alongside the other retired :action endpoints, also 410, whose message still interpolates an information_url set to the string "TODO". Two routes, two tombstones, one for the old upload host and one for the older one.

This guide will not pin a date on when uploads stopped. PyPI's upload and metadata API history gets its own entry in this series, and a migration timestamp is not the same thing as an announcement. What matters here is the shape: the write path is closed and answers 410, the read path is frozen, and conveyor serves the frozen result.

Warehouse still points at it. The Project model carries a has_docs boolean, and documentation_url returns None unless it is set. When it is set, the URL comes from the docs.url setting, which in the development environment reads https://pythonhosted.org/{project}/, and it surfaces as docs_url in the JSON API. A project's JSON can hand a client a link that only conveyor knows how to answer. That is the tidiest available statement of why this service is not optional.

The wiring

How requests reach a service with no end-user hostname

Conveyor has no hostname meant for a browser or a client to type. Its Cabotage ingress address (conveyor.ingress.us-east-2.pypi.io in production) does resolve and will answer a direct request, but nobody is meant to use it that way. It is a backend, declared twice in pypi/infra, and the two declarations differ in exactly one interesting way.

files.pythonhosted.org: default backend, no condition

The file-hosting module is instantiated in terraform/main.tf for files.pythonhosted.org, with a conveyor_address pointing at a Cabotage ingress host and a files_bucket of pypi-files. Inside terraform/file-hosting/fastly-service.tf there are three backends. Compare their headers:

The Package File condition is the modern URL regex from the top of this guide. The NeverReq condition is a deliberate contradiction, req.http.Fastly-Client-IP == "127.0.0.1" && req.http.Fastly-Client-IP != "127.0.0.1", which can never be true. The archive backend is entered only from the restart logic in files.vcl, which assigns req.backend = F_S3_Archive after a miss on primary storage.

That leaves Conveyor as the only backend with no condition on it, which makes it the fallback. Search files.vcl for the word “conveyor” and you find nothing, because nothing needs to route to it. A hash-shaped URL goes to object storage. Everything else falls through to the redirector by default. The old URL scheme is not handled by a rule. It is what is left over when the rules are done.

pythonhosted.org: one backend, and an apex

The docs-hosting module is simpler and stranger. One domain, one backend named Conveyor, a gzip policy, one VCL file, and a default_ttl of 86400. There is no second origin. Every request to pythonhosted.org is a conveyor request.

The apex handling is the detail worth stealing. The module derives a local from the domain's shape:

apex_domain = length(split(".", var.domain)) > 2 ? false : true

Two labels means an apex, and an apex cannot be a CNAME. So the Route 53 record is an A record pointing at Fastly's anycast addresses, plus a matching AAAA record created only when apex_domain is true. The same module instantiated for test.pythonhosted.org counts three labels, takes the CNAME branch, and skips the IPv6 record. One module, two DNS strategies, chosen by counting dots.

The deployment, in two files

The Procfile is one line:

web: gunicorn -c gunicorn.conf.py -k aiohttp.worker.GunicornWebWorker conveyor.app:application

And gunicorn.conf.py opens with the line that tells you which platform this runs on:

bind = "unix:/var/run/cabotage/cabotage.sock"

A Unix socket at that exact path is the Cabotage web-process contract, the same one camo meets with --socket-listen on its Go binary. Different language, different framework, identical shape. The rest of the file is unremarkable and worth reading anyway: max_requests = 2048 with a jitter of 128 recycles workers often enough that a slow leak never becomes an incident, timeout = 60 lines up with the first_byte_timeout of 60000 milliseconds on the Fastly backend, and when_ready touches /tmp/app-initialized as a readiness marker. Production and test both deploy through Cabotage, and the test service points at its own conveyor address in the same Terraform.

The map

Where everything is

If you're going to work on this

1. Run it before you change it. The README's four commands stand up a working copy against live PyPI, because the only configuration is a hostname and a public bucket. Almost any reported behaviour reproduces on a laptop.
2. Check the edge before you check the app. Half the questions that look like conveyor bugs (why does this URL 404, why is this redirect stale) are answered by which backend claimed the request and what TTL cached the answer. Open fastly-service.tf first.
3. Treat the JSON API as a hard dependency with no fallback. Conveyor's correctness sits entirely downstream of what /pypi/<project>/json returns. When a field there changes shape, conveyor changes behaviour without a commit. has_sig already proved it.
4. Do not add state. This service survives on four dependencies and no on-call story because a worker can be killed at any moment and lose nothing but an in-memory redirect map it rebuilds within ten minutes. Anything that would need a database belongs in Warehouse.
5. The unawaited _normalize_filename is a real bug with a small, self-contained, testable fix, sitting in a public repository. Start there.