PyPI · Infrastructure github.com/pypi/infra PyPI Onboarding · No. 07 7,285 words · ~36 min Edge

Infra: The Layer With a Three-Line README

Two DNS zones, five production Fastly services (nine counting TestPyPI), an email system, two object stores, and a web application firewall. About 3,200 lines of Terraform and VCL decide a great deal of what PyPI does before any Python runs.

Orientation

Eight modules, thirteen instantiations

The pypi/infra repository has 234 commits and a README of three lines. The orientation guide quotes it in full; there is no more of it than that. Under the README sits one directory, terraform/, and under that sits the shape of production.

Eight modules live there: dns, warehouse, file-hosting, docs-hosting, image-proxy, email, gmail, and inspector. The top-level main.tf, 289 lines, calls them thirteen times. Most things exist twice, once for production and once for TestPyPI, and the second copy is usually the same module with a different domain and a flag or two turned off.

InstantiationModuleServes
dnsdnsBoth Route 53 zones, DNSSEC, CAA, apex TXT
gmailgmailMX and one DKIM record for pypi.org
email, testpypi-emailemailSES identity, DKIM, DMARC, bounce notifications
pypi, test-pypiwarehouseThe Fastly service in front of the application
file-hosting, test-file-hostingfile-hostingfiles.pythonhosted.org and its buckets
docs-hosting, test-docs-hostingdocs-hostingpythonhosted.org, legacy user documentation
pypi-camo, test-pypi-camoimage-proxyThe Fastly service in front of camo
inspectorinspectorinspector.pypi.io

Two locals blocks at the top of main.tf do more work than their size suggests. The first, fastly_endpoints, lists four IPv4 addresses, four IPv6 addresses, and one CNAME target, all of them python.map.fastly.net. The second, domain_map, names seven hostnames and points every one of them at that same entry:

domain_map = {
  "pypi.org"                    = "python.map.fastly.net"
  "test.pypi.org"               = "python.map.fastly.net"
  "pythonhosted.org"            = "python.map.fastly.net"
  "test.pythonhosted.org"       = "python.map.fastly.net"
  "files.pythonhosted.org"      = "python.map.fastly.net"
  "test-files.pythonhosted.org" = "python.map.fastly.net"
  "inspector.pypi.io"           = "python.map.fastly.net"
}

Every module that creates a Route 53 record takes both maps as variables. Each one then computes the same local:

apex_domain = length(split(".", var.domain)) > 2 ? false : true

Count the dots. Two labels means an apex, and an apex cannot hold a CNAME, so the module writes A and AAAA records with the literal Fastly addresses. Three labels or more means a subdomain, so it writes a single CNAME and skips the AAAA record entirely. That is why pypi.org and pythonhosted.org carry hardcoded anycast addresses while files.pythonhosted.org carries a name. The same four lines appear in four modules, copied rather than shared.

One instantiation breaks the pattern. Inspector lives at inspector.pypi.io, in a zone the dns module does not create, so main.tf looks the zone up instead:

data "aws_route53_zone" "pypi_io" {
  name = "pypi.io"
}

The comment above it, typos and all, reads inspector dns zone is different fromd ns module one. The pypi.io zone is managed somewhere else. Terraform reads it and writes one record into it.

What is not here

The repository holds no secret values. Every credential arrives as a variable marked sensitive = true in config.tf and is supplied at apply time: the Fastly key, the AWS and Backblaze keys, the WAF token, the Datadog token, the Warehouse shared secret, the IP salt. Reading this Terraform tells you the shape of production and none of its contents. It also means you cannot run it. There is a Makefile, and its terraform target runs init and plan, never apply.

Names

One delegation set, two zones, one signature

terraform/dns/main.tf is 159 lines and creates almost everything a registrar cares about. It opens with an empty resource:

resource "aws_route53_delegation_set" "ns" {}

A delegation set is a fixed group of four Route 53 nameservers. Both zones are created with delegation_set_id pointing at it, so pypi.org and pythonhosted.org answer from the same four nameservers, and the module exports that list as its nameservers output. Without the shared set, Route 53 would pick a random four per zone and the registrar records would have to be updated separately.

The two zones are not treated alike.

aws_route53_delegation_set four shared nameservers zone: pypi.org SIGNED kms_key (ECC_NIST_P256, SIGN_VERIFY) route53_key_signing_key "pypi0" route53_hosted_zone_dnssec CAA: amazon, globalsign, letsencrypt TXT: SPF + 2 site verifications CNAME: strong1/strong2 HelpScout DKIM records written here by: gmail, email, testpypi-email, pypi, test-pypi zone: pythonhosted.org NOT SIGNED no KMS key no key signing key no DNSSEC resource CAA: amazon, globalsign, letsencrypt no apex TXT no DKIM records written here by: file-hosting, test-file-hosting, docs-hosting, test-docs-hosting
Fig. 1. The dns module creates both zones from one delegation set, then signs one of them. The zone that carries mail is signed. The zone that carries every package download is not.

The signing is real and it is only on the primary zone. The module builds a KMS key with customer_master_key_spec = "ECC_NIST_P256" and key_usage = "SIGN_VERIFY", attaches a policy granting dnssec-route53.amazonaws.com the right to Sign and to CreateGrant, then creates an aws_route53_key_signing_key and an aws_route53_hosted_zone_dnssec for aws_route53_zone.primary. The user-content zone gets a aws_route53_zone resource, a CAA record, and nothing else.

The Terraform proves only that the DNSSEC resources reference aws_route53_zone.primary and nothing on the user-content zone. Nothing in the repository documents why. A plausible reading follows from how DNSSEC fails: a signed zone fails closed, so if signing breaks, a validating resolver refuses the answer instead of returning it unsigned. On the zone that serves mail and the web site, that is the behaviour you want. On pythonhosted.org, which is where every pip install in the world fetches its files, a validation failure would be a global outage with no fallback, and the cost of that failure mode could outweigh the attack it prevents. Read that as an inference, though, not a decision this guide can point to in writing.

Both zones share the CAA configuration, built from a list and a report address in main.tf:

caa_report_uri = "mailto:infrastructure-staff@python.org"
caa_issuers    = ["amazon.com", "globalsign.com", "letsencrypt.org"]

The module turns that into 0 issue "<issuer>" records plus one 0 iodef record. Three issuers, because certificates come from three places: Amazon for the AWS-terminated services, Let's Encrypt and GlobalSign through Fastly.

The apex TXT record on pypi.org is a list of three strings, and the third one names the whole mail estate in a single line:

v=spf1 include:_spf.google.com include:amazonses.com include:helpscoutemail.com -all

Three senders are authorised for pypi.org. Google Workspace handles human mail. Amazon SES sends every notification the application generates. HelpScout runs the support desk. The -all at the end is a hard fail, not a soft one, so anything else claiming to be pypi.org fails SPF definitively; what a receiver does with that failure (reject it, quarantine it, deliver it with a warning header) is still its own local policy, not something SPF dictates. Two strong1 and strong2 DKIM CNAMEs pointing at helpscout.net sit alongside it, which is the only sign in the DNS module that a third mail system exists at all.

The big one

The warehouse module, before Pyramid sees anything

The warehouse module is 315 lines of Terraform plus a 615-line VCL file, and it is where most of the surprising behaviour lives. It builds one Fastly service per environment: the primary domain, five extra domains for production, one backend, one healthcheck, four log sinks, two synthetic response objects, and a set of snippets that inject secrets into VCL.

The domains, and the one that is special

The primary domain gets a domain block. The rest come from a dynamic "domain" loop over extra_domains, which for production is pypi.python.org, www.pypi.org, pypi.io, www.pypi.io, and warehouse.python.org. A comment sits above the list in main.tf:

“the first domain in extra_domains gets an XMLRPC exception/bypass in VCL”

pypi/infra → terraform/main.tf

The VCL is templated, so ${extra_domains[0]} is interpolated into it. Every host in that list normally gets a 301 for GET and HEAD and a 308 for anything else. The exception fires for a POST to pypi.python.org, at /pypi, with an XML content type. Those requests get their Host header rewritten and are passed through, because an XML-RPC client will not follow a redirect and the callers of that endpoint have not been updated since before Warehouse existed. Ordering in a Terraform list is load-bearing. Reorder the list and old XML-RPC clients break.

Two secrets injected as VCL snippets

Two snippets of type recv carry values into the edge:

set req.http.Warehouse-Token = "${var.warehouse_token}";
set var.Warehouse-Ip-Salt   = "${var.warehouse_ip_salt}";

The first is a shared secret. Fastly sets it on every request, the application checks it, and a request that arrives at the origin without it did not come through the CDN. The second never leaves the edge. In vcl_recv, the client IP is concatenated with the salt, hashed with digest.hash_sha256, and set as Warehouse-Hashed-IP. The plain Warehouse-IP header still goes to the backend, with a TODO beside it about removing it once the backend has caught up.

The hashed value is what the blocklist uses. The service declares a dictionary, a condition, and a response object:

dictionary { name = "masked_ip_blocklist" }

condition {
  name      = "masked_ip_blocklist"
  type      = "REQUEST"
  statement = "table.contains(masked_ip_blocklist, req.http.Warehouse-Hashed-IP)"
}

The matching response object returns a 403 with the body Your IP Address has been temporarily blocked. Terraform creates the dictionary but does not populate it, so blocking an address is an edge operation, not a deploy, and the record of who is blocked is a set of hashes rather than a set of addresses. Anyone with dictionary access can add an entry; nobody reading the dictionary can read back the IPs.

Killing an API method without touching the application

Five XML-RPC methods are dead, and they are dead in VCL. Each one is matched on the request body, which Fastly can inspect for these small payloads:

if ((req.url.path ~ "^/pypi$" || req.url.path ~ "^/pypi/$") &&
    req.http.Content-Type ~ "text/xml" &&
    req.body ~ "<methodName>search</methodName>") {
        error 666 "Disable XMLRPC search method";
}

The five error codes 665 through 669 cover list_packages, search, package_releases, release_urls, and release_data. Each one is caught in vcl_error and answered with a synthetic XML-RPC fault: status 200, content type text/xml, faultCode of -32500, and a faultString that names a replacement. The one for search is the message anyone who ran pip search in the last few years has seen. A 403 would have been simpler, but an XML-RPC client handles a fault and shows the text; it does not handle an HTTP error. So the edge speaks the protocol back.

Two more user-agent rules sit nearby, both written in frustration. A go-http-client agent requesting a /simple URL without a trailing slash gets error 670; a specific hardcoded Chrome 86 user-agent string doing the same gets 671. Both resolve to a 406 whose body explains that these clients were generating redirect storms and asks the operator to get in touch. A third rule lives in Terraform rather than VCL, as a condition plus a response object: any user-agent matching bandersnatch/1\.(0|1|2|3)\ gets a 403 reading Bandersnatch version no longer supported, upgrade to 1.4+.

Health, shields, and one deliberate cache hole

The backend is the application ingress, over TLS, with a five-second connect timeout, a sixty-second first-byte timeout, and a shield at iad-va-us. A healthcheck hits /_health/ every fifteen seconds and needs three of the last five checks to pass. The VCL then does something that looks like a mistake and is not:

if (req.url == "/_health/") {
    return(pass);
}

The health URL is never cached, so an external monitor can reach past Fastly and get an end-to-end answer. A cached health check tells you the CDN is up, which was never the question. Admin URLs get the same treatment, and POST requests skip the shield entirely with set req.backend = F_Application, because a write has nothing to gain from a second cache layer in front of it.

Storage

Three backends, one that can never be chosen

The file-hosting module serves files.pythonhosted.org. It declares three backends, and the way each one is selected is different.

BackendConditionReached when
ConveyornoneDefault. Anything not matched by another rule
B2Package FileURL matches the hash-path regex
S3_ArchiveNeverReqOnly after a restart sets the backend by hand

Conveyor has no request_condition at all, which makes it the default. The Conveyor guide covers what happens once a request lands there. The Package File condition is a single regex:

req.url ~ "^/packages/[a-f0-9]{2}/[a-f0-9]{2}/[a-f0-9]{60}/"

Two hex characters, two more, then sixty: that is a SHA-256 digest split three ways. Any URL of that shape is a real distribution file and goes to Backblaze. Anything else is an old-style URL and goes to conveyor. The routing decision is the file name.

S3_Archive is gated on a condition that cannot be true:

statement = "req.http.Fastly-Client-IP == \"127.0.0.1\" &&
             req.http.Fastly-Client-IP != \"127.0.0.1\""

Fastly's model requires every backend to have a condition, and there is no way to write “never”, so the module writes a contradiction. The backend is fully configured and unreachable by the normal path. It is reached by VCL setting req.backend = F_S3_Archive after a restart. The Conveyor guide names this idiom; here is the machinery it drives.

GET /packages/… vcl_recv condition "Package File" ^/packages/[a-f0-9]{2}/[a-f0-9]{2}/[a-f0-9]{60}/ match no match backend B2 Backblaze, shield iad-va-us SigV4 signed in vcl_miss backend Conveyor default, no condition old URLs, redirects vcl_fetch: status? 404 or 403, and req.restarts == 0 200 / 206 restart backend S3_Archive set by hand, condition NeverReq Fastly-Force-Shield = 1 Fallback-Backend = 1 deliver Cache-Control immutable max-age=365000000 x-amz-meta → x-pypi-file Datadog: "Storage Fallback success" if 200/206, "Storage Fallback failure" otherwise
Fig. 2. Backend selection and failover in file-hosting. Only the left-hand path involves a restart, and every restart is logged, because a fallback fetch means the primary store missed a file it should have had.

The failover trigger carries an apology in the comment above it:

# Note 403, which sucks but is apparently the response you get for a
# non existent file from B2's S3 interop  when auth'd???

An S3-compatible service that returns 403 for a missing object rather than 404 forces the VCL to treat both codes as “try the archive”. That has a cost. A genuine authorisation failure at B2, from an expired or revoked key, looks exactly like a missing file, and the whole service quietly fails over to Glacier while continuing to serve. The two Datadog conditions exist for that reason. Storage Fallback success and Storage Fallback failure both key on req.restarts > 0 && req.http.Fallback-Backend == "1", so a rise in either is the signal that something is wrong with the primary store.

The archive bucket is built for the case where it is almost never read. It has object_lock_enabled = true and one lifecycle rule:

rule {
  id = "to-cold-storage"
  transition {
    days          = 1
    storage_class = "GLACIER_IR"
  }
  status = "Enabled"
}

One day, then Glacier Instant Retrieval. Instant Retrieval is the Glacier tier that still answers in milliseconds, which is the only tier that would work here: a failover that took minutes to restore a file would be an outage rather than a fallback. The Warehouse overview covers why Backblaze is primary in the first place.

Signing requests in VCL

Neither bucket is public. Both are read with SigV4 signatures computed at the edge, in VCL, twice over: once in vcl_miss and once again in vcl_pass, because those are separate subroutines and the code cannot be shared between them. Each copy declares its locals, builds the canonical request, builds the string to sign, and calls one function:

set var.signature = digest.awsv4_hmac(
  var.B2SecretKey, var.dateStamp, var.B2Region, "s3", var.stringToSign
);

The result becomes an Authorization header, and four client headers are dropped before the request goes upstream: Accept, Accept-Language, User-Agent, and Fastly-Client-IP. Anything not in SignedHeaders is a liability, and the object store has no business seeing who asked. There is a related fix a few lines earlier in vcl_recv, filed as issue 219 in the repository: x-amz-cf-id is stripped unconditionally, because a client that sends it makes an S3-compatible backend fail when that header is not part of the signature.

Four near-identical blocks of signing code, roughly 250 lines in total, exist because VCL has no functions. That is not sloppiness. It is the shape of the language.

Checkpoint 1 · modules, zones, backends

Q1pythonhosted.org serves every package download and is not DNSSEC-signed, while pypi.org is. What does the Terraform actually show?

C. That's the whole verifiable fact: the KMS key, the aws_route53_key_signing_key, and the aws_route53_hosted_zone_dnssec resource all name aws_route53_zone.primary. The user-content zone gets a zone resource and a CAA record, nothing more. Why the maintainers chose that split isn't written down anywhere in the repository. It's plausible that a signed zone fails closed, and a validation failure on the file-serving zone would stop every install with no fallback, but that's this guide's inference, not a documented decision.

Q2A request for /packages/ab/cd/<60 hex>/foo-1.0.tar.gz returns 403 from Backblaze. What does the edge do?

B. vcl_fetch treats 404 and 403 from F_B2 the same way, because B2's S3 interop answers 403 for a missing object. The restart sets req.backend = F_S3_Archive, re-enables shielding with Fastly-Force-Shield, and sets Fallback-Backend so the two Datadog conditions can fire. C is the trap: Conveyor is the default backend for the service, but the restart path names the archive backend explicitly.

Q3Why does the S3_Archive backend carry a condition that compares a header to a value and to its negation?

D. The condition is named NeverReq and it is unsatisfiable by construction. Its job is to keep the backend configured, healthy, and out of automatic selection, so that the only way to reach it is VCL setting req.backend by hand. The same trick appears as a RESPONSE condition named Never, which registers the Linehaul log sink without ever firing it automatically.

The edge, on its own

Behaviour you will not find in warehouse

Several things PyPI does are implemented once, at the edge, and appear nowhere in the application. Reading warehouse alone will not show them to you.

  1. 01

    Segmented caching, except for metadata

    file-hosting/vcl/files.vcl · vcl_recv

    Package URLs turn on req.enable_segmented_caching, which lets Fastly cache a large file in pieces and serve range requests from them. Then one nested condition turns it back off for any URL ending in .metadata. Those are the PEP 658 metadata files, a few kilobytes each, and the point of PEP 658 is that a resolver can read a wheel's metadata without downloading the wheel. Splitting a two-kilobyte object into segments costs more than it saves. The same file gets special handling again in vcl_fetch: metadata is stored uncompressed, so the edge sets Vary: Accept-Encoding and compresses on the fly with gzip or brotli depending on what the client asked for.

  2. 02

    Immutability, asserted by the CDN

    file-hosting/vcl/files.vcl · vcl_fetch

    Object storage does not set cache headers on PyPI's files. The edge does, on any 200 or 206 for a package URL:

    set beresp.http.Cache-Control = "max-age=365000000, immutable, public";
    set beresp.ttl = 365000000s;

    That is about eleven and a half years. It is safe only because a package file is content-addressed by the hash in its own URL and PyPI does not permit re-uploading a deleted filename. The rule that makes this cacheable forever is a policy in the application; the caching itself is a decision made here.

  3. 03

    One hardcoded URL with a TTL of zero

    file-hosting/vcl/files.vcl · vcl_fetch

    Directly inside that block sits an exception for a single literal path: a hash of sixty a characters, ending in aaaaaaaaa-0.0.0.tar.gz. It gets max-age=0 and beresp.ttl = 0s. It is a probe. Without it there is no way to exercise the fetch path end to end without permanently caching the result at every point of presence for a decade.

  4. 04

    Conditional request headers, deleted so logging works

    file-hosting/vcl/files.vcl · vcl_recv

    On edge requests for package URLs, If-None-Match and If-Modified-Since are unset. The comment explains why: Fastly strips most response headers before vcl_deliver runs on a 304, so a 304 cannot be logged with the file metadata attached. Removing the headers that make a 304 possible guarantees a 200 with a full body. A measurable amount of bandwidth is being spent to keep download counts correct, and the comment says plainly that the code can be deleted if Fastly ever exposes the real headers on a 304.

  5. 05

    An admin override header

    file-hosting/vcl/files.vcl · vcl_recv

    A recv snippet injects var.X-PyPI-Admin from a Terraform variable. If a request carries a matching X-PyPI-Admin header, then X-Force-Backend: PRIMARY pins the request to B2 and X-Force-Backend: ARCHIVE pins it to the archive, and either way the request is a pass, so the cache is skipped in both directions. It is the tool for answering “is the file actually in the archive” without waiting for a failover.

  6. 06

    The header rename that feeds the download dataset

    file-hosting/vcl/files.vcl · vcl_deliver, vcl_log

    Object storage returns PyPI's own metadata as x-amz-meta-project, x-amz-meta-version, x-amz-meta-package-type, and x-amz-meta-python-version. On edge responses, the VCL copies each into an x-pypi-file-* header and unsets the original, along with x-guploader-uploadid and x-goog-storage-class. The renamed headers are then interpolated into the log line that goes to Google Cloud Storage, and that log line is the input format described in the Linehaul guide. Every row in the public download dataset starts as a string built here.

    One guard matters. The log call is wrapped in if (!segmented_caching.is_inner_req), so the per-segment fetches of a large file do not each count as a download.

  7. 07

    Content negotiation for the Simple API

    warehouse/vcl/main.vcl · vcl_recv, vcl_deliver

    PEP 691 negotiation happens at the edge. A ?format= query parameter wins outright; otherwise accept.media_lookup resolves the Accept header, with a comment several paragraphs long explaining the function's constraints and the text/invalid sentinel used to work around them. The two latest aliases are collapsed into their v1 equivalents and text/html is rewritten to application/vnd.pypi.simple.v1+html, which keeps the cache from holding four copies of one index page. The real requested type is stashed in Warehouse-Real-Content-Type and restored on the way out, so the client still sees the content type it asked for.

    The same subroutine also strips query strings from everything except a listed set of paths, sorts the surviving parameters with boltsort.sort, canonicalises project names in legacy JSON URLs, and reduces the locale cookie to a single PyPI-Locale header so that Vary does not have to name the whole cookie. All of it exists to keep the cache key small.

Where to look first

If a response header, a status code, or a cache lifetime does not match what the application returns, read the VCL before reading the Python. Security headers are a clear case: Strict-Transport-Security, X-Frame-Options, X-Content-Type-Options, and a long Permissions-Policy are all set in vcl_deliver, with a comment saying they are done there “so that we always get these headers, regardless of the origin server being used.” Change the application and they will not change.

The small modules

Mail, docs, and the proxy with no DNS record

email: 160 lines that make bounces a webhook

The email module runs twice, once for pypi.org and once for test.pypi.org, both in us-west-2. It creates an SES domain identity, a DKIM configuration whose three tokens become three CNAME records, and a custom MAIL FROM domain of ses.<domain> with its own MX and SPF records. DMARC is conditional: the record is only created when the dmarc variable is non-empty, and both instantiations supply one. The policy is the same for both.

v=DMARC1; p=quarantine; pct=100; rua=<report address>; fo=1; adkim=s; aspf=r

Quarantine rather than reject, applied to all mail, with strict DKIM alignment and relaxed SPF alignment. The two environments send their aggregate reports to different processors.

The other half of the module is the feedback loop. An SNS topic is created with an aggressive retry policy, 100 retries with exponential backoff, and subscribed over HTTPS to a single URL: https://pypi.org/_/ses-hook/. Three aws_ses_identity_notification_topic resources then wire deliveries, bounces, and complaints into that topic, all with include_original_headers = true. Every delivery event SES produces arrives back at the application as a POST. That is how PyPI knows an address is undeliverable without polling anything. The interfaces guide covers the sending side, where SES is one implementation of a swappable interface; this is the return path.

A TODO at the bottom of the file has outlived several provider releases. SES cannot be told through Terraform to stop sending its own bounce notifications by mail, so that setting is applied by hand in the console. The issue it links to is from the era when the AWS provider was still published under terraform-providers.

gmail: 38 lines, and one misnamed resource

The smallest module with any content. Five MX records pointing at Google, and one TXT record holding the Google Workspace DKIM key. It creates nothing else, because Google Workspace is configured in Google Workspace; only the DNS side belongs here. The resource holding the Google key is named primary_amazonses_dkim_record, copied from the email module and never renamed. Harmless, and a good reminder that resource names in Terraform state are labels, not documentation.

docs-hosting: one backend, and the apex trick

Legacy user documentation, uploaded before PyPI stopped accepting it. The module builds a Fastly service with a single backend, Conveyor, a default TTL of one day, a gzip policy, and a 154-line VCL that is the general-purpose file: force TLS, strip fragments and query strings, cache 303, 307, and 308, serve stale on a 5xx, retry once on a 502 or 503, and set the same security headers as everywhere else. There is no routing logic, because there is only one place a request can go.

Its DNS is the interesting part, and it is where the apex rule earns its keep. Production docs-hosting serves pythonhosted.org, two labels, an apex, so the module writes A records with the four Fastly IPv4 addresses and an AAAA record with the four IPv6 addresses. The test instantiation serves test.pythonhosted.org, three labels, so it writes one CNAME and the count on the AAAA record evaluates to zero. Same module, same variables, different record types, decided by counting dots.

image-proxy: 34 lines and no Route 53 record at all

The smallest module in the repository, and the only one that creates no DNS. It has three variables, one Fastly service, one backend, a default_ttl of 10 seconds, an HSTS header with max-age=300, and a request setting that forces TLS. That is the whole file.

The absence is explained by the domain it is given: pypi-camo.global.ssl.fastly.net. That name is served by Fastly's own DNS, so declaring it on the service is all the configuration there is. No zone, no record, no certificate to manage. The Camo guide reads this same service from the other side and finds a proxy with almost no edge configuration in front of it. This module is why. There is no missing piece; the piece is thirty-four lines long.

Operations

The firewall, the logs, and how a change ships

A firewall Terraform configures but does not own

Three services carry a Next-Gen WAF: pypi, test-pypi, and inspector, all at ngwaf_percent_enabled = 100. The wiring is a split of ownership, and it is worth reading closely because it is unusual.

The Fastly service declares an Edge_Security dictionary and four empty dynamic snippets, one each for init, miss, pass, and deliver, the last three at priority 9000 so they run after everything else. Terraform then fills them with a comment and stops:

content         = "### Terraform managed ngwaf_config_${each.key}"
manage_snippets = false

The manage_items = false and manage_snippets = false flags are the whole point. Terraform creates the slots and hands them over. The SignalSciences platform writes the real VCL into them, and the sigsci_edge_deployment_service_backend resource keeps that deployment synchronised against fastly_service_vcl.pypi.active_version. Change the Fastly service and the WAF re-syncs; change the WAF and Terraform does not notice, by design. The pypi service carries the same pattern once more in a lifecycle block that ignores changes to product_enablement, which is another setting managed outside this repository.

A leftover comment above the duplicated sigsci provider block in ngwaf.tf reads “if i dont add this it errors: Provider configuration not present”. The provider is already configured in config.tf, and it is configured again inside the module.

Five log sinks, four destinations, one game

The warehouse service declares five logging blocks that resolve to four actual destination systems. Two S3 sinks write to psf-fastly-logs-eu-west-1, the same bucket, one for all traffic and one for 5xx responses only, with a longer format that adds timing, fastly_info.state, restart count, and backend. A Datadog sink fires on the same 5xx condition with a JSON payload carrying geography, POP, protocol, TLS version, and the origin status carried through from the shield. The GCS sink is Linehaul, and it fires on a condition that reads like a specification of what counts as a download of an index page:

var.Ship-Logs-To-Line-Haul && !req.http.Fastly-FF &&
req.url.path ~ "^/simple/.+/" && req.request == "GET" &&
http_status_matches(resp.status, "200,304")

Not a shield request, a real project page, a GET, and either fresh or not-modified. Production has linehaul_enabled = true; TestPyPI has it set to false, and its GCS bucket is the staging one, so the switch is a boolean in main.tf rather than a separate configuration.

The fifth sink, an HTTPS logging block, is a curiosity. A Fastly-Top-POPS init snippet, enabled only in production, samples 3 percent of requests in thirteen named data centres and ships a short line to a Fastly Labs endpoint over it. The snippet says what it is for: a game shown at events to help people understand edge networks. It carries its own removal instructions and an email address to notify. It is the only logging destination in the repository that serves no operational purpose, and it is documented better than most code that does.

The worker queues

warehouse/worker.tf is 60 lines and creates the infrastructure half of the background-task story. Three SQS queues per environment, named from the module's name variable: pypi-worker, pypi-worker-default, and pypi-worker-malware. All three get visibility_timeout_seconds = 900, with a comment explaining the consequence: a task not finished in fifteen minutes becomes visible again and another worker picks it up. One IAM policy grants read and write on exactly those three ARNs, plus a global sqs:ListQueues.

The Warehouse overview covers the Celery side. This is where the queues those tasks land in are declared, and the fifteen-minute timeout is the number that decides whether a slow task runs twice.

How a change ships

The backend is Terraform Cloud, organisation psf, workspace pypi-infra, declared in a cloud block in config.tf. That is where state lives and where an apply runs, which is why the local Makefile stops at plan. Provider versions are pinned at the top level to AWS 5.13.0, Fastly 5.13.0, and sigsci 3.3.0, with a mise.toml pinning Terraform itself. The inspector module's versions.tf carries commented-out newer pins and one line of explanation: “vrsions in rest of repo need upgraded first to use these version specifiers”.

Two Fastly services carry the same comment above activate = true:

“Set to false for spicy changes”

pypi/infra → terraform/warehouse/main.tf, terraform/docs-hosting/main.tf

Fastly versions a service configuration. Setting activate = false uploads a new version without making it live, so it can be inspected, diffed, and activated by hand. It is the only safety mechanism in the repository, it is a one-word edit, and it is documented in a joke.

Checkpoint 2 · edge behaviour and operations

Q4You want to know why a .metadata file is served with Vary: Accept-Encoding when the object in storage has no such header. Where do you look?

A. Metadata files are stored uncompressed. The edge adds Vary: Accept-Encoding whether or not it compresses, then sets beresp.gzip or beresp.brotli based on what the client asked for. The same VCL also turns segmented caching off for these files, which is on for every other package URL. None of this is visible from the application, and none of it is a property of the stored object.

Q5Terraform creates four dynamic snippets on the pypi service whose content is a single comment, with manage_snippets = false. What is happening?

B. This is split ownership written into the configuration. manage_snippets = false and manage_items = false tell Terraform to create the resource and then stop reconciling its contents, so the SignalSciences platform can own them. The sigsci_edge_deployment_service_backend resource keeps the deployment tracking the Fastly service's active_version. If you diff the live snippets against the repository they will not match, and that is correct.

Q6pip search returns a readable error rather than an HTTP failure. Which layer produces it?

C. vcl_recv matches <methodName>search</methodName> in req.body on /pypi with an XML content type and raises error 666, which vcl_error answers with status 200 and a fault body carrying faultCode -32500. Four more methods are killed the same way. B is close but wrong for this case: response objects are used for the Bandersnatch and blocklist 403s, which need no protocol-specific body. An XML-RPC client needs a fault, so the edge writes one.

The map

Where everything is

ThingWhereWhy you'd look
Which modules exist, and how often terraform/main.tf Thirteen module blocks, the two locals maps, and every production value that is not a secret
Zones, DNSSEC, CAA, SPF terraform/dns/main.tf The delegation set, which zone is signed, and the three authorised mail senders
Edge behaviour for the web site and Simple API terraform/warehouse/vcl/main.vcl PEP 691 negotiation, query stripping, dead XML-RPC methods, blocked user-agents, security headers
Domains, blocklist, healthcheck, log sinks terraform/warehouse/main.tf The alias list whose order matters, the hashed-IP dictionary, the four logging destinations
Backend selection for file downloads terraform/file-hosting/fastly-service.tf The three backends and the conditions that pick them, including NeverReq
Failover, signing, caching, download logging terraform/file-hosting/vcl/files.vcl The 403 quirk, the SigV4 blocks, the immutable TTL, the x-pypi-file-* rename
Buckets and the archive lifecycle terraform/file-hosting/main.tf The B2 bucket and read key, the S3 archive bucket, the one-day Glacier IR transition
Transactional mail and bounce handling terraform/email/main.tf SES identity, DKIM, DMARC, and the SNS topic that POSTs delivery events to the application
The camo edge service terraform/image-proxy/main.tf Thirty-four lines and no DNS. See the Camo field guide
WAF wiring terraform/warehouse/ngwaf.tf, terraform/inspector/ngwaf.tf The dictionary and snippet slots Terraform creates and does not fill
Background-task queues terraform/warehouse/worker.tf Three SQS queues and the fifteen-minute visibility timeout
State and provider pins terraform/config.tf, terraform/versions.tf, terraform/mise.toml The Terraform Cloud workspace, the sensitive variable list, the pinned provider versions

If you're going to work on this

Five habits.

  1. When a response does not match what the application returns, read the VCL first. Status codes, cache lifetimes, security headers, and content types are all rewritten at the edge, and the application has no idea.
  2. Treat list order as configuration. The first entry of extra_domains gets an XML-RPC bypass. Nothing enforces that except a comment.
  3. Before editing a Fastly service, set activate = false. The new version uploads without going live, and you can look at it before it serves anyone.
  4. When you find something unreachable, check whether it is unreachable on purpose. NeverReq and Never are how this repository says “configured, but only I decide when”. Deleting them breaks failover and download logging respectively.
  5. Expect duplication and do not tidy it. The four SigV4 blocks exist because VCL has no functions. The repeated apex_domain local exists because Terraform modules do not share locals. Both are the language, not the author.

Pyramid terms, plain English

ACL / __acl__
A list of (allow or deny, who, permission) rules a resource can define. Pyramid's authorization layer checks it before letting a request through.
config.scan()
Walks the codebase looking for decorators like @view_config and registers whatever it finds. Without a scan, a decorator alone does nothing.
directive
A custom method bolted onto Pyramid's Configurator object (via add_directive), so a plugin can offer its own verb, like config.add_periodic_task(...), instead of every caller repeating the same boilerplate.
find_service()
Asks the registry for "whatever implements this interface" and gets an object back, without the caller knowing or caring which concrete class it is.
includeme(config)
The one function every Pyramid plugin must define. config.include("some.package") calls it and hands over the Configurator, so the plugin can register its own routes, views, and services.
predicate
A condition attached to a route or view, like request_method="POST", that Pyramid checks before it will match that route or view to an incoming request.
register_service_factory()
The other half of find_service(): tells the registry "when someone asks for this interface, call this factory to build the answer."
registry
Pyramid's central lookup table, assembled once at startup, holding every route, view, service, and setting the running app knows about.
resource
In traversal, an object reachable by walking a chain of __getitem__ calls from the root. The traversal equivalent of a URL path segment.
root factory
The function that builds the very first resource in a traversal chain, given a request. Every other resource is reached by indexing into it.
subscriber
A function registered to run automatically whenever a specific event fires, like a transaction commit, without the code that fires the event knowing the subscriber exists.
traversal
Pyramid's alternative to matching a URL against a pattern: walk a resource tree one path segment at a time via __getitem__, and whatever object you land on becomes the view's context.
tween
Short for "between." Middleware that sits between Pyramid's router and the view, in an explicit, orderable chain, wrapping the whole request/response cycle.
venusian
The library config.scan() actually uses. A decorator like @view_config just leaves a note for venusian to find later; it does nothing by itself until scanned.
view deriver
A wrapper Pyramid can apply to every view during registration, used for cross-cutting behavior (like injecting a metrics call) without every view author remembering to add it by hand.