Camo: One Secret, Two Codebases

The problem

An img tag you did not write

A project README is Markdown or reStructuredText that the uploader controls, and PyPI renders it into HTML on the project page. That HTML contains <img> tags pointing at hosts PyPI has never heard of: a badge service, someone's blog, a raw file on a git forge, a screenshot on an S3 bucket that stopped being paid for in 2019.

Serve those tags as written and two things go wrong at once.

The first is privacy. Every visitor to a project page makes a request straight to that third-party host, which learns the visitor's IP address, their browser, and (from the Referer) which PyPI project they were looking at. A badge on a popular project becomes a decent traffic-analysis feed for whoever runs the badge.

The second is mixed content. pypi.org is HTTPS. A README from 2014 links an image over plain http://. Browsers block that image or degrade the lock icon, and either way the page is worse.

The fix is a camo server. The upstream project describes itself plainly:

“A camo server is a special type of image proxy that proxies non-secure images over SSL/TLS, in order to prevent mixed content warnings on secure pages. The server works in conjunction with back-end code that rewrites image URLs and signs them with an HMAC.”

cactus/go-camo → README.adoc

Read the last sentence twice. The server is half of the design. The other half is “back-end code that rewrites image URLs,” and at PyPI that code lives in Warehouse, in a different language, in a different repository, on a different deploy schedule. The two halves agree on one shared secret and one URL format, and on nothing else.

Where the code came from

GitHub wrote the original camo in Ruby. Eli Janssen rewrote it in Go as cactus/go-camo. PyPI forked that. The fork's go.mod has never been renamed:

module github.com/cactus/go-camo/v2

Every import path inside pypi/camo still says cactus. That is not sloppiness, it is what a fork you intend to keep merging from upstream should look like. It does mean that reading the repository tells you almost nothing about what PyPI changed, which is the subject of a later section.

Warehouse's half

camoify: parse the HTML, rewrite every src

Warehouse does the rewriting in a Jinja filter. warehouse/filters.py defines camoify, which takes rendered README HTML, parses it into a DOM with html5lib, walks every img element, replaces the src, and serializes the tree back out:

for element in dom.getElementsByTagName("img"):
    src = element.getAttribute("src")
    if src:
        element.setAttribute("src", request.camo_url(src))

Note what is missing: a scheme test. The comment above the loop says the filter replaces “inline images that don't point to HTTPS,” but the code rewrites every image, HTTPS ones included. The comment describes an older intent; the behaviour is the safer one, since an HTTPS image still leaks the visitor's IP to whoever hosts it. Trust the loop, not the comment.

The filter is registered by name in warehouse/config.py, alongside the rest of Warehouse's Jinja environment, and three templates use it: the project description, the sponsor logos in the footer, and the sponsor grid on the sponsors page. Sponsors are the surprise. Their logos are PyPI's own content and still go through the proxy, because the pipe is cheaper to keep uniform than to special-case.

Avatars go through it too. warehouse/utils/gravatar.py builds a secure.gravatar.com URL from an MD5 of the email address and then hands it straight to the same request method:

return request.camo_url("?".join([url, urllib.parse.urlencode(params)]))

That URL is already HTTPS. It is proxied anyway, so that loading a PyPI page never tells Gravatar who is reading it.

The request method and the two settings

The signing itself is a Pyramid request method rather than a filter, added in the same module that registers the filters:

config.add_request_method(_camo_url, name="camo_url")

Anything with a request in hand can call request.camo_url(url). Two settings feed it, both read in warehouse/config.py from environment variables: CAMO_URL becomes camo.url, and CAMO_KEY becomes camo.key.

CAMO_URL is a format string, not a plain URL. _camo_url calls .format(request=request) on it, so the development value can follow whatever host the developer is using:

CAMO_URL={request.scheme}://{request.domain}:9000/
CAMO_KEY=insecurecamokey

That key matters in the next section. It is the whole of the contract between the two codebases.

The wire format

A digest, a slash, and a URL in hex

Here is _camo_url, near enough complete:

camo_url = request.registry.settings["camo.url"].format(request=request)
camo_key = request.registry.settings["camo.key"].encode("utf8")
url = url.encode("utf8")

path = "/".join(
    [
        hmac.new(camo_key, url, digestmod="sha1").hexdigest(),
        binascii.hexlify(url).decode("utf8"),
    ]
)

return urllib.parse.urljoin(camo_url, path)

Two hex strings joined by a slash. The first is an HMAC-SHA1 of the raw URL bytes under the shared key. The second is the same URL bytes, hex-encoded so they survive being a path segment. Nothing is escaped, nothing is base64, and no query string is involved.

Take the development key and a made-up image:

key: insecurecamokey
url: http://example.com/logo.png

a884a722bd10b5ae8a1bee587c87da7ad92d2de2
687474703a2f2f6578616d706c652e636f6d2f6c6f676f2e706e67

→ http://localhost:9000/a884a722bd10b5ae8a1bee587c87da7ad92d2de2/687474703a2f2f6578616d706c652e636f6d2f6c6f676f2e706e67

The digest is 40 hex characters because SHA-1 is 20 bytes. Remember that number.

The Go side, in reverse

ServeHTTP in pkg/camo/proxy.go splits the request path and takes components one and two as the signature and the encoded URL. Anything with fewer than three components is a 404 before any crypto runs. Then it calls the decoder, and a failure is a flat rejection:

sURL, ok := encoding.DecodeURL(p.config.HMACKey, sigHash, encodedURL)
if !ok {
    http.Error(w, "Bad Signature", http.StatusForbidden)
    return
}

DecodeURL lives in pkg/camo/encoding/url.go and picks an encoding by measuring the digest: exactly 40 characters means hex, anything else means base64. go-camo supports both, because different back ends emit different ones. Warehouse only ever emits hex, so PyPI only ever exercises the first branch.

HexDecodeURL then does the obvious three steps: hex-decode the URL, hex-decode the digest, and verify. The verification is the part worth reading closely:

mac := hmac.New(sha1.New, *hmackey)
mac.Write(*urlbytes)
macSum := mac.Sum(nil)

if len(macSum) != len(*macbytes) {
    return fmt.Errorf("mismatched length")
}

if subtle.ConstantTimeCompare(macSum, *macbytes) != 1 {
    return fmt.Errorf("invalid mac")
}

subtle.ConstantTimeCompare, not bytes.Equal. A byte-by-byte comparison that returns early leaks how many leading bytes were right, and an attacker who can time enough requests can walk a forged digest into place one byte at a time. The constant-time compare takes the same time whether the first byte differs or none of them do.

Same numbers, both languages

go-camo's own test file pins the encoding with a fixed vector: key test, URL http://golang.org/doc/gopher/frontpage.png, digest 0f6def1cb147b0e84f39cbddc5ea10c80253a6f3. Feed the same two inputs to Python's hmac.new(..., digestmod="sha1") and you get that string back, character for character. The scheme is HMAC-SHA1 and hex, and both are boring enough that a Go standard library and a Python standard library agree without anyone having written glue.

SHA-1 in 2026 deserves a sentence. The collision attacks that killed SHA-1 for certificates are attacks on the bare hash function. HMAC-SHA1 is not broken by them, and the threat here is forgery of a specific message by someone without the key, which is exactly what HMAC still resists. It would not be the choice for a new design. It is not a live problem in this one.

Why the signature exists

The alternative is an open proxy on PyPI's network

Delete the MAC check and camo becomes a service that fetches any URL you name, from inside PSF infrastructure, and streams the result back to you. That is a server-side request forgery machine with a public front door. The interesting targets are not on the public internet: cloud metadata endpoints, internal admin interfaces, anything bound to a private range that trusts its network position.

The signature stops the first half of that: nobody can hand camo an arbitrary URL and have it fetched, because Camo will only act on a URL that Warehouse already decided to sign, and Warehouse signs whatever URL appears in a README that someone uploaded. Everything else stops at 403 Bad Signature before a socket is opened.

It does not stop the second half. A README is uploader-controlled content, so a signed URL can still point at 169.254.169.254 or 127.0.0.1, because Warehouse signs whatever it's given without checking where it resolves. That is what the rest of this section is for: the Go proxy assumes the signing side will eventually sign something regrettable, and checks the address itself before it dials.

What checkURL rejects

checkURL runs on the decoded URL before any request is made. It requires the scheme to be http or https, so file:// and friends never get near the fetcher. It requires a non-empty hostname that survives IDNA normalization. It rejects localhost and localdomain by name through a small trie matcher. It runs whatever filter ruleset the operator configured.

Where the IP check actually happens

isRejectedIP in pkg/camo/misc.go rejects any address that is not global unicast, then checks two explicit lists. IPv4: 127.0.0.0/8, 169.254.0.0/16, and the three RFC 1918 ranges. IPv6 is longer, and includes ::1/128, the IPv4-mapped block ::ffff:0:0/96, unique local addresses fc00::/7, link-local fe80::/10, and the documentation prefix 2001:db8::/32.

The placement is the clever part. That check does not run when the URL is parsed. It runs inside the dialer's Control hook, on the address the connection is actually about to use. The source comment explains why, and it is worth quoting:

“Move ip filtering to dial.control, this avoids cases where an adversary may return an unblocked ip on name resolution the first time, and a blocked ip the second time.”

pkg/camo/proxy.go

That is DNS rebinding, and checking at parse time would not catch it. A hostname that resolves to a public address when validated and a private one when dialed passes the first check and fails the second, because the second happens after resolution and before the packet.

Redirects get the same treatment

An origin that returns 302 Found could otherwise walk the proxy anywhere it liked. The HTTP client sets CheckRedirect to re-run the full checkURL on every hop and to cap the chain at MaxRedirects, which defaults to three. A rejected hop returns Bad redirect, not a fetch.

Headers are allowlisted in both directions

pkg/camo/vars.go holds two maps, ValidReqHeaders and ValidRespHeaders. Only keys present and true are copied through, so the default for anything unlisted is to drop it. The entries set to false are the documentation:

X-Forwarded-For → false

Not blindly forwarded. Passing it would hand the origin the visitor IP that the proxy exists to hide.

Accept-Encoding → false

Images other than SVG do not compress usefully, so the proxy declines to negotiate it.

Server → false (response)

Replaced with camo's own server name rather than the origin's.

Range → true

Kept in both directions, with Accept-Ranges and Content-Range, because Safari needs byte-range requests to play video.

On top of that, the proxy sends Accept: image/* and checks the response media type against the same glob filter, so an origin that answers an image request with HTML gets rejected rather than relayed. Video and audio are opt-in flags that PyPI does not set.

One more, cheap and easy to miss: if an incoming request carries a Via header equal to the proxy's own server name, camo returns 404. That is loop protection, and it is why the Procfile bothers to pass --server-name.

The fork, audited

896 commits, eighteen of them PyPI's

The repository carries 896 commits, and reading that number as PyPI activity is the standard mistake. Almost all of it is upstream history that came along with the fork. The honest measure is the range from the merge base to HEAD, which is eighteen commits: eight merges pulling upstream forward, and ten that PyPI wrote.

Five of those ten change how the service behaves. They are the entire PyPI-specific surface.

1. 1

   Make it run under Cabotage

   June 2022 · Procfile, Dockerfile, pkg/router/router.go

   Rewrote the Procfile to bind a Unix socket, adjusted the container build, and added a /healthcheck route to the router so a load balancer can probe the process without signing anything.

2. 2

   Read max-size from the environment

   June 2022 · cmd/go-camo/main.go

   A twelve-factor accommodation. The size cap becomes $GOCAMO_MAXSIZE rather than a value baked into the command line.

3. 3

   Redirect instead of erroring when max-size is exceeded

   June 2022 · cmd/go-camo/main.go, pkg/camo/proxy.go

   Nine lines. When the origin's Content-Length exceeds the cap and MaxSizeRedirect is set, camo answers 302 to that URL instead of 404 Content length exceeded. PyPI wanted a visible “this image is too large” placeholder rather than a broken-image icon.

4. 4

   Set the server name

   2023 · Procfile

   Passes --server-name=https://github.com/pypi/camo. It goes into the Server response header and into the Via loop check described above.

5. 5

   Add Access-Control-Allow-Origin

   July 2024 · Procfile

   The most recent commit on the fork. Appends --header="Access-Control-Allow-Origin: $ALLOW_ORIGIN", using a flag that already existed upstream. No Go code changed.

The max-size-redirect provenance, precisely

PyPI wrote --max-size-redirect. Ee Durbin committed it to the fork in June 2022 on a branch named max_size_redirect, and it lived only there for two years.

Then it went home. The same author opened it as pull request 80 against cactus/go-camo, upstream merged it in June 2024, and it shipped in go-camo v2.5.0 later that month. So the flag is a PyPI-authored feature and a stock upstream feature, and has been both since 2024. If you are diffing the fork today looking for it as evidence of divergence, you will not find it there.

One artifact of that history survives in the fork. Its CLI struct declares the field with the old go-flags tag style:

MaxSizeRedirect string `long:"max-size-redirect" description:"URL to redirect when max-size is exceeded"`

Every neighbouring field uses kong's name: and help: tags, because upstream switched argument parsers underneath the fork and the merges never reconciled this one line. It still works: kong ignores tags it does not know and derives --max-size-redirect from the field name anyway. Upstream's own copy of the same field uses the modern tags.

Deployment

One Procfile line is the whole configuration

Camo has no config file. Everything it needs comes from four flags and one environment variable, and the flags are all in one place:

web: /bin/go-camo --socket-listen=/var/run/cabotage/cabotage.sock --server-name=https://github.com/pypi/camo --max-size=$GOCAMO_MAXSIZE --max-size-redirect=$GOCAMO_MAXSIZEREDIRECT --header="Access-Control-Allow-Origin: $ALLOW_ORIGIN"

That is the file. One line, no second process type, no worker.

--socket-listen is the Cabotage convention: a web process binds a Unix socket at a known path rather than a TCP port, and the platform's sidecar handles everything above it. go-camo's flag validation accepts a socket in place of --listen or --ssl-listen, so this is a supported mode upstream, not a patch.

Dockerfile.cabotage is an ordinary two-stage build: golang:alpine compiles with make build and the release version and git hash passed in as build arguments, then an alpine:latest stage copies the single binary out. No Go toolchain and no source ship in the running image.

The key never appears in the Procfile

main.go reads GOCAMO_HMAC from the environment, lets an explicit --key flag override it, and refuses to start if the result is empty:

if len(config.HMACKey) == 0 {
    mlog.Fatal("HMAC key required")
}

The Procfile passes no key, so production supplies it as an environment variable. Its value must equal Warehouse's CAMO_KEY, character for character. Nothing checks that, and nothing can. Rotate one and not the other and every README image on PyPI turns into a 403 at once, with no error anywhere except the proxy's response codes.

Development makes the pairing visible. Warehouse's dev/environment sets CAMO_KEY=insecurecamokey; its docker-compose.yml starts a camo container with GOCAMO_HMAC: "insecurecamokey". Same string, two files, two languages.

The front door

34 lines of Terraform, instantiated twice

Camo does not face the internet. A Fastly service does, and it is defined in pypi/infra at terraform/image-proxy/main.tf. The whole module is 34 lines plus an eight-line versions.tf: three variables, one fastly_service_vcl resource, one domain, one backend, one response header, one request setting.

The top-level main.tf instantiates it twice, once for each side of the world:

Neither domain is a PyPI hostname. Both are Fastly's own shared-certificate domains, which is the cheap way to get TLS on a service nobody types by hand. The CAMO_URL Warehouse is configured with points at one of them, and users only ever see it inside an img tag.

Three settings carry the entire edge policy:

default_ttl = 10

Ten seconds. It applies only when the origin sends no cache directive of its own, and camo passes through the origin's Cache-Control, Expires, Etag, and Last-Modified. A well-behaved image host keeps its own long TTL; a silent one gets cached for ten seconds and re-fetched.

Strict-Transport-Security: max-age=300

Set as a response header. Five minutes is short as HSTS goes, which is what you want on a domain that is not yours.

force_ssl = true

A request setting that redirects plain HTTP to HTTPS at the edge. The proxy that exists to fix mixed content declines to be reached over cleartext itself.

There is no custom VCL. That is what makes this module unusual rather than merely small. The warehouse module carries 450 lines of Terraform and a 615-line main.vcl; file-hosting carries 390 lines and a 618-line files.vcl. Camo's front door has none, because there is nothing to decide at the edge: every request is either a valid signed path or it is not, and only the proxy can tell which.

It is the smallest of the four Fastly modules in the repository. It is not the smallest module overall, which is gmail at 38 lines of MX and DKIM records that create no service at all. The full Terraform layer, VCL included, is entry No. 07 of this series.

The map

Where everything is

If you're going to work on this

Five habits.

1. Change the URL format on one side and you have broken the other. There is no version negotiation, no handshake, and no error that says “format mismatch.” There is only 403.
2. Before you attribute a feature to PyPI, check whether upstream has it. --max-size-redirect started here and shipped upstream two years later, and there may be more of that.
3. Diff against upstream rather than reading the log. git log <upstream>/master..HEAD turns 896 commits into eighteen, and eighteen into five that matter.
4. When you touch the header allowlists, remember which direction each map governs. X-Forwarded-For being false in ValidReqHeaders is a privacy control, not an oversight, and flipping it would undo the reason camo exists.
5. Merge commits can revert deliberate ones. The USER nobody line is the example in this repository; assume it is not the only one and check the working tree rather than the history when you need to know what runs.

Next in the series is conveyor, the redirector. Like camo, it has no end-user-facing hostname of its own. Unlike camo, what decides when to reach it is 600 lines of VCL.