Swappable Backends Field Guide

The trick

One call, six possible objects

The view that accepts every upload to PyPI puts the file away in two statements. Here they are, from warehouse/forklift/legacy.py:

storage = request.find_service(IFileStorage, name="archive")
storage.store(
    file_.path,
    os.path.join(tmpdir, filename),
    meta={
        "project": file_.release.project.normalized_name,
        "version": file_.release.version,
        "package-type": file_.packagetype,
        "python-version": file_.python_version,
    },
)

On a laptop, store copies bytes into a directory inside a container. In production it uploads an object to a bucket. The view does not know which happened, and there is nothing in those two statements that could tell it. IFileStorage is not a class you can instantiate. It has no code in it at all. It is a label, and something else decided at startup what the label points at.

Six classes in Warehouse claim that label. Two write to local disk, two talk to S3, one talks to Google Cloud Storage, and one talks to Backblaze B2. Any of them can be the answer, and picking is not the view's problem.

The obvious alternative would be a branch on the environment. Warehouse compares the running environment against a constant in exactly eleven places across the whole tree: a block of development-only settings, the debug toolbar include, two Content Security Policy rules, log formatting, Celery's broker TLS flag, one API detail, and four CLI commands that refuse to run outside development. None of those eleven choose a storage backend, a mail transport, or a rate limiter. There is no such comparison anywhere in warehouse/packaging/services.py, warehouse/email/services.py, or warehouse/rate_limiting/.

The choice runs the other way. The environment hands the application a dotted path, and the application imports whatever is on the end of it. This guide follows one interface, IFileStorage, from an environment variable through to a running object, then makes two shorter passes over email and rate limiting to show where the same pattern bends.

Two pieces of machinery sit underneath and are not re-explained here. pyramid_services supplies config.register_service_factory and request.find_service, and Pyramid's includeme convention is how each part of Warehouse gets a chance to register anything. Both are the subject of No. 02, Warehouse's Plugin Architecture. Read that first if config.include(".packaging") does not yet mean anything to you.

Vocabulary

An interface is a class with nothing in it

Warehouse gets its interfaces from zope.interface. Here is the whole of the storage contract, from warehouse/packaging/interfaces.py, trimmed only of docstring prose:

class IGenericFileStorage(Interface):
    def create_service(context, request):
        """Create the service, given the context and request..."""

    def get(path: str):
        """Return a file like object that can be read..."""

    def get_metadata(path: str):
        """Return a dictionary containing any user-created metadata..."""

    def get_checksum(path: str):
        """Return the md5 digest of the file at a given path..."""

    def store(path: str, file_path, *, meta=None):
        """Save the file located at file_path to the file storage..."""


class IFileStorage(IGenericFileStorage):
    pass


class ISimpleStorage(IGenericFileStorage):
    pass

Read the method signatures again. There is no self. There are no bodies. Nothing here is ever instantiated, subclassed for behaviour, or called. If you put a print in one of these methods it would never run.

The declaration does two jobs. It writes down the contract in one screen, which is the only place a reader can see what all six storage classes owe each other. And it produces an importable, hashable Python object, which makes it usable as a key. That second job is the load-bearing one.

Look at the two subclasses at the bottom. IFileStorage and ISimpleStorage add nothing. They have the same five methods and the same meaning for each. They exist because Warehouse needs to look up the store for distribution files separately from the store for the Simple API pages, and a registry keyed by object identity cannot tell two uses apart if they share a key. An empty subclass is the cheapest way to mint a second key with the same contract. IDocsStorage, in the same file, is a genuinely different contract: two methods, one of which is remove_by_prefix.

@implementer is a claim, not a check

On the implementation side the decorator looks like this:

@implementer(IFileStorage)
class LocalFileStorage(GenericLocalBlobStorage):
    @classmethod
    def create_service(cls, context, request):
        return cls(request.registry.settings["files.path"])

@implementer records a declaration on the class. It does not compare method names against the interface, and it will happily decorate a class that implements none of the five methods. Import succeeds either way.

The check exists, but it lives in the test suite. zope.interface.verify.verifyClass compares a class against an interface and raises if a method is missing or its signature does not fit. Warehouse calls it at 47 sites, covering 23 distinct interfaces. So the guarantee is real and it is enforced at the moment you would want it enforced, which is before merge, not at every import in production. The interfaces that no test verifies are worth knowing about: IRateLimiter is one of them, and it comes up again in section six.

The implementations

Six ways to store a file

All six live in warehouse/packaging/services.py. Each is a thin subclass of a shared generic base, and each contributes exactly one thing of its own: a create_service classmethod that knows which settings keys to read. The table adds a seventh row, LocalSimpleStorage, because it shows the pattern at its thinnest: same base, same methods, different settings key, different interface.

create_service is the factory, and it runs late

Every one of these classmethods takes (cls, context, request) and returns an instance. That signature is what pyramid_services wants from a factory. Nothing is built at startup: the registry stores the callable, and calls it the first time some code in a request asks for the interface. The result is cached for the rest of that request.

Late construction is what makes the cloud implementations affordable. S3FileStorage.create_service asks for another service by name, aws.session, and builds a bucket resource from it. B2FileStorage asks for b2.api. GCSFileStorage asks for gcloud.gcs and then does a live get_bucket call, wrapped in a retry that catches ServiceUnavailable. If those ran at import time, every management command and every test collection would need cloud credentials. Because they run per request and only on demand, a request that never touches a file never authenticates to anything.

The local one shouts about itself

The shared local base refuses to be quiet:

class GenericLocalBlobStorage:
    def __init__(self, base):
        # This class should not be used in production, it's trivial for it to
        # be used to read arbitrary files from the disk. ...
        warnings.warn(
            "LocalFileStorage is intended only for use in development, you "
            "should not use it in production due to the lack of safe guards "
            "for safely locating files on disk.",
            InsecureStorageWarning,
            stacklevel=2,
        )
        self.base = base

    @classmethod
    def create_service(cls, context, request):
        raise NotImplementedError

Two things to take from that. First, InsecureStorageWarning subclasses DevelopmentModeWarning, which subclasses UserWarning. Warehouse's pytest config sets filterwarnings = ["error", ...] and then adds one line ignoring DevelopmentModeWarning, so the warning is loud enough that a developer sees it and quiet enough that it does not fail a test run. Three other warnings in warehouse/utils/exceptions.py follow the same shape for the null OIDC publisher, the null integrity service, and the null OAuth provider.

Second, the base class's create_service raises. The base cannot be named in an environment variable, because service factories are stored at startup and only invoked lazily, on first lookup. Naming the base would resolve cleanly and fail later, on the first request that needs it, not at startup. Only the three concrete subclasses can be named, and each differs from the others by exactly one dictionary key.

Not every implementation implements everything

This is the sharpest limit of the pattern, and it is easier to see in the code than to be told:

class GenericGCSBlobStorage(GenericBlobStorage):
    def get(self, path: str):
        # Note: this is not actually used in to serve files, instead our CDN is
        # configured to connect directly to our storage bucket.
        raise NotImplementedError

    def get_metadata(self, path: str):
        raise NotImplementedError

    def get_checksum(self, path: str):
        raise NotImplementedError

GCSFileStorage passes verifyClass(IFileStorage, ...). All five method names are present with the right signatures. Three of them raise the moment you call them. The reasoning in the comment is sound: nothing in Warehouse reads a distribution back out through the storage service, because the CDN is pointed at the bucket directly, so a read path would be dead code with a credential requirement attached.

The S3 base makes the same argument and reaches the opposite conclusion. Its get carries a near-identical comment about the CDN, then implements the read anyway, translating NoSuchKey into FileNotFoundError. B2 implements all four. So among the cloud backends, one is read-write, one is read-write, and one is write-only with three landmines.

An interface guarantees that a method exists. It does not guarantee the method works. When you swap a backend, read the implementation you are swapping to, not just the interface it claims.

The reads are not hypothetical, either. reconcile_file_storages, a Celery task that runs every fifteen minutes, asks for both storages by name and calls get_checksum on each to decide whether a file needs copying from one to the other. That task is exactly the code that would break against a write-only backend.

The switch

One environment variable, parsed by shlex

Here is the line from dev/environment that decides everything above for a development checkout:

FILES_BACKEND=warehouse.packaging.services.LocalFileStorage path=/var/opt/warehouse/packages/ url=http://localhost:9001/packages/{path}

A dotted path, then keyword arguments in shell syntax. Seven lines of warehouse/config.py turn it into settings:

def maybe_set_compound(settings, base, name, envvar):
    if envvar in os.environ:
        value = shlex.split(os.environ[envvar])
        kwargs = dict(i.split("=") for i in value[1:])
        settings[f"{base}.{name}"] = value[0]
        for key, value in kwargs.items():
            settings[f"{base}.{key}"] = value

shlex.split applies the quoting rules you already know from a shell, so a value with spaces in it can be quoted and a long chain of keywords needs no escaping. The commented-out BILLING_BACKEND alternative in dev/environment carries five keyword arguments on one line. The first token becomes files.backend. Every remaining key=value becomes files.<key>. So one variable produces three settings:

That last row surprises people. files.url never reaches a storage class. routes.py reads it on its second line and uses it as the route pattern for packaging.file, plus a redirect from /packages/{path:.*}. One variable, two unrelated consumers: the class path goes to the service registry, the URL goes to the router. Change the backend and you are usually changing where files are served from as well, so the two travel together.

Fourteen backends are configured this way. The list is worth reading in one go, because it is the full inventory of what Warehouse considers swappable at deploy time:

maybe_set_compound(settings, "billing", "backend", "BILLING_BACKEND")
maybe_set_compound(settings, "files", "backend", "FILES_BACKEND")
maybe_set_compound(settings, "archive_files", "backend", "ARCHIVE_FILES_BACKEND")
maybe_set_compound(settings, "simple", "backend", "SIMPLE_BACKEND")
maybe_set_compound(settings, "docs", "backend", "DOCS_BACKEND")
maybe_set_compound(settings, "sponsorlogos", "backend", "SPONSORLOGOS_BACKEND")
maybe_set_compound(settings, "origin_cache", "backend", "ORIGIN_CACHE")
maybe_set_compound(settings, "mail", "backend", "MAIL_BACKEND")
maybe_set_compound(settings, "metrics", "backend", "METRICS_BACKEND")
maybe_set_compound(settings, "breached_emails", "backend", "BREACHED_EMAILS")
maybe_set_compound(settings, "breached_passwords", "backend", "BREACHED_PASSWORDS")
maybe_set_compound(settings, "domain_status", "backend", "DOMAIN_STATUS_BACKEND")
maybe_set_compound(settings, "github.oauth", "backend", "GITHUB_OAUTH_BACKEND")
maybe_set_compound(settings, "gitlab.oauth", "backend", "GITLAB_OAUTH_BACKEND")

Four more backends use the plain maybe_set instead: CAPTCHA_BACKEND, OIDC_BACKEND, INTEGRITY_BACKEND, and HELPDESK_BACKEND. Those take a bare dotted path and nothing else, because their implementations read no extra settings. The compound form exists only for backends that need arguments.

The dotted path becomes a class

Now the packaging plugin's includeme, the first four statements of it:

def includeme(config):
    files_storage_class = config.maybe_dotted(config.registry.settings["files.backend"])
    config.register_service_factory(
        files_storage_class.create_service, IFileStorage, name="cache"
    )

    archive_files_storage_class = config.maybe_dotted(
        config.registry.settings["archive_files.backend"]
    )
    config.register_service_factory(
        archive_files_storage_class.create_service, IFileStorage, name="archive"
    )

config.maybe_dotted is Pyramid's resolver. Given a string it imports the module and pulls the attribute; given anything else it returns it unchanged, which is how metrics can pass a class object as its default. The result is a class. Warehouse then reaches straight past it for one attribute, create_service, and hands that bound classmethod to the registry as the factory. The class itself is never stored.

Look at what those two blocks do together. IFileStorage gets registered twice, under two different names, from two different environment variables. Production runs two storage backends behind one interface at the same time: an archive that receives every upload, and a cache that the CDN serves from. reconcile_file_storages copies from one to the other. The upload view writes only to "archive". Nothing in any of that code names a vendor.

The same file registers ISimpleStorage and IDocsStorage with no name at all, from SIMPLE_BACKEND and DOCS_BACKEND. Four registrations, four environment variables, one includeme.

Second worked example

Email, where the implementations differ in capability

Storage shows the pattern at its cleanest, where every backend does the same job through a different vendor. Email shows what happens when they do not.

warehouse/email/interfaces.py is twenty-one lines and declares three methods:

class IEmailSender(Interface):
    def create_service(context, request):
        """Create the service, given the context and request..."""

    def send(recipient, message):
        """Sends an EmailMessage to the given recipient."""

    def last_sent(to, subject):
        """Determines when an email was last sent, if at all"""

The wiring is two lines at the bottom of warehouse/email/__init__.py, a file otherwise 1,193 lines long and full of message-sending functions:

email_sending_class = config.maybe_dotted(config.registry.settings["mail.backend"])
config.register_service_factory(email_sending_class.create_service, IEmailSender)

Same two moves as packaging, one interface, no name. Three classes can land on the other end.

SMTPEmailSender wraps pyramid_mailer and calls send_immediately. SESEmailSender builds a raw MIME message and calls send_raw_email on an Amazon SES client, which it gets from the aws.session service, the same one S3FileStorage uses. And ConsoleAndSMTPEmailSender subclasses the SMTP one, calls super().send(...), then prints the message to standard output with a pointer to a web inbox.

That third class is the development default:

MAIL_BACKEND=warehouse.email.services.ConsoleAndSMTPEmailSender host=mailpit port=1025 ssl=false sender=noreply@pypi.org

The keyword arguments here go to pyramid_mailer's own settings under mail., not to the sender class. mailpit is a container in docker-compose.yml listening on 1025 for SMTP and publishing its web UI on 1080, which is the URL the printed block tells you to open. So a developer gets the email three ways at once: delivered over SMTP, browsable in a fake inbox, and dumped in the application log.

last_sent is where the abstraction leaks

Compare the two implementations of the third method. SMTP:

def last_sent(self, to, subject):
    # We don't store previously sent emails, so nothing to compare against
    return None

SES:

def last_sent(self, to, subject):
    last_email = (
        self._db.query(SESEmailMessage)
        .filter(
            SESEmailMessage.to == to,
            SESEmailMessage.subject == subject,
        )
        .order_by(SESEmailMessage.created.desc())
        .first()
    )
    if last_email:
        return last_email.created
    return None

The SES sender writes an SESEmailMessage row on every send, so it can answer the question. The SMTP sender writes nothing, so it cannot, and it says so in a comment rather than pretending. This is not a vendor difference. It is a capability difference wearing the same interface.

It has a visible consequence. _send_email_to_user takes a repeat_window argument, and when one is set it does this:

sender = request.find_service(IEmailSender)
last_sent = sender.last_sent(to=email.email, subject=msg.subject)
if last_sent and (datetime.datetime.now() - last_sent) <= repeat_window:
    return "repeat-window"

In production that suppresses a duplicate. In development last_sent is always None, the condition is always false, and the suppression never fires. A developer testing a "we already emailed you about this" path locally will watch it not happen, and nothing in the log will explain why. The interface hid the difference from the caller, which is what it is for, and then the difference showed up anyway as behaviour.

Third worked example

Rate limiting, and the object that does nothing

The third variation drops the environment variable entirely. IRateLimiter declares five methods and, unlike the other two interfaces here, no create_service: test, hit, resets_in, clear, and get_window_stats. Two classes implement it, both in warehouse/rate_limiting/__init__.py. (There is no services.py in that package. The implementations sit in the package __init__ alongside the includeme.)

RateLimiter is real. It wraps a MovingWindowRateLimiter from the limits library over a storage built from the ratelimit.url setting, which points at Redis. Every public method carries the same decorator:

def _return_on_exception(rvalue, *exceptions):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(self, *args, **kwargs):
            try:
                return fn(self, *args, **kwargs)
            except exceptions as exc:
                logger.warning("Error computing rate limits", error=repr(exc))
                self._metrics.increment(
                    "warehouse.ratelimiter.error", tags=[f"call:{fn.__name__}"]
                )
                return rvalue

        return wrapper

    return deco

The return values are chosen one method at a time. test and hit return True on a RedisError, which means allow the action. clear and resets_in return None. get_window_stats returns an empty list. Warehouse fails open: if Redis is unreachable, rate limiting stops happening and uploads keep working, with a warning log and a warehouse.ratelimiter.error metric per call so somebody finds out. That is a deliberate trade, and the direction is the same one PyPI takes elsewhere. An index that rejects every upload is worse than an index that briefly forgets its limits.

Now read the other implementation:

@implementer(IRateLimiter)
class DummyRateLimiter:
    def test(self, *identifiers):
        return True

    def hit(self, *identifiers):
        return True

    def clear(self, *identifiers):
        return None

    def resets_in(self, *identifiers):
        return None

    def get_window_stats(self, *identifiers):
        return []

Five methods, five constants, and they match the fail-open values from the decorator exactly. DummyRateLimiter is the null object: an implementation whose entire content is the absence of behaviour, so that callers never need a None check. ProjectService leans on it directly, holding its limiters in a collections.defaultdict(DummyRateLimiter, ratelimiters), so asking for a limiter that was never configured returns a permissive one instead of raising KeyError.

A different registration style, for a different reason

Storage and email vary by environment, so their switch is an environment variable. Rate limiters vary by purpose: there is one for project creation by IP, another for project creation by user, and more elsewhere. All of them are the same class with different limit strings. So the registration is wrapped in a custom directive instead:

def _register_rate_limiter(config, limit_string, name):
    config.register_service_factory(
        RateLimit(limit_string, identifiers=[name]),
        IRateLimiter,
        name=name,
    )


def includeme(config):
    config.add_directive("register_rate_limiter", _register_rate_limiter)
    config.registry["ratelimiter.storage"] = storage_from_string(
        config.registry.settings["ratelimit.url"]
    )
    config.add_tween("warehouse.rate_limiting.headers.rate_limit_headers_tween_factory")

Callers then write one line each. Packaging's includeme registers two:

config.register_rate_limiter(project_create_user_limit_string, "project.create.user")
config.register_rate_limiter(project_create_ip_limit_string, "project.create.ip")

RateLimit is a small callable class, not a function, with __eq__ and __repr__ defined so tests can compare registrations by value. It also takes a limiter_class argument defaulting to RateLimiter, which is the seam: pass DummyRateLimiter and the same registration produces a limiter that permits everything.

The same move, at three different distances

Null implementations appear across Warehouse, and they are not all swapped the same way. Three examples mark out the range.

Metrics swap through an environment variable, with a twist. warehouse/metrics/__init__.py defaults to the class object itself rather than a string, which maybe_dotted passes through untouched:

metrics_class = config.maybe_dotted(
    config.registry.settings.get("metrics.backend", NullMetrics)
)
config.register_service_factory(metrics_class.create_service, IMetricsService)

So a deployment that sets nothing gets NullMetrics, whose methods all return None. Development, oddly, does not use it: dev/environment sets METRICS_BACKEND=warehouse.metrics.DataDogMetrics host=notdatadog, which sends real statsd packets to a hostname that does not resolve. Both approaches produce no metrics. Only one of them exercises the real code path.

Rate limiting swaps through a constructor argument and a defaultdict, as above. No environment variable exists for it.

Search swaps only in tests. SearchService and NullSearchService both implement ISearchService, and the second is a no-op, but warehouse/search/__init__.py registers only SearchService. The single place NullSearchService is used outside its own unit test is tests/conftest.py. There is no SEARCH_BACKEND variable to find, so do not go looking for one.

The pattern is a spectrum, not a rule. Ask where the swap happens before you assume it is configuration.

The payoff

What this buys, and what it costs

Three concrete returns, and one bill.

Tests get a registry made of dictionaries

Because every dependency arrives through find_service, a test can replace the whole service layer without touching Pyramid. tests/conftest.py does it in ten lines:

class _Services:
    def __init__(self):
        self._services = defaultdict(lambda: defaultdict(dict))

    def register_service(self, service_obj, iface=None, context=None, name=""):
        self._services[iface][context][name] = service_obj

    def find_service(self, iface=None, context=None, name=""):
        return self._services[iface][context][name]

A fixture then registers a stub per interface and assigns dummy_request.find_service = pyramid_services.find_service. A view under test asks for IEmailSender and gets a stub that records calls. No monkeypatch, no import surgery, no configuration. This only works because the view never named a class.

Development needs no credentials

A fresh checkout runs the full upload path, the full email path, and the full rate-limiting path against local disk, a local fake inbox, and a local Redis. Nobody hands a new contributor an AWS key. The parity table is short and worth memorising:

The production column deliberately says "an object-storage class" rather than naming one. Which class runs where is deployment configuration, and it is not in the repository. Read the code for the shape and the operators' configuration for the contents.

The bill: a bug report that says "the wrong backend"

Indirection costs you the ability to read the answer off the call site. When a file goes missing, an email does not arrive, or a limit does not apply, the stack trace names a generic base class and the view names an interface. Neither tells you what was running. The recovery is a fixed procedure, and it is short.

1. 01

   Find the interface and the name

   grep the failing view for find_service

   Two arguments matter: the interface, and the name=. IFileStorage alone is ambiguous, because there are two of them. Getting "cache" and "archive" the wrong way round explains a surprising share of confusion.

2. 02

   Find the registration

   grep for the interface name in the owning package's __init__.py

   You are looking for the register_service_factory call and the settings key it reads. That gives you the environment variable, which is the only thing that actually decides.

3. 03

   Read the variable in the failing environment

   dev/environment, docker-compose.yml, or the deployment's configuration

   The first token is the class. Now you know what was running. Everything before this step was navigation.

4. 04

   Read that class, not the interface

   warehouse/<package>/services.py

   Check for methods that raise, return constants, or swallow exceptions. GCSFileStorage.get, SMTPEmailSender.last_sent, and every method on DummyRateLimiter are all cases where the interface promises something the implementation does not deliver. The interface tells you what should happen. Only the class tells you what did.

Four steps, all greps, no debugger. Cheap for what it buys: the upload view has not needed to know a vendor's name in its whole history, and swapping one is a change to a deployment variable rather than a change to application code.

The map

Where everything is

If you're going to work on this

Five habits, in the order you will need them.

1. When you add an implementation, add a verifyClass assertion in the same commit. It is one line, and it is the only automated check that @implementer was telling the truth.
2. When you add a method to an interface, grep for every @implementer of it and decide, per class, what the honest answer is. Returning a constant is allowed. Returning a constant without a comment saying why is how last_sent would have become a mystery.
3. Do not write if statements about the environment in a service. If two environments need different behaviour, that is two classes and a settings key. The twelve existing environment comparisons are all about developer tooling, and they are the precedent to stay inside.
4. Before swapping a backend in any environment, read the class you are swapping to for methods that raise. A backend can satisfy the interface and still break a periodic task, which is how a change that passes review fails fifteen minutes after deploy.
5. When you register a second implementation of one interface, use name= and be exact about it. The "cache" and "archive" pair is the only case in the tree today, and it is worth reading reconcile_file_storages once so the two names stop being interchangeable in your head.

Next up

No. 05 in this series, Uploads, Then and Now, follows the other half of the upload path: the protocol a client speaks to forklift, why it looks the way it does, and what PEP 694 proposes to replace it with. The find_service(IFileStorage, name="archive") call that opened this guide is the last thing that view does. That guide covers everything before it.