A Backend Performance Playbook

Most performance work I've seen fails the same way: someone has a hunch, rewrites a function to be clever, and ships a change that makes the code harder to read and the system no faster. The bottleneck was somewhere else entirely. Performance is an empirical discipline, and the first skill is refusing to act on intuition.

This is the playbook I actually use, roughly in order. It leans on data-pipeline examples — market data, backtests, the kind of workloads where a slow query means a slow research loop — but the principles are general.

Measure before you optimize

The rule is absolute: profile first, always. Not because measuring is virtuous, but because your intuition about where time goes is usually wrong, and optimizing the wrong thing costs you twice — once to build it and again to maintain it.

Before touching code, I want three numbers: where wall-clock time goes, how many queries a request makes, and what the tail latency looks like (p95/p99), not just the average. Averages hide the requests that actually make users angry.

import time, logging
from contextlib import contextmanager

@contextmanager
def timed(label: str):
    start = time.perf_counter()
    try:
        yield
    finally:
        logging.info("%s took %.1fms", label, (time.perf_counter() - start) * 1000)

with timed("load_ohlcv"):
    candles = load_ohlcv(symbol, start, end)

That trivial helper has found more real bottlenecks for me than any fancy profiler, because it forces me to state what I think is slow and then read the actual number. Nine times out of ten the surprise is which line dominates.

Respect the cost hierarchy

When you do find the hot path, orders of magnitude matter more than cleverness. The rough cost hierarchy that governs almost every backend:

1. Network / cross-service calls — milliseconds, and unbounded when something upstream is slow.
2. Disk / database I/O — sub-millisecond to milliseconds, and it's where most real time goes.
3. Memory access — nanoseconds.
4. CPU work — nanoseconds, and rarely your problem.

The practical consequence: a single saved round trip beats almost any amount of in-process cleverness. Vectorizing a loop to shave CPU is pointless if the function makes fifty sequential database calls. I've watched people spend a day micro-optimizing a backtest's inner math when the real cost was refetching the same OHLCV series from the database on every iteration. The fix wasn't faster math — it was one query and a hold in memory.

Fast code that makes a slow call is slow code. Fix the call.

Kill N+1 before anything else

The N+1 query is the most common backend performance bug, and the most rewarding to fix because it converts N round trips into one. You fetch a list, then loop and issue a query per item. It looks innocent and it's death by a thousand round trips.

# N+1: one query for symbols, then one per symbol — N+1 round trips
symbols = db.query("SELECT id FROM symbols WHERE active = true")
for s in symbols:
    s.latest = db.query("SELECT * FROM candles WHERE symbol_id = %s "
                        "ORDER BY ts DESC LIMIT 1", s.id)

# Fixed: one round trip, let the database do the join
rows = db.query("""
    SELECT DISTINCT ON (symbol_id) * FROM candles
    WHERE symbol_id = ANY(%s) ORDER BY symbol_id, ts DESC
""", [s.id for s in symbols])

Related patterns worth internalizing: select only the columns you need, push filtering and aggregation into the database rather than pulling rows into app memory to loop over, and make sure the columns you filter and sort on are actually indexed. An EXPLAIN ANALYZE on your slowest query is often the highest-return ten minutes in the whole effort.

Cache deliberately, invalidate honestly

Caching is the highest-leverage tool here and the easiest to misuse. The mental model I use: a cache is a cache-first data service, not a bag you sprinkle over slow code. The read path checks the cache, falls back to the source, and populates on miss — in one place, so every caller benefits and there's exactly one policy to reason about.

async def get_daily_ohlcv(symbol: str, day: str) -> list[Candle]:
    key = f"ohlcv:{symbol}:{day}"
    if hit := await redis.get(key):
        return decode(hit)
    candles = await db.load_ohlcv(symbol, day)
    # historical bars never change — cache hard; today's are still forming
    ttl = 86_400 if day < today() else 60
    await redis.set(key, encode(candles), ex=ttl)
    return candles

The hard part is never storing — it's invalidation. My rules: cache immutable or slow-changing data aggressively (a completed trading day's bars are immutable — cache them for a day), give volatile data short TTLs, and prefer keying by a version or content identity so a new write lands on a new key instead of forcing you to hunt down and delete the old one. When you must actively invalidate, invalidate on write at the source, and accept that anything more elaborate is a distributed-systems problem in disguise.

Concurrency, pooling, and when Redis earns its place

Once single-request work is lean, throughput becomes the question, and the answer is usually about not blocking and not thrashing connections.

Move slow, non-urgent work off the request path. If a backtest takes thirty seconds, the HTTP handler should enqueue a job and return an id; a pool of workers runs it and the client polls or gets notified. Holding a request open for thirty seconds ties up a worker and a connection for no reason.

Pool your connections. Opening a Postgres connection is expensive, and an unbounded number of them will take the database down faster than any query. A bounded pool is both a performance optimization and a safety limit.

pool = asyncpg.create_pool(dsn, min_size=5, max_size=20)  # bound it, always

async def query(sql, *args):
    async with pool.acquire() as conn:   # borrow, use, return
        return await conn.fetch(sql, *args)

As for when to add Redis: add it when you have a genuine cross-process need — a shared cache, a rate limiter, a job queue, ephemeral session state — not because it's on the standard diagram. Redis is superb at those and it's also another service to run, monitor, and reason about when it's down. Earn the dependency.

Backpressure: the part people skip

The failure mode that separates systems that degrade gracefully from ones that fall over is missing backpressure. Every queue, pool, and buffer is bounded in reality; if your code pretends they're infinite, you don't avoid the limit, you just meet it as a crash instead of a slowdown.

Concretely: bound your queues and reject or shed load when they're full, set timeouts on every outbound call so one slow dependency can't exhaust your workers, and prefer failing fast with a clear signal (a 429, a circuit breaker) over accepting work you can't complete. A system that says "not now" under load stays alive; one that accepts everything falls over and takes the good requests down with the bad.

Closing

The playbook is almost boring in its order, and that's the point. Measure first so you're solving a real problem. Respect the cost hierarchy so you're solving the expensive problem. Kill N+1s, cache deliberately with an honest invalidation story, move slow work to workers behind bounded pools, add infrastructure like Redis only when it earns its keep, and put backpressure everywhere so load degrades instead of detonating.

None of it is clever. All of it compounds. The fastest backends I've worked on weren't full of tricks — they just never did unnecessary work, and never pretended their resources were infinite.