Caching in System Design: Strategies, Eviction & Pitfalls

11 min read

Posted byProPeers

Caching in system design means keeping a copy of frequently read data in a faster layer, usually memory, so most requests skip the slow trip to the database or another service. Adding a cache is the easy part. The real design work is deciding what to cache, how to keep it fresh, and what happens when the cache is wrong or empty.

That last part is where system design interviews go. Saying "add Redis" earns you nothing. Explaining how the cache stays consistent, what it evicts and how it fails is what separates a strong answer from a generic one.

Key takeaways

  • A cache trades freshness for speed. Decide up front how stale each piece of data is allowed to be.
  • Cache-aside (read from the cache, fall back to the database, then fill the cache) is the default pattern for read-heavy systems.
  • On writes, update the database and then delete the cached copy. Deleting is safer than updating it.
  • Eviction (LRU, LFU, TTL) decides what leaves when memory is full. LRU is the usual default.
  • Plan for failure: stampedes, hot keys and mass expiry hurt more in production than a missing cache ever did.

Where can a cache sit?

A request passes through several layers, and each can cache something different:

LayerWhat it cachesExamples
Browser or appPages, images, API responsesHTTP Cache-Control headers, local storage
CDNStatic files and cacheable pages near the userCloudflare, Akamai, CloudFront
Application processHot objects in the server's own memoryA map or LRU inside the service
Distributed cacheShared data for every serverRedis, Memcached
DatabaseRecently used pages and query plansMySQL buffer pool, PostgreSQL shared buffers

In interviews, "the cache" almost always means the distributed one: a separate Redis or Memcached cluster that all application servers share. An in-process cache is faster still, but each server holds its own copy, so keeping copies consistent gets harder as you add servers.

Why is a cache so much faster?

Because memory is far closer and quicker than disk or another machine. In Peter Norvig's widely quoted table of typical timings, fetching from main memory takes about 100 nanoseconds, while a disk seek takes about 8 milliseconds: roughly 80,000 times longer. A database query also pays for parsing, locking and often a network hop, so serving a hot record from memory instead of the database regularly cuts response time from milliseconds to well under one.

The second benefit is load. Every request the cache answers is one the database never sees, which is often what keeps a read-heavy system alive at peak traffic.

How does data get into the cache? Read strategies

There are two common ways.

Cache-aside (lazy loading). The application owns the logic. It checks the cache first; on a miss it reads the database and writes the result into the cache for next time. Only data that someone actually asked for gets cached, and if the cache goes down, the application still works (more slowly) by reading the database directly.

Read-through. The application only talks to the cache, and the cache loads missing data from the database itself. The code is simpler, but you need a cache that supports loaders, and every miss is still slow.

Here is cache-aside in Python. A dictionary stands in for Redis and a sleep stands in for a slow database:

Python
import time DB = {"user:42": {"name": "Asha", "plan": "pro"}} cache = {} # stands in for Redis: key -> (value, expires_at) TTL_SECONDS = 300 def slow_db_read(key): time.sleep(0.05) # pretend the database takes 50 ms return DB.get(key) def get_user(key): entry = cache.get(key) if entry and entry[1] > time.time(): return entry[0], "hit" value = slow_db_read(key) # miss: go to the source of truth cache[key] = (value, time.time() + TTL_SECONDS) return value, "miss" def update_user(key, value): DB[key] = value # 1. write the database first cache.pop(key, None) # 2. then delete the cached copy (don't update it) print(get_user("user:42")) print(get_user("user:42")) update_user("user:42", {"name": "Asha", "plan": "free"}) print(get_user("user:42"))

Output:

Text
({'name': 'Asha', 'plan': 'pro'}, 'miss') ({'name': 'Asha', 'plan': 'pro'}, 'hit') ({'name': 'Asha', 'plan': 'free'}, 'miss')

What happens on a write? Write strategies

Reads are easy. Writes are where caches go stale. You have three standard options:

StrategyHow a write worksStrengthWeaknessGood fit
Write-throughWrite the cache and the database together, synchronouslyCache is always freshEvery write is slower; you cache data nobody may readData that is read soon after being written
Write-back (write-behind)Write the cache now, flush to the database laterVery fast writesData is lost if the cache dies before flushingCounters, metrics, high write volume
Write-aroundWrite only the database; the cache fills on the next readNo cache pollutionThe first read after a write is a missData that is rarely re-read

In practice, most read-heavy services combine cache-aside reads with update the database, then delete the key. Why delete rather than update? Picture two requests updating the same user at nearly the same moment. Each writes the database and then the cache, but the cache writes can land in the opposite order, leaving the older value cached until it expires. Deleting avoids that race: the next read misses and fetches whatever the database holds now.

A smaller race remains. A slow reader can fetch the old value just before your write, then put it back into the cache just after your delete. Short TTLs bound how long that can last, and some teams delete the key a second time a moment after the write (a "delayed double delete") to close the gap.

What gets evicted when the cache is full?

Memory is finite, so the cache needs a rule for what to drop:

PolicyEvictsWorks well when
LRU (least recently used)The item not touched for the longest timeRecent data is likely to be reused (most web traffic)
LFU (least frequently used)The item used the fewest timesA stable set of popular items
FIFOThe oldest item insertedSimple, predictable workloads
TTL (time to live)Items past their expiry timeData that must not be older than a set limit

TTL usually works alongside one of the others: every key expires eventually, and LRU or LFU decides what to drop under memory pressure before that.

Redis exposes this choice through its maxmemory-policy setting, with options such as allkeys-lru, allkeys-lfu, volatile-lru, volatile-ttl and noeviction. Its own eviction documentation calls allkeys-lru "a good default option if you have no reason to prefer any others". One detail worth knowing: Redis approximates LRU by sampling a handful of keys rather than tracking exact order, which saves a lot of memory and comes very close to true LRU in practice.

Build an LRU cache (a classic interview question)

"Design an LRU cache with O(1) get and put" (LeetCode 146) tests whether you know which two data structures to combine: a hash map for O(1) lookup, and a doubly linked list that keeps keys in order of use, so moving a key to the front or dropping the oldest are both O(1).

Python's OrderedDict already is that pair:

Python
from collections import OrderedDict class LRUCache: def __init__(self, capacity): self.capacity = capacity self.items = OrderedDict() def get(self, key): if key not in self.items: return -1 self.items.move_to_end(key) # now the most recently used return self.items[key] def put(self, key, value): if key in self.items: self.items.move_to_end(key) self.items[key] = value if len(self.items) > self.capacity: self.items.popitem(last=False) # drop the least recently used cache = LRUCache(2) cache.put(1, "a") cache.put(2, "b") cache.get(1) # touching 1 makes 2 the least recently used cache.put(3, "c") # over capacity, so 2 is evicted print(cache.get(2), cache.get(1), cache.get(3)) # -1 a c

In C++ you build the pair yourself. list::splice moves a node to the front in constant time without copying it:

C++
#include <iostream> #include <list> #include <string> #include <unordered_map> #include <utility> using namespace std; class LRUCache { int capacity; list<pair<int, string>> order; // front = most recently used unordered_map<int, list<pair<int, string>>::iterator> where; public: explicit LRUCache(int capacity) : capacity(capacity) {} string get(int key) { auto it = where.find(key); if (it == where.end()) return "-1"; order.splice(order.begin(), order, it->second); // move to front in O(1) return it->second->second; } void put(int key, const string& value) { auto it = where.find(key); if (it != where.end()) { it->second->second = value; order.splice(order.begin(), order, it->second); return; } order.emplace_front(key, value); where[key] = order.begin(); if ((int)order.size() > capacity) { where.erase(order.back().first); // evict the least recently used order.pop_back(); } } }; int main() { LRUCache cache(2); cache.put(1, "a"); cache.put(2, "b"); cache.get(1); cache.put(3, "c"); cout << cache.get(2) << " " << cache.get(1) << " " << cache.get(3) << "\n"; // -1 a c }

Java's LinkedHashMap can keep entries in access order, and overriding removeEldestEntry turns it into an LRU cache in a few lines:

Java
import java.util.LinkedHashMap; import java.util.Map; public class LRUCache<K, V> extends LinkedHashMap<K, V> { private static final long serialVersionUID = 1L; private final int capacity; public LRUCache(int capacity) { super(16, 0.75f, true); // true = keep entries in access order this.capacity = capacity; } @Override protected boolean removeEldestEntry(Map.Entry<K, V> eldest) { return size() > capacity; // evict the least recently used } public static void main(String[] args) { LRUCache<Integer, String> cache = new LRUCache<>(2); cache.put(1, "a"); cache.put(2, "b"); cache.get(1); cache.put(3, "c"); System.out.println(cache.get(2) + " " + cache.get(1) + " " + cache.get(3)); // null a c } }

The Java version prints null rather than -1 for the evicted key, because that is what a Map returns for a missing entry. If an interviewer lets you use library classes, say so and use them. If not, write the C++-style version and explain why each operation is O(1).

How do you keep cached data fresh?

Cache invalidation is famously one of the two hard problems in computer science, and you usually combine several of these:

  • TTL on every key. The safety net: even if every other mechanism fails, stale data disappears within a bounded time.
  • Delete on write. As above: update the database, then delete the key.
  • Versioned keys. Put a version in the key, such as product:17:v5. A change bumps the version, and old entries simply expire unused.
  • Change events. Publish an event when data changes (or stream the database's change log) and let a consumer delete affected keys. This works across many services that cache the same data.

What goes wrong with caches in production?

ProblemWhat happensFix
Cache stampede (thundering herd)A hot key expires and hundreds of requests hit the database at once to rebuild itLet one request rebuild while others wait (a lock or request coalescing), or refresh shortly before expiry
Cache penetrationRequests for keys that don't exist always miss and always reach the databaseCache "not found" results with a short TTL, or check a Bloom filter first
Cache avalancheMany keys expire at the same moment, often after a restart or bulk loadAdd random jitter to TTLs so expiries spread out
Hot keyOne key (a celebrity profile, a flash-sale item) overloads a single cache nodeReplicate the key, or add a small in-process cache in front
Stale readsUsers see old data after an updateDelete on write, shorter TTLs, read your own writes from the database

Interviewers love the stampede. The strongest short answer: "Only one request recomputes the value. The rest either wait for it or serve the slightly stale copy, and TTLs get jitter so popular keys don't all expire together."

How should you talk about caching in an interview?

Walk through five points in order, and the conversation stays under your control:

  1. Access pattern. Is the data read far more often than it's written? Caching pays off most for read-heavy data.
  2. What and how. Which objects to cache, what the key looks like (user:{id}), and roughly how much memory that needs.
  3. Strategy. Cache-aside for reads, database first then delete on writes, unless the problem needs something stronger.
  4. Eviction and TTL. LRU plus a TTL that matches how stale the data may be: seconds for a price, hours for a profile picture.
  5. Failure and measurement. What happens when the cache is down or cold, how you prevent stampedes, and which number proves it works: the hit ratio.

The same instinct shows up at every level. Memoization in algorithms, covered in how to reduce time complexity, is caching applied to a single function. Retrieval systems for AI apps cache embeddings and search results for the same reason, as covered in what RAG is and how it works.

Caching, databases, queues and sharding are the core of the System Design module (weeks 12 to 15) of the DSA, System Design & AI Bootcamp, which is taught live by practising engineers.

Frequently asked questions

What is caching in system design?

Caching stores copies of frequently used data in a faster layer, usually memory, so most requests are served without querying the database or another service. It cuts latency and database load, at the cost of possibly serving slightly stale data.

What is the difference between cache-aside and write-through?

Cache-aside is a read strategy: the application checks the cache, reads the database on a miss and then fills the cache. Write-through is a write strategy: every write goes to the cache and the database together, so the cache is never stale but writes are slower.

Which cache eviction policy should I use?

Start with LRU (least recently used) plus a TTL on every key. It suits most web traffic, where recently used data is likely to be used again. Consider LFU when a stable set of items stays popular for a long time.

What is a cache stampede and how do you prevent it?

A stampede happens when a popular key expires and many requests rebuild it from the database at the same moment. Prevent it by letting one request rebuild the value while the others wait or serve the old copy, and by adding random jitter to TTLs.

Should I update or delete the cache when data changes?

Delete it. Write the database first, then delete the cached key so the next read loads the fresh value. Updating the cache directly can leave old data cached when two writes race.

2 stories