Caching in System Design: Strategies, Eviction & Pitfalls
11 min read
Caching in system design means keeping a copy of frequently read data in a faster layer, usually memory, so most requests skip the slow trip to the database or another service. Adding a cache is the easy part. The real design work is deciding what to cache, how to keep it fresh, and what happens when the cache is wrong or empty.
That last part is where system design interviews go. Saying "add Redis" earns you nothing. Explaining how the cache stays consistent, what it evicts and how it fails is what separates a strong answer from a generic one.
Key takeaways
- A cache trades freshness for speed. Decide up front how stale each piece of data is allowed to be.
- Cache-aside (read from the cache, fall back to the database, then fill the cache) is the default pattern for read-heavy systems.
- On writes, update the database and then delete the cached copy. Deleting is safer than updating it.
- Eviction (LRU, LFU, TTL) decides what leaves when memory is full. LRU is the usual default.
- Plan for failure: stampedes, hot keys and mass expiry hurt more in production than a missing cache ever did.
Where can a cache sit?
A request passes through several layers, and each can cache something different:
| Layer | What it caches | Examples |
|---|---|---|
| Browser or app | Pages, images, API responses | HTTP Cache-Control headers, local storage |
| CDN | Static files and cacheable pages near the user | Cloudflare, Akamai, CloudFront |
| Application process | Hot objects in the server's own memory | A map or LRU inside the service |
| Distributed cache | Shared data for every server | Redis, Memcached |
| Database | Recently used pages and query plans | MySQL buffer pool, PostgreSQL shared buffers |
In interviews, "the cache" almost always means the distributed one: a separate Redis or Memcached cluster that all application servers share. An in-process cache is faster still, but each server holds its own copy, so keeping copies consistent gets harder as you add servers.
Why is a cache so much faster?
Because memory is far closer and quicker than disk or another machine. In Peter Norvig's widely quoted table of typical timings, fetching from main memory takes about 100 nanoseconds, while a disk seek takes about 8 milliseconds: roughly 80,000 times longer. A database query also pays for parsing, locking and often a network hop, so serving a hot record from memory instead of the database regularly cuts response time from milliseconds to well under one.
The second benefit is load. Every request the cache answers is one the database never sees, which is often what keeps a read-heavy system alive at peak traffic.
How does data get into the cache? Read strategies
There are two common ways.
Cache-aside (lazy loading). The application owns the logic. It checks the cache first; on a miss it reads the database and writes the result into the cache for next time. Only data that someone actually asked for gets cached, and if the cache goes down, the application still works (more slowly) by reading the database directly.
Read-through. The application only talks to the cache, and the cache loads missing data from the database itself. The code is simpler, but you need a cache that supports loaders, and every miss is still slow.
Here is cache-aside in Python. A dictionary stands in for Redis and a sleep stands in for a slow database:
import time
DB = {"user:42": {"name": "Asha", "plan": "pro"}}
cache = {} # stands in for Redis: key -> (value, expires_at)
TTL_SECONDS = 300
def slow_db_read(key):
time.sleep(0.05) # pretend the database takes 50 ms
return DB.get(key)
def get_user(key):
entry = cache.get(key)
if entry and entry[1] > time.time():
return entry[0], "hit"
value = slow_db_read(key) # miss: go to the source of truth
cache[key] = (value, time.time() + TTL_SECONDS)
return value, "miss"
def update_user(key, value):
DB[key] = value # 1. write the database first
cache.pop(key, None) # 2. then delete the cached copy (don't update it)
print(get_user("user:42"))
print(get_user("user:42"))
update_user("user:42", {"name": "Asha", "plan": "free"})
print(get_user("user:42"))Output:
({'name': 'Asha', 'plan': 'pro'}, 'miss')
({'name': 'Asha', 'plan': 'pro'}, 'hit')
({'name': 'Asha', 'plan': 'free'}, 'miss')What happens on a write? Write strategies
Reads are easy. Writes are where caches go stale. You have three standard options:
| Strategy | How a write works | Strength | Weakness | Good fit |
|---|---|---|---|---|
| Write-through | Write the cache and the database together, synchronously | Cache is always fresh | Every write is slower; you cache data nobody may read | Data that is read soon after being written |
| Write-back (write-behind) | Write the cache now, flush to the database later | Very fast writes | Data is lost if the cache dies before flushing | Counters, metrics, high write volume |
| Write-around | Write only the database; the cache fills on the next read | No cache pollution | The first read after a write is a miss | Data that is rarely re-read |
In practice, most read-heavy services combine cache-aside reads with update the database, then delete the key. Why delete rather than update? Picture two requests updating the same user at nearly the same moment. Each writes the database and then the cache, but the cache writes can land in the opposite order, leaving the older value cached until it expires. Deleting avoids that race: the next read misses and fetches whatever the database holds now.
A smaller race remains. A slow reader can fetch the old value just before your write, then put it back into the cache just after your delete. Short TTLs bound how long that can last, and some teams delete the key a second time a moment after the write (a "delayed double delete") to close the gap.
What gets evicted when the cache is full?
Memory is finite, so the cache needs a rule for what to drop:
| Policy | Evicts | Works well when |
|---|---|---|
| LRU (least recently used) | The item not touched for the longest time | Recent data is likely to be reused (most web traffic) |
| LFU (least frequently used) | The item used the fewest times | A stable set of popular items |
| FIFO | The oldest item inserted | Simple, predictable workloads |
| TTL (time to live) | Items past their expiry time | Data that must not be older than a set limit |
TTL usually works alongside one of the others: every key expires eventually, and LRU or LFU decides what to drop under memory pressure before that.
Redis exposes this choice through its maxmemory-policy setting, with options such as allkeys-lru, allkeys-lfu, volatile-lru, volatile-ttl and noeviction. Its own eviction documentation calls allkeys-lru "a good default option if you have no reason to prefer any others". One detail worth knowing: Redis approximates LRU by sampling a handful of keys rather than tracking exact order, which saves a lot of memory and comes very close to true LRU in practice.
Build an LRU cache (a classic interview question)
"Design an LRU cache with O(1) get and put" (LeetCode 146) tests whether you know which two data structures to combine: a hash map for O(1) lookup, and a doubly linked list that keeps keys in order of use, so moving a key to the front or dropping the oldest are both O(1).
Python's OrderedDict already is that pair:
from collections import OrderedDict
class LRUCache:
def __init__(self, capacity):
self.capacity = capacity
self.items = OrderedDict()
def get(self, key):
if key not in self.items:
return -1
self.items.move_to_end(key) # now the most recently used
return self.items[key]
def put(self, key, value):
if key in self.items:
self.items.move_to_end(key)
self.items[key] = value
if len(self.items) > self.capacity:
self.items.popitem(last=False) # drop the least recently used
cache = LRUCache(2)
cache.put(1, "a")
cache.put(2, "b")
cache.get(1) # touching 1 makes 2 the least recently used
cache.put(3, "c") # over capacity, so 2 is evicted
print(cache.get(2), cache.get(1), cache.get(3)) # -1 a cIn C++ you build the pair yourself. list::splice moves a node to the front in constant time without copying it:
#include <iostream>
#include <list>
#include <string>
#include <unordered_map>
#include <utility>
using namespace std;
class LRUCache {
int capacity;
list<pair<int, string>> order; // front = most recently used
unordered_map<int, list<pair<int, string>>::iterator> where;
public:
explicit LRUCache(int capacity) : capacity(capacity) {}
string get(int key) {
auto it = where.find(key);
if (it == where.end()) return "-1";
order.splice(order.begin(), order, it->second); // move to front in O(1)
return it->second->second;
}
void put(int key, const string& value) {
auto it = where.find(key);
if (it != where.end()) {
it->second->second = value;
order.splice(order.begin(), order, it->second);
return;
}
order.emplace_front(key, value);
where[key] = order.begin();
if ((int)order.size() > capacity) {
where.erase(order.back().first); // evict the least recently used
order.pop_back();
}
}
};
int main() {
LRUCache cache(2);
cache.put(1, "a");
cache.put(2, "b");
cache.get(1);
cache.put(3, "c");
cout << cache.get(2) << " " << cache.get(1) << " " << cache.get(3) << "\n"; // -1 a c
}Java's LinkedHashMap can keep entries in access order, and overriding removeEldestEntry turns it into an LRU cache in a few lines:
import java.util.LinkedHashMap;
import java.util.Map;
public class LRUCache<K, V> extends LinkedHashMap<K, V> {
private static final long serialVersionUID = 1L;
private final int capacity;
public LRUCache(int capacity) {
super(16, 0.75f, true); // true = keep entries in access order
this.capacity = capacity;
}
@Override
protected boolean removeEldestEntry(Map.Entry<K, V> eldest) {
return size() > capacity; // evict the least recently used
}
public static void main(String[] args) {
LRUCache<Integer, String> cache = new LRUCache<>(2);
cache.put(1, "a");
cache.put(2, "b");
cache.get(1);
cache.put(3, "c");
System.out.println(cache.get(2) + " " + cache.get(1) + " " + cache.get(3)); // null a c
}
}The Java version prints null rather than -1 for the evicted key, because that is what a Map returns for a missing entry. If an interviewer lets you use library classes, say so and use them. If not, write the C++-style version and explain why each operation is O(1).
How do you keep cached data fresh?
Cache invalidation is famously one of the two hard problems in computer science, and you usually combine several of these:
- TTL on every key. The safety net: even if every other mechanism fails, stale data disappears within a bounded time.
- Delete on write. As above: update the database, then delete the key.
- Versioned keys. Put a version in the key, such as
product:17:v5. A change bumps the version, and old entries simply expire unused. - Change events. Publish an event when data changes (or stream the database's change log) and let a consumer delete affected keys. This works across many services that cache the same data.
What goes wrong with caches in production?
| Problem | What happens | Fix |
|---|---|---|
| Cache stampede (thundering herd) | A hot key expires and hundreds of requests hit the database at once to rebuild it | Let one request rebuild while others wait (a lock or request coalescing), or refresh shortly before expiry |
| Cache penetration | Requests for keys that don't exist always miss and always reach the database | Cache "not found" results with a short TTL, or check a Bloom filter first |
| Cache avalanche | Many keys expire at the same moment, often after a restart or bulk load | Add random jitter to TTLs so expiries spread out |
| Hot key | One key (a celebrity profile, a flash-sale item) overloads a single cache node | Replicate the key, or add a small in-process cache in front |
| Stale reads | Users see old data after an update | Delete on write, shorter TTLs, read your own writes from the database |
Interviewers love the stampede. The strongest short answer: "Only one request recomputes the value. The rest either wait for it or serve the slightly stale copy, and TTLs get jitter so popular keys don't all expire together."
How should you talk about caching in an interview?
Walk through five points in order, and the conversation stays under your control:
- Access pattern. Is the data read far more often than it's written? Caching pays off most for read-heavy data.
- What and how. Which objects to cache, what the key looks like (
user:{id}), and roughly how much memory that needs. - Strategy. Cache-aside for reads, database first then delete on writes, unless the problem needs something stronger.
- Eviction and TTL. LRU plus a TTL that matches how stale the data may be: seconds for a price, hours for a profile picture.
- Failure and measurement. What happens when the cache is down or cold, how you prevent stampedes, and which number proves it works: the hit ratio.
The same instinct shows up at every level. Memoization in algorithms, covered in how to reduce time complexity, is caching applied to a single function. Retrieval systems for AI apps cache embeddings and search results for the same reason, as covered in what RAG is and how it works.
Caching, databases, queues and sharding are the core of the System Design module (weeks 12 to 15) of the DSA, System Design & AI Bootcamp, which is taught live by practising engineers.
Frequently asked questions
What is caching in system design?
Caching stores copies of frequently used data in a faster layer, usually memory, so most requests are served without querying the database or another service. It cuts latency and database load, at the cost of possibly serving slightly stale data.
What is the difference between cache-aside and write-through?
Cache-aside is a read strategy: the application checks the cache, reads the database on a miss and then fills the cache. Write-through is a write strategy: every write goes to the cache and the database together, so the cache is never stale but writes are slower.
Which cache eviction policy should I use?
Start with LRU (least recently used) plus a TTL on every key. It suits most web traffic, where recently used data is likely to be used again. Consider LFU when a stable set of items stays popular for a long time.
What is a cache stampede and how do you prevent it?
A stampede happens when a popular key expires and many requests rebuild it from the database at the same moment. Prevent it by letting one request rebuild the value while the others wait or serve the old copy, and by adding random jitter to TTLs.
Should I update or delete the cache when data changes?
Delete it. Write the database first, then delete the cached key so the next read loads the fresh value. Updating the cache directly can leave old data cached when two writes race.