The requirement
User U follows users F1, F2, ..., FN. When U opens the app, show the most recent tweets from everyone they follow, ordered newest-first. Low read latency. Billions of users, power users with 100M followers, millions of writes per minute.
Fan-out on write (push)
When a user tweets, push the tweet into a materialized timeline for each of their followers.
Pros. Reads are cheap — just fetch the user's pre-computed timeline. Cons. A single tweet from a user with 100M followers produces 100M writes. Celebrity tweets would take hours to fan out.
Fan-out on read (pull)
On each timeline read, query "tweets from users U follows, ordered by time" across the tweet store.
Pros. Writes are cheap — one row per tweet. Celebrities are no problem. Cons. Reads are expensive. A user following 5000 people queries 5000 tweet feeds on every pull. Hot reads kill you at Twitter scale.
The hybrid answer
Split users by follower count:
- Regular users (< 10k followers). Fan-out on write. Pre-materialize timelines into Redis per follower. Reads are free.
- Celebrities (≥ 10k followers). Do not fan out. On read, merge the precomputed timeline with the celebrity tweets the user would have received.
readTimeline(userId):
base = redis.zrange("timeline:" + userId, 0, 100)
celebrityTweets = db.query("
SELECT * FROM tweets
WHERE author_id IN (celebrityIds followed by user)
ORDER BY created_at DESC LIMIT 100
")
return merge(base, celebrityTweets).sorted().take(100)
The hot path stays cheap for the 99.9% of users who follow only normal accounts. The expensive merge only kicks in when celebrities are involved, and even then only a small fraction of the timeline is celebrity-sourced.
The storage picture
- Tweets. Wide-column store like Cassandra or a sharded relational setup. Partition by tweet ID (which is time-sortable if using Snowflake IDs).
- Timelines (materialized). Redis sorted sets keyed by user ID, score by tweet timestamp, capped at ~800 entries. Trim on insert.
- Graph (follows). Dedicated graph-friendly store or a sharded relational table with heavy indexing. Queries are "who does U follow" and "who follows U."
The messaging backbone
Tweet writes go into Kafka. Multiple consumers:
- Indexer → search service.
- Timeline fan-out service → Redis (for non-celebrity authors).
- Analytics pipeline.
- Trend detection.
Kafka handles the throughput; each consumer scales independently. If the fan-out service is slow, reads still work via the merge path (degraded freshness, not broken).
Handling ordering and dedup
Tweet IDs are Snowflake-style — time-prefixed, so lexicographic sort is temporal sort. The merge step dedupes by ID (a user following both a celebrity and a normal account will not see the tweet twice).
Failure modes
- Fan-out lag. A surge of tweets causes the fan-out queue to back up. New tweets take minutes to appear. Mitigation: shed low-priority work, scale out consumers, show "live" indicator only when lag is under threshold.
- Timeline corruption. A bad deploy writes garbage into Redis timelines. Recovery: rebuild from the tweet store by replaying recent history.
- Hot cache keys. A viral tweet from a 9k-follower user who crosses the celebrity threshold mid-burst. Threshold should have hysteresis — demote slowly, not instantly.
The signal you want to send
Recognize that pure fan-out-on-write does not scale to celebrities. Recognize that pure fan-out-on-read does not scale to reads. Propose the hybrid with a crisp threshold. Name the consistency trade-offs — timelines are eventually consistent, and that is acceptable for this product. Close with monitoring: fan-out lag, cache hit rate, merge path latency.