The brief

Design a web crawler that can systematically browse the internet to build a search index, handling billions of pages with politeness constraints and distributed coordination.

The full worked answer — architecture, key decisions, trade-offs and failure modes — is part of Thita's paid system design course. See plans or read it inside the System Design course.

Design a Web Crawler — system design case study

A worked system design answer for a Web Crawler, timed at about 60 minutes — the length of the design portion of a real round. Reported asked at Google, Amazon, Microsoft.

It walks the same sequence an interviewer expects: clarify the requirements, size the load, settle the API, choose the storage and the caching, then defend the trade-offs. It builds on the Queues & Async family and belongs to Queues & Async.

Lessons alongside this case study

Queue Fundamentals — Core lesson.
Delivery Guarantees & Idempotency — Core lesson.
Event-Driven Patterns Deep Dive — Depth lesson.
Kafka Deep Dive — Depth lesson.
Stream Processing — Depth lesson.
Design YouTube — Core lesson, worked case study, about 60 minutes, asked at Google, Netflix.
Design a Webhook Delivery System — Depth lesson, worked case study, about 45 minutes, asked at Stripe.
Design a Notification System — Depth lesson, worked case study, about 60 minutes, asked at Meta, Amazon.
Design a Web Crawler — Depth lesson, worked case study, about 60 minutes, asked at Google, Amazon, Microsoft.
Design a Job Scheduler — Depth lesson, worked case study, about 60 minutes, asked at Airbnb, Uber, Stripe.

Other worked case studies

Design URL Shortener — about 45 minutes. Builds on Databases, Caching, Scalability.
Design Rate Limiter — about 40 minutes. Builds on Caching, Scalability.
Design Twitter Feed — about 45 minutes. Builds on Databases, Caching, Queues & Async.
Design WhatsApp Chat — about 45 minutes. Builds on Networking & APIs, Databases, Queues & Async.
Design YouTube — about 45 minutes. Builds on Databases, Caching, Scalability.
Design Uber — about 45 minutes. Builds on Search, Geo & Aggregation, Queues & Async, Databases.
Design Ticketmaster — about 45 minutes. Builds on Coordination & Transactions, Databases, Caching.
Design Payment System — about 45 minutes. Builds on Coordination & Transactions, Queues & Async, Databases.
Design a Key-Value Store — about 45 minutes. Builds on Databases, Scalability.
Design Dropbox — about 45 minutes. Builds on Databases, Queues & Async.
Design a Distributed Cache — about 45 minutes. Builds on Caching, Scalability.
Design a Webhook Delivery System — about 45 minutes. Builds on Queues & Async, Networking & APIs.
Design a Notification System — about 45 minutes. Builds on Queues & Async, Scalability.
Design URL Shortener at Massive Scale — about 45 minutes. Builds on Scalability, Search, Geo & Aggregation.
Design Ticketmaster at Scale — about 45 minutes. Builds on Coordination & Transactions, Scalability.
Design an Ad Click Aggregator — about 45 minutes. Builds on Search, Geo & Aggregation, Queues & Async.
Design Uber Deep Dive — about 45 minutes. Builds on Search, Geo & Aggregation, Scalability.
Design a Job Scheduler — about 60 minutes. Builds on Queues & Async.
Design Instagram — about 60 minutes. Builds on Scalability.
Design TikTok Feed — about 60 minutes. Builds on Scalability.
Design Google Docs — about 60 minutes. Builds on Coordination & Transactions.
Design a Stock Exchange — about 60 minutes. Builds on Coordination & Transactions.
Design a Top-K Leaderboard — about 60 minutes. Builds on Search, Geo & Aggregation.
Design Google Maps — about 60 minutes. Builds on Search, Geo & Aggregation.
Design a Recommendation System — about 60 minutes. Builds on AI Infrastructure.

The system design pattern families

Foundations — The 45-minute interview framework, estimation with modern numbers, and CAP theorem. 6 lessons · about 2 hours · Interview Soon · 1 worked case study.
Networking & APIs — Protocols (REST, gRPC, WebSocket, SSE), API design, pagination, and idempotency. 5 lessons · about 2 hours · Interview Soon · 1 worked case study.
Databases — SQL vs NoSQL decision tree, replication, sharding, and indexing strategies. 7 lessons · about 3 hours · Interview Soon · 2 worked case studies.
Caching — Cache-aside, write-through, failure modes, and when NOT to cache. 5 lessons · about 1.5 hours · Interview Soon · 2 worked case studies.
Scalability — Consistent hashing, read vs write scaling, fan-out strategies, and circuit breakers. 7 lessons · about 1.5 hours · Interview Soon · 4 worked case studies.
Coordination & Transactions — Raft consensus, distributed locking, saga pattern, and conflict resolution. 9 lessons · about 1.5 hours · Interview Soon · 5 worked case studies.
Search, Geo & Aggregation — Full-text search, geospatial indexing, stream processing, and probabilistic structures. 9 lessons · about 1.5 hours · Interview Soon · 5 worked case studies.
Reliability & Operations — Monitoring, observability, deployment strategies, and graceful degradation. 2 lessons · about 3 hours · Deep Dive.
Security & Auth — OAuth 2.0, JWT vs sessions, API security, and encryption. 2 lessons · about 3 hours · Deep Dive.
AI Infrastructure — LLM serving, inference batching, RAG pipelines, and vector search at scale. 6 lessons · about 6 hours · Deep Dive · 1 worked case study.
Real-World Architecture Studies — Deep dives into Discord, Uber, Stripe, Netflix, and Kafka production architectures. 5 lessons · about 4 hours · Deep Dive.

Related interview preparation

Looking for the practice sheet? — Track your progress against every family as a checklist, with company tags and AI interview practice.
Low Level Design course — The object-level companion: OOP, SOLID, the design patterns and the machine-coding round.
DSA patterns course — The coding round: the algorithmic patterns behind the problems asked alongside system design.
Latency and capacity numbers — The reference table of modern latency, throughput and storage figures used in estimation.
AI mock interviews — Practise the spoken answer against an interviewer that pushes back on the trade-offs.

Opening system design

Progress is saved as you study.