Part 1 followed system design from 1990s mainframes and client-server applications to the microservices of the 2010s. By 2015 the parts of a modern system existed: cloud infrastructure, containers, Kubernetes and a microservices playbook. This part covers what the 2020s added, then puts the eras side by side with one shop designed three times and a real project checked against its running cluster.
The 2020s through 2026: platform engineering and AI-native systems
Kubernetes becomes a platform, not a project
Running Kubernetes yourself turned out to be a full-time job, so most teams moved to managed Kubernetes: EKS, AKS or GKE run the control plane, and you run the workloads. The work that remained moved into practices and tools:
- GitOps. The desired state of every cluster lives in Git, and a controller such as Argo CD keeps the cluster matching it. A deploy is a merged pull request; a rollback is a revert. Argo graduated in the CNCF on 6 December 2022 (CNCF).
- Packaging. Helm charts and Kustomize overlays describe an application once and vary it per environment.
- Autoscaling at three levels. The Horizontal Pod Autoscaler adds pods when CPU or memory rises. KEDA scales on events such as queue length, including down to zero. Karpenter adds and removes nodes to fit pending pods.
- Service mesh and mTLS. Istio or Linkerd sidecars encrypt traffic between services, prove each service's identity, and add retries and traffic splitting without changing code.
- OpenTelemetry and distributed tracing. OpenTelemetry formed when OpenTracing and OpenCensus merged in 2019 (CNCF). It gives one standard for traces, metrics and logs, so a request can be followed across every service it touches.
- SRE practices. Service level objectives, error budgets and blameless incident reviews turn reliability into a measured, negotiated target instead of a feeling.
- Internal developer platforms. A platform team packages all of the above, with templates, pipelines, dashboards and guardrails, so product teams can ship without becoming Kubernetes experts.
- Policy as code and zero trust. Tools such as OPA Gatekeeper and Kyverno reject non-compliant manifests at admission time. Every request is authenticated by identity, not trusted because it came from inside the network.
None of this changes what the system does for its users. It changes how many people it takes to run it safely.
AI-native architecture
The newest layer is AI. A few terms first:
- Large language model (LLM): a model that takes text, or text plus images, and generates text. You call it through a provider's API or run it yourself.
- Embedding model: turns text into a vector of numbers so that similar meanings land close together.
- Vector database: stores those vectors and finds the nearest ones quickly.
- Retrieval-augmented generation (RAG): before asking the model, search your own documents and put the relevant passages in the prompt. The technique was described by Lewis et al. in 2020 (arXiv).
- Agent: a program in which the model chooses the next step, such as calling a tool, searching or answering, inside a loop the code controls.
- Model Context Protocol (MCP): an open protocol, introduced by Anthropic on 25 November 2024, that gives AI applications a standard way to connect to tools and data sources (Anthropic; specification).
- Orchestration frameworks: LangChain provides model and tool abstractions; LangGraph models an agent as an explicit state graph; CrewAI coordinates several role-based agents.
- LLM gateway and model routing: one internal service through which all model calls pass, applying authentication, budgets, redaction, caching and the choice of model per request.
- LLM observability: tools such as Langfuse record prompts, responses, token counts, cost and evaluation scores per trace.
AI adds design concerns that older systems never had:
| Concern | Why it is new | A common control |
|---|---|---|
| Non-deterministic output | The same input can give different answers | Validate structured output against a schema; never let free text drive a write |
| Token cost | You pay per token, and an agent loop can call the model many times | Per-request and per-user budgets in the gateway; caching; smaller models for easy steps |
| Latency | A model call takes seconds, not milliseconds | Stream responses, run steps in parallel, set hard deadlines |
| Hallucination | The model can state false things fluently | Ground answers in retrieved data; show sources; check facts in code |
| Prompt injection | Text in a document or web page can carry instructions | Treat retrieved text as data; keep tools least-privileged |
| Tool authorization | A tool call acts in the real world | Tools act as the calling user, and the backend re-checks every permission |
| Data privacy | Prompts may contain personal data and leave your network | Redact before sending; choose regions and providers deliberately |
| Evaluation and monitoring | "It works" has no unit test | Golden datasets, scored evaluations, regression runs before each model change |
| Agent reliability | Loops can stall, repeat or wander | Step limits, timeouts and a deterministic fallback path |
| Human oversight | Some actions should never be fully automatic | Approval queues for refunds, deletions and payments |
AI components are optional capabilities, not a replacement for application logic. In Figure 5 the left side works on its own. Take the AI zone away and customers can still browse, pay and get their orders. That is the right default: the model proposes, and deterministic code decides. I wrote more about that rule in Four AI systems, one cluster.
1992 versus 2026, side by side
| Area | Early 1990s | 2026 |
|---|---|---|
| Infrastructure | Mainframes and physical servers in your own building | Public cloud, private cloud and hybrid mixes |
| Architecture | Mainframe, two-tier client-server, monolith | Modular monoliths, microservices, event-driven systems |
| Deployment | Manual installation on servers and every desktop | Automated CI/CD and GitOps, many deploys a day |
| Scaling | Buy a bigger machine or add a server, weeks ahead | Elastic horizontal and vertical scaling in minutes |
| Databases | Mostly one central relational database | SQL, NoSQL, distributed SQL and vector databases |
| Networking | LANs and private leased lines | The Internet, APIs and service-to-service networking |
| Monitoring | System logs and vendor admin consoles | Metrics, logs, traces and continuous profiling |
| Security | Perimeter: trusted inside, guarded edge | Identity-centric, zero trust, encryption everywhere |
| Disaster recovery | Backup tapes and standby sites | Automated backups, replicas, multi-region failover |
| AI | Specialised expert systems and research | LLMs, RAG, agents and AI services as optional components |
These are representative patterns, not hard boundaries. Many organisations in 2026 still run mainframes, monoliths and single relational databases, and run them well. Some 1990s companies ran surprisingly distributed systems.
One e-commerce shop, built in three eras
Let's keep the requirements fixed and change only the year. The shop needs user registration, a product catalogue, a shopping cart, order processing, payments, inventory and shipping. All three figures use the same rows, from customers at the top to operations at the bottom, so you can compare them directly.
Version A: 1992
The "customer interface" is a telephone. Order-entry clerks use desktop clients connected to a central application and database. Card payments are settled in an overnight batch file, and shipping manifests are printed.
Trade-offs. It is simple, consistent and cheap to run for a fixed number of staff, and one database transaction covers an entire order. But capacity is the number of clerks, a release means a weekend of installs, and the database server is the single point of failure.
Version B: 2012
Customers now place orders themselves. One codebase runs on several virtual machines behind a load balancer. Sessions move into Memcached or Redis, so any server can handle any request and a server can die without losing carts. MySQL replicas absorb catalogue reads. A queue moves emails, labels and stock updates out of the checkout path. A hosted payment page keeps card numbers away from the shop's servers.
Trade-offs. This design scales well for most shops and is still a sensible choice in 2026. The costs are cache invalidation, replica lag (a customer may not see an order they just placed if the page reads a lagging replica), and a release process that still deploys everything at once.
Version C: 2026
The frontend is a React or Next.js application. Services run in containers on Kubernetes, one per business capability. PostgreSQL holds the relational core: users, orders, payments and stock. MongoDB is used only where it is justified, for a catalogue whose products have very different attributes. Redis holds carts. Kafka, or RabbitMQ for simpler work queues, carries order-placed events to shipping, email and analytics. Argo CD deploys from Git, and OpenTelemetry follows each request. An AI shopping assistant can answer questions about products, but it reads through the catalogue API and cannot place orders or change prices.
Trade-offs. Teams can ship independently, and only the busy services need to scale. But checkout now crosses orders, inventory, payments and shipping, so a single database transaction is no longer possible. You need an outbox, idempotent consumers and compensating actions. You also need tracing to debug, an on-call rota per service, and a cloud bill to watch.
For a team of five, Version B, or Version C as a modular monolith, is usually the better engineering decision.
Case study: RoboShop AI
RoboShop AI (source) is my demonstration shop for robots. Everything below was checked against commit 02a4478 and against the live mk4 Minikube cluster on 8 November 2026. Where the code supports something that is not switched on, I say so.
| Component | Why it exists and what it does | Talks to | Problem it solves | Operational and security trade-off |
|---|---|---|---|---|
web (NGINX) | Serves the TypeScript storefront and proxies /api/* and /moks/*; returns 403 for direct /api/chat | Traefik in front; ai-gateway and moks behind | One public entry point with security headers and strict routes | Simple and fast, but every route change is an NGINX config change |
ai-gateway (FastAPI) | The public API: authorization, server-side prices and stock, Razorpay order flow with an Idempotency-Key, chat history, order ledger | Every domain service over HTTP; RabbitMQ through its outbox | The browser is never trusted for prices, stock or payment status | The JSON ledger allows one writer, so the gateway runs one replica. A PostgreSQL ledger exists in code (COMMERCE_BACKEND=postgres) but is not enabled |
user-service (Python) | Accounts, hashed sessions, MFA, profiles, purchase history in MongoDB | Called by ai-gateway with a service token | Identity has one owner | Session checks fail closed: if it is down, signed-in actions stop |
catalogue-service (Node.js) | Read-only product API over MongoDB, plus import tools | ai-gateway and moks | Catalogue reads stay cheap and safe | Admin edits still go through the gateway, so two components touch product data |
cart-service (Python) | Redis carts with compare-and-set revisions in Lua and a seven-day sliding expiry | Only ai-gateway; no browser access | Concurrent cart edits cannot overwrite each other | Redis is a cache here; the gateway's ledger backs up quantities |
payment-service (Node.js) | Creates Razorpay orders and verifies captured payments; test mode by default | ai-gateway; Razorpay | Payment provider details live in one place | Real payments need deliberate merchant configuration |
shipping-service (Spring Boot) | Quotes from a PostgreSQL table of countries and cities; consumes paid-order events | RabbitMQ; its own PostgreSQL | Shipping data and logic have one owner | Deduplicates event IDs because delivery is at least once |
currency-service (Go) | Display prices from the ECB reference feed, refreshed hourly in the background | ai-gateway; ECB | Requests never wait on an outside API | Each replica has its own cache; rates are informational, never charged |
RabbitMQ and kafka-bridge | Durable queues for paid orders; the bridge copies them to Kafka and acks only after Kafka confirms | Gateway outbox → RabbitMQ → shipping and bridge → Kafka | Analytics can replay events without touching checkout | Two brokers to run; Kafka is not in the checkout path at all |
mcp-server (FastMCP) | MCP tools to search, compare and manage the signed-in user's cart | ai-gateway, with the caller's own session | AI clients get typed tools that cannot bypass authorization | Private: a ClusterIP Service and a NetworkPolicy, with no public route today |
moks (LangGraph) | Shopping assistant: a fixed three-step graph that searches the catalogue, grounds evidence and generates an answer | ai-gateway and catalogue-service; Amazon Bedrock or Ollama when configured | Product questions answered from real catalogue data | It has no tools and makes no writes. No model was configured on mk4, so the widget asks an admin to connect one |
| Kubernetes and Argo CD | Each component is an Argo CD Application with manual sync; images come from a Harbor registry | Git and Harbor | Repeatable deploys from Git | All Deployments run one replica, and there are no autoscalers on mk4 |
| OpenTelemetry | Tracing code exists in the services | An OpenTelemetry Collector, when enabled | One trace across services | Disabled on mk4 (OTEL_SDK_DISABLED=true), so no traces are collected today |
RoboShop AI is a learning and demonstration system, not a highly available production deployment. Its README says so too. The order ledger is the clearest example of a deliberate trade-off. A JSON file with an exclusive lock is easy to reason about and back up, but it pins the gateway to one replica until the PostgreSQL backend is switched on and migrated. The deeper dives are in the RoboShop AI architecture article and the MCP server article.
Compared with a 1990s retail system
Put Figure 9 next to Figure 6. The 1992 shop had one program and one database. Consistency was free, and deployment was painful. RoboShop AI has eleven application services, four data stores (MongoDB, Redis, PostgreSQL and the ledger file) and two message brokers, plus independent images and Git-driven deploys. Consistency is engineered: idempotency keys, an outbox, at-least-once delivery and deduplication.
The 1992 shop had no customer-facing interface at all. RoboShop has three: the storefront, the assistant and MCP. Each of the newer two is constrained so it can never do more than the signed-in user could do by hand.
How system design interviews have changed
Interviews in the 1990s and 2000s focused on a single system: CPU and memory sizing, schema design, normalisation, transactions and isolation levels, networking basics, capacity planning and availability through redundant hardware. Those questions have not gone away. Modern interviews add distributed systems, horizontal scaling, cache invalidation, event-driven design, Kubernetes, cloud networking, security, observability, cost and multi-region resilience. Increasingly, they also ask about AI workloads.
Five questions that show the shift:
- "Design a shopping cart." Traditional: tables, foreign keys, one transaction at checkout. Modern: where the cart lives (Redis, a database or the client), what happens during a partition (Dynamo's always-writable cart), and how concurrent edits are resolved.
- "How do you make checkout reliable?" Traditional: a database transaction with the right isolation level. Modern: idempotency keys, an outbox, at-least-once delivery, deduplicating consumers and compensation when a later step fails.
- "The system is slow. What do you do?" Traditional: check CPU, memory, disk and slow queries on the server. Modern: start from the trace of a slow request, find the hop that grew, then compare metrics and logs for that service.
- "How do you handle ten times the traffic?" Traditional: a bigger server and a faster disk, ordered months ahead. Modern: stateless services, horizontal autoscaling, caching, read replicas or partitioning, a queue to absorb bursts, and a cost estimate.
- "Add an AI assistant to the product." Traditional: not asked. Modern: RAG over which data, how the model's actions are authorized, token budgets and latency, prompt injection, evaluation and the fallback when the model is down.
What has not changed is the foundation: latency, throughput, consistency, fault tolerance and data modelling. A candidate who can reason about those can learn any new tool. For a full preparation plan, see the DevOps interview preparation guide.
What have we gained, and what have we lost?
Gained:
- Elasticity. Capacity in minutes, paid for by use.
- Faster delivery. Small, frequent, reversible deploys instead of weekend releases.
- Infrastructure automation. Environments described in code, reviewed and reproducible.
- Managed services. Backups, failover and patching handled by someone else.
- Independent deployments. Teams no longer queue behind each other.
- Better observability. Traces, metrics and logs answer questions nobody planned for.
- Global reach. CDNs and multi-region deployments put the service near users.
Lost, or at least made harder:
- Simplicity. Distributed systems fail in partial, confusing ways.
- Predictable cost. Cloud bills grow quietly; egress and idle resources add up.
- Independence from vendors. Managed services are convenient and hard to leave.
- Network reliability. Every service call can be slow, fail or arrive twice.
- Easy consistency. One transaction became sagas and reconciliation jobs.
- A small attack surface. More endpoints, more credentials, more dependencies.
- Easy troubleshooting. A bug can be spread across five services and a queue.
- Low platform overhead. Kubernetes itself needs upgrades, security patches and expertise.
- Cheap, dependable answers. AI inference costs money per request, and its output needs checking.
The best engineers I have worked with choose the simplest architecture that meets the actual requirements, and they write down what would make them change it. A well-built monolith on two virtual machines with a managed database beats a twenty-service platform that nobody fully understands.
What comes next?
These are directions, not established facts:
- AI-assisted software development. More code will be drafted by models, which makes review, tests and architecture decisions more important, not less.
- Autonomous operations with human oversight. Agents that diagnose incidents and propose fixes, with people approving changes to production.
- Agentic workflows. Multi-step business processes in which models choose some of the steps inside limits set by code.
- WebAssembly. Small, sandboxed, fast-starting modules on servers and at the edge.
- Edge computing. More logic running near users for latency and data-residency reasons.
- Platform engineering. Internal platforms treated as products, with paved roads for common tasks.
- Policy-driven infrastructure. Security, cost and compliance rules enforced automatically at every step.
- Sustainable computing. Carbon and energy treated as design constraints alongside cost.
- AI workload scheduling. GPUs scheduled as carefully as databases are capacity-planned today.
- Hybrid and sovereign clouds. Regulation keeping some data and workloads in particular countries or on premises.
What will probably stay: networks will still fail, data will still need an owner, users will still notice latency, and every design will still be a set of trade-offs.
Conclusion
System design has evolved from managing individual computers to coordinating distributed services, data, infrastructure, and increasingly intelligent software. But the fundamentals remain the same: build systems that are reliable, maintainable, secure, and appropriate for the problem.
The 1992 bank and the 2026 bank are both well designed if each fits its constraints. When you design your next system, start from the requirements and the trade-offs, not from the newest tool. Adopt microservices, Kubernetes or an AI agent when they solve a problem you actually have, and be ready to explain what each one costs.
Frequently asked questions
What is system design evolution?
It is the way software architecture changed as constraints changed: from centralised mainframes and client-server applications in the early 1990s, to three-tier web applications, distributed systems and SOA in the 2000s, to cloud, containers and microservices in the 2010s, and to platform engineering and optional AI components in the 2020s.
Is a monolith still a good choice in 2026?
Often, yes. A well-structured modular monolith is easier to build, test, deploy and debug than microservices, and one database transaction keeps data consistent. Microservices pay off when many teams need to deploy independently or when parts of the system have very different scaling needs.
What is the difference between client-server and three-tier architecture?
In two-tier client-server, a desktop program holds the user interface and much of the business logic and talks directly to a database. Three-tier architecture separates presentation (a web server or browser), business logic (application servers) and data (the database), so each tier can be secured and scaled on its own.
Does every modern application need AI?
It doesn't. AI components such as RAG, agents and MCP tools are optional capabilities. Use them where they add clear value, such as search, support or summarisation, and keep authorization, prices, payments and other writes in deterministic application code.
What should I learn first for system design interviews?
Start with the fundamentals that never changed: latency, throughput, consistency, availability, data modelling and caching. Then learn how they apply to distributed systems: replication, partitioning, queues, idempotency, observability and cost. Kubernetes and AI patterns build on those foundations.
Discussion comments
No comments yet. Start the discussion.