Kappa becomes interesting when a team is tired of paying for duplicate batch and speed paths and is willing to make the event log the center of the architecture.
In practice, this chapter helps decide when one streaming path truly simplifies the system and when historical replay, retention, and durable-log requirements cost more than the simplification is worth.
In interviews and architecture reviews, it is especially useful when you need to explain why the batch layer was removed, which complexity disappeared, and which new requirements around log storage, backfill, and materialized views appeared instead.
Practical value of this chapter
Design in practice
Helps design one streaming path without duplicating batch and stream logic.
Decision quality
Guides Kappa adoption where historical replay and fresh data matter more than the cost of a batch layer.
Interview articulation
Supports explaining backfill, bootstrap, and historical recomputation in Kappa.
Risk and trade-offs
Highlights retention, storage, and durable-log requirements of the model.
Related book
Big Data (Nathan Marz)
This is where Lambda Architecture was formulated — the starting point whose cost Kappa grew up to push back against.
Kappa Architecture builds processing around an event log: events are stored once, and serving views are recomputed from that log through the very path that handles the live stream.
Unlike Lambda Architecture, Kappa drops the separate batch layer and speed layer. The cost of the approach does not disappear — it moves into the durable log, the retention rules, and the discipline of reprocessing.
In practice the decision is made not by saying “we removed the batch layer” but in the details: stream processing, stateful processing, materialized views, historical replay, backfill, data retention, and delivery semantics.
Why Kappa emerged
One circuit instead of two
Lambda runs two compute paths — batch and speed — and the same logic has to live in two places. The cost of drift shows up fast: the two paths stop agreeing, and every bug gets caught twice.
Replay as a standard operation
Recomputation here is not a separate batch pipeline but a replay of events from the log through the same code that serves the live stream. So fixing the logic never calls for a second implementation.
Streaming-native platform
Mature streaming platforms keep stateful pipelines running without a separate batch stack. But the cost just shifts: the platform now has to provide a durable log, checkpoints, and predictable recovery.
Kappa flow
Kappa removes the separate batch layer: the live stream and historical replay move through the same processing path.
Lambda and Kappa: key differences
| Criterion | Lambda | Kappa |
|---|---|---|
| Compute model | Batch, speed, and serving layers. | One streaming path plus materialized views. |
| Code paths | Two separate paths for batch recomputation and fast updates. | One processing path for the live stream and historical replay. |
| Reprocessing | Often through a batch recompute over the full historical dataset. | Replay events from an immutable log through the same stream processor. |
| Latency | Low through the speed layer, then reconciled with the batch result. | Low if the stream processor and state store can handle the load. |
| Operational complexity | Higher because two stacks must keep their semantics aligned. | Fewer paths, but stricter requirements for the streaming platform. |
| Best fit | When batch, ETL, and streaming workloads are already mature and separate. | When the platform is built around an event log (Kafka/Pulsar). |
How Kappa is implemented in practice
- Make the event log the source of truth: events should be immutable and schemas should be versioned.
- Move key materialized views into the streaming processing path.
- Make historical replay and backfill standard operations that run through the same code.
- Separate stateful processing from serving APIs with clear data contracts.
- Define SLAs for late events, ordering, and exactly-once / at-least-once delivery guarantees.
When to choose and what to look for
Kappa is suitable if
- Core domain data is already generated as events.
- The system needs one logic path for live processing and historical recomputation.
- The team is ready to operate a streaming stack and stateful processors.
Risk areas
- Poor event schemas and weak schema governance.
- Heavy joins or windowed computations without state-size control.
- Underestimating historical replay costs: CPU, storage I/O, and backpressure.
References
Related chapters
- Big Data: Principles and best practices of scalable realtime data systems (short summary) - Kappa's roots in the Lambda comparison and the move from two processing paths to a single event log.
- Streaming Data (short summary) - Hands-on stream-processing practices: delivery semantics, windows, stream state, and operational limits.
- Kafka: The Definitive Guide, 2nd Edition (short summary) - Technology foundation for Kappa with partitioned logs, retention, and historical replay as a standard mechanism.
- Event-driven architecture: Event Sourcing, CQRS, Saga - Architectural integration layer where Kappa naturally builds on event logs and workflow patterns.
- Data Pipeline / ETL / ELT Architecture - How to embed Kappa pipelines into end-to-end data platforms with orchestration and quality controls.
- Distributed message queue - Queue design case covering event order, data durability, and consumer scaling under high load.
- Designing Data-Intensive Applications, 2nd Edition (short summary) - Core theory on stream/table duality, replication, and distributed data processing.
- Enterprise Integration Patterns (short summary) - Pattern language for robust service integration in streaming architectures and event workflows.
- Data Mesh in Action (short summary) - Organizational view on scaling Kappa-like platforms through domain-owned data products.
- Google Global Network: Evolution and Architectural Principles for the AI Era - Network context for cross-region stream processing and latency-sensitive workloads.