Most teams meet Apache Kafka the same way. Someone asks for a feature "in real time", an architect draws a broker in the middle of the diagram, and a year later the company owns a cluster that four services write to and nobody feels safe turning off. Deciding when to use Apache Kafka deserves more scrutiny than that drawing gets, because the bill arrives in headcount and cloud spend long after the proof of concept.
Kafka is not exotic. The Apache Software Foundation describes it as a distributed event streaming platform for data pipelines, streaming analytics, data integration and mission-critical applications, and states that more than 80% of Fortune 100 companies use it. Ubiquity says nothing about a specific workload, though. Plenty of pipelines running on Kafka today would run fine, and cheaper, on a scheduled job.
This article is the center of our streaming material: what the platform does, the four signals that justify adopting it, the workloads where a nightly batch still wins, what it costs to operate, and how to start without betting the data platform on a technology nobody on the team has run at 3 a.m.
What is Apache Kafka?
Kafka stores events in an append-only log. A producer writes a record to a topic, the topic is split into partitions, and each record lands at a numbered position called an offset. Records are never edited in place: they sit in order until the retention window expires.
That storage model explains most of Kafka's behavior. A consumer reads by advancing its own offset, so reading does not destroy the record and several independent applications read the same topic without coordinating. Consumers are organized in consumer groups, and Kafka assigns each partition to one member, which is how a single logical consumer scales across machines.
Retention is time-based by default. The broker configuration reference sets log.retention.hours to 168, which is seven days, and the value is a topic-level decision rather than a law of nature. Teams needing longer history extend retention or enable tiered storage, which pushes older segments into object storage. That feature reached general availability in Kafka 3.9, turning "keep a year of events" into a configuration instead of a project.
What separates Kafka from a queue: the record stays, and each consumer group tracks its own position. Source: BIX Tech, based on the Apache Kafka documentation.
One piece of context matters for anyone evaluating Kafka now. Version 4.0, released in March 2025, was the first major release to run entirely without ZooKeeper, on the built-in KRaft consensus protocol, and it made the rewritten consumer rebalance protocol generally available. Share groups became production-ready in 4.2, in February 2026. Anyone judging Kafka on a five-year-old memory of it is judging a different system.
When to use Apache Kafka: four signals
Adoption makes sense when at least two of the following are true. One signal alone rarely pays for a cluster.
Several consumers need the same event. An order confirmation feeds billing, inventory, customer notifications, the fraud model and the analytics warehouse. Point-to-point integrations between those five grow quadratically and break in different ways. A log each of them reads independently is the cheapest structure available.
History has to be replayable. A new service needs three months of events to build its state, a retrained model has to see exactly what the old one saw, or a bug corrupted a downstream table and the fix is to reprocess. Replay separates Kafka sharply from a traditional queue, where a consumed message is gone.
Producers and consumers change at different speeds. The team emitting events ships weekly, the team consuming them ships monthly, and a third team has not been hired yet. A topic with a documented schema lets those cadences diverge without a coordinated release.
Order matters per entity, at volume. Everything that happens to one account, device or shipment has to be processed in sequence, while millions of other entities run in parallel. Kafka delivers that through the partition key, a design decision rather than a default.
When none of this describes the workload, streaming is usually the expensive path to a result a scheduled job already produces.
When batch is enough
The test we apply before any architecture discussion: does the decision change if the data is six hours old? If the answer is no, latency is not the constraint, and the budget belongs somewhere else, such as data quality or the semantic layer.
Finance closes the month on records that settle overnight. A churn model retrained weekly does not improve because its features arrived in five seconds. Regulatory extracts have deadlines measured in days. For all of them, an orchestrated batch pipeline is cheaper to run and far easier to fix at 2 a.m. when a source system sends malformed data.
Streaming also carries costs that rarely appear in the first estimate. Late-arriving events force a windowing strategy, exactly-once processing requires idempotent consumers and transactional writes, and schema changes become a compatibility negotiation. Debugging moves from "read the failed job's log" to "reconstruct what the system believed at that offset". Our full comparison is in streaming versus batch processing; the architecture should match the value of time in the specific decision.
| Workload | Value of one hour of freshness | Reasonable first choice |
|---|---|---|
| Monthly financial close and regulatory extracts | None, the deadline is in days | Batch |
| Marketing attribution and model retraining | Low, the model is refreshed weekly | Batch |
| Executive dashboards reviewed each morning | Low, decisions happen daily | Batch or micro-batch |
| Inventory sync across sales channels | High, oversell risk is immediate | Streaming or change data capture |
| Fraud scoring during checkout | Total, the decision expires in milliseconds | Streaming |
| Operational monitoring of equipment or logistics | High, the alert drives an intervention | Streaming |
That table decides the first version, never the final one: mature platforms often run both paths from the same events.
What Kafka actually costs to run
Infrastructure. Brokers are stateful, replicated machines. A production cluster keeps three copies of every partition across availability zones, so the storage bill is roughly three times the raw volume, plus the cross-zone traffic replication generates. Partitions cost too, since each one consumes file handles, memory and rebalance time, which makes Kafka topic design an early cost decision disguised as a technical detail.
Retention. Seven days of a 200 GB per day feed, replicated three times, is over 4 TB of hot disk before anyone reads it. Tiered storage moves the cold portion to object storage and changes that arithmetic, at the cost of higher latency on old reads.
People. Somebody upgrades brokers, watches consumer lag, sizes partitions, rotates certificates and answers the page. Managed services absorb part of that, never all: topic design, schema evolution and consumer behavior stay on your side no matter who runs the brokers.
Managed pricing mirrors those surfaces. Amazon MSK bills provisioned clusters by broker instance-hour plus storage in GB-months, and MSK Serverless by cluster-hour, partition-hour and data volume. Confluent Cloud bills on elastic compute units, data transfer and storage. The shape is consistent: you pay for capacity, for movement and for keeping bytes around.
The four layers around the broker
Saying "we use Kafka" describes the transport and leaves out most of the work. Four layers sit around the broker, and each one is a separate decision.
Integration happens through Kafka Connect, which moves data between Kafka and external systems with configured connectors instead of custom producer code. Most database ingestion and warehouse delivery belongs there.
Processing splits into two very different options. Kafka Streams is a Java library that runs inside your application, with no cluster to operate, and fits Kafka-to-Kafka transformations with moderate state. Apache Flink is a distributed engine with its own runtime, and fits complex event processing, large state and sources beyond Kafka. We covered the pairing in how Apache Kafka and Flink work together; the honest summary is that Flink buys capability and charges operational complexity for it.
Contracts live in a schema registry, an ecosystem component rather than part of Apache Kafka itself. Without one, the topic becomes an undocumented interface between teams, and the first incompatible producer change teaches everyone why the piece existed.
Queue semantics now exist natively. Share groups, generally available since Kafka 4.2, let several consumers cooperatively process records from the same partition with per-record acknowledgement, covering the job-queue pattern that used to push teams toward a second piece of middleware.
Four decisions hide behind "we use Kafka". Source: BIX Tech.
Self-managed, managed, or Kafka-compatible
The protocol became a standard, so the question splits in two: who operates the brokers, and whether you need Kafka itself or only its API.
| Option | What you operate | How you pay | Fits when |
|---|---|---|---|
| Self-managed Apache Kafka | Brokers, KRaft controllers, upgrades, security, monitoring | Infrastructure plus the engineers | You already run stateful systems and need full control or on-premises |
| Amazon MSK | Topics, clients, schemas | Broker-hour plus storage, or serverless by cluster, partition and volume | The stack is on AWS and the team wants the open-source API without broker operations |
| Confluent Cloud | Topics, clients, schemas | Elastic compute units, transfer, storage | You want connectors, registry and stream processing as one product |
| Redpanda | Topics, clients, schemas | Cluster or cloud subscription | You want Kafka API compatibility in a single C++ binary, with no JVM to tune |
| Amazon Kinesis Data Streams | Streams, shards or on-demand capacity | Shard-hour or throughput, plus retention | The workload is AWS-native and the team prefers fewer knobs to fewer limits |
Kinesis deserves a note, because its limits are explicit in a way Kafka's are not. Each provisioned shard accepts up to 1 MB/s or 1,000 records per second of writes and serves 2 MB/s of reads, with retention from 24 hours to 365 days. Capacity planning becomes arithmetic instead of tuning, which some teams consider the main feature. The full comparison is in Kafka versus Kinesis.
BIX Tech works with all of these, on AWS, Azure, Google Cloud and on-premises, and the recommendation tracks a team's operating model far more than the feature matrix. An excellent platform that nobody on staff can debug is a liability.
How to start without betting the platform
Pick one use case with a deadline a human can feel: fraud checks, stock availability, an SLA alert. A vague "we want real-time data" produces a cluster with no owner and no measurable outcome.
Name the events before choosing the tooling: OrderPlaced, PaymentAuthorized, ShipmentDispatched, each with its fields and the entity it keys on. That vocabulary survives every technology decision that follows.
Start with a narrow topic domain and a deliberate partition key, since both are painful to change later, then set durability before the first production write rather than after the first data loss. Deploying Kafka in production safely covers replication factor, minimum in-sync replicas and producer acknowledgements, where most silent loss originates.
Instrument consumer lag from day one and alert on it, since lag is the one metric that says whether the system is keeping its promise. Then write down who owns the cluster outside business hours. That answer, more than any benchmark, decides whether streaming works in your company.
Where to go deeper
When the open question is whether to stream at all, start with the decision material. Batch versus stream processing frames the trade-off around pipeline design, real-time analytics and when it adds value is deliberately skeptical, and why latency matters in modern data pipelines argues the other side with the measurable cost of delay. Once the decision is made, the best architecture for real-time analytics lays out the stack options.
For the build itself, Apache Kafka for modern data pipelines is the reference pattern from ingestion to serving, and Apache Kafka explained covers the moving parts, delivery guarantees and tuning in more detail than this pillar allows. Teams running streaming alongside scheduled work will want automating real-time pipelines with Airflow, Kafka and Databricks, reliability and CI/CD included, while streaming MongoDB to ClickHouse shows the database-to-analytics path that skips broker operations entirely.
Comparing platforms is its own body of work. Event-driven architecture with Redpanda covers the Kafka-compatible route, and Apache Flink and Amazon Kinesis at scale documents the AWS-native pairing. Smaller workloads have a smaller answer: Redis beyond caching explains when Redis Streams handles the job without a broker cluster.
Streaming is also becoming infrastructure for AI systems. Building reactive agent pipelines with Kafka treats agents as producers and consumers of events, and ClickHouse for real-time analytics covers the serving layer that makes streamed data queryable at low latency.
Choosing Kafka well comes down to two honest answers: how many independent consumers genuinely need the same events, and how fast a decision loses its value. Say "several" and "minutes", and the platform pays off for years. Say "one" and "tomorrow", and you get a cluster to maintain plus a report that was already fine. The technology is mature enough that the risk sits entirely in that judgment.
If your company is weighing a streaming platform, or already running one nobody is comfortable operating, our specialists can review the use case, the cost model and the operating burden before the architecture hardens. Talk to the team and decide with numbers instead of diagrams.
Frequently asked questions about Apache Kafka
What is Apache Kafka used for?
Apache Kafka is a distributed event streaming platform used to move events between systems in real time and keep them replayable. Typical uses are decoupling microservices, feeding analytics and machine learning pipelines, change data capture from databases, and operational monitoring. The project reports that more than 80% of Fortune 100 companies run it.
When should you use Apache Kafka instead of a batch pipeline?
Use Kafka when several independent consumers need the same events, when history has to be replayable, when producers and consumers evolve at different speeds, or when per-entity ordering matters at high volume. If a decision would be identical with six-hour-old data, a scheduled batch pipeline is cheaper to build, run and debug.
Is Apache Kafka a message queue or a database?
Neither, though it borrows from both. Kafka stores an ordered, append-only log that consumers read by advancing their own offset, so records survive being read and can be replayed. It keeps data for a configured retention window, seven days by default, rather than indefinitely, and it offers no ad hoc query capability.
How much does it cost to run Apache Kafka?
The software is free under the Apache License, and the cost sits in replicated infrastructure, retention and operations. Production clusters keep three copies of each partition across zones, so storage and cross-zone traffic scale accordingly. Managed services such as Amazon MSK and Confluent Cloud bill on compute capacity, data transfer and stored volume.
Do you still need ZooKeeper to run Kafka?
No. Apache Kafka 4.0, released in March 2025, was the first major version to run entirely on KRaft, the built-in consensus protocol, and removed ZooKeeper support. New clusters run KRaft by default, which eliminates one distributed system from the deployment and simplifies operations.
