Stream Processing Frameworks: Apache Flink vs Kafka Streams vs Spark Streaming

In distributed streaming networks, events encounter variable network latency and mobile disconnections before arriving at the cluster:

• Event Time: The exact timestamp recorded when the event occurred on the client device (e.g., user clicked checkout at 12:00:01).
• Processing Time: The timestamp on the stream processing server when it physically executes the record (e.g., 12:00:09 due to cellular lag).
• Ingestion Time: The timestamp when Kafka persisted the message to a partition broker.

The Watermark Solution for Out-Of-Order Data:

To compute accurate windows without waiting indefinitely for delayed network packets, engines like Apache Flink use Watermarks:

• A Watermark is a temporal heuristic assertion flowing through the data stream signaling: "We guarantee with high probability that no further events with EventTime < T will arrive."
• When Watermark(T) passes an operator, the engine finalizes and closes all time windows ending at or before T, emitting results downstream while diverting late-arriving stragglers to a side output.