Article: From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline

Parveen Saini — Mon, 04 May 2026 11:00:00 GMT

This article describes how a production delta-index pipeline migrated from scheduled batch to micro-batch Spark Structured Streaming. It covers why record-level streaming was rejected, how partition-based watermarks replaced fragile S3 completion markers, overlap-window correctness, and restart-as-design strategies for better predictability in object-store–based ingestion systems.

By Parveen Saini

Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance

Leela Kumili — Fri, 01 May 2026 14:06:00 GMT

Confluent introduces a new approach in Apache Kafka that moves schema IDs from message payloads to record headers, aiming to simplify schema governance and evolution. The update integrates with Schema Registry, improves compatibility across serialization formats, and reduces coupling between data and metadata in event-driven architectures.

By Leela Kumili

InfoQ - Streaming

Article: From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline

Confluent Moves Schema IDs to Kafka Headers to Simplify Schema Governance