InfoQ - Apache Spark https://www.infoq.com InfoQ Apache Spark feed Article: From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline https://www.infoq.com/articles/micro-batch-streaming-lessons-learned/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Apache+Spark <img src="https://res.infoq.com/articles/micro-batch-streaming-lessons-learned/en/headerimage/micro-batch-streaming-lessons-learned-header-1777381781538.jpg"/><p>This article describes how a production delta-index pipeline migrated from scheduled batch to micro-batch Spark Structured Streaming. It covers why record-level streaming was rejected, how partition-based watermarks replaced fragile S3 completion markers, overlap-window correctness, and restart-as-design strategies for better predictability in object-store–based ingestion systems.</p> <i>By Parveen Saini</i> Apache Spark Spark Streaming AI, ML & Data Engineering Development article Mon, 04 May 2026 11:00:00 GMT https://www.infoq.com/articles/micro-batch-streaming-lessons-learned/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Apache+Spark Parveen Saini 2026-05-04T11:00:00Z /articles/micro-batch-streaming-lessons-learned/en