https://www.infoq.com InfoQ Big Data Articles feed https://www.infoq.com/articles/time-series-storage-design/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Big+Data-articles <img src="https://res.infoq.com/articles/time-series-storage-design/en/headerimage/Time-Series-Storage-Design-Choices-That-Shape-Cost-and-Performance-header-1778155792101.jpg"/><p>Every time-series database makes a set of storage design decisions: how to lay out rows, when to compress, what to partition on. These decisions determine cost and query performance more than the choice of database itself. This article works through those fundamentals from first principles, using widely available tools like PostgreSQL and Apache Parquet to make each trade-off measurable.</p> <i>By Nirmesh Khandelwal</i> Big Data Time Series Data AI, ML & Data Engineering article Tue, 12 May 2026 09:00:00 GMT https://www.infoq.com/articles/time-series-storage-design/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Big+Data-articles Nirmesh Khandelwal 2026-05-12T09:00:00Z /articles/time-series-storage-design/en https://www.infoq.com/articles/micro-batch-streaming-lessons-learned/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Big+Data-articles <img src="https://res.infoq.com/articles/micro-batch-streaming-lessons-learned/en/headerimage/micro-batch-streaming-lessons-learned-header-1777381781538.jpg"/><p>This article describes how a production delta-index pipeline migrated from scheduled batch to micro-batch Spark Structured Streaming. It covers why record-level streaming was rejected, how partition-based watermarks replaced fragile S3 completion markers, overlap-window correctness, and restart-as-design strategies for better predictability in object-store–based ingestion systems.</p> <i>By Parveen Saini</i> Spark Streaming Apache Spark AI, ML & Data Engineering Development article Mon, 04 May 2026 11:00:00 GMT https://www.infoq.com/articles/micro-batch-streaming-lessons-learned/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Big+Data-articles Parveen Saini 2026-05-04T11:00:00Z /articles/micro-batch-streaming-lessons-learned/en