InfoQ - Data Lake https://www.infoq.com InfoQ Data Lake feed Presentation: From S3 to GPU in One Copy: Rethinking Data Loading for ML Training https://www.infoq.com/presentations/vortex-columnar-file-format-gpu-streaming/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Data+Lake <img src="https://res.infoq.com/presentations/vortex-columnar-file-format-gpu-streaming/en/mediumimage/onur-satici-medium-1787813397611.jpeg"/><p>Onur Satici explains how Vortex, an open-source columnar file format under the Linux Foundation, revolutionizes high-throughput data loading. He details how cascading lightweight encodings, layout-based segment pruning, and zero-copy memory pipelines eliminate CPU/NVMe bottlenecks to stream S3 data straight to GPUs at speeds up to 60 Gbps without requiring upfront data reprocessing.</p> <i>By Onur Satici</i> Architecture Columnar Databases Data Lake GPU Machine Learning S3 Streaming Transcripts QCon London 2026 Data Pipelines CUDA Rust Performance AI, ML & Data Engineering Architecture & Design Development presentation Fri, 04 Sep 2026 11:00:00 GMT https://www.infoq.com/presentations/vortex-columnar-file-format-gpu-streaming/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Data+Lake Onur Satici 2026-09-04T11:00:00Z /presentations/vortex-columnar-file-format-gpu-streaming/en Article: Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake Pipelines at Petabyte Scale https://www.infoq.com/articles/beyond-offset-lag-kafka-apache-hudi/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Data+Lake <img src="https://res.infoq.com/articles/beyond-offset-lag-kafka-apache-hudi/en/headerimage/beyond-offset-lag-kafka-apache-hudi-header-1787577734265.jpg"/><p>In this article, author Srikanth Mamidala discusses the data lake architecture used for analytics, reporting, and machine learning and shows how to manage the consumer lag metrics when using Kafka and Apache Hudi.</p> <i>By Srikanth Mamidala</i> Streaming Data Pipelines Data Lake Messaging Apache Kafka AI, ML & Data Engineering article Wed, 26 Aug 2026 09:00:00 GMT https://www.infoq.com/articles/beyond-offset-lag-kafka-apache-hudi/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Data+Lake Srikanth Mamidala 2026-08-26T09:00:00Z /articles/beyond-offset-lag-kafka-apache-hudi/en