Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows

Leela Kumili — Fri, 11 Sep 2026 14:17:00 GMT

Netflix has reworked its Conductor workflow orchestration engine to handle larger workloads, increasing supported workflow size from about 2,500 to 30,000 tasks and reducing p99 workflow evaluation latency by about 40%. Conductor 4.0 separates workflow metadata from task data, moves evaluation to asynchronous processing, and introduces dynamic worker allocation and concurrency controls.

By Leela Kumili

Presentation: From S3 to GPU in One Copy: Rethinking Data Loading for ML Training

Onur Satici — Fri, 04 Sep 2026 11:00:00 GMT

Onur Satici explains how Vortex, an open-source columnar file format under the Linux Foundation, revolutionizes high-throughput data loading. He details how cascading lightweight encodings, layout-based segment pruning, and zero-copy memory pipelines eliminate CPU/NVMe bottlenecks to stream S3 data straight to GPUs at speeds up to 60 Gbps without requiring upfront data reprocessing.

By Onur Satici

InfoQ - S3

Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows

Presentation: From S3 to GPU in One Copy: Rethinking Data Loading for ML Training