Pinterest Uses Event Time Watermarks to Finalize Petabyte Scale Data Partitions

Pinterest has introduced event time-based partition finalization for its database ingestion framework. The mechanism uses Apache Flink to track event time and Apache Iceberg metadata to publish a non-regressing watermark, helping downstream jobs determine when continuously ingested CDC data is sufficiently complete for processing.

By Leela Kumili