Lineage in YTsaurus Flow
Lineage (from English, “genealogy”) is information about which input messages and timers a specific output result of a computation was derived from. You use this information during processing so the framework can compute metadata and ensure ordering guarantees. Note that this information isn’t stored in the output message itself.
Why you need lineage
The framework uses lineage for two purposes:
- Computing meta-fields. Based on parent messages, Flow automatically populates the meta-fields of output messages, such as
EventTimestamp,AlignmentTimestamp, and others. For Swift computations and passthrough,AlignmentTimestampis inherited from the parents unchanged. This ensures correct prioritization of messages in downstream computations. - Guaranteeing the order of derived messages. If two messages share the same grouping keys across the entire lineage chain from the source to the current computation, their relative processing order is preserved. For more details, see the section Message Processing Order.
Default behavior
In most cases, you don’t need to manage lineage explicitly—the framework automatically sets the parents:
| Function type | Parent of the output message |
|---|---|
IProcessFunction::ProcessMessage / RowFunction |
The current input message |
IBatchProcessFunction::Process / BatchFunction |
All messages in the current batch |
| Timer handler | The current timer |
When to set lineage explicitly
By default, the entire current batch is considered the parent of the output message. Setting lineage explicitly lets you narrow this set to a specific subset of input objects. This makes the computation of EventTimestamp and AlignmentTimestamp more precise.
In a Swift computation the narrowing is mandatory: every output message must have exactly one parent (multiple only with allow_batching_with_relaxed_guarantees). With per-message processing the framework assigns the parent automatically; in batch processing user code sets it. In a Transform computation explicit lineage is optional.
API
You set lineage using the SetParents / set_parent_ids / setParentIds / WithParentIDs method on the OutputCollector object. The method returns a new collector with the lineage context attached. All calls to AddMessage / add_message / addMessage on this collector will carry that lineage.
For more details on how to use this in each language:
Conversion statistics
Flow estimates the output-to-input ratio for each stream pair, separately for message count and byte size. These statistics describe processed messages, not guaranteed deliveries.