Glossary of YTsaurus Flow

Introduction

This article collects all the key concepts of YTsaurus Flow: from the basic data processing model to the architecture and mechanisms for ensuring correctness. The material is structured from simple to complex: first, stream processing is covered, followed by parallelism, external connections, system design, and pipeline management.

When you’re getting familiar with the system, read the article from start to finish, following the links for details. Later, this page will serve as a reference: other documentation sections often link here to remind you of term definitions.

Basic model

To understand Flow as a stream processing system, you can use an analogy with a factory assembly line:

• Pipeline — the entire factory that solves a specific business task.
• Stream — the conveyor belt that carries data.
• Message — an individual item on the belt.
• Computation — a workstation that takes items from one or several belts, processes them, and places the results on other belts.

Pipeline

This is a specific business task that is being executed. It can contain many different data grouping and processing nodes.

From the YTsaurus perspective, a pipeline is represented as a Cypress object of type pipeline. For details about the object structure, creation methods, and management, see the Pipeline Object section.

Stream

A stream is a schematized, named data flow that connects computations in the pipeline graph. Each stream has a fixed schema (a set of typed fields) and is identified by stream_id. A stream consists of messages. Each stream has exactly one writer — a computation that outputs data to it — and an arbitrary number of readers.

The order of messages in a stream is not guaranteed in general. However, for derived messages with matching keys along the entire lineage chain, there are guarantees of relative order. For more details, see the Message Processing Order section.

Computation

A computation is a node in the pipeline graph that performs a specific data transformation. It reads messages from input streams, processes them, and writes the results to output streams. It can have multiple inputs and multiple outputs. The input stream is split into partitions for parallel processing.

For more details, see the Computation section.

Passthrough Computation

This is a type of computation that doesn’t contain any user business logic: incoming messages are converted to the output stream schema and passed on unchanged. It’s used for simple schema alignment between streams.

For more details, see Computation.

Swift

This is a data processing principle in Flow where the result of a computation isn’t stored in YTsaurus. Instead, the transformation function must be strictly deterministic — if needed, the result is recomputed. This reduces the load on YTsaurus while maintaining exactly-once guarantees.

For more details, see the Swift section.

Message

A single message within a stream.

In the context of a single computation, there are several types of messages:

• input — messages from other computations; essentially, they’re the output of another computation, but with a grouped schema.
• source — messages from internal sources.
• timer — special internal timer messages.
• output — the result of a computation’s work, which becomes public and can go to an internal sink and to other computations.

Key

A set of message field values defined via group_by_schema in the spec of a computation. All messages with the same key are routed to a single partition — each partition is assigned a key range [LowerKey; UpperKey). The key is the isolation unit for state, timers, and ordering guarantees.

The concept of a key depends on the message type:

• For input messages, the key is defined via group_by_schema.
• For source messages, the key is the source’s partition key.
• For output messages, there is no key.

Lineage

The collection of input messages and timers from which specific output results of a computation were derived. The framework uses lineage to compute system output messages and to ensure ordering guarantees for derived messages.

For more details, see the Lineage section.

Timer

A delayed-call mechanism tied to a specific grouping key within TTransformComputation. A timer lets a computation say, “Wake me up when time X arrives.”

Each timer contains two timestamp fields:

• TriggerTimestamp — the trigger moment. When the EventWatermark (or another configured watermark) exceeds this value, the timer is passed to the computation for processing.
• EventTimestamp — the business time of the original event, “held” in the timer.

Timers are reliably stored in YTsaurus. The result of timer processing is recorded in the same transaction in which the timer is removed — this ensures exactly-once guarantees.

For more details, see the Timers section.

Resource

Resources exist to describe data shared across multiple jobs. They can be clients to YTsaurus and other systems, caches for calls to external systems, machine learning models, and so on.

Currently, resources can only be static (that is, unchangeable without a restart).

Distributed Throttler

In Flow, you can create named distributed throttlers that are shared across all jobs in a pipeline. They’re used to limit the load on external APIs, balance throughput between partitions of a single computation, or slow down reading from sources. They’re implemented as a token bucket on the controller; jobs request a quota before processing messages or manually from user code. For more details, see the Distributed Throttler section.

State

Persistent data associated with a specific grouping key within a computation. It’s stored in YTsaurus dynamic tables and updated atomically when an epoch is committed. An empty state value corresponds to the absence of a row in the table.

For more details, see the Stateful Processing section.

Parallelism and execution

To ensure scalability of the data processing workflow, Flow splits each computation into multiple partitions, which are processed in parallel within jobs.

Partition

The input stream to a specific computation can be quite large, so for parallel data processing, the computation is split into multiple partitions. Approximate stream limits per partition: no more than 1 MB/s and no more than 1000 messages per second.

Thanks to the group_by_schema field in the computation spec, the input stream is grouped, and data with the same key will always go to a specific partition. For this, each partition is assigned a key range [LowerKey; UpperKey).

Job

A specific run of a handler for a specific partition on a worker selected by the controller.

Epoch

A job processes data in epochs to regularly and reliably save the processing progress. The epoch mechanism helps maintain exactly-once guarantees with a reasonable load on YTsaurus, and also to minimize downtime when recovering from failures.

Some epochs may not contain commits to YTsaurus. Some computation implementations may not make any commits at all.

Layout

A list of all jobs and partitions in the system. In other words, it’s a description of what’s running where right now.

External connections

Flow interacts with external systems (queues, tables, etc.) via connectors. Each connector provides a source for reading and/or a sink for writing.

Connector

A component that connects the pipeline to an external system by reading messages from the connector’s source and/or writing messages to the connector’s sink. A connector can include both a source and a sink (for example, in the case of QYT) or only a source (for example, in the case of ServiceLog).

In code and specs, sources and sinks are described directly. The connector itself doesn’t exist as an entity; it’s more of a logical grouping element for sources and sinks by the external system that’s being interacted with.

For more details about connectors, see the Connectors Documentation.

Source

A component that lets a computation read messages from an external system. It’s implemented as a separate class and configured in the computation spec.

Sink

A component that lets a computation write messages to an external system. It’s implemented as a separate class and configured in the computation spec.

System architecture

A running pipeline consists of a set of processes with two roles — controller and worker. The controller manages work distribution, and the workers directly process the data.

Controller

Manages the entire pipeline. It assigns jobs to workers, provides an API for pipeline management, is responsible for synchronizing all settings, monitors the status of all workers, and collects their statuses. It can be run in multiple instances (recommended: 2–3); then, one instance will be the leader, and the others will wait in stand-by mode to ensure fault tolerance.

Specs for execution are provided to the controller via a special API.

Computation Controller

The controller for a specific computation: it’s responsible for creating the required number of partitions with the necessary settings. Don’t confuse it with the plain controller. Computation Controller is a class, while the controller is a program that manages the cluster operation and creates the necessary Computation Controllers.

JobManager

A module responsible for creating and distributing jobs across workers.

LeaseManager

A module responsible for creating and pinging Lease — master transactions that jobs use during commits as prerequisite_transaction_ids. They help ensure exactly-once guarantees.

Worker

One of the two main types of processes in the system, performing the actual computations within the jobs assigned to it. It regularly sends heartbeats to the controller, reporting the statuses of all jobs. In return, it receives the current system settings, in particular the Layout.

Heartbeat

A periodic message a worker sends to the controller reporting the statuses of all its jobs. The controller's response carries the current system settings, in particular the Layout; a worker that stops sending heartbeats is treated as unavailable.

Message Distributor

A worker module responsible for distributing messages between different jobs. It runs based on the system’s current layout. It sends a message until it receives confirmation from the recipient about processing, including the fact of a reliable commit of the processing result (referred to in the codebase as MarkPersisted).

Resource Manager

A worker component that creates resources for shared use. For example, several computations can use the same binary database or the same YTsaurus client.

BufferStateManager

A component responsible for creating and maintaining the current sizes of all (or almost all) buffers in the system. Each job has buffers for incoming and outgoing messages. The task of BufferStateManager is to ensure the buffer sizes necessary for uninterrupted operation.

Companion

A separate process launched on the same host as the worker and executing user code in Python, Go, Java or Kotlin, or C++. It interacts with the worker via gRPC. It runs computations outside the worker process, including non-C++ implementations and C++ code through the C++ companion.

For more details, see the Companion section.

Timestamps and watermarks

Correct time-based event processing is one of the key tasks of a streaming system. Flow tracks three types of timestamps for each message and maintains a watermark mechanism to determine processing progress.

EventTimestamp, SystemTimestamp, AlignmentTimestamp, StabilizedEventTimestamp, and Watermarks

For each message in Flow, the following timestamps can be used:

• EventTimestamp — the true time of the event associated with the message.
• SystemTimestamp — the time when the specific message was created. For messages generated inside Flow, SystemTimestamp takes the “YTsaurus time”. For source streams, it’s the time when the message appeared in the remote system.
• AlignmentTimestamp — a timestamp used to align the processing progress of partitions. It’s computed automatically.
• StabilizedEventTimestamp — a timestamp computed based on AlignmentTimestamp; it’s used as a non-decreasing approximation of EventTimestamp across the message stream (with a matching set of keys in the lineage).

The first three timestamps are stored in the message itself, while StabilizedEventTimestamp is computed on the fly when needed.

Watermark is a timestamp indicating that the system no longer expects older events to arrive. Each stream in the system maintains current SystemWatermark and EventWatermark values.

Ensuring Exactly Once

By default, Flow ensures exactly-once semantics for event processing. The mechanism is based on barrier Lease transactions, deduplication of input messages by message_id (for internal streams) and by offsets (for sources), and the atomicity of output data via epoch transactions. If needed, you can relax the semantics to at-least-once or at-most-once.

For more details, see the Processing Guarantees section.

Message deduplication and message processing order

The main way to deduplicate messages is by message_id, not by offset. The offset-based mechanism is used only when working with queues. Also, Flow doesn’t provide any guarantee about the order of event processing.

Manage the pipeline

This section describes the pipeline lifecycle operations: from configuring it via specs to release, migration, and running in different environments.

Spec and DynamicSpec

Any pipeline running on Flow is a combination of two YSON configs called Spec and DynamicSpec.

• Spec — defines the pipeline’s static properties, its topology, node properties, connections between them, object types, etc. You can change the static spec only if the pipeline is stopped.
• DynamicSpec — defines the pipeline’s dynamic properties.

Accordingly, the specs contain the properties of all system components, both from the user layer (for example, the list of computations and the stream schemas between them) and from the system-level settings (compression codecs in system tables, buffer sizes, etc.).

Pipeline release

Updating the pipeline, which includes changing its executable files.

Update pipeline specs

Update the static and/or dynamic spec of the pipeline.

Start, stop, and pause the pipeline

Change the pipeline’s target state and transition the pipeline to that state. The existing states are:

• unknown — the pipeline hasn’t started yet.
• working — the pipeline is running, and messages are being processed.
• stopped — the pipeline is stopped, and all messages have been fully processed.
• paused — the pipeline is paused (jobs are stopped, and intermediate messages might not be fully processed).
• draining — a transitional state: the pipeline is in the process of stopping.
• pausing — a transitional state: the pipeline is in the process of pausing.
• completed — a final state: all pipeline sources were finite (finite = true), and all messages from them have been processed. You can’t exit this state — you’ll need to recreate the pipeline. This state most often appears in integration tests, and it can also occur in production pipelines if the deployment steps are executed in the wrong order (for example, if a source was mistakenly marked as finite) or if an incorrect spec is set.

Internal pipeline tables

Service tables that are placed in the pipeline directory in YTsaurus. They are necessary for the pipeline to run (for example, to deduplicate incoming messages, store partition information, etc.).

The full list of internal tables and their purposes are provided in the section Pipeline Object → Internal tables.

User tables

Tables that users work with. Typically, their content has a product-related meaning, for example:

• Ordered tables — queues from which the user pipeline reads input events or writes output events.
• Sorted tables with entity states (banner/offer/user profiles, etc.) that the user builds while the pipeline is running.

Migration

The process of changing the structure of objects in YTsaurus — for example, adding new tables, changing the schema of existing tables, or deleting them.

The main reasons to run migrations are:

• Deploying a new pipeline version that isn’t compatible with the current state of objects in YTsaurus. Migration lets you update the objects to a state that the new pipeline version can work with, without losing data.
• Changing the table type (replicated/standalone/chaos) for performance or availability reasons.

Examples of changes:

Require migration Changing multiple tables (adding new ones or deleting old ones). Changing table schemas. Changing table types (replicated/standalone/chaos).

Do not require migration Changing most table attributes (for example, changing compaction settings, which only affect performance). Changing the list of replicas.

Environment

The installation where the pipeline runs — testing, pre-production, or production.

It’s a synonym for the term “stage”.

See also

• Computation
• Spec and DynamicSpec
• Watermarks and Timers
• Stateful processing
• Connectors

You can change it only if the pipeline is stopped.