Use a spec to describe the pipeline you want. Any high-level API to Flow generates a pipeline spec.
A pipeline has two specs: Spec and DynamicSpec:
Spec is a static object that contains the topology of the entire pipeline. You can change this object only if the pipeline is stopped. The system guarantees that at runtime, every part of the system has the same view of the Spec.
DynamicSpec is the dynamic part of the spec. You can change it at any time, and the changes apply asynchronously to all parts of the system. DynamicSpec usually stores buffer sizes, timeouts, the number of partitions, and so on.
The node’s Config is also an important part. Unlike the spec, you set the config exactly once per release, and you can’t change it dynamically. In the future, some config parameters might move to the specs.
Warning
All parameters in Flow have optimal default values. Don’t change the settings unless you’re sure what they do and how they can affect the system.
You can find the default values by searching for the parameter name in the Flow codebase. They’re mostly grouped in the file spec.cpp.
Spec
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
validate_binary_version
|
Type: bool
Default value: false
|
|
binary_version
|
Type: std::string
Default value: TString("")
|
|
validate_binary_checksum
|
Type: bool
Default value: false
|
|
binary_checksum
|
Type: std::string
Default value: TString("")
|
|
computations
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TComputationIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TComputationSpec>>
Default value: {}
A named enumeration of all pipeline nodes. Keys must match [0-9A-Za-z_-]+.
|
|
resources
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TResourceIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TResourceSpec>>
Default value: {}
A named enumeration of all resources. Keys must match [0-9A-Za-z_-]+.
|
|
streams
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TStreamSpec>>
Default value: {}
A named enumeration of all data streams. Keys must match [0-9A-Za-z_-]+.
|
Computation
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
computation_class_name
|
Type: std::string
Required parameter
The built-in Computation or process-function adapter class name.
For new C++ user logic, select one of the built-in TProcessFunction*Computation adapters. Register the function with YT_FLOW_DEFINE_PROCESS_FUNCTION and name it in processing_function. Don’t create custom Computation classes.
|
|
process_function_graph
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TProcessFunctionGraphSpec>
Several process functions connected by stream IDs and run in one TProcessFunctionComputation, TProcessFunctionSwiftMapComputation, or TProcessFunctionSourceComputation. Batch and per-message functions can be combined. Internal message IDs are derived from their boundary parents, output stream, and suffix; timestamps and external IDs follow the host computation's policy. Swift requires deterministic replay; batch functions set each result's parents with SetParents. Merging different input messages requires parameters/allow_batching_with_relaxed_guarantees = %true and provides at-least-once delivery. In TProcessFunctionSourceComputation, nodes read source_streams, internal messages keep the key of the source message they derive from, nodes cannot have timers, and each output must derive from one source message. A message added with distribute = %false still advances the watermark if it is written to an output of the computation; nodes reading its stream do not receive it. The runner replaces the graph with processing_function, processing_function_parameters, timer_streams, streams_dependency, and allow_timer_self_dependency, so do not set these fields together with the graph.
|
|
processing_function
|
Type: std::optional<std::string>
The fully qualified process-function name registered with YT_FLOW_DEFINE_PROCESS_FUNCTION. Required by TProcessFunction*Computation adapters unless process_function_graph is set.
|
|
processing_function_parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Static process-function parameters. Their type is declared in YT_FLOW_DEFINE_PROCESS_FUNCTION registration.
|
|
group_by_schema
|
Type: NYT::TIntrusivePtr<NYT::NTableClient::TTableSchema>
Default value: {'value': [], 'attributes': {'strict': true, 'unique_keys': false}}
A schema for grouping all input streams. It is a dynamic table schema. Schema properties and requirements:
- Does not contain any ordering rules.
- Can contain computed fields.
- Must be empty only for
SourceComputation.
- For other
Computation instances, the first column must be of type uint64. It is recommended that this column be a computed hash of the other columns (for example, farm_hash). Note that using the modulo operator % and bigb_hash in expression is prohibited.
|
|
experimental_enable_non_uint_key
|
Type: std::optional<bool>
|
|
input_stream_ids
|
Type: THashSet<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>>
Default value: []
Input streams.
The schemas of all streams must be castable to group_by_schema.
|
|
output_stream_ids
|
Type: THashSet<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>>
Default value: []
Output streams of the Computation. They can be used as inputs for other Computations.
|
|
streams_dependency
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, THashSet<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>>>
Default value: {}
A description of dependencies between streams, including internal source and timer streams.
Example: {stream1, {stream2, stream3}} — stream1 depends on streams stream2 and stream3.
Specified only for timer and output streams. If no description is provided for a stream, the dependency list is auto-generated:
- for
timer streams, the list includes all input and source streams;
- for
output streams, the list includes all timer, input, and source streams.
|
|
watermark_strategy
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TWatermarkStrategySpec>
Default value: {}
Settings related to EventWatermark.
An optional parameter. Must be specified within SourceComputation.
|
|
required_resource_ids
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TResourceIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TResourceDescription>>
Default value: {}
A list of resources required for the Computation to work.
Has an alias parameter for locally renaming a resource (for example, Computation expects a resource named YTClient, but the provided resource has a different name). The controller and worker parameters control whether the resource is created on the Controller or on the Worker.
|
|
parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Default value: {}
Custom parameters for Computation and ComputationController.
|
|
timer_streams
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TTimerSpec>>
Default value: {}
Settings for all timer streams. Keys must match [0-9A-Za-z_-]+.
|
|
key_visitor_streams
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TKeyVisitorStreamSpec>>
Default value: {}
Settings for all key_visitor streams. Keys must match [0-9A-Za-z_-]+.
|
|
source_streams
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TSourceSpec>>
Default value: {}
Settings for all source streams. Keys must match [0-9A-Za-z_-]+.
|
|
sinks
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TSinkIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TSinkSpec>>
Default value: {}
Settings for all sinks. Keys must match [0-9A-Za-z_-]+.
|
|
external_state_managers
|
Type: THashMap<std::string, NYT::TIntrusivePtr<NYT::NFlow::TExternalStateManagerSpec>>
Default value: {}
A declarative declaration of external state managers for this Computation. The key is the client name that the process function passes to InitExternalStateClient (must start with /, for example /state); the value contains the manager class name and its parameters.
|
|
external_state_joiners
|
Type: THashMap<std::string, NYT::TIntrusivePtr<NYT::NFlow::TExternalStateJoinerSpec>>
Default value: {}
A declarative declaration of external state joiners (read-only access to external states via key join) for this Computation. The key is the client name that the process function passes to InitExternalStateClient (must start with /, for example /state); the value contains the joiner class name and its parameters.
|
|
state_joiners
|
Type: THashMap<std::string, NYT::TIntrusivePtr<NYT::NFlow::TStateJoinerSpec>>
Default value: {}
A declarative declaration of state joiners (read-only access to the internal state of another Computation via key join) for this Computation. The key is the client name that the process function passes to InitClient (must start with /); the value specifies the target computation_id, its state_name, and join_on.
|
|
heavy_hitters
|
Type: NYT::NFlow::THeavyHittersSpec
Default value: {}
Settings for detecting high-frequency keys.
|
|
pivot_finder
|
Type: NYT::NFlow::TPivotFinderSpec
Default value: {}
|
|
input_ordering
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TInputOrderingSpec>
Default value: {}
Settings for the processing order of input streams.
|
|
distribution_ordering
|
Type: NYT::NFlow::EDistributionOrdering
Default value: strict
Settings for the distribution order of output messages. Controls whether a race condition is possible between stopping and running partitions from the perspective of the order of output messages generated from input messages with the same key.
|
|
worker_group
|
Type: NYT::TStrongTypedef<std::string, NYT::NFlow::TWorkerGroupIdTag, NYT::TStrongTypedefOptions{true}>
Default value: TString("")
|
|
allow_timer_self_dependency
|
Type: bool
Default value: false
|
|
use_compact_input_messages
|
Type: std::optional<bool>
Use the compact_input_messages table for input message deduplication instead of input_messages. The compact deduplication key stores only the hash of key[0] and CityHash128(message_id), which reduces the row size (~200 bytes → ~71 bytes).
An optional parameter. If not set, the table is chosen automatically: the compact table is always used except when experimental_enable_non_uint_key is enabled (in that case, the first key column may not be uint64, and deduplication requires the full key). An explicitly set value takes priority over automatic selection. Enabling use_compact_input_messages together with experimental_enable_non_uint_key is prohibited.
|
|
timer_store_backend
|
Type: NYT::NFlow::ETimerStoreBackend
Default value: in_memory
Timer store of this Computation: in_memory (all timers of a partition are resident in worker memory and persisted in the timers table) or paged (timers live in pages of a dynamic table, only the pages needed for firing are resident).
Static: a change means a stop and a conversion of the timers at the next start. The paged store does not support a watermark_percentile below 100 on a Computation with timer streams yet. The paged backend must be enabled on the controller with enable_paged_timer_store.
|
|
timer_page_size
|
Type: NYT::NYTree::TSize
Default value: 4Ki
Timer page size (number of timers) for the paged store. At least 16. Pages are split and merged gradually under the maintenance budget. Timer rows stay in their family.
|
WatermarkStrategy
Fill this for SourceComputation. Other Computation types might not support it, or might support it only partially.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
event_timestamp_assigner
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TEventTimestampAssignerSpec>
Default value: {}
Automatic population of EventTimestamp for all output messages based on the value of a specified column.
|
|
watermark_generator
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TWatermarkGeneratorSpec>
Settings for the EventWatermark generation algorithm. For a Computation with at least one source, omitting this parameter is equivalent to an empty block with default settings. A multi-source Computation is processed as independent single-source computations: for each source, unavailable-group, idle-partition, and late-data-partition heuristics run independently in that order, and the prepared partitions are merged only afterwards. idle_partitions.max_ratio is applied separately to each source, and availability-group suppression decisions in one source do not affect another. Therefore, a fully idle source is not treated as a partial-idle stall and does not by itself raise /idle_partitions_watermark_stall, even if it gates the merged Computation watermark while another source remains active. Availability-group identifiers do not need to match across sources.
|
|
watermark_alignment
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TWatermarkAlignmentSpec>
Settings for aligning source across different partitions, including those of different Computation instances.
|
|
watermark_percentile
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TWatermarkPercentileSpec>
Default value: {}
Settings for a "percentile" EventWatermark. Used to ignore the oldest events within a configured percentage.
|
EventTimestampAssigner
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
column
|
Type: std::optional<std::string>
The time column.
|
|
format
|
Type: NYT::NFlow::ETimestampFormat
Default value: seconds
The time format used (seconds and milliseconds).
|
|
limit_by_system_timestamp
|
Type: bool
Default value: false
Limit EventTimestamp to the SystemTimestamp value to protect against corrupted values from the future.
|
WatermarkGenerator
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
use_source_watermark
|
Type: bool
Default value: false
Use SourceWatermark for EventWatermark generation. This option completely disables all heuristics for determining EventWatermark. Use it strictly when the input stream contains special markers with EventWatermark.
|
|
out_of_orderness_bound
|
Type: TDuration
Default value: 0
The upper bound on possible event reordering, used for estimating EventWatermark. Event reordering is not accounted for by default.
|
|
idle_partitions
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TIdlePartitionsSpec>
Heuristic settings for ignoring partitions with zero write throughput.
|
|
late_data_partitions
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TLateDataPartitionsSpec>
|
Settings for IdlePartitions:
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
duration
|
Type: TDuration
Default value: 1m
The time a partition must be empty before it is considered to have no writes.
|
|
max_ratio
|
Type: double
Default value: 0.4
The proportion of partitions that can be excluded from the watermark calculation by this logic. The conceptual purpose of this option is to protect against watermark advancement when the data provider stops writing messages to the queue due to an incident on their side. Setting the option to 1.0 effectively disables this protection.
|
Settings for UnavailablePartitionGroups:
Source: yt/yt/flow/library/cpp/common/spec.h
This structure has no main parameters.
WatermarkAlignment
Use this module to align input streams. It can cause a complete read stop if there are availability issues or problems estimating the Watermark for individual partitions or SourceComputation.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
group_name
|
Type: NYT::TStrongTypedef<std::string, NYT::NFlow::TWatermarkAlignmentGroupTag, NYT::TStrongTypedefOptions{true}>
Default value: default-alignment-group
The group name.
|
|
drift_bound
|
Type: TDuration
Default value: 20m
The maximum possible lead of an individual partition's watermark over the group's watermark.
|
|
read_delays
|
Type: std::optional<THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, TDuration>>
An advanced use option. If the EventTimestamp of an output message is greater than EventWatermark - Delay for at least one specified stream, reading is paused. If misconfigured, this option can cause a complete read stall.
|
WatermarkPercentile
Use this module to ignore old data in streams when calculating EventWatermark. Each partition maintains a set of unprocessed messages for each outgoing stream. By default, the system calculates EventWatermark using the minimum EventTimestamp value across all messages (that is, it accounts for 100% of events).
If an event falls within the value percentile, it’s always included in the EventWatermark calculation. If an event doesn’t fall within the specified percentile but its EventTimestamp differs from those events by no more than delay, it’s also included in the EventWatermark calculation. So the calculation algorithm is EventWatermark = max(MinEventTimestamp, PercentileEventTimestamp - delay), where MinEventTimestamp is the minimum across all events, and PercentileEventTimestamp is the EventTimestamp value at the corresponding value percentile.
Keep in mind that the inflight set for a single stream within a partition can contain dozens of messages at a time. That’s why values like 99 or 99.9 can behave almost identically to 100—meaning they won’t produce the desired effect. Also, even if old events make up less than 1% of the entire stream, they might be unevenly distributed across partitions. That’s why it’s better to use values like 80 or 90. To avoid negative effects like ignoring recent events, there’s the delay parameter: it ensures that only truly old events are excluded from the calculation. If there are no such old events, the system works the same as when value is 100, without losing any information.
This module affects only the EventWatermark calculation and doesn’t affect SystemWatermark.
Warning
The settings don’t affect how the system aggregates information from different partitions. The system always uses the minimum value across all partitions.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
value
|
Type: NYT::TStrongTypedef<double, NYT::NFlow::TWatermarkPercentileTag, NYT::TStrongTypedefOptions{true}>
Default value: 100.0
The percentage of events unconditionally considered for EventWatermark calculation.
|
|
delay
|
Type: TDuration
Default value: 1m
The freshness threshold for events below the selected percentile.
|
LateDataPartitions
Use this module to ignore partitions that contain only old data. It works similarly to WatermarkPercentile, but at the level of all partitions, not individual messages inside a partition.
WatermarkPercentile lets you ignore old messages inside a partition, but if an entire partition contains only old data, it will still slow down the overall EventWatermark. The LateDataPartitions module solves this by letting you ignore such partitions entirely.
The calculation algorithm is FinalWatermark = max(MinPartitionWatermark, PercentilePartitionWatermark - Delay), where MinPartitionWatermark is the minimum watermark across all partitions, and PercentilePartitionWatermark is the partition’s watermark at the specified value percentile (partitions are sorted by watermark).
If a partition falls within the value percentile, it’s always included. If a partition doesn’t fall within the specified percentile but its watermark differs from the percentile value by no more than delay, it’s also included. Partitions with a watermark below the threshold (PercentileWatermark - Delay) are considered to contain only late data, and their watermark is “hidden” (set to FinalWatermark).
For example, with value=90 and 100 partitions: 90% of the partitions (those with the highest watermarks) are always included, and the remaining 10% (those with the lowest watermarks) are included only if their lag from the 90th percentile doesn’t exceed delay.
Warning
This module runs after IdlePartitions. Idle partitions are already excluded from the calculation before the LateDataPartitions logic applies.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
value
|
Type: NYT::TStrongTypedef<double, NYT::NFlow::TWatermarkPercentileTag, NYT::TStrongTypedefOptions{true}>
Default value: 100.0
|
|
delay
|
Type: TDuration
Default value: 1m
|
Timer
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
time_type
|
Type: NYT::NFlow::ETimeType
Default value: event_time
The time type used by the timer: event_time, system_time, real_time
|
|
streams
|
Type: std::optional<THashSet<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>>>
List of streams that the timer monitors.
|
|
streams_with_delays
|
Type: std::optional<THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, TDuration>>
List of streams that the timer monitors with an individual delay specified
|
|
deduplicate_equal_timestamps
|
Type: bool
Default value: true
Enables timer deduplication: if you try to create a timer with the same key and trigger_timestamp, only one timer remains — the one with the smaller EventTimestamp value
|
If you don’t specify streams and streams_with_delays, the timer monitors the streams according to streams_dependency with zero extra delay. For current_time timers, streams and streams_with_delays are ignored: such a timer triggers when the current time reaches TriggerTimestamp.
Source
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
source_class_name
|
Type: std::string
Default value: TString("")
Name of the corresponding class.
Must be registered using YT_FLOW_DEFINE_SOURCE.
|
|
parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Default value: {}
Arbitrary parameters for the corresponding class.
|
Built-in source implementations are described in the connector documentation.
Sink
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
sink_class_name
|
Type: std::string
Default value: TString("")
Name of the corresponding class.
Must be registered using YT_FLOW_DEFINE_SINK.
|
|
input_stream_ids
|
Type: THashSet<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>>
Default value: []
List of input streams.
Only streams specified in output_stream_ids of the entire Computation are allowed here.
|
|
parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Default value: {}
Arbitrary parameters for the corresponding class.
|
Built-in sink implementations are described in the connector documentation.
HeavyHitters
This module detects high-frequency keys.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
window
|
Type: TDuration
Default value: 5m
Time window.
|
|
threshold
|
Type: double
Default value: 0.01
Keys that appear in the stream more often than threshold are considered heavy hitters. The algorithm uses O(1 / threshold) memory.
|
|
limit
|
Type: long
Default value: 5
Maximum number of heavy hitter keys to show.
|
These settings control the order in which the system processes events from input streams. Events with the earliest timestamps are processed first.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
time_type
|
Type: NYT::NFlow::ETimeType
Default value: event_time
Time for sorting: event_time, system_time, real_time
|
|
stream_delays
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, TDuration>
Default value: {}
Delays for input streams. If a stream is not specified, there is no delay
|
Streams
Register only output streams in streams. Source streams exist only inside their corresponding partitions and aren’t persisted in YT. Timer streams have a fixed schema.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
class_name
|
Type: std::optional<std::string>
|
|
schema
|
Type: NYT::TIntrusivePtr<NYT::NTableClient::TTableSchema>
Required parameter
Stream schema. It is a dynamic table schema. Cannot contain sorting or computed columns.
We recommend making schema changes by completely stopping the pipeline. In this case, no output messages remain inside the pipeline.
|
|
migration_function
|
Type: std::string
Default value: NYT::NFlow::DefaultMigrationFunction
|
Resource
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
resource_class_name
|
Type: std::string
Required parameter
Name of the corresponding class.
Must be registered using the YT_FLOW_DEFINE_RESOURCE macro.
|
|
parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Default value: {}
Arbitrary parameters for the corresponding class.
|
|
file_providers
|
Type: THashMap<NYT::TStrongTypedef<std::string, NYT::NFlow::TFileProviderIdTag, NYT::TStrongTypedefOptions{true}>, NYT::TIntrusivePtr<NYT::NFlow::TFileProviderSpec>>
Default value: {}
Named file providers of the resource. The controller attaches a file snapshot to a target revision only after discovering every name; a controller-specific target spec may be published independently. Workers materialize the exact revisions from that target. Such resources are worker-only and must have controller = %false on every required_resource_ids path that can reach them. Keys must match [0-9A-Za-z_-]+.
|
|
dependencies
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TResourceIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TResourceDescription>>
Default value: {}
Resources that this resource depends on. Mirrors the structure of required_resource_ids from Computation.
|
|
required_capabilities
|
Type: THashMap<std::string, long>
Default value: {}
|
|
preload_required
|
Type: bool
Default value: false
|
|
always_on
|
Type: bool
Default value: false
If true, the resource loads eagerly when assigned to a unit where a computation requires it (see worker / controller in required_resource_ids). Worker instances remain in memory while assigned, including between jobs. Pausing or stopping the pipeline releases them after all jobs finish. Resources that no computation requires on the unit are not loaded. Mutually exclusive with preload_required.
|
DynamicSpec
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
target_state
|
Type: NYT::NFlow::EPipelineState
Default value: unknown
Target state of the system. Use the start-pipeline, stop-pipeline, and pause-pipeline commands to change it.
|
|
computations
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TComputationIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TDynamicComputationSpec>>
Default value: {}
Named list of all pipeline nodes.
|
|
resources
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TResourceIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TDynamicResourceSpec>>
Default value: {}
|
|
throttlers
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TThrottlerIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TDynamicThrottlerSpec>>
Default value: {}
A set of named distributed throttlers common to all Computation pipeline nodes. Keys are throttler IDs and must match [0-9A-Za-z_-]+. For more information, see Distributed Throttler.
|
|
job_manager
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicJobManagerSpec>
Default value: {}
JobManager settings.
|
|
message_distributor
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicMessageDistributorSpec>
Default value: {}
MessageDistributor settings.
|
|
job_tracker
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicJobTrackerSpec>
Default value: {}
JobTracker settings.
|
|
controller_connector
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicControllerConnectorSpec>
Default value: {}
ControllerConnector settings.
|
|
singletons
|
Type: NYT::TIntrusivePtr<NYT::TSingletonsDynamicConfig>
Default value: {}
|
|
enable_mermaid_graph_describe
|
Type: bool
Default value: false
|
|
enable_united_stream_metrics
|
Type: bool
Default value: true
Enables export of metrics for aggregated streams to Monitoring (true by default). Setting this option to false stops these exports, so dependent alerts, dashboards, and reports no longer receive the data. The option affects metric export only; metrics for individual streams continue to be exported.
|
|
flow_view_cache_codec
|
Type: NYT::NCompression::ECodec
Default value: zstd_2
|
Stream and Resource currently don’t have or don’t support dynamic parameters.
Computation
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
batch_duration
|
Type: TDuration
Default value: 1s
Maximum time to collect an input batch. For YTsaurus queues, a non-full batch keeps collecting until this time passes since the previous batch; after a non-full queue response, the next request is also made no earlier than that moment.
|
|
max_rows_per_batch
|
Type: NYT::NYTree::TSize
Default value: 1K
Maximum size of an input batch in rows. Calculated separately for input_streams, timer_streams, and for each of the source_streams. For YTsaurus queues, unpacked Flow messages are counted; a batch may exceed the limit by one queue row.
|
|
max_bytes_per_batch
|
Type: NYT::NYTree::TSize
Default value: 10Mi
Maximum size of an input batch in bytes. Calculated similarly to max_rows_per_batch. For YTsaurus queues, the size of unpacked messages is counted; the limit may be exceeded as for max_rows_per_batch.
|
|
max_keys_per_batch
|
Type: std::optional<NYT::NYTree::TSize>
|
|
source_read_delay_estimator
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TSourceReadDelayEstimatorSpec>
Default value: {}
Controls Offered and eligible unread Ready estimation in Swift ordered sources with nonempty watermark_alignment.read_delays. Estimation always applies; this section only changes its parameters. Learning follows the source key across partition restarts. Unknown estimates remain unknown.
|
|
draining
|
Type: bool
Default value: false
The Draining mode is used to drain the pipeline. In this mode, the Computation stops running timers and fetching new events from the Source.
The stop-pipeline command stops the entire pipeline through a full drain. You can set this mode for individual Computation nodes for debugging.
|
|
empty_batch_backoff
|
Type: TDuration
Default value: 250ms
If the epoch is empty, the Computation sleeps for the specified additional time. We do not recommend setting this parameter to less than 100 ms.
|
|
lease_check_period
|
Type: TDuration
Default value: 1m
|
|
retryable_request
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicRetryableRequestSpec>
Default value: {}
|
|
tracer
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicPartitionTracerSpec>
Default value: {}
|
|
parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Default value: {}
Arbitrary dynamic parameters of the Computation class.
|
|
processing_function_parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Dynamic process-function parameters. Their type is declared in YT_FLOW_DEFINE_PROCESS_FUNCTION registration. For a process function graph, it contains the dynamic parameters of the nodes in nodes.
|
|
source_streams
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TDynamicSourceSpec>>
Default value: {}
Dynamic parameters of all Sources.
|
|
key_visitor_streams
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TStreamIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TDynamicKeyVisitorStreamSpec>>
Default value: {}
|
|
sinks
|
Type: THashMap<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TSinkIdTag>, NYT::TIntrusivePtr<NYT::NFlow::TDynamicSinkSpec>>
Default value: {}
Dynamic parameters of all Sinks.
|
|
external_state_managers
|
Type: THashMap<std::string, NYT::TIntrusivePtr<NYT::NFlow::TDynamicExternalStateManagerSpec>>
Default value: {}
|
|
external_state_joiners
|
Type: THashMap<std::string, NYT::TIntrusivePtr<NYT::NFlow::TDynamicExternalStateJoinerSpec>>
Default value: {}
|
|
state_joiners
|
Type: THashMap<std::string, NYT::TIntrusivePtr<NYT::NFlow::TDynamicStateJoinerSpec>>
Default value: {}
|
|
state_manager
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicStateManagerSpec>
Default value: {}
|
|
input_store
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicInputStoreSpec>
Default value: {}
|
|
timer_store
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicTimerStoreSpec>
Default value: {}
|
|
output_store
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicOutputStoreSpec>
Default value: {}
|
|
timer_store_count_limit
|
Type: NYT::NYTree::TSize
Default value: 250K
|
|
timer_store_byte_size_limit
|
Type: NYT::NYTree::TSize
Default value: 100Mi
|
|
blocked_time_window
|
Type: TDuration
Default value: 10m
Averaging window for the share of time the job spent blocked on each limit (blocked_share in diagnostics). The share is normalized to the job lifetime, so a job blocked from the start shows ~1 regardless of its age.
|
|
input_rows_throttler_id
|
Type: std::optional<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TThrottlerIdTag>>
Throttler ID for limiting the message processing rate. If set, before each iteration the Computation waits for a quota equal to the number of messages in the input batch. The ID must be present in dynamic_spec/throttlers. For more information, see Distributed Throttler.
|
|
input_bytes_throttler_id
|
Type: std::optional<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TThrottlerIdTag>>
Similar to input_rows_throttler_id, but quotas the total byte_size of messages in the batch — the system size of their serialized representation.
|
|
input_rows_throttler_class_id
|
Type: std::optional<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TQuotaClassIdTag>>
Quota class for the input_rows_throttler_id throttler. The class must be declared by that throttler; default selects the reserved class with weight 1.0. Requires input_rows_throttler_id to be set.
|
|
input_bytes_throttler_class_id
|
Type: std::optional<NYT::NFlow::TStrongIdentifierTypedef<NYT::NFlow::TQuotaClassIdTag>>
Quota class for the input_bytes_throttler_id throttler. The class must be declared by that throttler; default selects the reserved class with weight 1.0. Requires input_bytes_throttler_id to be set.
|
|
skip_if_expression
|
Type: std::optional<std::string>
YTQL predicate for filtering (skipping) input messages before Computation processing. A message is skipped (dropped) if the predicate evaluates to boolean true.
The predicate is evaluated on the message payload columns plus the meta-columns $message_id, $stream_id, $system_timestamp, $event_timestamp, $alignment_timestamp. The result must be boolean: NULL or a non-boolean result causes an exception.
Applies to both input and source streams. The number of skipped messages is exported via the input_streams/skipped_by_expression_count and source_streams/skipped_by_expression_count metrics.
|
Source
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Default value: {}
Arbitrary dynamic parameters of the corresponding Source class.
|
|
draining
|
Type: bool
Default value: false
|
Sink
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
parameters
|
Type: NYT::TIntrusivePtr<NYT::NYTree::IMapNode>
Default value: {}
Arbitrary dynamic parameters of the corresponding Sink class.
|
JobManager
This is the config for the module that manages jobs on the controller.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
balancer_type
|
Type: NYT::NFlow::EJobBalancerType
Default value: cpu_aware
|
|
use_cpu_aware_balancer
|
Type: std::optional<bool>
Enables experimental CPU-aware balancing. Default balancing can lead to severe CPU resource imbalance among workers.
|
|
rebalance_delay_after_pipeline_sync
|
Type: TDuration
Default value: 30s
|
|
rebalance_target_deviation
|
Type: double
Default value: 0.1
Maximum acceptable deviation.
|
|
rebalance_hot_mode_coeff
|
Type: double
Default value: 2.0
|
|
rebalance_action_min_time
|
Type: TDuration
Default value: 4s
|
|
rebalance_action_max_time
|
Type: TDuration
Default value: 12s
|
|
rebalance_sync_period
|
Type: TDuration
Default value: 10s
Period of cpu_aware balancing. It is not recommended to set a value lower than 15min because the balancer looks at 10min metrics.
|
|
rebalance_count_exceeded_allowed
|
Type: double
Default value: 1.2
|
|
rebalance_even_load_thresholds
|
Type: THashMap<NYT::NFlow::EBalanceResource, NYT::TIntrusivePtr<NYT::NFlow::TEvenLoadThresholds>>
Default value: {}
Per-resource thresholds of the "load is even" gate, for example {cpu = {spread = 2.0; ratio = 1.5}}. Only resources with a non-zero weight are consulted. Unset fields fall back to the defaults: a spread of 1.0 core for CPU and 1 GB for memory, a ratio of 1.2.
|
|
rebalance_min_cpu_spread
|
Type: std::optional<double>
Deprecated: the same as spread of the cpu entry of rebalance_even_load_thresholds. A value set explicitly in rebalance_even_load_thresholds takes precedence.
|
|
rebalance_min_cpu_ratio
|
Type: std::optional<double>
Deprecated: the same as ratio of the cpu entry of rebalance_even_load_thresholds. A value set explicitly in rebalance_even_load_thresholds takes precedence.
|
|
balance_weights
|
Type: THashMap<NYT::NFlow::EBalanceResource, double>
Default value: {'cpu': 1.0, 'memory': 0.0}
Relative importance of the resources for cpu_aware balancing, for example {cpu = 80; memory = 20}. The given keys are merged into the default {cpu = 1; memory = 0}; the weights are normalized, so only the proportions matter. A resource with a zero weight does not take part in balancing, so by default only CPU is balanced. The rebalance_target_deviation threshold applies to the normalized weighted sum: adding a second resource proportionally shrinks the contribution of the first one. With a non-zero memory weight, partition memory includes the measured job memory and the input buffer bytes of its streams. Each input stream is charged the larger of its demand, as requested before the buffer pool scales it down, and its resident bytes. A stream without demand, for example one with an explicit limit override or on a worker without buffer manager v2, is charged its resident bytes.
|
|
balancer_metrics_source
|
Type: NYT::NFlow::EBalancerMetricsSource
Default value: job
Where the cpu_aware balancer takes a partition's CPU usage from. partition: the 10-minute rate of the running job, and until it exists the partition's persisted history (measured by the previous job, survives moves and restarts); 30-second and immediate rates are never used. job: the previous behaviour, only the running job's metrics with a fallback to the 30-second and immediate rates. Off by default while the change is rolled out pipeline by pipeline; temporary switch.
|
|
balance_warmup_protection
|
Type: bool
Default value: false
Never move a running job whose metrics are not mature yet: the 10-minute rate exists and one more 10-minute window has passed since it appeared. Such a job is still paying for its previous move and its weight is unknown. The protection applies to every balancing resource: a memory-only configuration waits for the CPU rate to mature as well. A worker over the count limit whose kick candidate is immature is skipped for the round. Jobs younger than two minutes stay movable: they have invested nothing yet, and on a pipeline restart the workers do not register all at once. Partitions without a job are placed as usual. Off by default while the change is rolled out pipeline by pipeline; temporary switch.
|
|
balance_warmup_idle_worker_share
|
Type: double
Default value: 0.2
Share of the group's workers without a job of the group at which warm-up protection is lifted: filling idle workers matters more than the metrics of the jobs that would move, e.g. when workers register minutes apart after a pipeline restart. Temporary switch for rollback.
|
|
worker_coef_mode
|
Type: NYT::NFlow::EWorkerCoefMode
Default value: legacy
How the cpu_aware balancer estimates the relative speed of workers. probing: from partitions that moved between workers, comparing CPU per message before and after the move; observations per worker pair accumulate in balancer_state and are solved together; without moves all coefficients are 1. legacy: the previous estimate from the current usage of each worker's partitions against the computation averages. Off by default while the change is rolled out pipeline by pipeline; temporary switch.
|
|
worker_coef_half_life
|
Type: TDuration
Default value: 1d
Half-life of an accumulated worker-pair observation: when a new observation merges in, the old one loses weight in proportion to the time passed. Without new observations the estimate does not change.
|
|
worker_coef_retention
|
Type: TDuration
Default value: 7d
How long the observations of a worker absent from the group are kept; a worker returning earlier gets its coefficient back.
|
|
worker_coef_prior_weight
|
Type: double
Default value: 0.05
Weight of the prior "the worker's coefficient is 1" in observation weight units (a move of one partition weighs 1 / N, N being the number of partitions on the receiving worker). Pins the group's mean coefficient to 1 and damps single observations.
|
|
worker_coef_max_ratio
|
Type: double
Default value: 4.0
Safety limit: worker coefficients are clamped to [1 / max_ratio, max_ratio].
|
|
disable_even_load_gate
|
Type: std::optional<bool>
|
|
async_balancing
|
Type: bool
Default value: true
|
|
graceful_move
|
Type: bool
Default value: true
|
|
zero_queue_latency
|
Type: TDuration
Default value: 1s
|
|
planning_horizon
|
Type: TDuration
Default value: 10m
|
|
preloading_timeout
|
Type: TDuration
Default value: 30m
How long the resource_queue balancer keeps a model preload that is still loading, counted from the request. Until then the preload is not released and the worker counts as a worker of the computations that need the model; after it the usual rules apply.
|
|
minimum_worker_count
|
Type: unsigned long
Default value: 1
Minimum number of workers to distribute the load across.
Must be filled in to avoid out-of-memory errors and overloading individual workers when the system starts. Recommended value is about 60% of the target number of workers (to survive the loss of a subset of workers, for example, if a data center goes offline).
|
|
lost_job_timeout
|
Type: TDuration
Default value: 10s
Time after which a Job is considered lost if no status update reaches the controller. The same timeout is used to consider a worker lost.
|
|
faulty_address_window
|
Type: TDuration
Default value: 5m
Window within which the exponentially decaying counter of connection losses between the controller and a worker is calculated. If this counter exceeds faulty_address_attempts, the worker is considered faulty and is ignored during load balancing. Solves the problem of workers that constantly lose connection to the controller. Also solves the problem of two workers with different incarnation IDs but the same address (when deployment has two pods that think they share the same FQDN).
|
|
faulty_address_attempts
|
Type: unsigned long
Default value: 5
See the description of the faulty_address_window parameter.
|
|
partitioning
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicPartitioningCoordinatorSpec>
Default value: {}
Pipeline partitioning settings.
|
|
worker_group_override
|
Type: THashMap<NYT::TStrongTypedef<std::string, NYT::NFlow::TWorkerGroupIdTag, NYT::TStrongTypedefOptions{true}>, NYT::TIntrusivePtr<NYT::NFlow::TDynamicJobManagerGroupSpec>>
Default value: {}
|
|
partition_history_limit
|
Type: long
Default value: 4096
Most partition histories the balancer keeps across job restarts, see balancer_metrics_source. The histories live in one persisted document with a size limit, so when the limit is reached the lightest history gives way to a heavier one and lighter ones are not saved. A pipeline stop saves no histories at all.
|
|
watermark_history_thinning_factor
|
Type: int
Default value: 4
Thinning factor K of the watermark histories the controller keeps to compute ready_time_lag for event-time and system-time timers. A history element spans at most 1/K of its age, so the lag may be over-reported by at most 1/K of its value and is never under-reported; a larger K keeps more elements. A change applies to new samples only: existing elements are never split.
|
JobTracker
This is the config for the module that manages jobs on the worker.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
job_control_threads
|
Type: int
Default value: 3
Number of control threads.
In most cases, the default value is sufficient.
|
|
job_threads
|
Type: std::optional<int>
Number of threads for performing jobs.
If not specified, the pool size is calculated automatically based on the node's CPU limit.
If the node limit cannot be calculated, 30 threads are created.
|
|
buffer_state_manager
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicBufferStateManagerSpec>
Default value: {}
Configuration of the buffer management module for incoming and outgoing messages.
|
|
load_throughput_throttler
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TLoadThroughputThrottlerSpec>
Default value: {'limit': 134217728.0}
|
|
state_cache
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicStateCacheSpec>
Default value: {}
|
|
mark_performance_metrics_steady_after_first_iteration
|
Type: bool
Default value: false
Report the job's rate counters (CPU, input messages, memory) as steady only once its first iteration with input has completed, so that the balancer reads windowed metrics such as cpu_usage_10m after the job's initialization (state download, index build) has decayed out of them; the counters themselves are never reset. A job that has received no input for a minute is reported as steady as well, so an idle job reports its near-zero rates instead of staying unmeasured. Off by default while the change is rolled out pipeline by pipeline; temporary switch.
|
BufferStateManager
This is the config for the module that manages buffer sizes.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
manage_period
|
Type: TDuration
Default value: 1s
Frequency of buffer size recalculation.
|
|
demand_window
|
Type: TDuration
Default value: 1m
Time window over which the utilization of a single buffer is estimated.
|
|
input_buffer
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicBufferStateManagerSpec::TOneSideBufferSpec>
Default value: {}
Input buffer settings.
|
|
output_buffer
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicBufferStateManagerSpec::TOneSideBufferSpec>
Default value: {}
Output buffer settings.
|
|
enable_v2
|
Type: bool
Default value: true
Enables the v2 buffer-sizing strategy: limit = peak usage + headroom, with a v2_gain_epochs × demand × epoch bandwidth-delay-product floor. The demand × max_duration cap is raised by announced input backlog or, for a producing output, to an equal share of half fair_share_pool because producer epochs do not reveal the downstream acknowledgement period. Demand-backed limits are allocated before speculative output probes. On both sides, Σ(limits) ≤ fair_share_pool including other streams' in-flight bytes; job_limit remains the per-stream bound. Streams with job_overrides remain fully outside the pool as in v1, so worker buffer memory should cover fair_share_pool plus the actual in-flight bytes of overridden streams. Enabled by default; set this option to false to use the previous v1 formula.
|
Additional parameters
|
epoch_cycle_window_samples
|
Type: int
Default value: 16
Number of samples in the window used to estimate the median job epoch cycle.
|
|
max_rate_estimator_buckets
|
Type: int
Default value: 8
Number of buckets in the windowed-max drain-rate estimator; changing it resets the estimate.
|
|
warmup_refresh_period
|
Type: TDuration
Default value: 30s
How often a job polls the converged warmup state for persistence.
|
|
v2_gain_epochs
|
Type: double
Default value: 2.0
Minimum buffer target in units of job epochs, used as a bandwidth-delay-product floor. Applies only when enable_v2 is set.
|
|
v2_use_offered_rate
|
Type: bool
Default value: true
Whether announced backlog rate contributes to demand. Disable it when a producer systematically overstates its offered rate. Applies only when enable_v2 is set.
|
|
v2_floor
|
Type: NYT::NYTree::TSize
Default value: 2Mi
Minimum grant for a stream with backlog; it must fit the largest message. Applies only when enable_v2 is set.
|
|
v2_headroom_growth_factor
|
Type: double
Default value: 2.0
Headroom growth factor per management tick while utilization is high. Applies only when enable_v2 is set.
|
|
v2_high_utilization_threshold
|
Type: double
Default value: 0.65
Utilization threshold above which headroom grows; below half this value headroom decays. Applies only when enable_v2 is set.
|
|
v2_publish_threshold
|
Type: double
Default value: 0.25
Suppress a new limit when its relative increase is smaller than this value; decreases are always published. Applies only when enable_v2 is set.
|
All dimensional parameters in this config (measured in number of items or bytes) can be parsed from strings like "50K".
ControllerConnector
This is the config for managing the connection between the worker and the controller.
Don’t change it unless necessary.
Source: yt/yt/flow/library/cpp/common/spec.h
|
Parameter
|
Description
|
|
controller_wait_timeout
|
Type: TDuration
Default value: 5m
Time after which the worker forcibly disconnects all jobs in case of loss of connection with the controller.
|
|
controller_discover_period
|
Type: TDuration
Default value: 10s
Period for rediscovering the controller.
|
|
controller_heartbeat_period
|
Type: TDuration
Default value: 1s
Period of worker heartbeats to the controller.
|
|
worker_statistics_report_period
|
Type: TDuration
Default value: 30s
|
|
controller_heartbeat_rpc_timeout
|
Type: TDuration
Default value: 10s
Heartbeat timeout.
|
|
controller_heartbeat_failure_backoff
|
Type: TDuration
Default value: 1s
Backoff time in case of handshake failure.
|
|
controller_handshake_rpc_timeout
|
Type: TDuration
Default value: 10s
Handshake timeout.
|
|
controller_handshake_failure_backoff
|
Type: TDuration
Default value: 1s
|
|
orchid_update_period
|
Type: TDuration
Default value: 1s
Period for updating orchid on the controller.
|
Parameters
The parameters field exists in the Computation, Source, Sink, and Resource specs. It contains parameters of the selected built-in implementation, such as a process-function adapter. Check the selected class’s documentation for supported fields, and don’t use this block for user-logic parameters.
Set C++ process-function parameters separately in processing_function_parameters. Declare a regular yson struct, register its type with the function, and read it through the runtime context:
struct TMyParameters
: public NYTree::TYsonStruct
{
i64 Threshold;
REGISTER_YSON_STRUCT(TMyParameters);
static void Register(TRegistrar registrar);
};
YT_FLOW_DEFINE_PROCESS_FUNCTION(TMyFunction, TMyParameters);
"processing_function_parameters" = {
"threshold" = 10;
};
Static parameters are available in Init through initContext->GetParameters<TMyParameters>(); dynamic parameters are available through context->GetDynamicParameters<TMyParameters>(). For more details, see Process functions.
Config
This is the config for a Flow cluster node.
Source: yt/yt/flow/library/cpp/runner/config.h
|
Parameter
|
Description
|
|
logging
|
Type: NYT::TIntrusivePtr<NYT::NLogging::TLogManagerConfig>
Default value:
{
"high_backlog_watermark" = 100000;
"low_backlog_watermark" = 100000;
"min_disk_space" = 0;
"rules" = [
{
"exclude_categories" = [];
"max_level" = "maximum";
"min_level" = "info";
"writers" = [
"Stderr";
];
};
];
"writers" = {
"Stderr" = {
"common_fields" = {};
"enable_host_field" = %false;
"enable_native_tags" = %false;
"enable_source_location" = %false;
"enable_span_id_field" = %false;
"enable_system_fields" = %true;
"format" = "plain_text";
"type" = "stderr";
"yson_format" = "text";
};
};
}
Logging settings.
|
|
jaeger
|
Type: NYT::TIntrusivePtr<NYT::NTracing::TJaegerTracerConfig>
Default value: {}
|
|
pipe_io_dispatcher
|
Type: NYT::TIntrusivePtr<NYT::NPipeIO::TPipeIODispatcherConfig>
Default value: {}
|
|
solomon_registry
|
Type: NYT::TIntrusivePtr<NYT::NProfiling::TSolomonRegistryConfig>
Default value: {}
|
|
error_backtrace_enricher
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TBacktraceEnricherSpec>
Default value: {}
|
|
query_engine_config
|
Type: NYT::TIntrusivePtr<NYT::NQueryClient::TQueryEngineConfig>
Default value: {}
|
|
cluster_url
|
Type: std::string
Required parameter
YTsaurus cluster to use.
|
|
path
|
Type: TString
Required parameter
Working directory of the pipeline on the cluster_url cluster.
|
|
proxy_role
|
Type: std::optional<std::string>
rpc proxy role.
|
|
rpc_port
|
Type: int
Default value: 0
Main port for communication between workers and controllers, can be different for each instance.
|
|
monitoring_port
|
Type: int
Default value: 0
API for obtaining metrics.
|
|
companion
|
Type: NYT::TIntrusivePtr<NYT::NFlow::NCompanion::TCompanionConfig>
Companion process parameters. Needed by any worker that runs a companion (Python, Java, Go, C++ companion). Filled in automatically for a vanilla launch.
|
|
controller
|
Type: NYT::TIntrusivePtr<NYT::NFlow::NController::TControllerConfig>
Default value: {}
Controller parameters.
|
|
worker
|
Type: NYT::TIntrusivePtr<NYT::NFlow::NWorker::TWorkerConfig>
Default value: {}
Worker parameters.
|
|
authenticator
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TAuthenticatorConfig>
Default value: {}
|
|
bus_server
|
Type: NYT::TIntrusivePtr<NYT::NBus::NTcp::TBusServerConfig>
Default value: {}
|
|
rpc_server
|
Type: NYT::TIntrusivePtr<NYT::NRpc::TServerConfig>
Default value: {}
|
|
core_dumper
|
Type: NYT::TIntrusivePtr<NYT::NCoreDump::TCoreDumperConfig>
|
|
solomon_exporter
|
Type: NYT::TIntrusivePtr<NYT::NProfiling::TSolomonExporterConfig>
Default value: {'enable_solomon_aggregates': true}
|
|
solomon_proxy
|
Type: NYT::TIntrusivePtr<NYT::NProfiling::TSolomonProxyConfig>
Default value: {}
|
|
http_client_config
|
Type: NYT::TIntrusivePtr<NYT::NHttp::TClientConfig>
Default value: {}
|
|
https_client_config
|
Type: NYT::TIntrusivePtr<NYT::NHttps::TClientConfig>
Default value: {}
|
|
http_poller_threads
|
Type: int
Default value: 1
|
|
abort_on_unrecognized_options
|
Type: bool
Default value: true
When set to true, Flow Node will not start if there are unknown options in the config.
|
|
ignore_singletons_dynamic_config
|
Type: bool
Default value: false
|
|
enable_porto_resource_tracker
|
Type: bool
Default value: true
|
Additional parameters
Controller
Source: yt/yt/flow/library/cpp/controller/config.h
|
Parameter
|
Description
|
|
controller_threads
|
Type: int
Default value: 5
Number of threads.
|
|
orchid_update_period
|
Type: TDuration
Default value: 1s
orchid recalculation period.
|
|
warm_up_time
|
Type: TDuration
Default value: 5s
Warm-up period when the controller starts.
|
|
scheduler_period
|
Type: TDuration
Default value: 5s
Scheduler invocation period.
|
|
cache_period
|
Type: TDuration
Default value: 1s
|
|
feedback_period
|
Type: TDuration
Default value: 1s
|
|
metrics_period
|
Type: TDuration
Default value: 5s
|
|
write_own_retryable_errors_period
|
Type: TDuration
Default value: 5s
|
|
publish_retry_period
|
Type: TDuration
Default value: 5s
Period during which the controller attempts to tell YTsaurus that it is the leader.
|
|
publish_timeout
|
Type: TDuration
Default value: 2h
|
|
election_manager
|
Type: NYT::NYTree::TPolymorphicYsonStruct<NYT::NYTree::NDetail::TPolymorphicMapping<&NYT::NFlow::NController::ElectionBackendDiscriminator.<char const at offset 0>, NYT::NFlow::NController::EElectionBackend, NYT::NYTree::NDetail::TOptionalValue<NYT::NFlow::NController::EElectionBackend, (NYT::NFlow::NController::EElectionBackend)0>, NYT::NFlow::NController::TElectionBackendConfigBase, NYT::NYTree::NDetail::TLeafTag<(NYT::NFlow::NController::EElectionBackend)0, NYT::NFlow::NController::TCypressElectionBackendConfig>, NYT::NYTree::NDetail::TLeafTag<(NYT::NFlow::NController::EElectionBackend)1, NYT::NFlow::NController::TDyntableElectionBackendConfig>, NYT::NYTree::NDetail::TLeafTag<(NYT::NFlow::NController::EElectionBackend)2, NYT::NFlow::NController::TChaosElectionBackendConfig>>>
Default value:
{
"backend" = "cypress";
"leader_cache_update_period" = 1000;
"leader_lease_ping_period" = 1000;
"leader_lease_ttl" = 5000;
"lock_acquisition_period" = 1000;
}
Leader election settings for multiple controllers.
|
|
persisted_state_manager
|
Type: NYT::TIntrusivePtr<NYT::NFlow::NController::TPersistedStateManagerConfig>
Default value: {}
PersistedStateManager settings.
|
|
lease_manager
|
Type: NYT::TIntrusivePtr<NYT::NFlow::NController::TLeaseManagerConfig>
Default value: {}
LeaseManager settings.
|
|
controller_service
|
Type: NYT::TIntrusivePtr<NYT::NFlow::NController::TControllerServiceConfig>
Default value: {}
ControllerService settings.
|
|
bus
|
Type: NYT::TIntrusivePtr<NYT::NBus::NTcp::TBusConfig>
Default value: {}
|
Additional parameters
|
publish_request_timeout
|
Type: TDuration
Default value: 1s
Timeout of a single request of a publication attempt. A failed attempt is retried, so this only decides how long one attempt may hang before the retry.
|
PersistedStateManager
Source: yt/yt/flow/library/cpp/controller/config.h
|
Parameter
|
Description
|
|
timeout
|
Type: TDuration
Default value: 5s
Save timeout in YTsaurus.
|
|
max_reads_per_transaction
|
Type: long
Default value: 10000
|
|
max_writes_per_transaction
|
Type: long
Default value: 10000
|
LeaseManager
Source: yt/yt/flow/library/cpp/controller/config.h
|
Parameter
|
Description
|
|
lease_timeout
|
Type: TDuration
Default value: 10m
How long a job stays fenced without the controller refreshing the fence. With the Cypress election backend this is the timeout of the job's master lease transaction, with the chaos one the timeout of its chaos lease, and with the dyntable one the ttl of the pipeline-wide deadline row that gates every worker commit.
|
|
lease_ping_period
|
Type: TDuration
Default value: 30s
Period at which the leader prolongs the leases. Every backend prolongs from the leader and none from the worker, but the machinery differs: the chaos leases are pinged by one periodic executor, the Cypress ones by a client-side pinger per lease transaction, and the shared dyntable deadline is simply rewritten.
|
|
max_concurrent_requests
|
Type: long
Default value: 500
Maximum number of concurrent requests the manager keeps in flight when it attaches to leases or terminates them. Do not increase to avoid overloading the master. Prolongation is not capped by it: the chaos pings are dispatched in one fire-and-forget round, and the Cypress ones belong to the lease transactions themselves.
|
ControllerService
Source: yt/yt/flow/library/cpp/controller/config.h
|
Parameter
|
Description
|
|
set_spec_retry_count
|
Type: int
Default value: 3
Number of attempts to update the spec in case of errors.
|
|
set_spec_retry_period
|
Type: TDuration
Default value: 5s
Time between attempts.
|
|
enable_paged_timer_store
|
Type: bool
Default value: false
Allows pipeline specs that select the paged timer store backend (timer_store_backend). Disabled by default: a spec that resolves paged for any computation is rejected. The check applies when a spec is set: a pipeline whose stored spec already selects paged keeps running.
|
|
tables_throttler
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TLoadThroughputThrottlerSpec>
Default value: {}
|
Plus a small buffer for the case where some live partitions also won’t receive any writes.
Runner config
This is the config for running a pipeline. The format of this config isn’t standardized; you can use whatever config works best for your tasks.
Flow provides a simple version of this config, NYT::NFlow::TSimpleRunnerConfig, and a program, NYT::NFlow::TSimpleRunnerProgram, to run a pipeline using this config. The Flow test framework relies on this config format to inject the cluster and the pipeline path in the local YTsaurus, and to apply test default parameters for faster test runs.
SimpleRunnerConfig
Source: yt/yt/flow/library/cpp/runner/simple_runner_program.h
|
Parameter
|
Description
|
|
logging
|
Type: NYT::TIntrusivePtr<NYT::NLogging::TLogManagerConfig>
Default value:
{
"high_backlog_watermark" = 100000;
"low_backlog_watermark" = 100000;
"min_disk_space" = 0;
"rules" = [
{
"exclude_categories" = [];
"max_level" = "maximum";
"min_level" = "info";
"writers" = [
"Stderr";
];
};
];
"writers" = {
"Stderr" = {
"common_fields" = {};
"enable_host_field" = %false;
"enable_native_tags" = %false;
"enable_source_location" = %false;
"enable_span_id_field" = %false;
"enable_system_fields" = %true;
"format" = "plain_text";
"type" = "stderr";
"yson_format" = "text";
};
};
}
Logging settings.
|
|
jaeger
|
Type: NYT::TIntrusivePtr<NYT::NTracing::TJaegerTracerConfig>
Default value: {}
|
|
pipe_io_dispatcher
|
Type: NYT::TIntrusivePtr<NYT::NPipeIO::TPipeIODispatcherConfig>
Default value: {}
|
|
solomon_registry
|
Type: NYT::TIntrusivePtr<NYT::NProfiling::TSolomonRegistryConfig>
Default value: {}
|
|
error_backtrace_enricher
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TBacktraceEnricherSpec>
Default value: {}
|
|
query_engine_config
|
Type: NYT::TIntrusivePtr<NYT::NQueryClient::TQueryEngineConfig>
Default value: {}
|
|
cluster_url
|
Type: std::string
Required parameter
Name of the cluster where the pipeline is located. The tail after the last / is treated as a proxy role and takes precedence over proxy_role — e.g. example-cluster/flow means cluster example-cluster with role flow.
|
|
proxy_role
|
Type: std::optional<std::string>
Required parameter
The proxy role to use for performing operations on the pipeline.
|
|
path
|
Type: TString
Required parameter
Path to the pipeline on YTsaurus.
|
|
spec
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TPipelineSpec>
Required parameter
Static spec with which to start the pipeline.
|
|
dynamic_spec
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDynamicPipelineSpec>
Default value: {}
Dynamic spec with which to start the pipeline.
|
|
abort_on_unrecognized_options
|
Type: bool
Default value: false
|
|
abort_on_specs_parseability_error
|
Type: bool
Default value: false
|
|
vanilla
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TVanillaConfig>
If set and enable=%true, the runner starts a vanilla operation on YTsaurus and runs the federation inside it, instead of starting the controller/workers locally.
|
|
set_flow_core_target
|
Type: bool
Default value: true
|
|
direct_controller_commands
|
Type: NYT::TIntrusivePtr<NYT::NFlow::TDirectControllerCommandsConfig>
Default value: {}
|
Additional parameters
See also