---
metadata:
  - name: generator
    content: Diplodoc Platform v5.70.0
alternate:
  - https://ytsaurus.tech/docs/en/flow/concepts/spec.md
  - https://ytsaurus.tech/docs/ru/flow/concepts/spec.md
  - href: https://ytsaurus.tech/docs/en/flow/concepts/spec.md
    type: text/markdown
    title: Markdown version
  - href: https://ytsaurus.tech/docs/en/llms.txt?revision=d1460a800885385aa850a1578c0d21951c4247ed
    rel: describedby
---
> **Documentation Index:** Fetch the complete configuration index at https://ytsaurus.tech/docs/en/llms.txt?revision=d1460a800885385aa850a1578c0d21951c4247ed

<!-- source: en/_includes/flow/concepts/spec.md -->
# Spec, DynamicSpec, and Config in YTsaurus Flow

Use a spec to describe the pipeline you want. Any high-level API to Flow generates a pipeline spec.

A pipeline has two specs: `Spec` and `DynamicSpec`:

- `Spec` is a static object that contains the topology of the entire pipeline. You can change this object only if the pipeline is stopped. The system guarantees that at runtime, every part of the system has the same view of the `Spec`.
- `DynamicSpec` is the dynamic part of the spec. You can change it at any time, and the changes apply asynchronously to all parts of the system. `DynamicSpec` usually stores buffer sizes, timeouts, the number of partitions, and so on.

The node’s `Config` is also an important part. Unlike the spec, you set the config exactly once per release, and you can’t change it dynamically. In the future, some config parameters might move to the specs.

{% note warning %}

All parameters in Flow have optimal default values. Don’t change the settings unless you’re sure what they do and how they can affect the system.

{% endnote %}

You can find the default values by searching for the parameter name in the Flow codebase. They’re mostly grouped in the file [spec.cpp](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.cpp).

## Spec

<!-- source: en/flow/generated_docs/NYT_NFlow_TPipelineSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `validate_binary_version` | **Type**: `bool`
**Default value**: `false`
 ||
|| `binary_version` | **Type**: `std::string`
**Default value**: `TString("")`
 ||
|| `validate_binary_checksum` | **Type**: `bool`
**Default value**: `false`
 ||
|| `binary_checksum` | **Type**: `std::string`
**Default value**: `TString("")`
 ||
|| `computations` | **Type**: `THashMap<`[NYT::NFlow::TComputationId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TComputationId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TComputationSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TComputationSpec)`>>`
**Default value**: `{}`
A named enumeration of all pipeline nodes. Keys must match `[0-9A-Za-z_-]+`. ||
|| `resources` | **Type**: `THashMap<`[NYT::NFlow::TResourceId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TResourceId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TResourceSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TResourceSpec)`>>`
**Default value**: `{}`
A named enumeration of all resources. Keys must match `[0-9A-Za-z_-]+`. ||
|| `streams` | **Type**: `THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TStreamSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamSpec)`>>`
**Default value**: `{}`
A named enumeration of all data streams. Keys must match `[0-9A-Za-z_-]+`. ||
|| `failure_handler` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TFailureHandlerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TFailureHandlerSpec)`>`
**Default value**: `{}`
Explicit failed-message capture into the pipeline failed_messages table. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TPipelineSpec.md -->

### Computation

<!-- source: en/flow/generated_docs/NYT_NFlow_TComputationSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `computation_class_name` | **Type**: `std::string`
**Required parameter**
The built-in Computation or process-function adapter class name.

For new C++ user logic, select one of the built-in `TProcessFunction*Computation` adapters. Register the function with `YT_FLOW_DEFINE_PROCESS_FUNCTION` and name it in `processing_function`. Don’t create custom Computation classes. ||
|| `process_function_graph` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TProcessFunctionGraphSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TProcessFunctionGraphSpec)`>`
Several process functions connected by stream IDs and run in one `TProcessFunctionComputation`, `TProcessFunctionSwiftMapComputation`, or `TProcessFunctionSourceComputation`. Batch and per-message functions can be combined. Internal message IDs are derived from their boundary parents, output stream, and suffix; timestamps and external IDs follow the host computation's policy. Swift requires deterministic replay; batch functions set each result's parents with `SetParents`. Merging different input messages requires `parameters/allow_batching_with_relaxed_guarantees = %true` and provides at-least-once delivery. In `TProcessFunctionSourceComputation`, nodes read `source_streams`, internal messages keep the key of the source message they derive from, nodes cannot have timers, and each output must derive from one source message. A message added with `distribute = %false` still advances the watermark if it is written to an output of the computation; nodes reading its stream do not receive it. The runner replaces the graph with `processing_function`, `processing_function_parameters`, `timer_streams`, `streams_dependency`, and `allow_timer_self_dependency`, so do not set these fields together with the graph. ||
|| `processing_function` | **Type**: `std::optional<std::string>`
The fully qualified process-function name registered with `YT_FLOW_DEFINE_PROCESS_FUNCTION`. Required by `TSwiftStableBatchSourceComputation`, and by the `TProcessFunction*Computation` adapters unless `process_function_graph` is set. ||
|| `processing_function_parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
Static process-function parameters. Their type is declared in `YT_FLOW_DEFINE_PROCESS_FUNCTION` registration. ||
|| `group_by_schema` | **Type**: `NYT::TIntrusivePtr<NYT::NTableClient::TTableSchema>`
**Default value**: `{'value': [], 'attributes': {'strict': true, 'unique_keys': false}}`
A schema for grouping all input streams. It is a dynamic table schema. Schema properties and requirements:

* Does not contain any ordering rules.
* Can contain computed fields.
* Must be empty only for `SourceComputation`.
* For other `Computation` instances, the first column must be of type `uint64`. It is recommended that this column be a computed hash of the other columns (for example, `farm_hash`). Note that using the modulo operator `%` and `bigb_hash` in `expression` is prohibited. ||
|| `experimental_enable_non_uint_key` | **Type**: `std::optional<bool>`
 ||
|| `input_stream_ids` | **Type**: `THashSet<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`>`
**Default value**: `[]`
Input streams.

The schemas of all streams must be castable to `group_by_schema`. ||
|| `output_stream_ids` | **Type**: `THashSet<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`>`
**Default value**: `[]`
Output streams of the Computation. They can be used as inputs for other Computations. ||
|| `streams_dependency` | **Type**: `THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, THashSet<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`>>`
**Default value**: `{}`
A description of dependencies between streams, including internal `source` and `timer` streams.

Example: `{stream1, {stream2, stream3}}` &mdash; `stream1` depends on streams `stream2` and `stream3`.

Specified only for `timer` and `output` streams. If no description is provided for a stream, the dependency list is auto-generated:

* for `timer` streams, the list includes all `input` and `source` streams;
* for `output` streams, the list includes all `timer`, `input`, and `source` streams. ||
|| `watermark_strategy` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TWatermarkStrategySpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWatermarkStrategySpec)`>`
**Default value**: `{}`
Settings related to `EventWatermark`.

An optional parameter. Must be specified within `SourceComputation`. ||
|| `required_resource_ids` | **Type**: `THashMap<`[NYT::NFlow::TResourceId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TResourceId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TResourceDescription](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TResourceDescription)`>>`
**Default value**: `{}`
A list of resources required for the `Computation` to work.

Has an `alias` parameter for locally renaming a resource (for example, `Computation` expects a resource named `YTClient`, but the provided resource has a different name). The `controller` and `worker` parameters control whether the resource is created on the `Controller` or on the `Worker`. ||
|| `parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
**Default value**: `{}`
Custom parameters for `Computation` and `ComputationController`. ||
|| `timer_streams` | **Type**: `THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TTimerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TTimerSpec)`>>`
**Default value**: `{}`
Settings for all `timer` streams. Keys must match `[0-9A-Za-z_-]+`. ||
|| `key_visitor_streams` | **Type**: `THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TKeyVisitorStreamSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TKeyVisitorStreamSpec)`>>`
**Default value**: `{}`
Settings for all `key_visitor` streams. Keys must match `[0-9A-Za-z_-]+`. ||
|| `source_streams` | **Type**: `THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TSourceSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TSourceSpec)`>>`
**Default value**: `{}`
Settings for all `source` streams. Keys must match `[0-9A-Za-z_-]+`. ||
|| `sinks` | **Type**: `THashMap<`[NYT::NFlow::TSinkId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TSinkId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TSinkSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TSinkSpec)`>>`
**Default value**: `{}`
Settings for all sinks. Keys must match `[0-9A-Za-z_-]+`. ||
|| `external_state_managers` | **Type**: `THashMap<std::string, NYT::TIntrusivePtr<`[NYT::NFlow::TExternalStateManagerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TExternalStateManagerSpec)`>>`
**Default value**: `{}`
A declarative declaration of [external state](https://ytsaurus.tech/docs/en/flow/cpp/state.md#external-state) managers for this `Computation`. The key is the client name that the process function passes to `InitExternalStateClient` (must start with `/`, for example `/state`); the value contains the manager class name and its parameters. ||
|| `external_state_joiners` | **Type**: `THashMap<std::string, NYT::TIntrusivePtr<`[NYT::NFlow::TExternalStateJoinerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TExternalStateJoinerSpec)`>>`
**Default value**: `{}`
A declarative declaration of [external state](https://ytsaurus.tech/docs/en/flow/cpp/state.md#external-state-joiner) joiners (read-only access to external states via key join) for this `Computation`. The key is the client name that the process function passes to `InitExternalStateClient` (must start with `/`, for example `/state`); the value contains the joiner class name and its parameters. ||
|| `state_joiners` | **Type**: `THashMap<std::string, NYT::TIntrusivePtr<`[NYT::NFlow::TStateJoinerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStateJoinerSpec)`>>`
**Default value**: `{}`
A declarative declaration of [state](https://ytsaurus.tech/docs/en/flow/cpp/state.md#state-joiner) joiners (read-only access to the internal state of another `Computation` via key join) for this `Computation`. The key is the client name that the process function passes to `InitClient` (must start with `/`); the value specifies the target `computation_id`, its `state_name`, and `join_on`. ||
|| `heavy_hitters` | **Type**: [NYT::NFlow::THeavyHittersSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_THeavyHittersSpec)
**Default value**: `{}`
Settings for detecting high-frequency keys. ||
|| `pivot_finder` | **Type**: [NYT::NFlow::TPivotFinderSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TPivotFinderSpec)
**Default value**: `{}`
 ||
|| `input_ordering` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TInputOrderingSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TInputOrderingSpec)`>`
**Default value**: `{}`
Settings for the processing order of input streams. ||
|| `distribution_ordering` | **Type**: [NYT::NFlow::EDistributionOrdering](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_EDistributionOrdering)
**Default value**: `strict`
Settings for the distribution order of output messages. Controls whether a race condition is possible between stopping and running partitions from the perspective of the order of output messages generated from input messages with the same key. ||
|| `worker_group` | **Type**: [NYT::NFlow::TWorkerGroupId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWorkerGroupId)
**Default value**: `TString("")`
 ||
|| `allow_timer_self_dependency` | **Type**: `bool`
**Default value**: `false`
 ||
|| `use_compact_input_messages` | **Type**: `std::optional<bool>`
Use the `compact_input_messages` table for [input message deduplication](https://ytsaurus.tech/docs/en/flow/concepts/guarantees.md) instead of `input_messages`. The compact deduplication key stores only the hash of `key[0]` and `CityHash128(message_id)`, which reduces the row size (~200 bytes → ~71 bytes).

An optional parameter. If not set, the table is chosen automatically: the compact table is always used except when `experimental_enable_non_uint_key` is enabled (in that case, the first key column may not be `uint64`, and deduplication requires the full key). An explicitly set value takes priority over automatic selection. Enabling `use_compact_input_messages` together with `experimental_enable_non_uint_key` is prohibited. ||
|| `timer_store_backend` | **Type**: [NYT::NFlow::ETimerStoreBackend](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_ETimerStoreBackend)
**Default value**: `in_memory`
Timer store of this `Computation`: `in_memory` (all timers of a partition are resident in worker memory and persisted in the `timers` table) or `paged` (timers live in pages of a dynamic table, only the pages needed for firing are resident).

Static: a change means a stop and a conversion of the timers at the next start. The `paged` store does not support a `watermark_percentile` below 100 on a `Computation` with timer streams yet. ||
|| `timer_page_size` | **Type**: [NYT::NYTree::TSize](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYTree_TSize)
**Default value**: `4Ki`
Timer page size (number of timers) for the `paged` store. At least 16. Pages are split and merged gradually under the maintenance budget. Timer rows stay in their family. ||
|#


{% cut "**Additional parameters**" %}


#|
|| `max_local_message_byte_size` | **Type**: `std::optional<long>`
Raises the output message and string value size limit up to this value for streams consumed only by sinks of this computation. Such a stream cannot be an input of another computation. ||
|#


{% endcut %}
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TComputationSpec.md -->

#### WatermarkStrategy {#watermark-strategy}

Fill this for `SourceComputation`. Other `Computation` types might not support it, or might support it only partially.

<!-- source: en/flow/generated_docs/NYT_NFlow_TWatermarkStrategySpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `event_timestamp_assigner` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TEventTimestampAssignerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TEventTimestampAssignerSpec)`>`
**Default value**: `{}`
Automatic population of `EventTimestamp` for all `output` messages based on the value of a specified column. ||
|| `watermark_generator` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TWatermarkGeneratorSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWatermarkGeneratorSpec)`>`
Settings for the `EventWatermark` generation algorithm. For a `Computation` with at least one `source`, omitting this parameter is equivalent to an empty block with default settings. A multi-source `Computation` is processed as independent single-source computations: for each `source`, unavailable-group, idle-partition, and late-data-partition heuristics run independently in that order, and the prepared partitions are merged only afterwards. `idle_partitions.max_ratio` is applied separately to each `source`, and availability-group suppression decisions in one `source` do not affect another. Therefore, a fully idle `source` is not treated as a partial-idle stall and does not by itself raise `/idle_partitions_watermark_stall`, even if it gates the merged `Computation` watermark while another `source` remains active. Availability-group identifiers do not need to match across sources. ||
|| `watermark_alignment` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TWatermarkAlignmentSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWatermarkAlignmentSpec)`>`
Settings for aligning `source` across different partitions, including those of different `Computation` instances. ||
|| `watermark_percentile` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TWatermarkPercentileSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWatermarkPercentileSpec)`>`
**Default value**: `{}`
Settings for a "percentile" `EventWatermark`. Used to ignore the oldest events within a configured percentage. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TWatermarkStrategySpec.md -->

##### EventTimestampAssigner {#event-timestamp-assigner}

<!-- source: en/flow/generated_docs/NYT_NFlow_TEventTimestampAssignerSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `column` | **Type**: `std::optional<std::string>`
The time column. ||
|| `format` | **Type**: [NYT::NFlow::ETimestampFormat](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_ETimestampFormat)
**Default value**: `seconds`
The time format used (`seconds` and `milliseconds`). ||
|| `limit_by_system_timestamp` | **Type**: `bool`
**Default value**: `false`
Limit `EventTimestamp` to the `SystemTimestamp` value to protect against corrupted values from the future. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TEventTimestampAssignerSpec.md -->

##### WatermarkGenerator {#watermark-generator}

<!-- source: en/flow/generated_docs/NYT_NFlow_TWatermarkGeneratorSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `use_source_watermark` | **Type**: `bool`
**Default value**: `false`
Use `SourceWatermark` for `EventWatermark` generation. This option completely disables all heuristics for determining `EventWatermark`. Use it strictly when the input stream contains special markers with `EventWatermark`. ||
|| `out_of_orderness_bound` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `0`
The upper bound on possible event reordering, used for estimating `EventWatermark`. Event reordering is not accounted for by default. ||
|| `idle_partitions` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TIdlePartitionsSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TIdlePartitionsSpec)`>`
Heuristic settings for ignoring partitions with zero write throughput. ||
|| `late_data_partitions` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TLateDataPartitionsSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TLateDataPartitionsSpec)`>`
 ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TWatermarkGeneratorSpec.md -->

Settings for `IdlePartitions`:

<!-- source: en/flow/generated_docs/NYT_NFlow_TIdlePartitionsSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `duration` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1m`
The time a partition must be empty before it is considered to have no writes. ||
|| `max_ratio` | **Type**: `double`
**Default value**: `0.4`
The proportion of partitions that can be excluded from the watermark calculation by this logic. The conceptual purpose of this option is to protect against watermark advancement when the data provider stops writing messages to the queue due to an incident on their side. Setting the option to `1.0` effectively disables this protection. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TIdlePartitionsSpec.md -->

Settings for `UnavailablePartitionGroups`:

<!-- source: en/flow/generated_docs/NYT_NFlow_TUnavailablePartitionGroupsSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

**This structure has no main parameters.**
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TUnavailablePartitionGroupsSpec.md -->

##### WatermarkAlignment {#watermark-alignment}

Use this module to align input [streams](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#stream-and-computation). It can cause a complete read stop if there are availability issues or problems estimating the `Watermark` for individual [partitions](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#partition) or `SourceComputation`.

<!-- source: en/flow/generated_docs/NYT_NFlow_TWatermarkAlignmentSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `group_name` | **Type**: [NYT::NFlow::TWatermarkAlignmentGroup](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWatermarkAlignmentGroup)
**Default value**: `default-alignment-group`
The group name. ||
|| `drift_bound` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `20m`
The maximum possible lead of an individual partition's watermark over the group's watermark. ||
|| `read_delays` | **Type**: `std::optional<THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, `[TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)`>>`
An advanced use option. If the `EventTimestamp` of an output message is greater than `EventWatermark - Delay` for at least one specified stream, reading is paused. If misconfigured, this option can cause a complete read stall. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TWatermarkAlignmentSpec.md -->

##### WatermarkPercentile {#watermark-percentile}

Use this module to ignore old data in streams when calculating `EventWatermark`. Each partition maintains a set of unprocessed messages for each outgoing stream. By default, the system calculates `EventWatermark` using the minimum `EventTimestamp` value across all messages (that is, it accounts for `100%` of events).

If an event falls within the `value` percentile, it’s always included in the `EventWatermark` calculation. If an event doesn’t fall within the specified percentile but its `EventTimestamp` differs from those events by no more than `delay`, it’s also included in the `EventWatermark` calculation. So the calculation algorithm is `EventWatermark = max(MinEventTimestamp, PercentileEventTimestamp - delay)`, where `MinEventTimestamp` is the minimum across all events, and `PercentileEventTimestamp` is the `EventTimestamp` value at the corresponding `value` percentile.

Keep in mind that the `inflight` set for a single stream within a partition can contain dozens of messages at a time. That’s why values like `99` or `99.9` can behave almost identically to `100`—meaning they won’t produce the desired effect. Also, even if old events make up less than 1% of the entire stream, they might be unevenly distributed across partitions. That’s why it’s better to use values like `80` or `90`. To avoid negative effects like ignoring recent events, there’s the `delay` parameter: it ensures that only truly old events are excluded from the calculation. If there are no such old events, the system works the same as when `value` is `100`, without losing any information.

This module affects only the `EventWatermark` calculation and doesn’t affect `SystemWatermark`.

{% note warning %}

The settings don’t affect how the system aggregates information from different partitions. The system always uses the minimum value across all partitions.

{% endnote %}

<!-- source: en/flow/generated_docs/NYT_NFlow_TWatermarkPercentileSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `value` | **Type**: [NYT::NFlow::TWatermarkPercentile](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWatermarkPercentile)
**Default value**: `100.0`
The percentage of events unconditionally considered for `EventWatermark` calculation. ||
|| `delay` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1m`
The freshness threshold for events below the selected percentile. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TWatermarkPercentileSpec.md -->

##### LateDataPartitions {#late-data-partitions}

Use this module to ignore partitions that contain only old data. It works similarly to `WatermarkPercentile`, but at the level of all partitions, not individual messages inside a partition.

`WatermarkPercentile` lets you ignore old messages **inside a partition**, but if an entire partition contains only old data, it will still slow down the overall `EventWatermark`. The `LateDataPartitions` module solves this by letting you ignore such partitions entirely.

The calculation algorithm is `FinalWatermark = max(MinPartitionWatermark, PercentilePartitionWatermark - Delay)`, where `MinPartitionWatermark` is the minimum watermark across all partitions, and `PercentilePartitionWatermark` is the partition’s watermark at the specified `value` percentile (partitions are sorted by watermark).

If a partition falls within the `value` percentile, it’s always included. If a partition doesn’t fall within the specified percentile but its watermark differs from the percentile value by no more than `delay`, it’s also included. Partitions with a watermark below the threshold (`PercentileWatermark - Delay`) are considered to contain only late data, and their watermark is “hidden” (set to `FinalWatermark`).

For example, with `value=90` and 100 partitions: 90% of the partitions (those with the highest watermarks) are always included, and the remaining 10% (those with the lowest watermarks) are included only if their lag from the 90th percentile doesn’t exceed `delay`.

{% note warning %}

This module runs **after** `IdlePartitions`. Idle partitions are already excluded from the calculation before the `LateDataPartitions` logic applies.

{% endnote %}

<!-- source: en/flow/generated_docs/NYT_NFlow_TLateDataPartitionsSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `value` | **Type**: [NYT::NFlow::TWatermarkPercentile](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWatermarkPercentile)
**Default value**: `100.0`
 ||
|| `delay` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1m`
 ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TLateDataPartitionsSpec.md -->

#### Timer

<!-- source: en/flow/generated_docs/NYT_NFlow_TTimerSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `time_type` | **Type**: [NYT::NFlow::ETimeType](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_ETimeType)
**Default value**: `event_time`
The time type used by the timer: `event_time`, `system_time`, `real_time` ||
|| `streams` | **Type**: `std::optional<THashSet<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`>>`
List of streams that the timer monitors. ||
|| `streams_with_delays` | **Type**: `std::optional<THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, `[TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)`>>`
List of streams that the timer monitors with an individual delay specified ||
|| `deduplicate_equal_timestamps` | **Type**: `bool`
**Default value**: `true`
Enables timer deduplication: if you try to create a timer with the same key and `trigger_timestamp`, only one timer remains — the one with the smaller `EventTimestamp` value ||
|| `swift` | **Type**: `bool`
**Default value**: `false`
A swift timer is acknowledged once its ephemeral descendants complete, e.g. the tasks of `IOutputCollector::Add`, rather than in the epoch that processed it. Requires disabled `deduplicate_equal_timestamps` and the in-memory timer store. The stream must not share its time type and watermark inputs with other timer streams, and it must be the only timer stream with the `current_time` time type; `trigger_timestamp` must not exceed the current timestamp of the epoch that sets the timer. Outputs may be added only by the processing step that adds no new task. After a job restart, the timer fires again with the state committed by the earlier steps, so the steps must tolerate being repeated ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TTimerSpec.md -->

If you don’t specify `streams` and `streams_with_delays`, the timer monitors the streams according to `streams_dependency` with zero extra delay. For `current_time` timers, `streams` and `streams_with_delays` are ignored: such a timer triggers when the current time reaches `TriggerTimestamp`.

#### Source {#source_spec}

<!-- source: en/flow/generated_docs/NYT_NFlow_TSourceSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `source_class_name` | **Type**: `std::string`
**Default value**: `TString("")`
Name of the corresponding class.

Must be registered using `YT_FLOW_DEFINE_SOURCE`. ||
|| `parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
**Default value**: `{}`
Arbitrary parameters for the corresponding class. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TSourceSpec.md -->

Built-in [source](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#source) implementations are described in the [connector documentation](https://ytsaurus.tech/docs/en/flow/connectors/about.md).

#### Sink {#sink_spec}

<!-- source: en/flow/generated_docs/NYT_NFlow_TSinkSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `sink_class_name` | **Type**: `std::string`
**Default value**: `TString("")`
Name of the corresponding class.

Must be registered using `YT_FLOW_DEFINE_SINK`. ||
|| `input_stream_ids` | **Type**: `THashSet<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`>`
**Default value**: `[]`
List of input streams.

Only streams specified in `output_stream_ids` of the entire `Computation` are allowed here. ||
|| `parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
**Default value**: `{}`
Arbitrary parameters for the corresponding class. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TSinkSpec.md -->

Built-in [sink](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#sink) implementations are described in the [connector documentation](https://ytsaurus.tech/docs/en/flow/connectors/about.md).

#### HeavyHitters

This module detects high-frequency keys.

<!-- source: en/flow/generated_docs/NYT_NFlow_THeavyHittersSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `window` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5m`
Time window. ||
|| `threshold` | **Type**: `double`
**Default value**: `0.01`
Keys that appear in the stream more often than `threshold` are considered heavy hitters. The algorithm uses `O(1 / threshold)` memory. ||
|| `limit` | **Type**: `long`
**Default value**: `5`
Maximum number of heavy hitter keys to show. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_THeavyHittersSpec.md -->

#### InputOrdering

These settings control the order in which the system processes events from input streams. Events with the earliest timestamps are processed first.

<!-- source: en/flow/generated_docs/NYT_NFlow_TInputOrderingSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `time_type` | **Type**: [NYT::NFlow::ETimeType](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_ETimeType)
**Default value**: `event_time`
Time for sorting: `event_time`, `system_time`, `real_time` ||
|| `stream_delays` | **Type**: `THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, `[TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)`>`
**Default value**: `{}`
Delays for input streams. If a stream is not specified, there is no delay ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TInputOrderingSpec.md -->

### Streams

Register only `output` streams in `streams`. `Source` streams exist only inside their corresponding partitions and aren’t persisted in YT. `Timer` streams have a fixed schema.

<!-- source: en/flow/generated_docs/NYT_NFlow_TStreamSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `class_name` | **Type**: `std::optional<std::string>`
 ||
|| `schema` | **Type**: `NYT::TIntrusivePtr<NYT::NTableClient::TTableSchema>`
**Required parameter**
Stream schema. It is a dynamic table schema. Cannot contain sorting or computed columns.

We recommend making schema changes by completely stopping the pipeline. In this case, no `output` messages remain inside the pipeline. ||
|| `migration_function` | **Type**: `std::string`
**Default value**: `NYT::NFlow::DefaultMigrationFunction`
 ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TStreamSpec.md -->

### Resource

<!-- source: en/flow/generated_docs/NYT_NFlow_TResourceSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `resource_class_name` | **Type**: `std::string`
**Required parameter**
Name of the corresponding class.

Must be registered using the `YT_FLOW_DEFINE_RESOURCE` macro. ||
|| `parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
**Default value**: `{}`
Arbitrary parameters for the corresponding class. ||
|| `file_providers` | **Type**: `THashMap<`[NYT::NFlow::TFileProviderId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TFileProviderId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TFileProviderSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TFileProviderSpec)`>>`
**Default value**: `{}`
Named file providers of the resource. The controller attaches a file snapshot to a target revision only after discovering every name; a controller-specific target spec may be published independently. Workers materialize the exact revisions from that target. Such resources are worker-only and must have `controller = %false` on every `required_resource_ids` path that can reach them. Keys must match `[0-9A-Za-z_-]+`. ||
|| `dependencies` | **Type**: `THashMap<`[NYT::NFlow::TResourceId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TResourceId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TResourceDescription](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TResourceDescription)`>>`
**Default value**: `{}`
Resources that this resource depends on. Mirrors the structure of `required_resource_ids` from `Computation`. ||
|| `required_capabilities` | **Type**: `THashMap<std::string, long>`
**Default value**: `{}`
 ||
|| `preload_required` | **Type**: `bool`
**Default value**: `false`
 ||
|| `always_on` | **Type**: `bool`
**Default value**: `false`
If `true`, the resource loads eagerly when assigned to a unit where a computation requires it (see `worker` / `controller` in `required_resource_ids`). Worker instances remain in memory while assigned, including between jobs. Pausing or stopping the pipeline releases them after all jobs finish. Resources that no computation requires on the unit are not loaded. Mutually exclusive with `preload_required`. With `cpu_aware` balancing, a worker without jobs in its current incarnation waits for all assigned `always_on` resources. Jobs are assigned once all these resources report `Loaded`, including resources required by other computations. Workers with jobs in their current incarnation continue to participate in balancing. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TResourceSpec.md -->

## DynamicSpec

<!-- source: en/flow/generated_docs/NYT_NFlow_TDynamicPipelineSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `target_state` | **Type**: [NYT::NFlow::EPipelineState](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_EPipelineState)
**Default value**: `unknown`
Target state of the system. Use the `start-pipeline`, `stop-pipeline`, and `pause-pipeline` commands to change it. ||
|| `computations` | **Type**: `THashMap<`[NYT::NFlow::TComputationId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TComputationId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicComputationSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicComputationSpec)`>>`
**Default value**: `{}`
Named list of all pipeline nodes. ||
|| `resources` | **Type**: `THashMap<`[NYT::NFlow::TResourceId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TResourceId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicResourceSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicResourceSpec)`>>`
**Default value**: `{}`
 ||
|| `throttlers` | **Type**: `THashMap<`[NYT::NFlow::TThrottlerId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TThrottlerId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicThrottlerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicThrottlerSpec)`>>`
A set of named distributed throttlers common to all `Computation` pipeline nodes. Keys are throttler IDs and must match `[0-9A-Za-z_-]+`. For more information, see [Distributed Throttler](https://ytsaurus.tech/docs/en/flow/concepts/distributed_throttler.md). The `failed_messages` throttler is added automatically; see [Capture rate limit](https://ytsaurus.tech/docs/en/flow/concepts/failed-messages.md#throttling). ||
|| `job_manager` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicJobManagerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicJobManagerSpec)`>`
**Default value**: `{}`
`JobManager` settings. ||
|| `message_distributor` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicMessageDistributorSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicMessageDistributorSpec)`>`
**Default value**: `{}`
`MessageDistributor` settings. ||
|| `job_tracker` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicJobTrackerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicJobTrackerSpec)`>`
**Default value**: `{}`
`JobTracker` settings. ||
|| `controller_connector` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicControllerConnectorSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicControllerConnectorSpec)`>`
**Default value**: `{}`
`ControllerConnector` settings. ||
|| `singletons` | **Type**: `NYT::TIntrusivePtr<`[NYT::TSingletonsDynamicConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_TSingletonsDynamicConfig)`>`
**Default value**: `{}`
 ||
|| `enable_mermaid_graph_describe` | **Type**: `bool`
**Default value**: `false`
 ||
|| `enable_united_stream_metrics` | **Type**: `bool`
**Default value**: `true`
Enables export of metrics for aggregated streams to Monitoring (`true` by default). Setting this option to `false` stops these exports, so dependent alerts, dashboards, and reports no longer receive the data. The option affects metric export only; metrics for individual streams continue to be exported. ||
|| `flow_view_cache_codec` | **Type**: [NYT::NCompression::ECodec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NCompression_ECodec)
**Default value**: `zstd_2`
 ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TDynamicPipelineSpec.md -->

`Stream` and `Resource` currently don’t have or don’t support dynamic parameters.

### Computation

<!-- source: en/flow/generated_docs/NYT_NFlow_TDynamicComputationSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `batch_duration` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
Maximum time to collect an input batch. For YTsaurus queues, a non-full batch keeps collecting until this time passes since the previous batch; after a non-full queue response, the next request is also made no earlier than that moment. ||
|| `max_rows_per_batch` | **Type**: [NYT::NYTree::TSize](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYTree_TSize)
**Default value**: `1K`
Maximum size of an input batch in rows. Calculated separately for `input_streams`, `timer_streams`, and for each of the `source_streams`. For YTsaurus queues, unpacked Flow messages are counted; a batch may exceed the limit by one queue row. ||
|| `max_bytes_per_batch` | **Type**: [NYT::NYTree::TSize](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYTree_TSize)
**Default value**: `10Mi`
Maximum size of an input batch in bytes. Calculated similarly to `max_rows_per_batch`. For YTsaurus queues, the size of unpacked messages is counted; the limit may be exceeded as for `max_rows_per_batch`. ||
|| `max_keys_per_batch` | **Type**: `std::optional<`[NYT::NYTree::TSize](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYTree_TSize)`>`
 ||
|| `source_read_delay_estimator` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TSourceReadDelayEstimatorSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TSourceReadDelayEstimatorSpec)`>`
**Default value**: `{}`
Controls Offered and eligible unread Ready estimation in Swift ordered sources with nonempty `watermark_alignment.read_delays`. Estimation always applies; this section only changes its parameters. Learning follows the source key across partition restarts. Unknown estimates remain unknown. ||
|| `draining` | **Type**: `bool`
**Default value**: `false`
The `Draining` mode is used to drain the pipeline. In this mode, the `Computation` stops running timers and fetching new events from the `Source`.

The `stop-pipeline` command stops the entire pipeline through a full drain. You can set this mode for individual `Computation` nodes for debugging. ||
|| `empty_batch_backoff` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `250ms`
If the epoch is empty, the `Computation` sleeps for the specified additional time. We do not recommend setting this parameter to less than 100 ms. ||
|| `lease_check_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1m`
 ||
|| `retryable_request` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicRetryableRequestSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicRetryableRequestSpec)`>`
**Default value**: `{}`
 ||
|| `tracer` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicPartitionTracerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicPartitionTracerSpec)`>`
**Default value**: `{}`
 ||
|| `parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
**Default value**: `{}`
Arbitrary dynamic parameters of the `Computation` class. ||
|| `processing_function_parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
Dynamic process-function parameters. Their type is declared in `YT_FLOW_DEFINE_PROCESS_FUNCTION` registration. For a process function graph, it contains the dynamic parameters of the nodes in `nodes`. ||
|| `source_streams` | **Type**: `THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicSourceSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicSourceSpec)`>>`
**Default value**: `{}`
Dynamic parameters of all `Sources`. ||
|| `key_visitor_streams` | **Type**: `THashMap<`[NYT::NFlow::TStreamId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TStreamId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicKeyVisitorStreamSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicKeyVisitorStreamSpec)`>>`
**Default value**: `{}`
 ||
|| `sinks` | **Type**: `THashMap<`[NYT::NFlow::TSinkId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TSinkId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicSinkSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicSinkSpec)`>>`
**Default value**: `{}`
Dynamic parameters of all `Sinks`. ||
|| `external_state_managers` | **Type**: `THashMap<std::string, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicExternalStateManagerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicExternalStateManagerSpec)`>>`
**Default value**: `{}`
 ||
|| `external_state_joiners` | **Type**: `THashMap<std::string, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicExternalStateJoinerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicExternalStateJoinerSpec)`>>`
**Default value**: `{}`
 ||
|| `state_joiners` | **Type**: `THashMap<std::string, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicStateJoinerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicStateJoinerSpec)`>>`
**Default value**: `{}`
 ||
|| `async_processors` | **Type**: `THashMap<std::string, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicAsyncProcessorSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicAsyncProcessorSpec)`>>`
**Default value**: `{}`
Limits of the async processors of the `Computation` by processor name, that is, the name passed to `IJobInitContext::InitAsyncProcessor`, or to `IRuntimeInitContext::InitAsyncProcessor` in a process function. A processor without an entry uses the default limits. A job fails if an entry names a processor that the `Computation` does not register. ||
|| `state_manager` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicStateManagerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicStateManagerSpec)`>`
**Default value**: `{}`
 ||
|| `input_store` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicInputStoreSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicInputStoreSpec)`>`
**Default value**: `{}`
 ||
|| `timer_store` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicTimerStoreSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicTimerStoreSpec)`>`
**Default value**: `{}`
 ||
|| `output_store` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicOutputStoreSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicOutputStoreSpec)`>`
**Default value**: `{}`
 ||
|| `timer_store_count_limit` | **Type**: [NYT::NYTree::TSize](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYTree_TSize)
**Default value**: `250K`
 ||
|| `timer_store_byte_size_limit` | **Type**: [NYT::NYTree::TSize](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYTree_TSize)
**Default value**: `100Mi`
 ||
|| `blocked_time_window` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `10m`
Averaging window for the share of time the job spent blocked on each limit (`blocked_share` in diagnostics). The share is normalized to the job lifetime, so a job blocked from the start shows `~1` regardless of its age. ||
|| `input_rows_throttler_id` | **Type**: `std::optional<`[NYT::NFlow::TThrottlerId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TThrottlerId)`>`
Throttler ID for limiting the message processing rate. If set, before each iteration the `Computation` waits for a quota equal to the number of messages in the input batch. The ID must be present in `dynamic_spec/throttlers`. For more information, see [Distributed Throttler](https://ytsaurus.tech/docs/en/flow/concepts/distributed_throttler.md). ||
|| `input_bytes_throttler_id` | **Type**: `std::optional<`[NYT::NFlow::TThrottlerId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TThrottlerId)`>`
Similar to `input_rows_throttler_id`, but quotas the total `byte_size` of messages in the batch &mdash; the system size of their serialized representation. ||
|| `input_rows_throttler_class_id` | **Type**: `std::optional<`[NYT::NFlow::TQuotaClassId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TQuotaClassId)`>`
Quota class for the `input_rows_throttler_id` throttler. The class must be declared by that throttler; `default` selects the reserved class with weight `1.0`. Requires `input_rows_throttler_id` to be set. ||
|| `input_bytes_throttler_class_id` | **Type**: `std::optional<`[NYT::NFlow::TQuotaClassId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TQuotaClassId)`>`
Quota class for the `input_bytes_throttler_id` throttler. The class must be declared by that throttler; `default` selects the reserved class with weight `1.0`. Requires `input_bytes_throttler_id` to be set. ||
|| `skip_if_expression` | **Type**: `std::optional<std::string>`
YTQL predicate for filtering (skipping) input messages before `Computation` processing. A message is skipped (dropped) if the predicate evaluates to boolean `true`.

The predicate is evaluated on the message `payload` columns plus the meta-columns `$message_id`, `$stream_id`, `$system_timestamp`, `$event_timestamp`, `$alignment_timestamp`. The result must be boolean: `NULL` or a non-boolean result causes an exception.

Applies to both `input` and `source` streams. The number of skipped messages is exported via the `input_streams/skipped_by_expression_count` and `source_streams/skipped_by_expression_count` metrics. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TDynamicComputationSpec.md -->

#### Source

<!-- source: en/flow/generated_docs/NYT_NFlow_TDynamicSourceSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
**Default value**: `{}`
Arbitrary dynamic parameters of the corresponding `Source` class. ||
|| `draining` | **Type**: `bool`
**Default value**: `false`
 ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TDynamicSourceSpec.md -->

#### Sink

<!-- source: en/flow/generated_docs/NYT_NFlow_TDynamicSinkSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `parameters` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::IMapNode>`
**Default value**: `{}`
Arbitrary dynamic parameters of the corresponding `Sink` class. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TDynamicSinkSpec.md -->

### JobManager

This is the config for the module that manages [jobs](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#job) on the [controller](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#controller).

<!-- source: en/flow/generated_docs/NYT_NFlow_TDynamicJobManagerSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `balancer_type` | **Type**: [NYT::NFlow::EJobBalancerType](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_EJobBalancerType)
**Default value**: `cpu_aware`
 ||
|| `use_cpu_aware_balancer` | **Type**: `std::optional<bool>`
Enables experimental CPU-aware balancing. Default balancing can lead to severe CPU resource imbalance among workers. ||
|| `rebalance_delay_after_pipeline_sync` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `30s`
 ||
|| `rebalance_target_deviation` | **Type**: `double`
**Default value**: `0.1`
Maximum acceptable deviation. ||
|| `rebalance_hot_mode_coeff` | **Type**: `double`
**Default value**: `2.0`
 ||
|| `rebalance_action_min_time` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `4s`
 ||
|| `rebalance_action_max_time` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `12s`
 ||
|| `rebalance_sync_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `10s`
Period of `cpu_aware` balancing. It is not recommended to set a value lower than `15min` because the balancer looks at `10min` metrics. ||
|| `rebalance_count_exceeded_allowed` | **Type**: `double`
**Default value**: `1.2`
 ||
|| `rebalance_even_load_thresholds` | **Type**: `THashMap<`[NYT::NFlow::EBalanceResource](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_EBalanceResource)`, NYT::TIntrusivePtr<`[NYT::NFlow::TEvenLoadThresholds](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TEvenLoadThresholds)`>>`
**Default value**: `{}`
Per-resource thresholds of the "load is even" gate, for example `{cpu = {spread = 2.0; ratio = 1.5}}`. Only resources with a non-zero weight are consulted. Unset fields fall back to the defaults: a spread of `1.0` core for CPU and `1 GB` for memory, a ratio of `1.2`. ||
|| `rebalance_min_cpu_spread` | **Type**: `std::optional<double>`
Deprecated: the same as `spread` of the `cpu` entry of `rebalance_even_load_thresholds`. A value set explicitly in `rebalance_even_load_thresholds` takes precedence. ||
|| `rebalance_min_cpu_ratio` | **Type**: `std::optional<double>`
Deprecated: the same as `ratio` of the `cpu` entry of `rebalance_even_load_thresholds`. A value set explicitly in `rebalance_even_load_thresholds` takes precedence. ||
|| `balance_weights` | **Type**: `THashMap<`[NYT::NFlow::EBalanceResource](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_EBalanceResource)`, double>`
**Default value**: `{'cpu': 1.0, 'memory': 0.0}`
Relative importance of the resources for `cpu_aware` balancing, for example `{cpu = 80; memory = 20}`. The given keys are merged into the default `{cpu = 1; memory = 0}`; the weights are normalized, so only the proportions matter. A resource with a zero weight does not take part in balancing, so by default only CPU is balanced. The `rebalance_target_deviation` threshold applies to the normalized weighted sum: adding a second resource proportionally shrinks the contribution of the first one. With a non-zero `memory` weight, partition memory includes the measured job memory and the input buffer bytes of its streams. Each input stream is charged the larger of its `demand`, as requested before the buffer pool scales it down, and its resident bytes. A stream without `demand`, for example one with an explicit limit override or on a worker without buffer manager v2, is charged its resident bytes. ||
|| `balancer_metrics_source` | **Type**: [NYT::NFlow::EBalancerMetricsSource](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_EBalancerMetricsSource)
**Default value**: `job`
Where the `cpu_aware` balancer takes a partition's CPU usage from. `partition`: the 10-minute rate of the running job, and until it exists the partition's persisted history (measured by the previous job, survives moves and restarts); 30-second and immediate rates are never used. `job`: the previous behaviour, only the running job's metrics with a fallback to the 30-second and immediate rates. Off by default while the change is rolled out pipeline by pipeline; temporary switch. ||
|| `balance_warmup_protection` | **Type**: `bool`
**Default value**: `false`
Never move a running job whose metrics are not mature yet: the 10-minute rate exists and one more 10-minute window has passed since it appeared. Such a job is still paying for its previous move and its weight is unknown. The protection applies to every balancing resource: a memory-only configuration waits for the CPU rate to mature as well. A worker over the count limit whose kick candidate is immature is skipped for the round. Jobs younger than two minutes stay movable: they have invested nothing yet, and on a pipeline restart the workers do not register all at once. Partitions without a job are placed as usual. Off by default while the change is rolled out pipeline by pipeline; temporary switch. ||
|| `balance_warmup_idle_worker_share` | **Type**: `double`
**Default value**: `0.2`
Share of the group's workers without a job of the group at which warm-up protection is lifted: filling idle workers matters more than the metrics of the jobs that would move, e.g. when workers register minutes apart after a pipeline restart. Temporary switch for rollback. ||
|| `worker_coef_mode` | **Type**: [NYT::NFlow::EWorkerCoefMode](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_EWorkerCoefMode)
**Default value**: `legacy`
How the `cpu_aware` balancer estimates the relative speed of workers. `probing`: from partitions that moved between workers, comparing CPU per message before and after the move; observations per worker pair accumulate in `balancer_state` and are solved together; without moves all coefficients are 1. `legacy`: the previous estimate from the current usage of each worker's partitions against the computation averages. Off by default while the change is rolled out pipeline by pipeline; temporary switch. ||
|| `worker_coef_half_life` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1d`
Half-life of an accumulated worker-pair observation: when a new observation merges in, the old one loses weight in proportion to the time passed. Without new observations the estimate does not change. ||
|| `worker_coef_retention` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `7d`
How long the observations of a worker absent from the group are kept; a worker returning earlier gets its coefficient back. ||
|| `worker_coef_prior_weight` | **Type**: `double`
**Default value**: `0.05`
Weight of the prior "the worker's coefficient is 1" in observation weight units (a move of one partition weighs `1 / N`, `N` being the number of partitions on the receiving worker). Pins the group's mean coefficient to 1 and damps single observations. ||
|| `worker_coef_max_ratio` | **Type**: `double`
**Default value**: `4.0`
Safety limit: worker coefficients are clamped to `[1 / max_ratio, max_ratio]`. ||
|| `disable_even_load_gate` | **Type**: `std::optional<bool>`
 ||
|| `async_balancing` | **Type**: `bool`
**Default value**: `true`
 ||
|| `graceful_move` | **Type**: `bool`
**Default value**: `true`
 ||
|| `zero_queue_latency` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
 ||
|| `planning_horizon` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `10m`
 ||
|| `preloading_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `30m`
How long the `resource_queue` balancer keeps a model preload that is still loading, counted from the request. Until then the preload is not released and the worker counts as a worker of the computations that need the model; after it the usual rules apply. ||
|| `minimum_worker_count` | **Type**: `unsigned long`
**Default value**: `1`
Minimum number of workers to distribute the load across.

Must be filled in to avoid out-of-memory errors and overloading individual workers when the system starts. Recommended value is about 60% of the target number of workers (to survive the loss of a subset of workers, for example, if a data center goes offline). ||
|| `lost_job_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `10s`
Time after which a `Job` is considered lost if no status update reaches the controller. The same timeout is used to consider a worker lost. ||
|| `faulty_address_window` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5m`
Window within which the exponentially decaying counter of connection losses between the controller and a worker is calculated. If this counter exceeds `faulty_address_attempts`, the worker is considered `faulty` and is ignored during load balancing. Solves the problem of workers that constantly lose connection to the controller. Also solves the problem of two workers with different incarnation IDs but the same address (when deployment has two pods that think they share the same FQDN). ||
|| `faulty_address_attempts` | **Type**: `unsigned long`
**Default value**: `5`
See the description of the `faulty_address_window` parameter. ||
|| `partitioning` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicPartitioningCoordinatorSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicPartitioningCoordinatorSpec)`>`
**Default value**: `{}`
Pipeline partitioning settings. ||
|| `worker_group_override` | **Type**: `THashMap<`[NYT::NFlow::TWorkerGroupId](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TWorkerGroupId)`, NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicJobManagerGroupSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicJobManagerGroupSpec)`>>`
**Default value**: `{}`
 ||
|| `partition_history_limit` | **Type**: `long`
**Default value**: `4096`
Most partition histories the balancer keeps across job restarts, see `balancer_metrics_source`. The histories live in one persisted document with a size limit, so when the limit is reached the lightest history gives way to a heavier one and lighter ones are not saved. A pipeline stop saves no histories at all. ||
|| `watermark_history_thinning_factor` | **Type**: `int`
**Default value**: `4`
Thinning factor K of the watermark histories the controller keeps to compute `ready_time_lag` for event-time and system-time timers. A history element spans at most 1/K of its age, so the lag may be over-reported by at most 1/K of its value and is never under-reported; a larger K keeps more elements. A change applies to new samples only: existing elements are never split. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TDynamicJobManagerSpec.md -->

### JobTracker

This is the config for the module that manages jobs on the [worker](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#worker).

<!-- source: en/flow/generated_docs/NYT_NFlow_TDynamicJobTrackerSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `job_control_threads` | **Type**: `int`
**Default value**: `3`
Number of control threads.

In most cases, the default value is sufficient. ||
|| `job_threads` | **Type**: `std::optional<int>`
Number of threads for performing jobs.

If not specified, the pool size is calculated automatically based on the node's CPU limit.
If the node limit cannot be calculated, 30 threads are created. ||
|| `buffer_state_manager` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicBufferStateManagerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicBufferStateManagerSpec)`>`
**Default value**: `{}`
Configuration of the buffer management module for incoming and outgoing messages. ||
|| `load_throughput_throttler` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TLoadThroughputThrottlerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TLoadThroughputThrottlerSpec)`>`
**Default value**: `{'limit': 134217728.0}`
 ||
|| `state_cache` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicStateCacheSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicStateCacheSpec)`>`
**Default value**: `{}`
 ||
|| `mark_performance_metrics_steady_after_first_iteration` | **Type**: `bool`
**Default value**: `false`
Report the job's rate counters (CPU, input messages, memory) as steady only once its first iteration with input has completed, so that the balancer reads windowed metrics such as `cpu_usage_10m` after the job's initialization (state download, index build) has decayed out of them; the counters themselves are never reset. A job that has received no input for a minute is reported as steady as well, so an idle job reports its near-zero rates instead of staying unmeasured. Off by default while the change is rolled out pipeline by pipeline; temporary switch. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TDynamicJobTrackerSpec.md -->

#### BufferStateManager

This is the config for the module that manages buffer sizes.

<!-- source: en/flow/generated_docs/NYT_NFlow_TDynamicBufferStateManagerSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

V2 can select a time window used to calculate buffer size for each stream. Set `min_duration` smaller than `max_duration` in the input or output buffer settings; those settings also describe the calculation and examples.

#|
|| **Parameter** | **Description** ||
|| `manage_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
Frequency of buffer size recalculation. ||
|| `demand_window` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1m`
Time window over which the utilization of a single buffer is estimated. ||
|| `input_buffer` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicBufferStateManagerSpec::TOneSideBufferSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicBufferStateManagerSpec_TOneSideBufferSpec)`>`
**Default value**: `{}`
Input buffer settings. ||
|| `output_buffer` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicBufferStateManagerSpec::TOneSideBufferSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicBufferStateManagerSpec_TOneSideBufferSpec)`>`
**Default value**: `{}`
Output buffer settings. ||
|| `enable_v2` | **Type**: `bool`
**Default value**: `true`
Enables the v2 buffer-sizing strategy: limit = peak usage + headroom, with a `v2_gain_epochs × demand × epoch` bandwidth-delay-product floor. The `demand × buffer-sizing window` cap (`max_duration`, or the automatic window when enabled) is raised by announced input backlog or, for a producing output without automatic window selection, to an equal share of half `fair_share_pool` because producer epochs do not reveal the downstream acknowledgement period. Demand-backed limits are allocated before speculative output probes. On both sides, Σ(limits) ≤ `fair_share_pool` including other streams' in-flight bytes; `job_limit` remains the per-stream bound. Streams with `job_overrides` remain fully outside the pool as in v1, so worker buffer memory should cover `fair_share_pool` plus the actual in-flight bytes of overridden streams. Enabled by default; set this option to `false` to use the previous v1 formula. ||
|#


{% cut "**Additional parameters**" %}


#|
|| `epoch_cycle_window_samples` | **Type**: `int`
**Default value**: `16`
Number of samples in the window used to estimate the median job epoch cycle. ||
|| `max_rate_estimator_buckets` | **Type**: `int`
**Default value**: `8`
Number of buckets in the windowed-max drain-rate estimator; changing it resets the estimate. ||
|| `warmup_refresh_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `30s`
How often a job polls the converged warmup state for persistence. ||
|| `v2_gain_epochs` | **Type**: `double`
**Default value**: `2.0`
Minimum buffer target in units of job epochs, used as a bandwidth-delay-product floor. Applies only when `enable_v2` is set. ||
|| `v2_use_offered_rate` | **Type**: `bool`
**Default value**: `true`
Whether announced backlog rate contributes to demand. Disable it when a producer systematically overstates its offered rate. Applies only when `enable_v2` is set. ||
|| `v2_floor` | **Type**: [NYT::NYTree::TSize](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYTree_TSize)
**Default value**: `2Mi`
Minimum grant for a stream with backlog; it must fit the largest message. Applies only when `enable_v2` is set. ||
|| `v2_headroom_growth_factor` | **Type**: `double`
**Default value**: `2.0`
Headroom growth factor per management tick while utilization is high. Applies only when `enable_v2` is set. ||
|| `v2_high_utilization_threshold` | **Type**: `double`
**Default value**: `0.65`
Utilization threshold above which headroom grows; below half this value headroom decays. Applies only when `enable_v2` is set. ||
|| `v2_publish_threshold` | **Type**: `double`
**Default value**: `0.25`
Suppress a new limit when its relative increase is smaller than this value; decreases are always published. Applies only when `enable_v2` is set. ||
|#


{% endcut %}
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TDynamicBufferStateManagerSpec.md -->

All dimensional parameters in this config (measured in number of items or bytes) can be parsed from strings like "50K".

### ControllerConnector

This is the config for managing the connection between the worker and the controller.

Don’t change it unless necessary.

<!-- source: en/flow/generated_docs/NYT_NFlow_TDynamicControllerConnectorSpec.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/common/spec.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/spec.h)

#|
|| **Parameter** | **Description** ||
|| `controller_wait_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5m`
Time after which the worker forcibly disconnects all jobs in case of loss of connection with the controller. ||
|| `controller_discover_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `10s`
Period for rediscovering the controller. ||
|| `controller_heartbeat_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
Period of worker heartbeats to the controller. ||
|| `worker_statistics_report_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `30s`
 ||
|| `controller_heartbeat_rpc_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `10s`
Heartbeat timeout. ||
|| `controller_heartbeat_failure_backoff` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
Backoff time in case of handshake failure. ||
|| `controller_handshake_rpc_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `10s`
Handshake timeout. ||
|| `controller_handshake_failure_backoff` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
 ||
|| `orchid_update_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
Period for updating `orchid` on the controller. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TDynamicControllerConnectorSpec.md -->

## Parameters {#parameters}

The `parameters` field exists in the `Computation`, `Source`, `Sink`, and `Resource` specs. It contains parameters of the selected built-in implementation, such as a process-function adapter. Check the selected class’s documentation for supported fields, and don’t use this block for user-logic parameters.

Set C++ process-function parameters separately in `processing_function_parameters`. Declare a regular [yson struct](https://ytsaurus.tech/docs/en/user-guide/storage/data-types.md#yson_struct), register its type with the function, and read it through the runtime context:

```cpp
struct TMyParameters
    : public NYTree::TYsonStruct
{
    i64 Threshold;

    REGISTER_YSON_STRUCT(TMyParameters);

    static void Register(TRegistrar registrar);
};

YT_FLOW_DEFINE_PROCESS_FUNCTION(TMyFunction, TMyParameters);
```

```yson
"processing_function_parameters" = {
    "threshold" = 10;
};
```

Static parameters are available in `Init` through `initContext->GetParameters<TMyParameters>()`; dynamic parameters are available through `context->GetDynamicParameters<TMyParameters>()`. For more details, see [Process functions](https://ytsaurus.tech/docs/en/flow/cpp/process-functions.md#parameters).

## Config

This is the config for a Flow cluster node.

<!-- source: en/flow/generated_docs/NYT_NFlow_TFlowNodeConfig.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/runner/config.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/runner/config.h)

#|
|| **Parameter** | **Description** ||
|| `logging` | **Type**: `NYT::TIntrusivePtr<`[NYT::NLogging::TLogManagerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NLogging_TLogManagerConfig)`>`
**Default value**:

```yson
{
    "high_backlog_watermark" = 100000;
    "low_backlog_watermark" = 100000;
    "min_disk_space" = 0;
    "rules" = [
        {
            "exclude_categories" = [];
            "max_level" = "maximum";
            "min_level" = "info";
            "writers" = [
                "Stderr";
            ];
        };
    ];
    "writers" = {
        "Stderr" = {
            "common_fields" = {};
            "enable_host_field" = %false;
            "enable_native_tags" = %false;
            "enable_source_location" = %false;
            "enable_span_id_field" = %false;
            "enable_system_fields" = %true;
            "format" = "plain_text";
            "type" = "stderr";
            "yson_format" = "text";
        };
    };
}
```
Logging settings. ||
|| `jaeger` | **Type**: `NYT::TIntrusivePtr<`[NYT::NTracing::TJaegerTracerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NTracing_TJaegerTracerConfig)`>`
**Default value**: `{}`
 ||
|| `pipe_io_dispatcher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NPipeIO::TPipeIODispatcherConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NPipeIO_TPipeIODispatcherConfig)`>`
**Default value**: `{}`
 ||
|| `solomon_registry` | **Type**: `NYT::TIntrusivePtr<`[NYT::NProfiling::TSolomonRegistryConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NProfiling_TSolomonRegistryConfig)`>`
**Default value**: `{}`
 ||
|| `error_backtrace_enricher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TBacktraceEnricherSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TBacktraceEnricherSpec)`>`
**Default value**: `{}`
 ||
|| `query_engine_config` | **Type**: `NYT::TIntrusivePtr<`[NYT::NQueryClient::TQueryEngineConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NQueryClient_TQueryEngineConfig)`>`
**Default value**: `{}`
 ||
|| `cluster_url` | **Type**: `std::string`
**Required parameter**
YTsaurus cluster to use. ||
|| `path` | **Type**: `TString`
**Required parameter**
Working directory of the pipeline on the `cluster_url` cluster. ||
|| `proxy_role` | **Type**: `std::optional<std::string>`
`rpc proxy` role. ||
|| `rpc_port` | **Type**: `int`
**Default value**: `0`
Main port for communication between workers and controllers, can be different for each instance. ||
|| `monitoring_port` | **Type**: `int`
**Default value**: `0`
API for obtaining metrics. ||
|| `companion` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::NCompanion::TCompanionConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_NCompanion_TCompanionConfig)`>`
Companion process parameters. Needed by any worker that runs a companion (Python, Java, Go, C++ companion). Filled in automatically for a vanilla launch. ||
|| `controller` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::NController::TControllerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_NController_TControllerConfig)`>`
**Default value**: `{}`
`Controller` parameters. ||
|| `worker` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::NWorker::TWorkerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_NWorker_TWorkerConfig)`>`
**Default value**: `{}`
`Worker` parameters. ||
|| `authenticator` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TAuthenticatorConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TAuthenticatorConfig)`>`
**Default value**: `{}`
 ||
|| `bus_server` | **Type**: `NYT::TIntrusivePtr<`[NYT::NBus::NTcp::TBusServerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NBus_NTcp_TBusServerConfig)`>`
**Default value**: `{}`
 ||
|| `rpc_server` | **Type**: `NYT::TIntrusivePtr<`[NYT::NRpc::TServerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NRpc_TServerConfig)`>`
**Default value**: `{}`
 ||
|| `core_dumper` | **Type**: `NYT::TIntrusivePtr<`[NYT::NCoreDump::TCoreDumperConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NCoreDump_TCoreDumperConfig)`>`
 ||
|| `solomon_exporter` | **Type**: `NYT::TIntrusivePtr<`[NYT::NProfiling::TSolomonExporterConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NProfiling_TSolomonExporterConfig)`>`
**Default value**: `{'enable_solomon_aggregates': true}`
 ||
|| `solomon_proxy` | **Type**: `NYT::TIntrusivePtr<`[NYT::NProfiling::TSolomonProxyConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NProfiling_TSolomonProxyConfig)`>`
**Default value**: `{}`
 ||
|| `http_client_config` | **Type**: `NYT::TIntrusivePtr<`[NYT::NHttp::TClientConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NHttp_TClientConfig)`>`
**Default value**: `{}`
 ||
|| `https_client_config` | **Type**: `NYT::TIntrusivePtr<`[NYT::NHttps::TClientConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NHttps_TClientConfig)`>`
**Default value**: `{}`
 ||
|| `http_poller_threads` | **Type**: `int`
**Default value**: `1`
 ||
|| `abort_on_unrecognized_options` | **Type**: `bool`
**Default value**: `true`
When set to true, Flow Node will not start if there are unknown options in the config. ||
|| `ignore_singletons_dynamic_config` | **Type**: `bool`
**Default value**: `false`
 ||
|| `enable_porto_resource_tracker` | **Type**: `bool`
**Default value**: `true`
 ||
|#


{% cut "**Additional parameters**" %}


#|
|| `address_resolver` | **Type**: `NYT::TIntrusivePtr<`[NYT::NNet::TAddressResolverConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NNet_TAddressResolverConfig)`>`
**Default value**: `{}`
 ||
|| `grpc_dispatcher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NRpc::NGrpc::TDispatcherConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NRpc_NGrpc_TDispatcherConfig)`>`
**Default value**: `{}`
 ||
|| `tcmalloc` | **Type**: `NYT::TIntrusivePtr<`[NYT::NTCMalloc::TTCMallocConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NTCMalloc_TTCMallocConfig)`>`
**Default value**: `{'guarded_sampling_rate': 16777216}`
 ||
|| `fiber_manager` | **Type**: `NYT::TIntrusivePtr<`[NYT::NConcurrency::TFiberManagerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NConcurrency_TFiberManagerConfig)`>`
**Default value**: `{}`
 ||
|| `resource_tracker` | **Type**: `NYT::TIntrusivePtr<`[NYT::NProfiling::TResourceTrackerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NProfiling_TResourceTrackerConfig)`>`
**Default value**: `{}`
 ||
|| `tcp_dispatcher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NBus::NTcp::TDispatcherConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NBus_NTcp_TDispatcherConfig)`>`
**Default value**: `{}`
 ||
|| `rpc_dispatcher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NRpc::TDispatcherConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NRpc_TDispatcherConfig)`>`
**Default value**: `{}`
 ||
|| `protobuf_interop` | **Type**: `NYT::TIntrusivePtr<`[NYT::NYson::TProtobufInteropConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYson_TProtobufInteropConfig)`>`
**Default value**: `{}`
 ||
|| `clients_cache` | **Type**: `NYT::TIntrusivePtr<`[NYT::NClient::NCache::TClientsCacheConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NClient_NCache_TClientsCacheConfig)`>`
**Default value**: `{}`
Connection settings of the node's YT clients cache. It backs the pipeline connector, the queue log writer and everything that takes a client from `TComputationContext::ClientsCache` &mdash; computations, connectors and state managers. The pipeline authenticator builds its own client and is not governed by these settings. ||
|| `clients_cache_factory` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::INode>`
Parameters of the root YT clients cache factory installed via `SetRootClientsCacheFactory()`. Their schema is defined by the factory itself; the built-in factory ignores this block. ||
|| `enable_phdr_cache` | **Type**: `bool`
**Default value**: `true`
 ||
|#


{% endcut %}
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TFlowNodeConfig.md -->

### Controller

<!-- source: en/flow/generated_docs/NYT_NFlow_NController_TControllerConfig.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/controller/config.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/controller/config.h)

#|
|| **Parameter** | **Description** ||
|| `controller_threads` | **Type**: `int`
**Default value**: `5`
Number of threads. ||
|| `orchid_update_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
`orchid` recalculation period. ||
|| `warm_up_time` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5s`
Warm-up period when the controller starts. ||
|| `scheduler_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5s`
Scheduler invocation period. ||
|| `no_spec_iteration_backoff` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
 ||
|| `cache_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
 ||
|| `feedback_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
 ||
|| `metrics_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5s`
 ||
|| `write_own_retryable_errors_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5s`
 ||
|| `publish_retry_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5s`
Period during which the controller attempts to tell YTsaurus that it is the leader. ||
|| `publish_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `2h`
 ||
|| `election_manager` | **Type**: `NYT::NYTree::TPolymorphicYsonStruct<NYT::NYTree::NDetail::TPolymorphicMapping<&NYT::NFlow::NController::ElectionBackendDiscriminator.<char const at offset 0>, NYT::NFlow::NController::EElectionBackend, NYT::NYTree::NDetail::TOptionalValue<NYT::NFlow::NController::EElectionBackend, (NYT::NFlow::NController::EElectionBackend)0>, NYT::NFlow::NController::TElectionBackendConfigBase, NYT::NYTree::NDetail::TLeafTag<(NYT::NFlow::NController::EElectionBackend)0, NYT::NFlow::NController::TCypressElectionBackendConfig>, NYT::NYTree::NDetail::TLeafTag<(NYT::NFlow::NController::EElectionBackend)1, NYT::NFlow::NController::TDyntableElectionBackendConfig>, NYT::NYTree::NDetail::TLeafTag<(NYT::NFlow::NController::EElectionBackend)2, NYT::NFlow::NController::TChaosElectionBackendConfig>>>`
**Default value**:

```yson
{
    "backend" = "cypress";
    "leader_cache_update_period" = 1000;
    "leader_lease_ping_period" = 1000;
    "leader_lease_ttl" = 5000;
    "lock_acquisition_period" = 1000;
}
```
Leader election settings for multiple controllers. ||
|| `persisted_state_manager` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::NController::TPersistedStateManagerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_NController_TPersistedStateManagerConfig)`>`
**Default value**: `{}`
`PersistedStateManager` settings. ||
|| `lease_manager` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::NController::TLeaseManagerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_NController_TLeaseManagerConfig)`>`
**Default value**: `{}`
`LeaseManager` settings. ||
|| `controller_service` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::NController::TControllerServiceConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_NController_TControllerServiceConfig)`>`
**Default value**: `{}`
`ControllerService` settings. ||
|| `bus` | **Type**: `NYT::TIntrusivePtr<`[NYT::NBus::NTcp::TBusConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NBus_NTcp_TBusConfig)`>`
**Default value**: `{}`
 ||
|#


{% cut "**Additional parameters**" %}


#|
|| `publish_request_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `1s`
Timeout of a single request of a publication attempt. A failed attempt is retried, so this only decides how long one attempt may hang before the retry. ||
|#


{% endcut %}
<!-- endsource: en/flow/generated_docs/NYT_NFlow_NController_TControllerConfig.md -->

#### PersistedStateManager

<!-- source: en/flow/generated_docs/NYT_NFlow_NController_TPersistedStateManagerConfig.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/controller/config.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/controller/config.h)

#|
|| **Parameter** | **Description** ||
|| `timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5s`
Save timeout in YTsaurus. ||
|| `max_reads_per_transaction` | **Type**: `long`
**Default value**: `10000`
 ||
|| `max_writes_per_transaction` | **Type**: `long`
**Default value**: `10000`
 ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_NController_TPersistedStateManagerConfig.md -->

#### LeaseManager

<!-- source: en/flow/generated_docs/NYT_NFlow_NController_TLeaseManagerConfig.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/controller/config.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/controller/config.h)

#|
|| **Parameter** | **Description** ||
|| `lease_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `10m`
How long a job stays fenced without the controller refreshing the fence. With the Cypress election backend this is the timeout of the job's master lease transaction, with the chaos one the timeout of its chaos lease, and with the dyntable one the ttl of the pipeline-wide deadline row that gates every worker commit. ||
|| `lease_ping_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `30s`
Period at which the leader prolongs the leases. Every backend prolongs from the leader and none from the worker, but the machinery differs: the chaos leases are pinged by one periodic executor, the Cypress ones by a client-side pinger per lease transaction, and the shared dyntable deadline is simply rewritten. ||
|| `max_concurrent_requests` | **Type**: `long`
**Default value**: `500`
Maximum number of concurrent requests the manager keeps in flight when it attaches to leases or terminates them. Do not increase to avoid overloading the master. Prolongation is not capped by it: the chaos pings are dispatched in one fire-and-forget round, and the Cypress ones belong to the lease transactions themselves. ||
|| `termination_backoff` | **Type**: `NYT::TExponentialBackoffOptions`
**Default value**:

```yson
{
    "backoff_jitter" = 0.1;
    "backoff_multiplier" = 1.5;
    "invocation_count" = 5;
    "max_backoff" = 5000;
    "min_backoff" = 500;
}
```
Retries of the termination of the leases whose jobs the controller has removed, within one scheduling iteration. The iteration is not persisted until every such lease is terminated, so this bounds the number of attempts made while the chaos cell or master is unavailable; each attempt also waits for its requests up to the client RPC timeout. Used by the Cypress and chaos election backends only. ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_NController_TLeaseManagerConfig.md -->

#### ControllerService

<!-- source: en/flow/generated_docs/NYT_NFlow_NController_TControllerServiceConfig.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/controller/config.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/controller/config.h)

#|
|| **Parameter** | **Description** ||
|| `set_spec_retry_count` | **Type**: `int`
**Default value**: `3`
Number of attempts to update the spec in case of errors. ||
|| `set_spec_retry_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration)
**Default value**: `5s`
Time between attempts. ||
|| `tables_throttler` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TLoadThroughputThrottlerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TLoadThroughputThrottlerSpec)`>`
**Default value**: `{}`
 ||
|#
<!-- endsource: en/flow/generated_docs/NYT_NFlow_NController_TControllerServiceConfig.md -->


Plus a small buffer for the case where some live partitions also won’t receive any writes.

## Runner config {#runner-config}

This is the config for running a pipeline. The format of this config isn’t standardized; you can use whatever config works best for your tasks.

Flow provides a simple version of this config, `NYT::NFlow::TSimpleRunnerConfig`, and a program, `NYT::NFlow::TSimpleRunnerProgram`, to run a pipeline using this config. The Flow test framework relies on this config format to inject the cluster and the pipeline path in the local YTsaurus, and to apply test default parameters for faster test runs.

### SimpleRunnerConfig

<!-- source: en/flow/generated_docs/NYT_NFlow_TSimpleRunnerConfig.md -->
<!-- This file is generated by yt/yt/flow/yandex/tools/generate_yson_struct_doc/generate.sh script -->
<!-- Before using doc generation tool check readme: yt/yt/flow/yandex/tools/generate_yson_struct_doc/README.md -->
Source: [yt/yt/flow/library/cpp/runner/simple_runner_program.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/runner/simple_runner_program.h)

#|
|| **Parameter** | **Description** ||
|| `logging` | **Type**: `NYT::TIntrusivePtr<`[NYT::NLogging::TLogManagerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NLogging_TLogManagerConfig)`>`
**Default value**:

```yson
{
    "high_backlog_watermark" = 100000;
    "low_backlog_watermark" = 100000;
    "min_disk_space" = 0;
    "rules" = [
        {
            "exclude_categories" = [];
            "max_level" = "maximum";
            "min_level" = "info";
            "writers" = [
                "Stderr";
            ];
        };
    ];
    "writers" = {
        "Stderr" = {
            "common_fields" = {};
            "enable_host_field" = %false;
            "enable_native_tags" = %false;
            "enable_source_location" = %false;
            "enable_span_id_field" = %false;
            "enable_system_fields" = %true;
            "format" = "plain_text";
            "type" = "stderr";
            "yson_format" = "text";
        };
    };
}
```
Logging settings. ||
|| `jaeger` | **Type**: `NYT::TIntrusivePtr<`[NYT::NTracing::TJaegerTracerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NTracing_TJaegerTracerConfig)`>`
**Default value**: `{}`
 ||
|| `pipe_io_dispatcher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NPipeIO::TPipeIODispatcherConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NPipeIO_TPipeIODispatcherConfig)`>`
**Default value**: `{}`
 ||
|| `solomon_registry` | **Type**: `NYT::TIntrusivePtr<`[NYT::NProfiling::TSolomonRegistryConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NProfiling_TSolomonRegistryConfig)`>`
**Default value**: `{}`
 ||
|| `error_backtrace_enricher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TBacktraceEnricherSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TBacktraceEnricherSpec)`>`
**Default value**: `{}`
 ||
|| `query_engine_config` | **Type**: `NYT::TIntrusivePtr<`[NYT::NQueryClient::TQueryEngineConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NQueryClient_TQueryEngineConfig)`>`
**Default value**: `{}`
 ||
|| `cluster_url` | **Type**: `std::string`
**Required parameter**
Name of the cluster where the pipeline is located. The tail after the last `/` is treated as a proxy role and takes precedence over `proxy_role` &mdash; e.g. `example-cluster/flow` means cluster `example-cluster` with role `flow`. ||
|| `proxy_role` | **Type**: `std::optional<std::string>`
**Required parameter**
The [proxy role](https://ytsaurus.tech/docs/en/user-guide/proxy/about.md#rpc_proxy) to use for performing operations on the pipeline. ||
|| `path` | **Type**: `TString`
**Required parameter**
Path to the pipeline on YTsaurus. ||
|| `spec` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TPipelineSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TPipelineSpec)`>`
**Required parameter**
Static spec with which to start the pipeline. ||
|| `dynamic_spec` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDynamicPipelineSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDynamicPipelineSpec)`>`
**Default value**: `{}`
Dynamic spec with which to start the pipeline. ||
|| `abort_on_unrecognized_options` | **Type**: `bool`
**Default value**: `false`
 ||
|| `abort_on_specs_parseability_error` | **Type**: `bool`
**Default value**: `false`
 ||
|| `vanilla` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TVanillaConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TVanillaConfig)`>`
If set and `enable=%true`, the runner starts a vanilla operation on YTsaurus and runs the federation inside it, instead of starting the controller/workers locally. ||
|| `set_flow_core_target` | **Type**: `bool`
**Default value**: `true`
 ||
|| `direct_controller_commands` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TDirectControllerCommandsConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TDirectControllerCommandsConfig)`>`
**Default value**: `{}`
 ||
|#


{% cut "**Additional parameters**" %}


#|
|| `address_resolver` | **Type**: `NYT::TIntrusivePtr<`[NYT::NNet::TAddressResolverConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NNet_TAddressResolverConfig)`>`
**Default value**: `{}`
 ||
|| `grpc_dispatcher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NRpc::NGrpc::TDispatcherConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NRpc_NGrpc_TDispatcherConfig)`>`
**Default value**: `{}`
 ||
|| `tcmalloc` | **Type**: `NYT::TIntrusivePtr<`[NYT::NTCMalloc::TTCMallocConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NTCMalloc_TTCMallocConfig)`>`
**Default value**: `{}`
 ||
|| `fiber_manager` | **Type**: `NYT::TIntrusivePtr<`[NYT::NConcurrency::TFiberManagerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NConcurrency_TFiberManagerConfig)`>`
**Default value**: `{}`
 ||
|| `resource_tracker` | **Type**: `NYT::TIntrusivePtr<`[NYT::NProfiling::TResourceTrackerConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NProfiling_TResourceTrackerConfig)`>`
**Default value**: `{}`
 ||
|| `tcp_dispatcher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NBus::NTcp::TDispatcherConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NBus_NTcp_TDispatcherConfig)`>`
**Default value**: `{}`
 ||
|| `rpc_dispatcher` | **Type**: `NYT::TIntrusivePtr<`[NYT::NRpc::TDispatcherConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NRpc_TDispatcherConfig)`>`
**Default value**: `{}`
 ||
|| `protobuf_interop` | **Type**: `NYT::TIntrusivePtr<`[NYT::NYson::TProtobufInteropConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NYson_TProtobufInteropConfig)`>`
**Default value**: `{}`
 ||
|| `clients_cache` | **Type**: `NYT::TIntrusivePtr<`[NYT::NClient::NCache::TClientsCacheConfig](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NClient_NCache_TClientsCacheConfig)`>`
**Default value**: `{}`
Connection settings of the YT clients cache the runner uses to talk to the pipeline cluster. They do not extend to the vanilla launcher, which builds its own clients. ||
|| `clients_cache_factory` | **Type**: `NYT::TIntrusivePtr<NYT::NYTree::INode>`
Parameters of the root YT clients cache factory installed via `SetRootClientsCacheFactory()`. Their schema is defined by the factory itself; the built-in factory ignores this block. ||
|#


{% endcut %}
<!-- endsource: en/flow/generated_docs/NYT_NFlow_TSimpleRunnerConfig.md -->

## See also

- [Autoscaler](https://ytsaurus.tech/docs/en/flow/concepts/autoscaler.md)
- [Computation](https://ytsaurus.tech/docs/en/flow/concepts/computation.md)
- [Watermarks and Timers](https://ytsaurus.tech/docs/en/flow/concepts/watermarks.md)
- [Connectors](https://ytsaurus.tech/docs/en/flow/connectors/about.md)
<!-- endsource: en/_includes/flow/concepts/spec.md -->

[*idle_partitions_max_ratio_default_value]: Why 0.4: if the source stops writing to one of the data centers, about a third of the partitions will become `idle`.

[*paramsClasses]: `Computation`, `Source`, `Sink`, `Resource`