Run a pipeline in a Vanilla operation

This is the simplest way to run Flow: you don’t need a separate long-running deployment of controllers and workers — they start inside a single YTsaurus Vanilla operation. To enable this type of run, just add the vanilla block to pipeline.yson.

What you’ll need

You need a configuration file:

• pipeline.yson — runner config with the pipeline spec. For the pipeline to run in a Vanilla operation, it must include a vanilla block with enable = %true (see How to enable). You don’t need a separate config.yson — the node config inside the jobs is built automatically.

And binaries — their roles depend on the language:

How to enable

Add the vanilla block to your pipeline config:

"vanilla" = {
    "enable" = %true;
    "pool" = "<your-pool>";
    "worker" = {"count" = 5};
};

The required fields are pool and worker.count. By default, the controller has one job, and each controller and worker job gets cpu_limit = 6 and memory_limit = 18 GiB. Without a network project, the launcher asks YTsaurus to allocate job ports. With a network project, each job has its own IP and uses fixed ports unless its launcher requests job ports: rpc_port = 10080, monitoring_port = 10081, and companion.port = 10082. See Additional parameters for port allocation details.

If needed, you can override the resources explicitly:

"vanilla" = {
    "enable" = %true;
    "pool" = "<your-pool>";
    "controller" = {"count" = 1; "cpu_limit" = 2; "memory_limit" = "8g"};
    "worker" = {"count" = 5; "cpu_limit" = 8; "memory_limit" = "32g"};
};

For the full list of fields, see TVanillaConfig and TVanillaTaskConfig.

When you run it, flow_server validates the spec itself, creates a Vanilla operation with two tasks (controller and worker), sets the pipeline spec, and starts it.

Additional parameters

The less frequently used fields of the vanilla block:

And the task fields (controller/worker):

On a shared-network host, fixed ports of neighboring jobs can collide. The launcher therefore sets port_count = 2 for the controller and port_count = 3 for the worker when no network project is configured. An explicit 0 selects fixed ports again unless a companion launcher raises worker.port_count. Allocated ports take priority over the node config: YT_PORT_0 is rpc_port and bus_server.port, YT_PORT_1 is monitoring_port, and YT_PORT_2 is companion.port. A controller or a worker without a companion needs two ports; a worker with a companion needs three. With fewer ports, some remain fixed and can still collide.

The Go and Python companion launchers raise worker.port_count to at least 3 when Vanilla is enabled, including when the value is omitted or explicitly set to 0. Their worker ports are allocated even with a network project; the controller follows the rules above.

Run the pipeline

./pipeline --config pipeline.yson

./pipeline --config pipeline.yson --flow-bin flow_server

./run.sh com.example.pipeline.PipelineMain --config pipeline.yson --flow-bin flow_server

./pipeline --config pipeline.yson --flow-bin flow_server

After the start, the runner by default (YT_FLOW_WAIT=1) waits until the pipeline reaches the completed state, printing new records of the controller’s public log all the while — those that appeared since the wait began; earlier ones are not shown. You can interrupt the runner; that does not affect the running operation. With YT_FLOW_WAIT=0, the runner exits right after the launch.

Validate the spec only

The C++ and Java runners support --validate-only: they parse and validate the static and dynamic specs locally without sending them to the controller. An invalid spec makes the runner fail; a valid one exits successfully. The Python and Go launchers do not forward this flag to flow_server and may start or update the pipeline instead. The command below uses the C++ runner.

./pipeline --config pipeline.yson --validate-only

See also

• Basic rollout rules
• Basic pipeline operations
• Updates and releases
• Security and access
• Logs
• Spec and DynamicSpec