FAQ for YTsaurus Flow

Q: Can you update the pipeline binary files without stopping or pausing?

A: The short answer is no. There are many compatibility nuances between the old and new binaries at the user code level, and the Flow team isn’t ready to take responsibility for saying that this is safe.

When you roll out a new binary with a stop but without drain:

Intermediate messages from the old binary may remain on YTsaurus, and the new binary might not be ready to handle them (for example, if some invariant has changed).
You might get duplicate events or lose events if the logic for generating output messages in Swift computations changes.
The diff between the two binaries might include commits from Flow itself. These changes might be unsafe to roll out without draining.

You can still update binary files without draining in these cases:

In a dev environment that you don’t mind completely breaking. If it breaks (or you notice anything odd), you can just redeploy it from scratch.

The Flow team reserves the right not to help resolve any pipeline issues that stem from such unsafe operations in the pipeline’s history.

Q: My pipeline is in the Completed state—how did this happen?

A: The most common situation is launching a new pipeline before setting the spec. A less common situation is when the pipeline is launched with all finite sources (finite=true in the spec), as in tests, and the sources have been processed.

In both cases, you can take the pipeline out of the Completed state only by fully recreating it. That means you need to stop the controllers and workers, delete the pipeline object on YTsaurus, and then create and launch everything again.

Q: How do you run Flow in an IPv4-only environment?

A: By default, Flow uses IPv6 for inter-node communication (Controller ↔ Worker, Worker ↔ Worker). To run in an IPv4-only environment, you need to configure the address_resolver section:

{
    cluster_url = "your-cluster";
    path = "//home/your-pipeline";

    address_resolver = {
        enable_ipv4 = %true;
        enable_ipv6 = %false;
    };

    # ... other config parameters
}

Warning

Exactly one protocol must be enabled: either enable_ipv4 = %true or enable_ipv6 = %true. Enabling both or disabling both will cause an error when the node starts.

Important details

The address_resolver setting is a global singleton and affects all DNS resolutions in the process, including:

Resolving addresses of other Flow nodes (Controller, Worker).
Resolving addresses of the YTsaurus RPC proxy.
Any other DNS queries within the process.

This means that with enable_ipv6 = %false, the DNS resolver will request only A records (IPv4). Make sure that all hosts the pipeline interacts with (including the YTsaurus RPC proxy) have A records in DNS.

Example tasks

Configuration reference