Running C++ process functions in a companion

The C++ companion SDK hosts process functions in a separate process managed by a Flow worker. It is useful when your user code needs a separate executable and address space. Build the companion separately from the worker and the runner; the worker sends batches to it through the companion protocol.

This reference assumes you already have a C++ process function and know how to configure and deploy a pipeline. The maintained resource companion executable, its build file and pipeline spec show the registration and wiring together.

Register hosted functions and resources

Use NYT::NFlow::NCompanionServer::TPipeline from yt/yt/flow/library/cpp/companion/server/pipeline.h. Populate it before starting the server:

  • AddTransform<TFunction, TStaticParameters, TDynamicParameters>(computationId) declares a hosted transform computation and registers its process function and parameter schemas.
  • AddSource<TFunction, TStaticParameters, TDynamicParameters>(computationId) declares a hosted source computation. Both parameter types default to TEmptyProcessFunctionParameters.
  • AddResource<TResource>() registers a companion-hosted resource class. Resource ids are declared in the spec, not in this call.

The computation id must match the spec. The spec's processing_function still selects the registered function by its C++ type name. Typed function registration replaces YT_FLOW_DEFINE_PROCESS_FUNCTION for that declaration; AddResource replaces YT_FLOW_DEFINE_RESOURCE. Avoid duplicate registration through both mechanisms. Duplicate computation ids are rejected.

Pass the populated pipeline to NYT::NFlow::NCompanionServer::RunCompanionMain(argc, argv, std::move(pipeline)), declared in companion_main.h. The executable's ya.make must depend on yt/yt/flow/library/cpp/companion/server; add the dependencies required by your functions and resources. The maintained build file above contains the complete dependency list for its example.

Connect the executable to the worker

  1. Make the companion executable and its runtime dependencies available in the worker job environment. Use the existing deployment instructions for the worker; a path in the spec does not upload the executable by itself.
  2. Declare a NYT::NFlow::NCompanion::TCompanionManager resource. In its parameters.entrypoint, set executable to the executable's absolute path in that environment. args supplies command-line arguments and env supplies additional environment variables.
  3. For each hosted computation, select a companion computation class in computation_class_name, set processing_function, and require the manager on the worker under alias CompanionManager (worker = %true, controller = %false). Keep the computation id consistent with AddTransform or AddSource.
  4. When using the standard flow_server runner with function types registered only in the companion, set the runner configuration's abort_on_specs_parseability_error to %false. The runner cannot parse a function type absent from its own registry. The companion executable must still register the selected function and parameter schemas correctly.

The C++ companion test harness shows this separation: flow_server acts as the runner, while the manager entrypoint points to the C++ executable. Its integration test checks output that includes values loaded in the serving companion process.

Startup and execution boundaries

RunCompanionMain starts the companion server. It requires worker-supplied YT_FLOW_MODE=Worker and YT_FLOW_COMPANION_CONFIG containing a positive RPC port. Use the runner to launch the pipeline; the companion entry point does not implement the runner's pipeline-launch command.

The C++ companion uses one multithreaded process. Make shared objects safe for concurrent use.

The supported interfaces and their boundaries are described in C++ companion limitations. Select the hosting mode explicitly rather than assuming every native process-function facility is available out of process.

Companion-hosted resources and HTTP clients

Register hosted resource classes with AddResource<TResource>() and connect worker proxies using Companion resource wiring. In a process function's Init, IRuntimeInitContext::GetStaticResource(alias) returns the bound companion-side resource. The maintained mapper demonstrates this path. A resource that exists only in the worker is not transferred into the companion automatically.

IRuntimeInitContext::GetHttpClient() and GetHttpsClient() provide shared HTTP clients. Configure them through http_client_config, https_client_config and http_poller_threads in the worker's companion block; see TCompanionConfig.

Source and tests

Previous
Resources
Next
Testing