YQL over YTsaurus Flow pragma reference

Syntax

YQL over Flow pragma names start with Ytflow.

An exception is Engine, common pragma for selecting query execution engine:

PRAGMA Ytflow.<pragma_name> = '<value>';
PRAGMA Engine = 'ytflow';

Engine

Selects Flow for query execution.

You must specify this pragma to run query through Flow.

PRAGMA Engine = 'ytflow';

Ytflow.Cluster

YTsaurus cluster name for internal pipeline tables.

PRAGMA Ytflow.Cluster = '<cluster-name>';

Ytflow.RuntimeCluster

Cluster name for running pipeline's vanilla operation.

Controllers and workers run as two task types (controller and worker) within a single operation; job counts and resources are configured separately for each task type.

PRAGMA Ytflow.RuntimeCluster = '<cluster-name>';

Ytflow.PipelineDirectory

Directory containing pipelines in Cypress on Ytflow.Cluster, used together with Ytflow.PipelineName.

Alternatively, set Ytflow.PipelinePath alone.

Example Cypress paths start with // and have no trailing /.

PRAGMA Ytflow.PipelineDirectory = '//home/my-project/pipelines';

Ytflow.PipelineName

Pipeline name, used to create a subdirectory within Ytflow.PipelineDirectory,
where internal pipeline tables are stored.

Alternatively, explicitly set Ytflow.PipelinePath.

PRAGMA Ytflow.PipelineName = 'my-pipeline';

Ytflow.PipelinePath

Full pipeline path in Cypress on Ytflow.Cluster;
this directory stores internal pipeline tables.

Alternatively, set Ytflow.PipelineDirectory and Ytflow.PipelineName separately.

If Ytflow.PipelinePath is specified, it takes precedence over those two pragmas.

PRAGMA Ytflow.PipelinePath = '//home/my-project/pipelines/my-pipeline';

Ytflow.Auth

Name of a YQL secret in category ytflow containing a token for authentication in YTsaurus.

This token must grant access to pipeline tables on Ytflow.Cluster
and permission to start vanilla operation on Ytflow.RuntimeCluster.

For more information about authentication, see Authentication.

PRAGMA Ytflow.Auth = 'my_ytflow_credentials';

Ytflow.WorkerCount

Number of worker jobs in vanilla operation on Ytflow.RuntimeCluster.

PRAGMA Ytflow.WorkerCount = '5';

Ytflow.WorkerCpuLimit

Guaranteed CPU allocation per worker job in vanilla operation on Ytflow.RuntimeCluster.

Actual CPU consumption may be higher if spare resources are available on a job's host.

Unit: one CPU core.

PRAGMA Ytflow.WorkerCpuLimit = '10.0';

Ytflow.WorkerMemoryLimit

Memory limit per worker job in vanilla operation on Ytflow.RuntimeCluster.

PRAGMA Ytflow.WorkerMemoryLimit = '20G';

Ytflow.EnableComputationPatternResources

Enables reuse of computation patterns across graphs in one worker.

PRAGMA Ytflow.EnableComputationPatternResources = 'true';

Ytflow.ControllerCount

Number of controller jobs in vanilla operation on Ytflow.RuntimeCluster.

One controller becomes leader; others remain on standby.

With a value of 1, there is no standby controller, and pipeline management is temporarily unavailable while this sole controller restarts.

PRAGMA Ytflow.ControllerCount = '3';

Ytflow.ControllerCpuLimit

Guaranteed CPU allocation per controller job in vanilla operation on Ytflow.RuntimeCluster.

Actual CPU consumption may be higher if spare resources are available on job's host.

Unit: one CPU core.

PRAGMA Ytflow.ControllerCpuLimit = '3.0';

Ytflow.ControllerMemoryLimit

Memory limit per controller job in vanilla operation on Ytflow.RuntimeCluster.

PRAGMA Ytflow.ControllerMemoryLimit = '5G';

Ytflow.GracefulUpdate

Pipeline update mode:

• true — gracefully stop and finish processing internal streams (stop)
• false — pause immediately (pause).

Once pipeline reaches its target state, its previous vanilla operation on Ytflow.RuntimeCluster is aborted and new version starts.

For restrictions on updating without draining, see Hotfix; for stop commands, see CLI reference.

PRAGMA Ytflow.GracefulUpdate = 'true';

Ytflow.UpdateTimeout

Timeout for waiting for previous pipeline version to stop or pause during an update.

If pipeline does not reach its target state within this time, query execution fails. This update does not stop its previous vanilla operation on Ytflow.RuntimeCluster or start new version.

Any stop or pause command already sent is not canceled.

PRAGMA Ytflow.UpdateTimeout = '600s';

Ytflow.TabletCellBundle

Tablet cell bundle (tablet_cell_bundle) for newly created tables.

PRAGMA Ytflow.TabletCellBundle = 'my-bundle';

Ytflow.Account

Account for newly created tables.

PRAGMA Ytflow.Account = 'my-account';

Ytflow.PrimaryMedium

Primary medium (primary_medium) for newly created tables.

PRAGMA Ytflow.PrimaryMedium = 'default';

Ytflow.Pool

Compute pool for pipeline's vanilla operation on Ytflow.RuntimeCluster.

PRAGMA Ytflow.Pool = 'my-pool';

Ytflow.YtPartitionCount

Number of partitions in newly created output queues in YTsaurus.

PRAGMA Ytflow.YtPartitionCount = '4';

Ytflow.YtTtl

Retention period for rows in newly created output ordered dynamic tables (queues) in YTsaurus.

Sets max_data_ttl and also sets:

• min_data_versions = 0
• max_data_versions = 1
• min_data_ttl = 0
• auto_compaction_period to half its TTL

For more information, see Automatic removal of old rows.

PRAGMA Ytflow.YtTtl = '7d';

Ytflow.YtConsumerVital

vital flag used when registering consumer with input queues in YTsaurus.

When automatic consumer-based queue trimming is enabled, queues respect vital consumer offsets and do not delete rows these consumers have not read yet.

Non-vital consumer offsets do not prevent deletion of rows these consumers have not processed.

PRAGMA Ytflow.YtConsumerVital = 'true';

Ytflow.YtConsumerDirectory

YTsaurus consumer directory.

PRAGMA Ytflow.YtConsumerDirectory = '//home/my-project/consumers';

Ytflow.YtConsumerName

YTsaurus consumer name.

PRAGMA Ytflow.YtConsumerName = 'my-consumer';

Ytflow.YtProducerDirectory

YTsaurus producer directory.

PRAGMA Ytflow.YtProducerDirectory = '//home/my-project/producers';

Ytflow.YtProducerName

YTsaurus producer name.

PRAGMA Ytflow.YtProducerName = 'my-producer';