YT pragmas

YT pragmas are a namespace for pragmas that configure YTsaurus‑specific parameters for YQL queries.

Syntax

YT pragma names include the yt prefix:

PRAGMA yt.<pragma_name> = '<value>';

Scope and features

By scope, you can divide YT pragmas into static and dynamic ones.

Static pragmas:

• They are initialized once at the earliest stage of query processing.
• They apply to all expressions in the current module where they are declared.
• If you specify a static pragma multiple times in a query, only its last set value will be applied.

Dynamic pragmas:

• They are initialized at the query execution stage, after optimization and execution plan creation.
• They are valid until the next identical pragma or until the end of the query.
• Only for a dynamic pragma can you reset its value to the default by assigning default.

Per‑cluster support

Some pragmas support a special operating mode — per‑cluster. This mode lets you set different pragma values for different clusters in a single query. For example, you can specify: “Run the query on cluster A with settings X, and on cluster B with settings Y.” This is useful for distributed queries where different clusters require different execution conditions.

The per‑cluster mode is available for all dynamic pragmas and for some static ones — in the documentation below, such pragmas are explicitly marked with the “per‑cluster” label.

How to use

By default, YT pragmas are written with the yt prefix — this means the setting will apply to the current cluster where the query is running. To apply the same pragma to another cluster with a different value, replace yt with the name of that cluster. For example, let’s set different temporary directories for two clusters in a query:

PRAGMA yt.TmpFolder = "//tmp/my_folder";            -- current cluster
PRAGMA cluster_2.TmpFolder = "//tmp/other_folder";  -- cluster cluster_2

... -- query body

As a result, when the query runs, temporary files will be saved to //tmp/my_folder on the current cluster and to //tmp/other_folder on the cluster_2 cluster.

yt.Annotations

This lets you set arbitrary structured information related to the operation. It’s useful for searching and identifying operations in the archive (you can search it via API). For more details, see the Operation settings section.

Signature

PRAGMA yt.Annotations = '{
    "name" = "login";
    "time" = "20.02.2002";
}';

Result

When you search for an operation in the archive, the information set in the Map will be visible.

yt.Auth

Use authentication data other than the default ones.

yt.AutoMerge / yt.TemporaryAutoMerge / yt.PublishedAutoMerge

Control the одноименной setting of YTsaurus that helps reduce quota consumption for the number of chunks. yt.TemporaryAutoMerge applies to all YT operations, except for merge inside a YtPublish node.

yt.PublishedAutoMerge applies only to merge inside a YtPublish node (if it runs there). yt.AutoMerge sets this setting’s value simultaneously for all YTsaurus operations in the query.

yt.BatchListFolderConcurrency

Set the number of concurrent directory listing operations.

yt.BinaryExpirationInterval

Let you manage the TTL of cached binary artifacts. It works only together with yt.BinaryTmpFolder. Each use of a binary artifact in a query extends its TTL.

yt.BinaryTmpFolder

Set a separate path on the cluster where the query’s binary artifacts (UDF and job binary) will be cached. Artifacts are saved in the root of the directory with a name equal to the artifact’s md5. Saving and using artifacts in this directory happens outside the transaction, even if the query includes the yt.ExternalTx pragma.

yt.BufferRowCount

Limit the number of records that JobProxy can buffer.

yt.ColumnGroupMode

Set the mode for computing column groups for the query’s intermediate tables. In disable mode, column groups aren’t used. In single mode, one group is created for all columns in the table. In perusage mode, granular column groups are created based on their consumers. All columns in one group are used simultaneously by one or more consumers. For example, if an intermediate table has columns [a, b, c, d, e, f] and two operations use it with column selections [a, b, c, d] and [c, d, e, f] respectively, the table will have three column groups: [a, b], [c, d], and [e, f]. If the intermediate table is used for publishing to an output table (i.e., the consumer is a YtPublish node), column groups aren’t applied, except when you explicitly set the column_groups modifier. In the latter case, the intermediate table uses the modifier’s column groups.

yt.CombineCoreLimit

Set the memory buffer size for running a CombineCore node.

yt.CommonJoinCoreLimit

Set the memory buffer size for running a CommonJoinCore node (it runs in a job when you choose the common JOIN strategy).

yt.ConvertDynamicTablesToStatic

Add preliminary conversion of dynamic tables to static tables. In join mode, only tables that are inputs to joins are converted. This lets you use the map join strategy on such tables. In all mode, all dynamic tables in the query are converted to static tables. In disable mode, no conversion happens.

yt.CostBasedOptimizerPartial

Controls when the cost-based optimizer starts for queries with multiple JOIN inputs. Use this pragma to avoid waiting for every input to be materialized when preserving computation parallelism is more important than fully reordering the joins.

The value specifies how many JOIN inputs must be ready and materialized before reordering:

• 0: Wait for all inputs and perform full reordering. This gives the optimizer the most information, but reduces parallelism and may prevent operation fusion.
• 1: Do not wait; optimize only ready subtrees. This preserves parallelism, but may reorder a smaller part of the plan.
• N > 1: Experimental mode that starts optimization when at least N inputs are ready.

Example

PRAGMA CostBasedOptimizer = 'native';
PRAGMA yt.CostBasedOptimizerPartial = '1';
PRAGMA yt.ExtendedStatsMaxChunkCount = '10000';

Result

The optimizer reorders ready JOIN subtrees without waiting for the remaining inputs. Extended statistics are requested only when the total number of table chunks on a cluster does not exceed 10,000.

Features and limitations

• The pragma applies only when CostBasedOptimizer is enabled.
• Values greater than 1 enable an experimental mode: the threshold semantics and query planning behavior may change in future YQL versions. For long-lived queries, use 0 or 1.
• Partial optimization preserves more parallelism, but may produce a less efficient JOIN order than full reordering after every input is ready.

yt.CoreDumpPath

Let you save the coredump from failed MapReduce operation jobs to a separate table.

yt.DataSizePerJob / yt.DataSizePerMapJob

Control how MapReduce operations are split into jobs; the larger the number, the fewer the jobs. For computationally expensive jobs, it’s recommended to decrease the value, and for jobs that quickly scan lots of data (in particular, user_sessions) — to increase it.

You can use K, M, and G suffixes to specify values in kilobytes, megabytes, and gigabytes, respectively.

yt.DataSizePerPartition

Control the size of partitions in MapReduce operations.

You can use K, M, and G suffixes to specify values in kilobytes, megabytes, and gigabytes, respectively.

yt.DataSizePerSortJob

Control how sort jobs are split in MapReduce operations.

You can use K, M, and G suffixes to specify values in kilobytes, megabytes, and gigabytes, respectively.

yt.DefaultCalcMemoryLimit

This sets the memory limit for computations that aren’t related to table access.

You can use the K, M, and G suffixes to specify values in kilobytes, megabytes, and gigabytes, respectively.

yt.DefaultLocalityTimeout

This sets the locality_timeout setting in the operation spec (this setting isn’t documented yet).

yt.DefaultMapSelectivityFactor

This sets the approximate ratio of output to input for the map stage in a combined MapReduce operation. See the documentation.

yt.DefaultMaxJobFails

This is the number of failed MapReduce jobs that, once reached, stops further retry attempts for the request. The request is then considered unsuccessful.

yt.DefaultMemoryDigestLowerBound

This sets the user_job_memory_digest_lower_bound setting in the operation spec. You can read about this setting in the documentation.

yt.DefaultMemoryLimit

This sets the memory limit, in bytes, that jobs consume. The limit is requested when you start MapReduce operations.

You can use the K, M, and G suffixes to specify values in kilobytes, megabytes, and gigabytes, respectively.

yt.DefaultMemoryReserveFactor

This sets the memory reservation factor for jobs. See the documentation.

yt.DefaultOperationWeight

This sets the weight of all MapReduce operations you launch within the selected compute pool.

yt.Description

This sets the information that’s displayed in the web interface on the operation page. See the documentation.

yt.DisableJobSplitting

Use this to prevent the YTsaurus Scheduler from adaptively splitting long-running user jobs further.

yt.DisableOptimizers

This disables the specified optimizers.

yt.DockerImage

You can specify a Docker image to create the environment where user jobs run.

yt.DontForceTransformForInputTables

This disables the forced data transformation for user tables with storage settings (erasure_codec, compression_codec, primary_medium, media, column groups) that differ from the default settings.

Forced transformation is applied to input tables if they’re used to write to output tables only via the YtMerge operation.

yt.ErasureCodecCpu

This is the multiplier for estimating CPU consumption when processing tables that are compressed with an erasure codec. It affects how MapReduce operations are split into jobs.

yt.EvaluationTableSizeLimit

This sets the maximum total size of tables used at the evaluation stage.

yt.ExpirationDeadline / yt.ExpirationInterval

This lets you manage the TTL of tables created by the operation.

yt.ExtendedStatsMaxChunkCount

Limits collection of extended column statistics for the cost-based optimizer. Use this pragma to prevent statistics collection for tables with many chunks from delaying query optimization.

The value specifies the maximum total number of input-table chunks on one cluster:

• At or below the threshold, YQL requests extended statistics.
• Above the threshold, statistics are not requested and the optimizer uses less accurate estimates.
• 0 removes the chunk-count limit.
• If the pragma is not set, extended statistics are not requested.

Example

PRAGMA CostBasedOptimizer = 'native';
PRAGMA yt.ExtendedStatsMaxChunkCount = '10000';

Result

The optimizer receives extended column statistics when the input tables have no more than 10,000 chunks in total on a cluster. For larger input sets, optimization continues without these statistics.

Features and limitations

• The pragma affects only statistics collection and does not restrict reading the tables themselves.
• A value of 0 may make statistics collection slow for very large tables.
• The recommended initial value is 10000; adjust it according to the number of input chunks and acceptable optimization time.

yt.ExternalTx

Use this to specify an external transaction in the context of which the request will run. For example, you might use this if the transaction was created via the CLI. This can be useful to perform several actions atomically, such as two YQL requests or a YQL request and a table change.

This pragma also lets you read intermediate data in transactions, for example, in a long operation that has temporary files.

Example

PRAGMA yt.ExternalTx = 'a3d149bd-674dfa1-fa68f314-2c22562';   -- Transaction ID (you can view it via the CLI)

Result

The request will run within the specified transaction.

Features and limitations

Specifying the pragma without a request body will cause an error.

yt.ExtraTmpfsSize

This lets you increase the tmpfs size in addition to the total size of all explicitly used files (specify the value in megabytes). This can be useful if you create new files locally from a UDF. This is ignored without UseTmpfs.

yt.FileCacheTtl

This lets you manage the TTL of the YTsaurus file cache. A value of 0 disables the use of TTL for the file cache.

yt.FolderInlineDataLimit

This sets the maximum data size for an inline list that’s the result of a Folder computation. If the size is larger, a temporary file is used.

yt.FolderInlineItemsLimit

This sets the maximum number of items in an inline list that’s the result of a Folder computation. If the number is larger, a temporary file is used.

yt.ForceJobSizeAdjuster

This sets the "force_job_size_adjuster" option in the operation settings.

yt.HybridDqExecution

This enables hybrid query execution via DQ.

yt.IgnoreTypeV3

When you read tables with a type_v3 schema, all fields with complex types appear as Yson fields in the query. Complex types include all non-data types and data types with more than one level of optionality.

yt.IgnoreWeakSchema

This tells the system to ignore the table’s weak schema (which is generated by sorting a non-schema table by a set of fields).

Used together with yt.InferSchema, this lets you infer the schema from the data for such tables.

yt.IgnoreYamrDsv

This tells the system to ignore _format=yamred_dsv if it’s specified in the input table’s metadata.

yt.InferSchema / yt.ForceInferSchema

This infers the data schema from the content of the table’s first rows. If you specify the PRAGMA without a value, the system assumes one first row. If you specify multiple rows and the column has different data types, the types expand up to Yson.

InferSchema only infers the data schema for tables that don’t have a schema specified in the metadata. ForceInferSchema ignores the data schema from the metadata, except for the list of key columns for sorted tables.

In addition to the detected columns, the system generates an _other dictionary column (string per row) with values from columns that weren’t present in the first row but were found later. This lets you use WeakField on such tables.

Because of the wide range of possible issues, this mode isn’t recommended and is disabled by default.

yt.InferSchemaTableCountThreshold

If the number of tables for which the schema is inferred from the content exceeds this value, schema inference runs as a separate operation on YTsaurus, which can be much faster.

yt.IntermediateAccount

This lets you use your own account for intermediate data within a fused MapReduce operation.

By default, the system uses a shared account, which might fill up at an inconvenient time.

If you specify the PRAGMA yt.TmpFolder, the system uses the account specified on the temporary directory instead of the shared one by default.

yt.IntermediateDataMedium

This sets the medium used for intermediate data in operations (Sort, MapReduce). For details, see the documentation.

yt.IntermediateReplicationFactor

This sets the replication factor for intermediate data.

yt.JobEnv

This sets the environment variables for the map and reduce jobs in the operation. The keys in the map set the environment variable names, and the values in the map set the environment variable values.

yt.JoinAllowColumnRenames

This enables column renaming when you use the Ordered JOIN strategy (it uses the rename_columns attribute). If you disable this option, the Ordered JOIN strategy is only selected when the column names on the left and right sides match.

yt.JoinCollectColumnarStatistics

This controls the use of columnar statistics to accurately estimate JOIN inputs and choose the appropriate strategy. Async enables asynchronous collection of columnar statistics.

yt.JoinColumnarStatisticsFetcherMode

This controls the mode for requesting columnar statistics from YTsaurus to accurately estimate JOIN inputs. The from_nodes mode gives an accurate estimate but might miss timeouts for large tables. The from_master mode is very fast but gives coarse statistics. The fallback mode works as a combination of the previous two.

yt.JoinMergeForce

This forces the selection of the Ordered JOIN strategy. If you set the flag to True, the Ordered JOIN strategy is selected even if one or both JOIN sides aren’t sorted. In this case, the unsorted sides are sorted beforehand. The limits on the maximum size of an unsorted table (see yt.JoinMergeUnsortedFactor) are ignored in this case.

yt.JoinMergeReduceJobMaxSize

This sets the maximum allowed size of the Reduce job when a small table is selected as the primary one in the Ordered JOIN strategy. If the resulting size exceeds this value, the Reduce operation repeats with a larger table as the primary one.

yt.JoinMergeTablesLimit

This sets the total allowed number of tables on the left and right sides to enable the Ordered JOIN strategy.

You can completely disable this strategy by setting the value to 0.

yt.JoinMergeUnsortedFactor

This sets the minimum ratio of the unsorted JOIN side’s size to the sorted side’s size for its additional sorting and the selection of the Ordered JOIN strategy.

yt.JoinMergeUseSmallAsPrimary

This explicitly controls the selection of the primary table in the Reduce operation for the Ordered JOIN strategy. If you set it to true, the smaller side is always selected as the primary table. If the flag’s value is false, the larger side is selected, except when the larger side has unique keys. Selecting the larger table as the primary one is safe even if it has monster keys, but it’s slower. If you don’t set this pragma, the primary table is selected automatically based on the maximum size of the resulting jobs (see yt.JoinMergeReduceJobMaxSize).

yt.LayerPaths

This lets you specify the sequence of porto layers to form the environment in which user jobs will run.

yt.LLVMMemSize

This sets the fixed memory size required for compiling LLVM code in jobs.

yt.LLVMPerNodeMemSize

This sets the memory size per computation graph node required for compiling LLVM code in jobs.

yt.LookupJoinLimit

A table can be used as a map in the Lookup JOIN strategy if its size doesn’t exceed the minimum of yt.LookupJoinLimit and yt.EvaluationTableSizeLimit.

yt.LookupJoinMaxRows

This sets the maximum number of rows in a table that can act as a map in the Lookup JOIN strategy.

yt.MapJoinLimit

This limits the size of the smaller table in a JOIN, which triggers the Map-side strategy (creating an in-memory map from the smaller table and using it in the Map over the larger table).

You can completely disable this strategy by setting the value to 0.

yt.MapJoinShardCount

The Map-side JOIN strategy can run in a sharded mode: the smaller side splits into N shards (where N is less than or equal to this PRAGMA’s value), each of which independently and in parallel joins with the larger side. The JOIN result is then the concatenation of the JOINs with the shards.

yt.MapJoinShardMinRows

This sets the minimum number of records per shard in the Map-side JOIN strategy.

yt.MapLocalityTimeout

This sets the map_locality_timeout setting in the operation specification (the setting is not yet documented).

yt.MaxColumnGroups

This sets the maximum number of column groups for the intermediate request table. If the calculated number of groups exceeds this limit, no groups are created for this table.

yt.MaxExtraJobMemoryToFuseOperations

This is the maximum amount of memory consumption by jobs allowed after operations are fused by optimizers.

yt.MaxInputTables

This limits the number of tables provided as input to each specific MapReduce operation.

yt.MaxInputTablesForSortedMerge

This limits the number of tables provided as input to a sorted merge operation.

yt.MaxJobCount

This is the maximum number of jobs within a single YTsaurus operation. It applies only to single-stage map, reduce, merge, and similar operations. If you specify both yt.DataSizePerJob and yt.MaxJobCount, job slicing will take yt.DataSizePerJob into account. Even if the resulting value N exceeds yt.MaxJobCount, N jobs will be launched. yt.MaxJobCount will only affect whether jobs are split after their number reaches a certain threshold.

yt.MaxKeyWeight

Increase the limit on the maximum length of table keys in YTsaurus by which the table is sorted.

yt.MaxOutputTables

This limits the number of output tables for each specific MapReduce operation.

yt.MaxReplicationFactorToFuseOperations

This is the maximum data replication factor allowed after operations are fused by optimizers.

yt.MaxRowWeight

Increase the limit on the maximum length of a table row in yt.

yt.MaxSpeculativeJobCountPerTask

This sets the number of speculative jobs in YTsaurus operations. By default, the YTsaurus cluster settings are used.

yt.MinColumnGroupSize

This sets the minimum size of a column group. If the calculated group contains fewer columns than the pragma value specifies, the group is not created.

yt.MinLocalityInputDataWeight

This sets the min_locality_input_data_weight setting in the operation specification (the setting is not yet documented).

yt.MinPublishedAvgChunkSize

If the average chunk size in the resulting output table is smaller than the specified setting, an additional YTsaurus Merge operation starts to enlarge the chunks to the specified size. A value of 0 has a special meaning — in this case, the merge always starts and enlarges the chunks to 1G.

If a compression codec is used for the table, the output chunk size may differ from the specified value by the compression ratio. Essentially, this pragma sets the data size per merge job. After compression, the output size may be significantly smaller. In this case, you should increase the pragma value by the expected compression ratio.

yt.MinTempAvgChunkSize

This setting is similar to yt.MinPublishedAvgChunkSize, but it works for intermediate temporary tables.

yt.NetworkProject

This sets the use of the specified network project in jobs for regular operations in the request.

yt.NightlyCompress

This controls the process of background table compression so that tables take up less space.

A true value sets the table attribute @force_nightly_compress to true.
A false value sets the table attribute @nightly_compression_settings with the child value enabled to false.

This setting applies only to tables newly created by a YQL request (and to tables overwritten using INSERT INTO ... WITH TRUNCATE).
This setting does not apply to temporary tables.

yt.OmitInaccessibleRows

This controls the behavior when reading tables with row-level ACL (RLS).

By default, reading a table with row-level ACL set results in an authorization error if the user doesn’t have the full_read permission. The yt.OmitInaccessibleRows pragma changes this behavior: when enabled, rows without access are skipped, and the query completes successfully. Only rows allowed by the RLS predicate are included in the result.

Example

PRAGMA yt.OmitInaccessibleRows = "true";

SELECT *
FROM `//path/to/table_with_rls`;

Result

The query will return only the rows accessible to the current user according to the row-level ACL. Rows without access will be skipped without an error.

Restrictions

• You can’t specify row_index in ranges when reading a table with row-level ACL — the query will result in an error. Row indexes in ranges are counted relative to the physical rows on disk, not the rows accessible to the user. For example, //path/to/table[:#100] will return up to 100 rows from the disk, some of which may be inaccessible and will be filtered out.
• RLS is not supported for dynamic tables — any read will return an error.

yt.OperationReaders

This lets you grant read access to created MapReduce operations in YTsaurus to other users besides the YQL operation owner.

yt.OperationSpec

This sets a map of operation settings. It lets you specify settings that don’t have pragma equivalents. Settings defined via specialized pragmas have higher priority and override values in this map.

yt.OptimizeFor

This controls the optimize_for attribute on created tables.

yt.Owners

This lets you grant access to manage MapReduce operations in YTsaurus (cancel, pause, run-job-shell, etc.) to other users besides the user who launched the request.

Example

PRAGMA yt.Owners = 'ivanov petrov';    -- user logins separated by a space

Result

The specified users will be able to manage MapReduce operations.

yt.ParallelOperationsLimit

This sets the maximum number of YTsaurus operations that can run simultaneously within a request.

yt.Pool

This is used to override the compute pool, which by default is the current user’s login or was set by the yt.StaticPool pragma.

Signature

PRAGMA yt.StaticPool = '<pool_1>';
PRAGMA yt.Pool = '<pool_2>';

Result

After specifying the yt.StaticPool pragma, the request will run in pool <pool_1>, and after the yt.Pool pragma — in <pool_2>.

yt.PoolTrees

You can choose pool trees that differ from the default one.

yt.PrimaryMedium

This sets the primary medium in YTsaurus for Published and Temporary tables, as well as for intermediate data in operations. It’s the same as setting the yt.IntermediateDataMedium, yt.PublishedPrimaryMedium, and yt.TemporaryPrimaryMedium pragmas at the same time.

yt.PublishedCompressionCodec / yt.TemporaryCompressionCodec

These settings configure the compression for tables created via YQL.

Published tables are those specified in INSERT INTO; all others are considered Temporary. The codec specified as Temporary is also used for intermediate data within a single YTsaurus operation, for example, a fused MapReduce.

yt.PublishedErasureCodec / yt.TemporaryErasureCodec

Erasure coding is disabled by default. To enable it, we recommend using the lrc_12_2_2 value.

The difference between Published and Temporary is the same as for CompressionCodec.

yt.PublishedMedia / yt.TemporaryMedia

This sets the @media attribute on newly created tables. If present, it specifies which media in YTsaurus will store the table’s chunks.

Published tables are those specified in INSERT INTO; all others are considered Temporary.

yt.PublishedPrimaryMedium / yt.TemporaryPrimaryMedium

This sets the @primary_medium attribute on newly created tables. If present, it specifies the primary medium in YTsaurus where the chunks will be written. By default, YTsaurus sets the primary medium to "default".

Published tables are those specified in INSERT INTO; all others are considered Temporary.

yt.PublishedReplicationFactor / yt.TemporaryReplicationFactor

This sets the replication factor for tables created via YQL.

Published tables are those specified in INSERT INTO; all others are considered Temporary.

yt.PythonCpu

This is a multiplier for estimating CPU consumption for Python UDF. It affects how MapReduce operations are split into jobs.

yt.QueryCacheChunkLimit

Use this pragma to control how tables are written to the cache, depending on the number of chunks in the table: with the concatenate command (as-is) or with the merge command (with merging).

Example

PRAGMA yt.QueryCacheChunkLimit = '100000';

Result

If the number of chunks in the table is less than the set limit, the table is written to the cache with the concatenate command, as-is.

If the number of chunks exceeds the set limit, the table is written to the cache with the merge command (with chunk merging).

So, with the default value of 0, tables are written to the cache using the merge command.

Features and limitations

The concatenate command — writing as-is — is faster than merge, which needs time to merge the chunks and then write them. It’s more efficient to write tables with a relatively small number of chunks to the cache using concatenate.

yt.QueryCacheIgnoreTableRevision

Use this pragma to avoid clearing the cache when table data changes. This speeds up debugging complex queries on large, changing tables when the query logic doesn’t depend on data changes.

By default, when calculating the hash from table metadata, the revision number — the YTsaurus revision, a non-negative integer that increases with each table change — is included along with other data. For more details about the revision attribute, see the Metadata Tree section.

With this pragma, you can control cache clearing: include or exclude the revision number when calculating the hash from table metadata.

Example

PRAGMA yt.QueryCacheIgnoreTableRevision = 'true';

Result

If the flag is set to true, the YTsaurus revision number is excluded when calculating the hash from table metadata, and the Query Cache isn’t cleared when the content of input tables changes.

Features and limitations

yt.QueryCacheMode

Use the yt.QueryCacheMode pragma to control caching of MapReduce operation results in YTsaurus.

The cache stores results of previous operations: if the same operation was already run in a previous query, it won’t be started again. Instead, YTsaurus will take the ready result from the cache. This speeds up queries that include calculations identical to those in other queries.

Query Cache is also useful when you’re debugging or making relatively small changes to a query: in this case, the result of a large part of the operations is taken from the cache.

Example

PRAGMA yt.QueryCacheMode = 'disable';

Result

Depending on the selected mode, YTsaurus will use, ignore, or update the cache:

• disable — the cache is disabled. YTsaurus doesn’t check the cache and doesn’t write results to it. This mode is suitable for production where data changes often: it saves resources and avoids unnecessary cache read and write operations;
• readonly — read-only mode. YTsaurus takes the result from the cache if it exists, but doesn’t save new results to it. This is suitable for testing or debugging: you can use old results and avoid cluttering the cache with new entries;
• refresh — write-only mode. YTsaurus saves results to the cache but doesn’t read from it. This is used to update the cache: run a query without reading from the cache and save its result for future use;
• normal — the default mode. YTsaurus both reads from and writes to the cache. This is especially useful during development and debugging when you run the same query multiple times to speed up iterations.

In normal and refresh modes, the operation result is also saved to the path //<tmp_folder>/query_cache/<hash>, where:

• tmp_folder — the temporary directory. By default, it’s tmp/<login>; you can set it using the yt.TmpFolder pragma;
• hash — the hash of the significant metadata and data of the input tables and the logical program that was run in the operation.

yt.QueryCacheTtl

Use this pragma to set the time that operation-created tables are stored in the query cache — TTL (Time to Live) — in the directory <tmp_folder>/query_cache/<hash>, where tmp_folder is the temporary directory. By default, it’s tmp/<login>; you can set it using the yt.TmpFolder pragma.

Example

PRAGMA yt.QueryCacheTtl = '3h';

Result

If you explicitly set a value in the pragma, the Query Cache will be cleared after the specified interval. The interval is counted from the moment the table is created in the query cache or from the moment the table was last used (see the yt.QueryCacheUseExpirationTimeout pragma).

If you don’t set an interval, the Query Cache will be cleared automatically after 7 days (by default).

yt.QueryCacheUseExpirationTimeout

This pragma defines the mode for counting the TTL interval for tables in the query cache.

Example

PRAGMA yt.QueryCacheUseExpirationTimeout = 'true';

Result

With the default value of false, the TTL is counted from the moment the table is created in the query cache.

If you set the value to true, the TTL is counted from the moment the table was last used.

Features and limitations

yt.ReduceLocalityTimeout

This sets the reduce_locality_timeout setting in the operation specification (the setting isn’t documented yet).

yt.ReleaseTempData

Use this pragma to control when temporary objects (for example, tables) that are created during query execution are deleted:

• immediate — delete objects as soon as they are no longer needed.
• finish — delete at the end of the entire YQL query execution.
• never — never delete.

yt.SamplingIoBlockSize

This pragma sets the minimum block size for large-block sampling.

yt.ScriptCpu

This is a multiplier for estimating CPU consumption by script UDFs (including Python UDF. It affects how MapReduce operations are split into jobs. You can override it with specialized pragmas yt.PythonCpu / yt.JavascriptCpu for a specific UDF type.

yt.SortLocalityTimeout

This pragma sets the sort_locality_timeout setting in the operation specification (this setting is not yet documented).

yt.StartedBy

This pragma sets a map that describes the client through which the operation was started. See the documentation.

yt.StaticPool

Use this pragma to override the compute pool, which by default is the current user’s login.

You can set only one new value for yt.StaticPool. If you specify the static pragma multiple times, its last value will be used. If you need to override the value for the next query, set it using the dynamic yt.Pool pragma.

Signature

PRAGMA yt.StaticPool = '<pool_1>';

Result

After you specify the pragma, the query will run in the <pool_1> pool.

Features and limitations

• Specifying a non-existent pool will cause a query execution error.

• If you specify yt.StaticPool multiple times, the last pragma value is used for all queries. For example, both queries written after the pragma with the pool_1 value and queries after the pragma with the pool_2 value will run in pool_2:

  PRAGMA yt.StaticPool = '<pool_1>';
  PRAGMA yt.StaticPool = '<pool_2>';
  

yt.SuspendIfAccountLimitExceeded

Suspend the operation if the “Account limit exceeded” error occurs in the jobs, see the documentation.

yt.SwitchLimit

This pragma sets the memory buffer size for running the Switch node.

yt.TableContentCompressLevel

This pragma sets the compression level for table content delivered via a file (when yt.TableContentDeliveryMode="file").

yt.TableContentDeliveryMode

If you set the value to native, the table content is delivered to jobs using the native YTsaurus mechanisms. If you set the value to file, the table content is first downloaded on the YQL server and then delivered to jobs as a regular file.

yt.TableContentMaxChunksForNativeDelivery

This pragma sets the maximum number of chunks in a table for delivering it to jobs using the native YTsaurus mechanisms. If this number is exceeded, the table is delivered via a file.

yt.TableContentMaxInputTables

This pragma sets the maximum number of tables for delivering them to jobs using the native YTsaurus mechanisms. If this number is exceeded, a preliminary merge is inserted.

yt.TableContentMinAvgChunkSize

This pragma sets the minimum average chunk size in a table for delivering it to jobs using the native YTsaurus mechanisms. A preliminary merge is inserted for chunks that are not large enough.

yt.TableContentTmpFolder

This pragma specifies the directory where temporary files for tables delivered via a file (when yt.TableContentDeliveryMode="file") will be stored. If you don’t specify it, the default YTsaurus file cache is used.

yt.TableContentUseSkiff

This pragma enables the Skiff format for delivering a table to operation jobs.

yt.TablesTmpFolder

This pragma specifies the directory for storing temporary tables. It takes precedence over yt.TmpFolder.

yt.TempTablesTtl

Use this pragma to manage the TTL of temporary tables. It affects tables with the full result; all other temporary tables are unconditionally deleted when the query finishes, regardless of this pragma.

yt.TentativePoolTrees

This pragma lets you cautiously extend operations to pool trees that differ from the standard ones.

yt.TentativeTreeEligibilityMaxJobDurationRatio

This pragma takes effect only if you set the yt.TentativePoolTrees pragma. It sets the allowed slowdown ratio for jobs in an alternative pool tree.

yt.TentativeTreeEligibilityMinJobDuration

This pragma takes effect only if you set the yt.TentativePoolTrees pragma. It sets the minimum average job duration in an alternative pool tree.

yt.TentativeTreeEligibilitySampleJobCount

This pragma takes effect only if you set the yt.TentativePoolTrees pragma. It sets the number of jobs in the sample.

yt.TmpFolder

Use this pragma to specify the directory for storing temporary tables and files. For more details, see the Temporary data section.

Example

PRAGMA yt.TmpFolder = '//tmp/yql/ivanov/folder';

Result

Temporary tables and files will be saved to the specified directory.

yt.TopSortMaxLimit

This is the maximum LIMIT value used in combination with ORDER BY that triggers the TopSort optimization.

yt.TopSortRowMultiplierPerJob

This pragma sets the expected number of rows per job in a TopSort operation, calculated as LIMIT * yt.TopSortRowMultiplierPerJob.

yt.TopSortSizePerJob

This pragma sets the expected data size per job in a TopSort operation.

yt.UseColumnarStatistics

Enable the use of columnar statistics to accurately estimate job sizes when running operations on tables with columnar selections. See the documentation.

In auto mode, the system automatically disables statistics usage for operations that include tables with optimize_for=lookup.

yt.UseDefaultTentativePoolTrees

This pragma sets the use_default_tentative_pool_trees option in the operation specification.

yt.UseNativeYtTypes

Enable writing values of complex types to tables using the native support for complex types in YTsaurus.

yt.UseQLFilter

Passes the compatible part of a WHERE condition to YTsaurus through input_query. Using min/max statistics, YTsaurus can skip chunks and blocks where the condition is known to be false. The pragma is useful for selective reads from large static tables with a strict schema.

Example

PRAGMA yt.UseQLFilter;

SELECT
    key AS key,
    value AS value
FROM `//path/to/table`
WHERE key >= 1000 AND key < 2000;

Result

Compatible comparisons from WHERE are passed in input_query. If table statistics prove that a chunk or block has no key values in the specified range, YTsaurus skips it without reading from disk. The remaining part of the condition is still evaluated by YQL.

Features and limitations

• Supported types are numeric types, Bool, String, Utf8, and their Optional variants.
• Supported expressions are <, <=, >, >=, ==, and != comparisons between a column and a constant expression, and AND, OR, NOT, EXISTS, and COALESCE.
• The table must have a strict schema.
• Dynamic tables are not supported.
• The pragma does not apply to tables with a custom schema or columns specified through WITH SCHEMA or WITH COLUMNS.

yt.UserSlots

This pragma sets an upper limit on the number of jobs that can run in parallel within a MapReduce operation.

yt.UseSkiff

Enable the Skiff format for input and output in operation jobs.

yt.UseTmpfs

Mount tmpfs to the _yql_tmpfs folder in the MapReduce job sandbox. This pragma isn’t recommended for use.