Commands

This section lists all the commands available in the YT system API and includes a full description of their options.

Structure

To put it simply, each command has the following structure:

• Command name (string).
• Description of the input and output formats (a YSON string with attributes).
• Parameters of executed action (a YSON structure).
• Input and output data streams.

Each command also defines the type of its input and output data. Below all the possible options for the input and output data:

• No data (null).
• The data is a binary stream. For example, file contents.
• The data is structured (structured). For example, regular Cypress nodes.
• The data is a binary stream. For example, table contents.

In addition to that, each command can be:

• Mutating or not (whether it changes anything in the metastate or not).
• Light or heavy (light commands only transmit command parameters within a query, but heavy commands write or read the data stream).

Formats

If your command works with structured or tabular data, you need to specify a format for it. For the input stream, use the input_format parameter, and for the output stream, use the output_format parameter, respectively.

For structured data formats, YSON (default) and JSON formats are supported. For tabular data, there exist many formats.

Retries

A command retry is an option to repeat the query in the event of transient (intermittent) errors. It is expected that the response to a query retry would be indistinguishable from the response to the original query, as if there were no transient errors. It doesn't mean, however, that the response to a retry will be totally identical to the original query response (that is, only standard isolation is guaranteed).

Availability of retry options and the mechanism of retries depends on the command properties.

Light commands

For non-mutating light commands, you can repeat the original query.

The mutating light commands change the system status. That's why you need to hint the system that such a query has already been made. To do this, before executing the command, generate the mutation_id for the command. This is a standard GUID that consists of four 32-bit numbers in the HEX format separated by a dash (-).

Specify the mutation_id generated both in the parameters of both the original query and in the retries. In addition, you need to add, to the original query, the retry parameter with the value of false. To the retried queries, add the retry parameter with the value of true. Some light mutating commands do not support retries (an example is concatenate). If you need to retry such commands, you can use transactions.

Heavy commands

You can't retry heavy commands However, you can implement a retry mechanism by using transactions.

Transactions

Transactions are an integral Cypress property. Many commands that interact with Cypress this way or another, are transactional. Each command or group of commands has a separate indication of whether it is transactional or not. If the command is transactional, it supports the following parameters:

Other query parameters

You can also specify prerequisites for your queries. Moreover, for light non-mutating queries, you can also specify where to read the data from. For more information, see Query parameters.

Working with transactions

For more information about transactions, see the Transactions article.

start_tx

Command properties: Mutating, Light.

Semantics:

• Begin a new transaction in the context of the current transaction.
• The new transaction is a nested (internal) transaction for the given one.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the new transaction.

Example:

PARAMETERS { "transaction_id" = "0-54b-1-36223d20" }
OUTPUT "0-54c-1-bb49086d"

ping_tx

Command properties: Mutating, Light.

Semantics:

• Update the transaction.

Detailed description.

• ping_tx pings the transaction on the server (including all the parent transactions if ping_ancestorsis specified). This way you can extend the TTL for the transaction.
• If the transaction started at the time s with the timeout (TTL) of t, then the transaction will complete at s+t by default.
• If you ping the transaction at the time r (s < r < s + t), it will be extended until r + t.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "transaction_id" = "0-54b-1-36223d20" }

commit_tx

Command properties: Mutating, Light.

Semantics:

• Complete the transaction successfully.
• While there are some incompleted internal transactions, the outer transaction can't complete.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "transaction_id" = "0-54b-1-36223d20" }

abort_tx

Command properties: Mutating, Light.

Semantics:

• Abort the transaction.
• All the active internal transactions are aborted as well.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "transaction_id" = "0-54b-1-36223d20" }

Working with Cypress

For more information about the metainformation tree, see Cypress.

create

Command properties: Mutating, Light.

Semantics:

• Create a node of a specified type in Cypress.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the created node.

Examples:

PARAMETERS { "path" = "//tmp/table" ; "type" = "table" }
OUTPUT "0-4-191-6c07cd58"

PARAMETERS { "type" = "user" ; attributes = { name = 'kulichek-robot' } }
OUTPUT "7727-417b1-1f5-d4f116ca"

PARAMETERS { "path" = "//tmp/document" ; "type" = "document" ; attributes = { value = {} } }
OUTPUT "7727-417b1-1f5-d4f116ca"

remove

Command properties: Mutating, Light.

Semantics:

• Delete the Cypress node.
• Deletes the node successfully, even if a non-empty subtree is growing from it.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//tmp/table" }

set

Command properties: Mutating, Light.

Semantics:

• Write new content into the Cypress node.

Parameters:

Input data:

• Type: structured.
• Value: Node content.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//tmp/node" }
INPUT {
INPUT   "my_integer" = 4 ;
INPUT   "my_double" = 2.718281828 ;
INPUT   "map" = { "a" = 1 ; "b" = 2 } ;
INPUT   "list" = [ 1, 2, 3 ]
INPUT }

multiset_attributes

Command properties: Mutating, Light.

Semantics:

• Set several attributes at the specified path (if the attributes exist already, they are overwritten).

Parameters:

Input data:

• Type: structured.
• Value: A map-node including new attribute values.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//tmp/node/@" }
INPUT {
INPUT   "attribute1" = 4 ;
INPUT   "attribute2" = "attribute2 value";
INPUT }

get

Command properties: Non-mutating, Light.

Semantics:

• Get the content of the Cypress node.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: Node content.

Example:

PARAMETERS { "path" = "//tmp/node" }
OUTPUT {
OUTPUT   "my_integer" = 4 ;
OUTPUT   "my_double" = 2.718281828 ;
OUTPUT   "map" = { "a" = 1 ; "b" = 2 } ;
OUTPUT   "list" = [ 1, 2, 3 ]
OUTPUT }

list

Command properties: Non-mutating, Light.

Semantics:

• Get a list of descendants for the Cypress node.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured
• Value: A list of the node's descendants.

Example:

PARAMETERS { "path" = "//tmp/node" }
OUTPUT [
OUTPUT   "home" ;
OUTPUT   "sys" ;
OUTPUT   "statbox" ;
OUTPUT   "tmp"
OUTPUT ]

lock

Command properties: Mutating, Light.

Semantics:

• Set a lock on the Cypress node.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the created lock and ID of the branched node.

Example:

PARAMETERS { "path" = "//tmp/node" }
OUTPUT {
OUTPUT   "lock_id" = "0-1-3fe00c8-353e9ba4";
OUTPUT   "node_id" = "0-1-3fe012f-9ad48d90";
OUTPUT }

unlock

Command properties: Mutating, Light.

Semantics:

• Release all the locks set for the selected transaction on the Cypress node.
• You can only release explicit locks.
• This command releases both already taken and enqueued locks.
• If the node is not locked, the command is considered successful.
• If the locked node version includes any changes compared to the original version, the unlock fails.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//tmp/node" }

copy

Command properties: Mutating, Light.

Semantics:

• Copy the Cypress node to the new address.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the created node.

Example:

PARAMETERS {
PARAMETERS   "source_path" = "//tmp/from" ;
PARAMETER    "destination_path" = "//tmp/to" ;
PARAMETERS }
OUTPUT "0-4-191-6c07cd58"

move

Command properties: Mutating, Light.

Semantics:

• Move the Cypress node to a new path.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS {
PARAMETERS   "source_path" = "//tmp/from" ;
PARAMETER    "destination_path" = "//tmp/to" ;
PARAMETERS }

link

Command properties: Mutating, Light.

Semantics:

• Create a symbolic link to the object at the new path.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the created node.

Example:

PARAMETERS {
PARAMETERS   "target_path" = "//tmp/from" ;
PARAMETER    "link_path" = "//tmp/to" ;
PARAMETERS }
OUTPUT "0-4-191-6c07cd58"

exists

Command properties: Non-mutating, Light.

Semantics:

• Checks what are the node exists.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: true or false string.

Example:

PARAMETERS {
PARAMETERS   "path" = "//tmp/my_table/@_format" ;
PARAMETERS }

concatenate

Semantics:

• Merge the set of files or tables (in the order in which their paths are listed).

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS {
PARAMETERS   "source_paths" = ["//tmp/file1"; "//tmp/file2"];
PARAMETERS   "destination_path" = "//tmp/file";
PARAMETERS }

Access control

For more information about access permissions, see Access control.

add_member

Command properties: Mutating, Light.

Semantics:

• Add to a group a user or another group.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS {
PARAMETERS   "group" = "admins";
PARAMETERS   "member": "devs";
PARAMETERS   }

remove_member

Command properties: Mutating, Light.

Semantics:

• Add to a group a user or another group.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS {
PARAMETERS   "group" = "admins";
PARAMETERS   "member": "devs";
PARAMETERS }

check_permission

Command properties: Non-mutating, Light.

Semantics:

• Check if a user has a certain permission to access a certain Cypress node.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//sys/accounts/statbox";
PARAMETERS   "permission": "use";
PARAMETERS   "user": "vasya";
PARAMETERS }

Working with files

For more information about files, see the Files article.

write_file

Command properties: Mutating, Heavy.

Synonyms (supported, but not recommended).

• upload.

Semantics:

• Upload the content to the file.
• The file must exist.
• If the file path includes the append=%true attribute, the data is appended to the file; otherwise, the file is overwritten.

Parameters:

Input data:

• Type: binary.
• Value: File content.

Output data:

• Type: null.

Example:

PARAMETERS {"path" = "//tmp/file"}
INPUT this is sample file content

read_file

Command properties: Non-mutating, Heavy.

Synonyms (supported, but not recommended).

• download.

Semantics:

• Get the content of the Cypress node.

Parameters:

Input data:

• Type: null.

Output data:

• Type: binary.
• Value: File content.

Example:

PARAMETERS { "path" = "//tmp/file" }
OUTPUT this is sample file content

Working with file cache

To learn more about the file cache, see the File cache section.

put_file_to_cache

Command properties: Mutating, Light.

Semantics:

• Add the file to the cache.
• Checks that, at the provided path, there's a file with the MD5 hash given in the command options.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: Path to the cached file.

Example:

PARAMETERS {
PARAMETERS   "path" = "//tmp/file";
PARAMETERS   "md5" = "a3dcb4d229de6fde0db5686dee47145d";
PARAMETERS   "cache_path" = "//tmp/yt_wrapper/file_storage/new_cache";
PARAMETERS }
OUTPUT "//tmp/yt_wrapper/file_storage/new_cache/5d/a3dcb4d229de6fde0db5686dee47145d"

get_file_from_cache

Command properties: Non-mutating, Light.

Semantics:

• Get a path to the cached file.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: Path to the cached file.

Example:

PARAMETERS {
PARAMETERS   "md5" = "a3dcb4d229de6fde0db5686dee47145d";
PARAMETERS   "cache_path" = "//tmp/yt_wrapper/file_storage/new_cache";
PARAMETERS }
OUTPUT "//tmp/yt_wrapper/file_storage/new_cache/5d/a3dcb4d229de6fde0db5686dee47145d"

Working with tables

To learn more about static tables, see the Static tables section.

To learn more about dynamic tables, see the Dynamic tables section.

write_table

Command properties: Mutating, Heavy.

Scope: Static tables.

Synonyms (supported, but not recommended).

• write.

Semantics:

• Add new entries to a static table.
• The table must exist.
• If the table path includes the append=%true attribute, the entries are appended to the table; otherwise, the table is overwritten.
• If the table path includes the sorted_by attribute, the system checks that the data is sorted by the specified set of keys and the resulting table is labeled as sorted. You can write sorted data (with a sorted_by attribute specified) only if any new key is greater than or equal to any old key.
• You can write unsorted data (that is, without the sorted_by attribute) to any table, but the table's sort flag will be removed.
• The command can be nested in a transaction.

Parameters:

Input data:

• Type: tabular.
• Value: Table content.

Output data:

• Type: null.

Example:

PARAMETERS {
PARAMETERS   "path" = "//tmp/node" ;
PARAMETERS   "table_writer" = { "codec_id" = "gzip" };
PARAMETERS }
INPUT { "id" = 1; "value" = 1.125; };
INPUT { "id" = 2; "value" = 2.000; };
INPUT { "id" = 3; "value" = 3.850; };

read_table

Command properties: Non-mutating, Heavy.

Scope: Static and dynamic tables.

Synonyms (supported, but not recommended).

• read.

Semantics:

• Retrieve entries from a Cypress table.
• The command can be nested in a transaction.

Parameters:

Input data:

• Type: null.

Output data:

• Type: tabular.
• Value: Table content.

Example:

PARAMETERS { "path" = "//tmp/node" }
OUTPUT { "id" = 1; "value" = 1.125; };
OUTPUT { "id" = 2; "value" = 2.000; };
OUTPUT { "id" = 3; "value" = 3.850; };

read_blob_table

Command properties: Non-mutating, Heavy.

Scope: Static tables with binary data.

Semantics:

• Get a binary data stream from a specific column in a Cypress table.
• The command can be nested in a transaction.

Parameters:

Input data:

• Type: null.

Output data:

• Type: binary.
• Value: Contents of a data column in the table.

Example:

PARAMETERS { "path" = "//tmp/node" }
OUTPUT "Hello world"

select_rows

Command properties: Non-mutating, Heavy.

Scope: Dynamic tables.

Semantics:

• Execute an SQL-like query for a dynamic table in accordance with the supported features.
• The transaction can be executed against a data snapshot with a specified timestamp.

Parameters:

Input data:

• Type: null.

Output data:

• Type: tabular.
• Value: Set of rows with the result.

Example:

PARAMETERS { "query" = "key, value from [//tmp/sometable]" }
OUTPUT { "key" = 1; "value" = "hello"; };
OUTPUT { "key" = 2; "value" = "world; };

insert_rows

Command properties: Mutating, Heavy.

Scope: Dynamic tables.

Semantics:

• Write rows to a dynamic table.

Parameters:

Input data:

• Type: tabular.
• Value: Table content.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table" }
INPUT { "id" = 1; "value" = 1.125; };
INPUT { "id" = 2; "value" = 2.000; };
INPUT { "id" = 3; "value" = 3.850; };

delete_rows

Command properties: Mutating, Heavy.

Scope: Dynamic tables.

Semantics:

• Delete all rows with the specified keys from the dynamic table.

Parameters:

Input data:

• Type: tabular.
• Value: Set of rows with the keys.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//tmp/sometable" }
INPUT { "id" = 1; };
INPUT { "id" = 2; };
INPUT { "id" = 3; };

lock_rows

Command properties: Mutating, Heavy.

Scope: Dynamic tables.

Semantics:

• Lock writing to the rows in the dynamic table while the current transaction is running.

Parameters:

A shared lock can be taken from multiple different transactions at the same time. An exclusive lock can be taken from a single transaction that usually updates this row.

When using read locks (in contrast to write locks), one row can be locked by multiple transactions. That's why, when there is a continuous stream of transactions that take a shared lock on a certain row, the row becomes locked permanently, and you won't be able to update it. This effect is referred to as write starvation. However this is never the case for write-write conflicts because they have an exclusive lock that is released every time a write transaction is complete.

To alleviate the effect of write starvation, you can decrease isolation of shared locks. For this, you can specify the weak or strong lock mode. The distinction of the weak mode is that when the transaction is complete, the timestamps until which the rows were locked by shared locks, aren't saved. The practical implication is that a write operation isn't locked if nested in a transaction that overlaps with the current transaction in time but completes later. However, if the write transaction completes earlier, the transaction with a shared lock isn't applied. In the weak mode, the shared lock becomes asymmetric.

Input data:

• Type: tabular.
• Value: Set of rows with the keys.

lookup_rows

Command properties: Non-mutating, Heavy.

Scope: Dynamic tables.

Semantics:

• Select the rows with the specified keys from the table.
• The command can be executed against a data snapshot with a specified timestamp.
• It is guaranteed that the relative order of rows retrieved in the response will be the same as the order of keys in the query.

Parameters:

Input data:

• Type: tabular.
• Value: Set of rows with the keys.

Output data:

• Type: tabular.
• Value: A set of rows with the specified keys and queried columns.

Example:

PARAMETERS { "path" = "//tmp/sometable" }
INPUT { "id" = 1; };
INPUT { "id" = 2; };
INPUT { "id" = 3; };
OUTPUT { "id" = 1; "value" = 1.125; };
OUTPUT { "id" = 2; "value" = 2.000; };
OUTPUT { "id" = 3; "value" = 3.850; };

trim_rows

Command properties: Mutating, Heavy.

Scope: Ordered dynamic tables.

Semantics:

• Remove rows from the beginning of the tablet of an ordered dynamic table . After that, the deleted data can no longer be read by the select_rows command. Numbering of other rows is unchanged in this case.
• The command is executed outside of transactions.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table"; tablet_index = 10; trimmed_row_count = 43 }

mount_table

Command properties: Mutating, Light.

Scope: Dynamic tables.

Semantics:

• Mounts the tablets of the dynamic table.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table" }

unmount_table

Command properties: Mutating, Light.

Scope: Dynamic tables.

Semantics:

• Unmounts the tablets of the dynamic table.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table"; "first_tablet_index" = 10; "last_tablet_index" = 20; }

remount_table

Command properties: Mutating, Light.

Scope: Dynamic tables.

Semantics:

• Enables you to update certain settings of dynamic tables without unmounting them.
• In the case of such remounting, the table remains accessible for reading and writing.
• The settings are taken from the table attributes and eventually reach the node where the tablets are processed.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table" }

freeze_table

Command properties: Mutating, Light.

Scope: Dynamic tables.

Semantics:

• Changes the tablets of the dynamic table to a frozen state, flushing all the data to the disk. You can read data in this state, but you can't write new entries. However, all the other written data exist in chunks and are available to map-reduce operations, for example (even without using the enable_dynamic_store_read option).

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table" }

unfreeze_table

Command properties: Mutating, Light.

Scope: Dynamic tables.

Semantics:

• Unfreezes a table that was frozed (for example, by a freeze_table command), making the table write-accessible.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table" }

reshard_table

Command properties: Mutating, Light.

Scope: Dynamic tables.

Semantics:

• Reshards the dynamic table (that is, changes the set of its tablets).
• The resharded tablets must be unmounted.
• Be sure to specify tablet_count for an ordered table. For a sorted table, you can specify both tablet_count and pivot_keys. The resharded tablets are replaced by a set of new tablets.
• In the case of a sorted table:
  • When passing pivot_keys, the first key in pivot_keys must match the first key of the first resharded tablet. The number of pivot_keys is equal to the number of new tablets that the resharded tablets are split into.
  • When passing tablet_count, the system will select pivot keys based on the data available in the table as evenly as possible. If the table isn't large enough, you might get less tablets then requested as a result. At default settings, your resulting tablets can't be smaller than about 200 MB each. For smaller slicing, use the optionenable_slicing.
  • If the first key column of the table has an integer type, then along with tablet_count, you can use uniform=True. In this case, uniform values from the range of the appropriate type will be selected as pivot keys. 0, 2^64/n, 2^64\*2/n, ... for an unsigned 64-bit type and -2^63, -2^63 + 2^64/n, -2^63 + 2^64\*2/n, ... for a signed 64-bit type.
• For an ordered table, table_count specifies the number of new tablets that the sharded tablets are split into. In this case, if the resulting tablets are higher in numbers than the old ones, new empty tablets are created. If the resulting tablets are smaller in numbers, the corresponding number of source trailing tablets are merged into a single tablet in their natural order.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table"; "schema" = [{"name" = "key"; type = "int64"; "sort_order" = "ascending"}; {"name" = "value"; type = "string"}]; }

reshard_table_automatic

Command properties: Mutating, Light.

Scope: Sorted dynamic tables.

Semantics:

• Forces an iteration of a background tablet balancer, that is, reshards the tablets that go beyond the min_tablet_size–max_tablet_size interval.
• In contrast to reshard_table, it can work with mounted tables. In the process of its operation, it can amount tablets.
• It's started asynchronously by default, but if you specify keep_actions, it returns a list of IDs that you can use to track the progress. Most APIs do it automatically when passed the sync=True flag.
• The command does not support replicated tables (you can use it independently on different replicas, however).

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: A list of tablet_action_id if keep_actions=True. Otherwise, it returns an empty list.

Example:

PARAMETERS { "path" = "//home/user/table"; }
OUTPUT [
OUTPUT     "11-22-33-44";
OUTPUT ]

alter_table

Command properties: Mutating, Light.

Scope: Static and dynamic tables.

Semantics:

• Changes the scheme and other table settings (both static and dynamic).
• You can usually use alter_table to change the settings that require complex joint validation. The attributes are used to change the recommended settings that only affect new data.
• Changing the schema involves various validation checks because the system has to make sure that the existing data matches the schema.
• You can change a static table to a dynamic table, but not vice versa.
• You can change the schema of a dynamic table, as well as upstream_replica_id, only if the table is unmounted.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "path" = "//home/user/table"; "schema" = [{"name" = "key"; type = "int64"; "sort_order" = "ascending"}; {"name" = "value"; type = "string"}]; }

alter_table_replica

Command properties: Mutating, Light.

Scope: Replicated dynamic tables.

Semantics:

• Changes the replica's properties: enables/disables it or changes its mode.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "replica_id" = "730e-8611b-3ff02c5-f647333f"; "enabled" = %true; }

get_table_columnar_statistics

Command properties: Non-mutating, Light.

Scope: Static and dynamic tables.

Semantics:

• Get statistics on the set of columns in the given set of tables (taken completely or partially, by ranges).
• The statistics includes:
  • The total data_weight for each of the requested columns.
  • The total data_weight for all the old chunks (that the metainformation about each column hasn't been saved to because the chunk has been generated before column-by-column statistics were supported).
  • The total weight of all the timestamps of rows in a dynamic table.
• The paths should always include the column selectors.
• The command can be nested in a transaction.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Values: Column-by-column statistics for the requested columns.

Example:

PARAMETERS {
PARAMETERS   "paths" = ["//tmp/table1{a,b,c}"; "//tmp/table2{a,b,c}"];
PARAMETERS   "transaction_id" = "1234-abcd-abcd-7890"
PARAMETERS }
OUTPUT {
OUTPUT    "column_data_weights" = {
OUTPUT        "a" = 8124;
OUTPUT        "b" = 124241241;
OUTPUT        "c" = 3121414;
OUTPUT    };
OUTPUT    "legacy_chunks_data_weight" = 100242;
OUTPUT    "timestamp_total_weight" = 50056;
OUTPUT }

Running operations

For more information about running data processing operations, see Data processing.

All the operations are run asynchronously, the specified commands only launch them. To find out whether the operation is complete or not, request the operation status using the get_operation command.
All the commands used to work with operations are also transactional. It means that everything you do with tables in an operation will be executed within the specified transaction when you run the operation. The node responsible for the operation (//sys/operations/<OP_ID>) is updated by the scheduleroutside of any transactions.

start_operation

Command properties: Mutating, Light.

Semantics:

• Run the operation of the specified type.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {  "operation_type" = "map_reduce";
PARAMETERS    "spec" = {
PARAMETERS      "input_table_paths" = [ "//tmp/table_in", "//tmp/table_in" ] ;
PARAMETERS      "output_table_path" = "//tmp/table_out"
PARAMETERS    }
PARAMETERS }
OUTPUT "37878b-ba919c15-cdc97f3a-8a983ece"

merge

Command properties: Mutating, Light.

Semantics:

• Start merging the source tables.
• Detailed description of all specification parameters.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {
PARAMETERS    "spec" = {
PARAMETERS      "input_table_paths" = [ "//tmp/table_in", "//tmp/table_in" ] ;
PARAMETERS      "output_table_path" = "//tmp/table_out"
PARAMETERS    }
PARAMETERS }
OUTPUT "37878b-ba919c15-cdc97f3a-8a983ece"

erase

Command properties: Mutating, Light.

Semantics:

• Start erasing data from the source table.
• Detailed description of all specification parameters.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {
PARAMETERS    "spec" = {
PARAMETERS      "table_path" = "//tmp/table[#0:#500]" ;
PARAMETERS    }
PARAMETERS }
OUTPUT "3f9e62-ce8d2965-6350842b-3e4628d2"

map

Command properties: Mutating, Light.

Semantics:

• Run the map operation on the source tables, writing the output to the output tables.
• Detailed description of all specification parameters.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {
PARAMETERS    "spec" = {
PARAMETERS      "mapper" = {
PARAMETERS          "command" = "cat"
PARAMETERS      } ;
PARAMETERS      "input_table_paths"  = [ "//tmp/table_in", "//tmp/table_in" ] ;
PARAMETERS      "output_table_paths" = [ "//tmp/table_out" ]
PARAMETERS    }
PARAMETERS }
OUTPUT "33ab3f-bf1df917-b35fe9ed-c70a4bf4"

reduce

Command properties: Mutating, Light.

Semantics:

• Run the reduce operation on the source tables, writing the output to the output tables.
• Detailed description of all specification parameters.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {
PARAMETERS    "spec" = {
PARAMETERS      "reducer" = {
PARAMETERS          "command" = "cat" ;
PARAMETERS      } ;
PARAMETERS      "input_table_paths"  = [ "//tmp/table_in", "//tmp/table_in" ] ;
PARAMETERS      "output_table_paths" = [ "//tmp/table_out" ] ;
PARAMETERS      "reduce_by"          = [ "my_key" ] ;
PARAMETERS }
OUTPUT "33ab3f-bf1df917-b35fe9ed-c70a4bf4"

join_reduce

Command properties: Mutating, Light.

Semantics:

• Run the join_reduce operation on the source tables, writing the output to the output tables.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {
PARAMETERS    "spec" = {
PARAMETERS      "reducer" = {
PARAMETERS          "command" = "cat" ;
PARAMETERS      } ;
PARAMETERS      "input_table_paths"  = [ "//tmp/table_in", "//tmp/table_in" ] ;
PARAMETERS      "output_table_paths" = [ "//tmp/table_out" ] ;
PARAMETERS      "join_by"          = [ "my_key" ] ;
PARAMETERS }
OUTPUT "33ab3f-bf1df917-b35fe9ed-c70a4bf4"

sort

Command properties: Mutating, Light.

Semantics:

• Start sorting the source tables.
• Detailed description of all the specification parameters.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {
PARAMETERS    "spec" = {
PARAMETERS      "input_table_paths" = [ "//tmp/table_in", "//tmp/table_in" ] ;
PARAMETERS      "output_table_path" = "//tmp/table_out" ;
PARAMETERS      "sort_by"           = [ "mykey" ];
PARAMETERS    }
PARAMETERS }
OUTPUT "37878b-ba919c15-cdc97f3a-8a983ece"

map_reduce

Command properties: Mutating, Light.

Semantics:

• Run Map-Reduce on the source tables.
• Detailed description of all the specification parameters.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {
PARAMETERS    "spec" = {
PARAMETERS      "mapper" = {
PARAMETERS          "command" = "cat"
PARAMETERS      } ;
PARAMETERS      "reducer" = {
PARAMETERS          "command" = "cat" ;
PARAMETERS      } ;
PARAMETERS      "input_table_paths" = [ "//tmp/table_in", "//tmp/table_in" ] ;
PARAMETERS      "output_table_path" = "//tmp/table_out" ;
PARAMETERS      "reduce_by" = [ "my_key" ] ;
PARAMETERS    }
PARAMETERS }
OUTPUT "37878b-ba919c15-cdc97f3a-8a983ece"

remote-copy

Command properties: Mutating, Light.

Semantics:

• Copy the source tables from the specified cluster.
• Detailed description of all specification parameters.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Value: ID of the running operation.

Example:

PARAMETERS {
PARAMETERS    "spec" = {
PARAMETERS      "input_table_paths" = [ "//tmp/table_in", "//tmp/table_in" ] ;
PARAMETERS      "output_table_path" = "//tmp/table_out" ;
PARAMETERS      "cluster_name"           = <cluster-name>;
PARAMETERS    }
PARAMETERS }
OUTPUT "37878b-ba919c15-cdc97f3a-8a983ece"

Working with operations

list_operations

Command properties: Non-mutating, Light.

Semantics:

• Get a list of operations that match the filters.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Returns a dictionary with the following fields:
  • operations: List with explicit descriptions of operations. Each operation described is a dictionary that includes the selected operation attributes: id, type, state, authenticated_user, brief_progress, brief_spec, start_time, suspended, weight. The weight, brief_progress, and brief_spec attributes are optional.
  • incomplete: Whether the list of operations is complete (that is, whether all the operations in the range from_time — to_time are listed).
  • pool_counts: Statistics on pools.
  • user_counts: Statistics on users.
  • state_counts: Statistics on operation states.
  • type_counts: Statistics on operation type.
  • failed_jobs_count: Count of failed jobs for the operations.

Example:

PARAMETERS { }
OUTPUT {
OUTPUT     "operations" = [
OUTPUT         {
OUTPUT             "id" = "7001208d-fef089b3-3fe03e8-453d99a1";
OUTPUT             "type" = "remote-copy";
OUTPUT             "state" = "initializing";
OUTPUT             "authenticated_user" = "user-name";
OUTPUT             "brief_progress" = {};
OUTPUT             "brief_spec" = {
OUTPUT                 ...
OUTPUT             };
OUTPUT             "start_time" = "2018-02-06T11:06:34.200591Z";
OUTPUT             "suspended" = %false;
OUTPUT             "weight" = 1.;
OUTPUT         };
OUTPUT     ];
OUTPUT     "incomplete" = %true;
OUTPUT     "pool_counts" = {
OUTPUT         "pool-counts-example" = 2;
OUTPUT         "user-name-1" = 2;
OUTPUT         ...
OUTPUT     };
OUTPUT     "user_counts" = {
OUTPUT         "yql" = 52;
OUTPUT         "user-name-1" = 2;
OUTPUT     };
OUTPUT     "state_counts" = {
OUTPUT         "materializing" = 10;
OUTPUT         "pending" = 763;
OUTPUT         "running" = 1848;
OUTPUT         "completed" = 6654;
OUTPUT         "aborted" = 37;
OUTPUT         "failed" = 98;
OUTPUT     };
OUTPUT     "type_counts" = {
OUTPUT         "map" = 4294;
OUTPUT         "merge" = 1337;
OUTPUT         "erase" = 97;
OUTPUT         "sort" = 1126;
OUTPUT         "reduce" = 886;
OUTPUT         "map_reduce" = 1609;
OUTPUT         "remote-copy" = 24;
OUTPUT         "join_reduce" = 37;
OUTPUT     };
OUTPUT     "failed_jobs_count" = 109;
OUTPUT }

get_operation

Command properties: Non-mutating, Light.

Semantics:

• Get information about an operation.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.
• Returns a dictionary with the requested operation attributes.

Example:

PARAMETERS {  "operation_id" = "33ab3f-bf1df917-b35fe9ed-c70a4bf4"; attributes = [ "state" ] }
OUTPUT {
OUTPUT     "state" = "running";
OUTPUT }

abort_operation

Command properties: Mutating, Light.

Semantics:

• Abort the operation.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "operation_id" = "33ab3f-bf1df917-b35fe9ed-c70a4bf4" }

complete_operation

Command properties: Mutating, Light.

Semantics:

• Instantly complete the operation, saving the currently calculated result.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "operation_id" = "33ab3f-bf1df917-b35fe9ed-c70a4bf4" }

suspend_operation

Command properties: Mutating, Light.

Semantics:

• Suspend the operation.
• This command is executed instantly. After that, no more jobs are started for the operation, and the running jobs are optionally aborted.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "operation_id" = "33ab3f-bf1df917-b35fe9ed-c70a4bf4" }

resume_operation

Command properties: Mutating, Light.

Semantics:

• Resume the suspended operation.
• This command is executed instantly.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "operation_id" = "33ab3f-bf1df917-b35fe9ed-c70a4bf4" }

update_operation_parameters

Command properties: Mutating, Light.

Semantics:

• Update the parameters of a running operation.

Parameters:

Scheduler settings for the tree:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS {"`operation_id`" = "33ab3f-bf1df917-b35fe9ed-c70a4bf4"; "parameters" = {"pool" = "username"; "scheduling_options_per_pool_tree" = {"physical" = {"weight" = 2; "resource_limits" = { "user_slots" = 1; "cpu" = 0.5; "network" = 10; "memory" = 1000000000}}}}}

Working with jobs

get_job

Command properties: Non-mutating, Light.

Semantics:

• Getting information about a job. The command can run both for running and completed jobs (if information about the jobs was saved).

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured. Map with the job's attributes.

Example:

PARAMETERS { "operation_id" = "e13c5406-e5dd6f5d-3fe03e8-fe05f0d3"; "job_id" = "f11ae559-a0375703-3fe0384-8f1"}
OUTPUT {
OUTPUT     "operation_id" = "e13c5406-e5dd6f5d-3fe03e8-fe05f0d3";
OUTPUT     "job_id" = "f11ae559-a0375703-3fe0384-8f1";
OUTPUT     "state" = "completed";
OUTPUT     "start_time" = "2018-02-06T09:37:02.858492Z";
OUTPUT     "finish_time" = "2018-02-06T09:42:19.185525Z";
OUTPUT     "address" = "<cluster-name>:9012";
OUTPUT     "statistics" = {
OUTPUT         "data" = {
OUTPUT              ...
OUTPUT         };
OUTPUT     };
OUTPUT     "events" = [
OUTPUT         ...
OUTPUT     ];
OUTPUT }

list_jobs

Command properties: Non-mutating, Light.

Semantics:

• Get all the jobs that match the filters for a given operation.

Parameters:

The job_type, job_state, address, and has_stderr parameters define the job filter. The response will only include the jobs that meet the filtering criteria.

The sort_field andsort_order define the order of jobs in the response. In this case, the limit and offset parameters define the slice (subset) of jobs in the response: the first offset jobs are skipped, and then limit of the remaining jobs is selected.

The data_source parameter regulates the source of data from which the jobs are taken:

• In the runtime mode, jobs are retrieved from the controller agent and Cypress and then merged.
• In the archive mode, jobs are retrieved from the archive and the controller agent and then merged.
• The auto mode automatically determines the source of jobs based on availability of operations in the controller agent.

Input data:

• Type: null.

Output data:

• Type: structured.

• Structure with the fields jobs, cypress_job_count, controller_agent_job_count, and archive_job_count.

• *_count fields: Counter of jobs found for a given operation in the appropriate data sources, without filtering. If all three numbers are zeros, it means that there's no information about the operation's jobs. If all the three numbers aren't zeros, but the jobs response is empty, it means that all the jobs were filtered out. If you get null instead of a number, the corresponding data source wasn't polled.

• jobs field: List of structures that describe each job. Each job can have the following fields:

  • id (guid), type (string), state (string), address (string): Required fields.
  • start_time (instant), finish_time (instant), progress (double), stderr_size (int): Optional fields.
  • error, brief_statistics, input_paths, core_infos: Optional fields.

Example:

PARAMETERS {operation_id" = "4505e8eb-28fa88e2-3fe03e8-c6fcd8fa"; }
OUTPUT {
OUTPUT     "jobs" = [
OUTPUT         {
OUTPUT             "id" = "55aff293-7ef14284-3fe0384-3e07";
OUTPUT             "type" = "map";
OUTPUT             "state" = "failed";
OUTPUT             "address" = "fb-sas2-0511-sas-yt-prod-portovm-14323.gencfg-c.host-name.net:9012";
OUTPUT             "start_time" = "2018-05-05T00:41:27.433832Z";
OUTPUT             "finish_time" = "2018-05-05T00:49:04.288196Z";
OUTPUT             "fail_context_size" = 973230u;
OUTPUT             "error" = {
OUTPUT                 "code" = 1205;
OUTPUT                 "message" = "User job failed";
OUTPUT                 ...
OUTPUT             };
OUTPUT             ...
OUTPUT         };
OUTPUT         ...
OUTPUT         {
OUTPUT             "id" = "69ae20a7-887b25ab-3fe0384-3cff";
OUTPUT             "type" = "map";
OUTPUT             "state" = "running";
OUTPUT             "address" = "fb-sas2-0101-sas-yt-prod-portovm-14323.gencfg-c.host-name.net:9012";
OUTPUT             "start_time" = "2018-05-07T13:04:03.339873Z";
OUTPUT             "progress" = 0.;
OUTPUT             "brief_statistics" = <
OUTPUT                 "timestamp" = "2018-05-07T13:04:08.431740Z";
OUTPUT             > {
OUTPUT                 "processed_input_compressed_data_size" = 0;
OUTPUT                 "processed_input_data_weight" = 0;
OUTPUT                 "processed_output_uncompressed_data_size" = 0;
OUTPUT                 "processed_output_compressed_data_size" = 0;
OUTPUT                 "processed_input_uncompressed_data_size" = 0;
OUTPUT                 "processed_input_row_count" = 0;
OUTPUT             };
OUTPUT         };
OUTPUT     ];
OUTPUT     "cypress_job_count" = 200;
OUTPUT     "scheduler_job_count" = 208;
OUTPUT     "archive_job_count" = #;
OUTPUT     "type_counts" = {
OUTPUT         "map" = 408;
OUTPUT     };
OUTPUT     "state_counts" = {
OUTPUT         "running" = 208;
OUTPUT         "failed" = 200;
OUTPUT     };
OUTPUT }

abandon_job

Command properties: Mutating, Light.

Semantics:

• Abort the job and consider that its input data was processed successfully.

Parameters:

Input data:

• Type: null

Output data:

• Type: null.

Example:

PARAMETERS { "job_id" = "1225d-1f2fb8c4-f1075d39-5fb7cdff" }

abort_job

Command properties: Mutating, Light.

Semantics:

• Abort the job, and let it later be restarted by the scheduler (as any aborted operation job).

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "job_id" = "1225d-1f2fb8c4-f1075d39-5fb7cdff" }

dump_job_context

Command properties: Non-mutating, Light.

Semantics:

• Get the input context of the job.

Parameters:

Input data:

• Type: null.

Output data:

• Type: null.

Example:

PARAMETERS { "job_id" = "1225d-1f2fb8c4-f1075d39-5fb7cdff"; "path" = "//tmp/input_context" }

get_job_input

Command properties: Non-mutating, Heavy.

Semantics:

• Get the full input for the job. The command is used both with running and failed jobs with a saved specification and all the input data available.

Parameters:

Input data:

• Type: null.

Output data:

• Type: binary.
• Value: Job input.

Example:

PARAMETERS { "job_id" = "1225d-1f2fb8c4-f1075d39-5fb7cdff"}

get_job_fail_context

Command properties: Non-mutating, Heavy.

Semantics:

• Get the fail context of the job. The command is used with failed jobs.

Parameters:

Input data:

• Type: null.

Output data:

• Type: binary.
• Value: Fail context of the job.

Example:

PARAMETERS { "operation_id" = "33ab3f-bf1df917-b35fe9ed-c70a4bf4"; "job_id" = "1225d-1f2fb8c4-f1075d39-5fb7cdff"}

get_job_stderr

Command properties: Non-mutating, Heavy.

Semantics:

• Get the stderr of the job. The command is used with failed jobs.

Parameters:

Input data:

• Type: null.

Output data:

• Type: binary.
• Value: The stderr of the job.

Example:

PARAMETERS { "operation_id" = "33ab3f-bf1df917-b35fe9ed-c70a4bf4"; "job_id" = "1225d-1f2fb8c4-f1075d39-5fb7cdff"; "offset" = 500; "limit" = 100 }
OUTPUT {
OUTPUT   "total_size" = 1000;
OUTPUT   "end_offset" = 600;
OUTPUT }

Other

parse_ypath

Command properties: Non-mutating, Light.

Semantics:

• Perform parsing of the passed YPath by putting all IDs of the complex YPath into attributes.

Parameters:

Input data:

• Type: null.

Output data:

• Type: structured.

Example:

PARAMETERS { "path" = "//tmp/table[#1:#2]" }
OUTPUT { "path" = "<ranges=[{lower_limit={row_index=1};upper_limit={row_index=2}}]>//tmp/table" }

execute_batch

Command properties: Mutating if the set includes mutating commands, Light.

Semantics:

• Use a single query to execute the set of commands passed in the parameters.
• The command can (and will be) executed in parallel. It means that if a set includes both writing to and reading from the node, the reading result can either be the older value or the updated one.
• The set can only include light commands.
• The set can only include commands with the input type of null or structured.
• The set can only include commands with the output type of null or structured.

Parameters:

Input data:

• Type: structured.
• The queries executed are listed in the requests parameter, which is a list.
• Each element in this list is a dictionary including the following fields:

Output data:

• Type: structured.
• The output produces a list of the same length as at the input.
• Each list item describes the result of a single query execution. This is a dictionary of the following format:

Example:

PARAMETERS {
PARAMETERS    "requests" = [
PARAMETERS      {
PARAMETERS        "command" = "set";
PARAMETERS        "parameters" = {"path" = "//tmp/a"};
PARAMETERS        "input" = "value_a";
PARAMETERS      };
PARAMETERS      {
PARAMETERS        "command" = "get";
PARAMETERS        "parameters" = {"path" = "//tmp/b"};
PARAMETERS      };
PARAMETERS      {
PARAMETERS        "command" = "get";
PARAMETERS        "parameters" = {"path" = "//nonexisting"};
PARAMETERS      };
PARAMETERS    ];
PARAMETERS }
OUTPUT [
OUTPUT   { };
OUTPUT   { output = "value_b"; };
OUTPUT   { error = {...} };
OUTPUT ]

get_supported_features

Command properties: Non-mutating, Light.

Semantics:

• Returns a dictionary with the features field describing elementary data types, compression codecs, erasure codecs, and other features supported by the cluster.

Parameters:

• No

Input data:

• Type: null.

Output data:

• Type: structured. The features key in the response includes a dictionary with the following fields.

Example:

PARAMETERS { }
OUTPUT {
OUTPUT   "features" = {
OUTPUT     "primitive_types" = ["int8"; "int16"; ... ];
OUTPUT     "erasure_codecs" = ["lrc_12_2_2"; "reed_solomon_6_3"; ... ];
OUTPUT     "compression_codecs" = ["none"; "snappy"; "brotli_1"; ... ];
OUTPUT   };
OUTPUT }

generate_timestamp

Command properties: Non-mutating, Light.

Semantics:

• Generates a monotonous timestamp.

Parameters:

• No

Input data:

• Type: null.

Output data:

• Type: structured. The timestamp key in the response stores a uint64 value.

Example:

PARAMETERS { }
OUTPUT {
OUTPUT   "timestamp" = 1723665447133469427u;
OUTPUT }