Data types

This section describes the data types supported by YTsaurus and the way they are described in the schema and represented in the formats.

Overview

YTsaurus supports a number of primitive types:

• string;
• integer;
• boolean;
• float;
• double;
• date;
• datetime;
• timestamp;
• interval.

As well as the following composite (complex) types:

• optional;
• list;
• struct;
• tuple;
• variant;
• tagged.

You can use one of the following two methods to specify a type in a table schema:

• Using the type and (optionally) the required keys: historically, the first method but it is only good for defining primitive or optional primitive types.
• Using the type_v3 key.

The type key always expects a string.
The type_v3 key expects either a string for primitive types or a YSON dictionary.
A YSON dict always has the type_name key that stores the type name.
The remaining keys depend on the specific type and are described below.

Describing types in a schema

Primitive types

You can define primitive types in a schema both using the type and the type_v3 keys.

If primitive type T is defined using type, YTsaurus will additionally check the required key in the schema:

• required=%true — the column will have a strictly defined type. The Null value or a missing value is not allowed;
• required=%false — the column will have the optional<T> type, allowing all primitive type values and the Null value.

The table lists the supported types and their representation in the type/type_v3 keys.

Schema example:

type_v3=utf8

type_v3=bool

type_v3=yson

Temporal types

Temporal types in YTsaurus are categorized into two groups. Historically, the first ones to appear in the system were date, datetime, timestamp, and interval. They are used to represent times from the beginning of 1970 to the end of 2105. They were followed by date32, datetime64, timestamp64, and interval64. These types can be used to represent times over a wider range, about 145,000 years into the past and into the future. We recommend using the latter types, because they have a wider range of values.

All dates and times assume the Gregorian calendar. When working with points in time in the distant past, keep in mind that YTsaurus does not account for the adoption of the Gregorian calendar, which occurred at different times in different countries. YTsaurus assumes that the Gregorian calendar has always been used.

Time zones

Warning: support for the data types described in this section is incomplete and will likely change in a backward-incompatible way in the future. These types must not be used in production workflows.

Types tz_timestamp64, tz_datetime64, tz_date32, tz_timestamp, tz_datetime, and tz_date store time information incorporating time zone details. Logically, these types store the pair:

• a timestamp represented by an integer of the corresponding "timezone-free" type; the number represents a point in time in the UTC
• The time zone id, an unsigned 16-bit number (see the list of time zones; numbering is zero-based: "GMT" = 0, "Europe/Moscow" = 1).

The internal representation of values for these types is described below. Certain higher-level tools offer a convenient way to work with these types.

This pair is serialized into a string as follows:

• The timestamp is written in presorted representation (see below).
• The time zone id is written in presorted representation (see below).

The presorted representation of an integer is obtained as follows:

1. The number is written in big-endian format.
2. If the number has a signed type, the most significant (sign) bit is inverted. Otherwise, this step is skipped.

Example. To store the point in time 2025-01-01T00:00:00 in the Moscow time zone using the tz_datetime64 type, follow these steps:

1. Convert this point in time to UTC. The result is 2024-12-31T21:00:00Z.
2. Convert the UTC time to a Unix timestamp. The result is 1735678800.
3. Write this timestamp in big-endian representation. The result is "\x00\x00\x00\x00\x67\x74\x5b\x50".
4. Because the tz_datetime64 type is based on the signed Int64 type, invert the most significant bit. The result is "\x80\x00\x00\x00\x67\x74\x5b\x50".
5. Convert the "Europe/Moscow" time zone name to its id. The result is 1.
6. Write the id in big-endian representation. The result is "\x00\x01".
7. Append the time zone id to the timestamp. The result is "\x80\x00\x00\x00\x67\x74\x5b\x50\x00\x01".

Decimal

The values of type decimal(p, s) are real numbers with the specified precision.

To define this type in a schema, specify the following keys:

• type_name: value of decimal.
• precision: total number of decimal digits in a numeric value, precision must be in the range [1, 35].
• scale: number of digits to the right of the decimal point in a numeric value, scale must be in the range [0, precision].

Schema example:

type_v3={
    type_name=decimal;
    precision=10;
    scale=2;
}

The values 3.14, -2.71, 9.99 may be of type decimal(3, 2) (precision=3, scale=2).

The type supports a number of special values, such as nan, +inf, -inf.

Description of binary representation

decimal numbers have a special binary representation,
which is used by default in many formats, such as yson.

For the purposes of this representation, the values of decimal(p, s) types are maintained as binary strings. Binary string length
depends on precision.

You need to perform the following steps to obtain a binary representation of a decimal number. These steps will be illustrated with the values 3.1415, -2.7182 of type decimal(5, 4).

1. Take an integer made up of the value's digits. The number of bits is taken from precision in the table above. In this example, 32-bit numbers 31415, -27182.
2. Write the number as a big-endian sequence. In this example, the strings are \x00\x00\x7A\xB7, \xFF\xFF\x95\xD2.
3. Invert the most significant bit. In this example, the strings are \x80\x00\7A\xB7, \x7F\xFF\x95\xD2.

The integer representations of the special values of nan, +inf, -inf for the first step are shown in the table below:

Optional type

The optional<T> type means that a value may be of type T or be empty.

Legacy columns containing the type=T;required=false attributes correspond to type optional<T> defined using type_v3.

To define type optional, specify the keys below:

• type_name: value of optional.
• item: element type description.

Schema example:

type_v3={
  type_name=optional;
  item=string;
}

type_v3={
  type_name=optional;
  item={
    type_name=optional;
    item=bool;
  }
}

List

Values of type list<T> are lists of elements of type T.

To define the type in the schema, specify the keys below:

• type_name: value of list.
• item: element type description.

Schema example:

type_v3={
  type_name=list;
  item=string;
}

type_v3={
  type_name=list;
  item={
    type_name=list;
    item=double;
  }
}

Struct

A collection of named fields with specified value types.

To define this type in a schema, specify the following keys:

• type_name: value of struct.
• members: list of dictionaries with keys:
  • name: field name, must be a non-empty utf8 string.
  • type: field type.

Schema example:

type_v3={
  type_name=struct;
  members=[
    {
      name=foo;
      type=int32;
    };
    {
      name=bar;
      type={
        type_name=optional;
        item=string;
      }
    };
  ]
}

Tuple

A collection of unnamed fields of certain predefined types.

To define this type, you need to specify the following keys in the schema:

• type_name: value of tuple.
• elements: list of dictionaries with keys:
• type: element type.

Schema example:

type_v3={
  type_name=tuple;
  elements=[
     {
       type=double;
     };
     {
       type=double;
     };
  ]
}

Variant

Variant is strictly a single value from a defined collection of types.
A variant may be of one of two types:

• Variant over struct. Each type has a name (as in a struct), and each variant value is labeled with the name of the relevant variant element value.
• Variant over a tuple. In this case, all elements are unnamed, and each value is identified by an index.

To define this type, specify the following keys in a schema:

• type_name: value of variant.
• elements or members (not both): the keys have a structure similar to these keys in tuple / struct:

• elements: for the option with unnamed elements with the key itself containing a list of dictionaries with keys:
  • type: element type.
• members: for the option with named elements with the key itself containing a list of dictionaries with keys:
  • name: element name, must be a non-empty utf8 string.
  • type: description of element type.

Schema example:

type_v3={
  type_name=variant;
  members=[
     {
       name=int_field;
       type=int64;
     };
     {
       name=string_field;
       type=string;
     };
  ]
}

type_v3={
  type_name=variant;
  elements=[
     {
       type=int32;
     };
     {
       type=string;
     };
     {
       type=double;
     };
  ]
}

Dict

A dict is a sequence of key/value pairs.
YTsaurus does not check the keys for uniqueness or order.
However, most clients will upload data to an actual dictionary while processing, and the value for non-unique keys will be lost.

To define this type, specify the following keys in a schema:

• type_name: value of dict.
• key: description of key type.
• value: value type description.

Schema example:

type_v3={
  type_name=dict;
  key=int64;
  value={
    type_name=optional;
    item=string;
  };
}

Tagged

The tagged type helps annotate other types with a string. Any value of type T can serve as a value for type tagged<TAG_NAME,T>,
however, the types themselves will be considered different wherever YTsaurus compares schemas. For instance, when the possibility of merging two tables into one is being checked.

To define this type, specify the following keys in a schema:

• type_name: value of tagged.
• tag: tag name, must be a non-empty utf8 string.
• item: element type description.

Media in the web UI

The web UI can display images and play audio when viewing a table or YQL query results in Query Tracker. To enable this, set the column type to tagged<string> via type_v3, and specify the media tag in the tag field:

type_v3={
  type_name=tagged;
  tag="image/svg";
  item="string";
}

View the full example

The image/svg value indicates that the column contains a Base64-encoded SVG image. For other supported media types, select a tag value from the table:

Limits

Two limits apply when viewing a table in the web UI:

Cell data loading limit. The web UI limits the amount of data downloaded from each table cell to avoid loading large values into memory along with the entire page. By default, this limit is 1 KiB and applies to all values, including Base64 strings.

If the size of a Base64 string exceeds the current cell data loading limit, the UI receives only part of the value, marks it as incomplete, and displays a preview button in the cell. To view the image, click this button:

Media preview limit. A separate limit of 16 MiB applies to the size of the Base64 string used for previews. It applies to the encoded value in the cell, not to the size of the original file.

If the string exceeds this limit, you cannot view the image or play the audio. This limit cannot be changed in the settings.

Thus, media data within the cell limit (1 KiB by default, up to 64 KiB) is displayed immediately; data above the cell limit and up to 16 MiB is displayed after you click the preview button ; and data larger than 16 MiB is not displayed.

Example

In this example, you will create a table with a column for a PNG image. You will then write the YTsaurus logo to it in Base64 and verify the result in the web interface.

1. Save the YTsaurus logo to the current directory as logo.png.

2. Create the //tmp/tagged-media table with a column for a PNG image:

   $ yt create table  --attributes '{
       schema=[
           {name=image; type_v3={item=string; type_name=tagged; tag="image/png"}};
       ]
   }' //tmp/tagged-media
   

3. Encode the image contents in Base64 without the Data URL prefix and write the string to the table:

   $ IMAGE_BASE64=$(base64 < logo.png | tr -d '\r\n')
   $ printf '{"image":"%s"}\n' "$IMAGE_BASE64" \
       | yt write-table --format json //tmp/tagged-media
   

4. Open the //tmp/tagged-media table in the web interface. The image column should display the YTsaurus logo.

Representing compound types in formats

Formats are used to read and write tables.
Some formats do not support composite data, and some, such as dsv / schemaful_dsv, will return an error in response to an attempt to read a composite value. For example, Values of type "any" are not supported by the chosen format.

YSON

There are two YSON representations of composite types. The representations of the struct and variant types differ:
the default representation is more convenient to use,
while the alternative representation is somewhat more efficient for storage and processing.

You can switch between representations using the complex_type_mode flag. Possible values: named / positional.
Type representation descriptions are provided below. Unless otherwise specified, the type representation does not depend on the complex_type_mode setting.

The dict type with string keys has two representations. By default, the positional representation is used as a list (because this is how data is stored in YTsaurus). For readability, you can enable named mode using the string_keyed_dict_mode flag.

Primitive types

Primitive types have a linear representation as a single YSON value.
The table shows a mapping between the primitive and the YSON types.

Time types

The YSON format has a time_mode option that controls the representation of the date, datetime, and timestamp types. Its possible values are:

• binary (default) — uses an unsigned number representing the number of days / seconds / milliseconds since the Unix epoch.
• text — uses a string representation. The representations of date, datetime, and timestamp are as follows, respectively:
  2022-01-02, 2022-01-02T03:04:05Z, 2022-01-02T03:04:05.123456Z.

UUID

The YSON format has a uuid_mode option that controls the representation of the uuid type. Its possible values are:

• binary (default) — uses a 16-byte binary representation.
• text_yt — uses a string representation in the YTsaurus format consisting of 4 groups, for example, 61626364-65666768-696a6b6c-6d6e6f70;
• text_yql — uses a string representation in the YQL format consisting of 5 groups and more closely resembling the RFC representation, for example, 64636261-6665-6867-696a-6b6c6d6e6f70.

Decimal

The YSON format has a decimal_mode option that controls the representation of the decimal type. Its possible values are:

• binary (default) — encoded as a binary string containing the binary representation of a decimal number.
• text — uses the textual representation of a decimal number.

Optional

The representation of the optional type depends on its inner type.
This is required for backward compatibility with columns that have required=%false set.
If T is an arbitrary type other than optional, then optional<T> is represented as follows:

• Null value of optional is represented as #;
• otherwise, the standard representation of a value of type T is used.

If T is an optional type, then optional<T> is represented as follows:

• Null value of the outer optional is represented as #;
• otherwise, the [ v ] representation is used (a YSON list of length 1), where v is the YSON representation of a value of type T.

Example values of type optional<int64>:

#

-42

Example values of type optional<optional<int64>>:

#

[ # ]

[ -42 ]

List

The list<T> type is encoded as a YSON list whose elements are encoded representations of elements of type T.

Example values of type list<int64>:

[]

[42; -1;]

Struct

The structure representation depends on the value of the complex_type_mode flag.

Named representation (default)

The representation described here applies when the YSON format option
complex_type_mode is not set or is set to complex_type_mode=named.

The struct is represented by a YSON dictionary where field names serve as keys and the contents of these fields are the values.

Example values for the struct<Foo:int64;Bar:optional<utf8>> type:

{Foo=42;Bar=#;}

{Foo=-5;Bar="minus five";}

Positional representation

If the YSON format option complex_type_mode=positional is set, a different representation is used.

A struct is encoded as a YSON list with the i-th position containing a YSON representation of the struct's i-th field.
The list may be shorter than the number of fields in the structure. In this case, the remaining types must be of type optional<T>,
and the fields are considered to have an empty optional value.

Example values for the struct<Foo:int64;Bar:optional<utf8>> type:

[42; #;]
[42]

[-5;"minus five";]

Tuple

The tuple type is encoded as a fixed-length YSON list. The i-th position contains the i-th field's encoded value.

Example values for the tuple<int64;optional<utf8>> type:

[42; #;]

[-5;"minus five";]

Variant

Unnamed variant

The unnamed option is represented by a YSON list of length 2 that includes the following elements:

• Alternative number (indexed at 0).
• An encoded value for the relevant alternative.

Example values for the variant<int64;optional<utf8>> type:

[0; 42]

[1; #]

[1; "foo bar";]

Named variant option

Named representation (default)

The representation described here applies when the YSON format option complex_type_mode is not set or is set to complex_type_mode=named.

The named option is represented by a YSON list of length 2 that includes the following elements:

• Alternative name.
• An encoded value for the relevant alternative.

Example values for the variant<Foo:int64;Bar:optional<utf8>> type:

[Foo; 42]

[Bar; #]

[Bar; "foo bar";]

Positional representation

If the format option complex_type_mode=positional is set.

The named option is represented by a YSON list of length 2 that includes the following elements:

• Index of alternative.
• An encoded value for the relevant alternative.

Example values for the variant<Foo:int64;Bar:optional<utf8>> type:

[0; 42]

[1; #]

[1; "foo bar";]

Dict

By default, the dict type is represented as a YSON list in which each item is a two-item YSON list containing a key and a value.

Example values of the dict<int32;string> type:

[[1;"one"];[4;"four"]]

[]

dict can be represented as a YSON map. However, a YSON map supports only strings as keys, whereas dict also supports other key types.

Dict with string keys

The representation of a dictionary with string keys depends on the value of the string_keyed_dict_mode flag.

Positional representation (default)

The representation described here applies when the YSON format option string_keyed_dict_mode is not set or is set to string_keyed_dict_mode=positional.

See above

Example values of the dict<string;int32> type:

[["one";1];["four";4]]

Named representation

If the YSON format option string_keyed_dict_mode=named is set, a different representation is used.

dict is encoded as a YSON map.

Example values of the dict<string;int32> type:

{one=1; four=4}

Tagged

The tagged type does not change the YSON representation of its element.