Dynamic tables are a type of table in YTsaurus that implement an interface to read and write data by key. They support transactions and their own SQL dialect.
Dynamic tables add the capabilities of typical key-value storage to the YTsaurus system.
Elementary operations with such tables are writing, deleting, and reading rows by key. You can also execute SQL queries to work with row ranges.
Dynamic tables support MVCC for transactions. Working with them does not require communication with the YTsaurus master server, which provides additional scalability.
The closest analogs to dynamic tables are:
- Apache HBase: In terms of data storage organization.
- Spanner: In terms of transaction implementation.
- Impala: In terms of the query calculation model.
The similarities with these systems will be described below.
Types of dynamic tables
Architecturally, the way dynamic tables are stored in YTsaurus uses the Log-structured merge tree idea. This makes it similar to the BigTable and HBase systems. To ensure fault tolerance and consistency,
Hydra state replication library is used (analog of Raft). To support distributed transactions, the two-phase commit that is similar to Google Spanner is used.
Besides the attributes inherent to all tables, dynamic tables have a number of unique attributes, some of which are represented in the table:
|chunk_row_count||Int||Total number of non-combined rows in all table chunks.|
|atomicity||Atomicity||Atomicity mode. By default:
|in_memory_mode||InMemoryMode||In-memory mode for tables. By default:
chunk_row_count attribute can be used to evaluate the table size, but the evaluation is approximate for the following reasons:
- It does not take into account the rows that were recently written and have not yet been flushed into chunks, residing only in nodes memory.
- Different versions of a single row can be counted multiple times until corresponding chunks are combined.
- Rows may be logically marked as deleted or obsolete but will still be counted by the attribute until physically deleted.
Individual attributes of a dynamic table affect its behavior. In particular, you can use individual attributes to:
- Set up background compaction.
- Set TTL: Time to live for individual values.
- Manage automatic sharding.
A number of size and content type limitations are imposed on rows and the schema of a dynamic table:
- The number of columns in a dynamic table cannot exceed 1024, of which no more than 32 can be key columns.
- The maximum length of the
Stringvalue in a dynamic table is 16 megabytes.
Besides that, dynamic tables inherit the limitations of static tables.
In-memory tables (in-memory)
Accessing data in RAM is much faster than accessing data on the disk. In the YTsaurus system, you can set up a table so that its data is permanently stored in memory. This drastically increases the speed of table operations. This method has certain limitations:
The RAM capacity on all cluster nodes is limited, so this option should only be used when really needed.
Any sorted dynamic table can be switched to in-memory mode, as well as taken out of this mode. Run
remount_tableto make the changes take effect.
The use of dynamic in-memory tables does not affect the levels of isolation, consistency, or fault tolerance. The data of such tables is still stored in chunks of the distributed storage, in-memory mode only means that in addition to the disk the table data is also stored in RAM.
When in-memory mode is enabled, data is uploaded into the cluster node memory. This may take some time.
In-memory mode is configured using the
in_memory_mode attribute for the table. Possible attribute values:
|None||Disabled.||Table data is written to chunks and their blocks, compressed and uncompressed, and cached on a common basis. This mode is suitable for large tables that cannot be fixed in memory.|
|Compressed||Enabled.||The compressed table data — the contents of the chunk blocks — is permanently present in memory, but the data blocks must be unpacked for read accesses. Unpacked blocks are cached on a common basis. This mode is a good combination in terms of speed-to-memory capacity ratio.|
|Uncompressed||Enabled.||Uncompressed table data is permanently present in memory, no disk accesses or data unpacking are needed for read accesses. This mode provides maximum performance at the cost of RAM.|
Hash tables to read by key
The read by key operation searches for the table data keys using a binary search. You can achieve some acceleration by building hash tables for the keys. To do this, set the
enable_lookup_hash_table = %true attribute for the table.
Hash tables only make sense in
uncompressed mode and they are not allowed in other modes.
Table storage formats
The YTsaurus system offers several ways of table data storage in chunks. Switching between them is managed by the
optimize_for attribute on the table. This attribute can be changed on an existing table, but the effect will only be noticeable for newly created chunks. Taking into account compaction, old data can also be recompressed in background mode.
If the table is already mounted, run the
remount_table command to inform the tablets of the format change.
Possible values of the
lookup: The format, which enables rows to be efficiently read from a table by individual keys.
scanformat uses column-by-column storage, as well as adaptive column-by-column compression algorithms. The vertical data partitioning method is less suitable for random reads by keys, because more blocks will need to be read and more CPU performance will be required.
The access control system applies to dynamic tables:
- To read data from a dynamic table (using the
lookup_rowsqueries), the user must have the
readpermission for the table.
- To write data to a dynamic table (using the
delete_rowscommands), the user must have the
writepermission for the table.
- To run the
reshard_tablecommands, the user must have the
mountpermission for the table.
- Similar to accounts, in order to create tables linked to a bundle, the user must have the
usepermission for the bundle. The same rule is required to change the table bundle.