--- metadata: - name: generator content: Diplodoc Platform v5.50.6 alternate: - https://ytsaurus.tech/docs/en/user-guide/data-processing/chyt/reference/configuration.md - https://ytsaurus.tech/docs/ru/user-guide/data-processing/chyt/reference/configuration.md --- > **Documentation Index:** Fetch the complete configuration index at https://ytsaurus.tech/docs/en/llms.txt # Clique configuration The entire clique configuration is described by a single [YSON document](https://ytsaurus.tech/docs/en/user-guide/storage/yson-docs.md) called a *speclet*. The clique speclet is stored in Cypress at `//sys/strawberry/chyt//speclet`. You can also view its contents in the [web interface](https://ytsaurus.tech/docs/en/user-guide/data-processing/chyt/cliques/ui.md) on the **Speclet** tab or retrieve it by running the CLI command `yt clickhouse ctl get-speclet`. A speclet is a set of [options](https://ytsaurus.tech/docs/en/user-guide/data-processing/chyt/cliques/configs.md#options) that are interpreted by the controller at clique startup. In particular, the controller generates the final configuration of CHYT instances based on these options. An important difference between CHYT and original ClickHouse is that the settings are specified not in XML format, but in YSON format that is native for YTsaurus. The principle by which the settings are interpreted in this format is described [below](#clickhouse_config). ## How the configuration is formed { #how-form } The following information is used to form a configuration: 1. The basic config from Cypress: `//sys/clickhouse/config`. 2. [Options](https://ytsaurus.tech/docs/en/user-guide/data-processing/chyt/cliques/configs.md#options) described in the speclet. 3. The native cluster connection configuration taken from the contents of `//sys/@cluster_connection`. {% note warning "Attention" %} All clique instances always work with the same configuration and it is impossible to change the configuration without restarting the clique. {% endnote %} ## Advanced configuration of the instance's YTsaurus part { #yt_config } The YTsaurus part of the instance configuration resides in the `yt_config` option. With this option, you can configure advanced settings that don't exist as individual speclet options. The following sub-options are supported: - `settings`: CHYT‑specific default query [settings](https://ytsaurus.tech/docs/en/user-guide/data-processing/chyt/reference/settings.md). Changing the setting in this section changes the default value for this setting for all queries in the clique. - `table_writer`: [Table Writer configuration](https://ytsaurus.tech/docs/en/user-guide/storage/io-configuration.md#table_writer). - `table_attribute_cache`: Table attribute cache configuration. This cache significantly improves the responsiveness of CHYT, though at the moment it may lead to non‑consistent reads (when data has already appeared in the table, but CHYT does not see it yet). To disable this cache, use the following configuration: ``` {read_from=follower;expire_after_successful_update_time=0;expire_after_failed_update_time=0;refresh_time=0;expire_after_access_time=0} ``` - `create_table_default_attributes` [`{optimize_for = scan}`]: The default attributes with which tables will be created during `CREATE` queries in CHYT. - `health_checker`: The Health Checker configuration. Consists of 3 fields: — `queries` [``["select * from `//sys/clickhouse/sample_table`"]``]: A list of test queries whose performance will be checked regularly. {% note warning "Attention" %} Queries from Health Checker are currently executed on behalf of user `yt-clickhouse` who must have the `read` permissions to access the table. {% endnote %} — `period` [60000]: The test triggering period in milliseconds. — `timeout` [`0.9 * period / len(queries)`]: The triggering timeout for each of the configured queries. If the query does not fit into the timeout, it is considered `failed`. - `subquery`: The configuration of the main system part which coordinates the execution of ClickHouse queries on top of YTsaurus tables. Before configuring this part, read the article about [query execution within a clique](https://ytsaurus.tech/docs/en/user-guide/data-processing/chyt/queries/anatomy.md#query-execution). — `min_data_weight_per_thread` [64 MiB]: (in bytes) When splitting the input into subqueries, the coordinator will try to give no less than the given amount to each core of each instance. — `max_data_weight_per_subquery` [50 GiB]: (in bytes) The maximum allowable amount of data to be processed on one core of one instance. This restriction is protective and protects clique users from accidentally running a huge query that processes petabytes. The constant of 50 GiB is selected, because such amount is processed on a single core in about dozens of minutes. - `show_tables`: The `SHOW TABLES` query behavior configuration. Enables you to configure the list of directories in which `SHOW TABLES` will show a list of tables. — `roots`: The list of YPath paths of directories in Cypress from which the tables will be collected for `SHOW TABLES`. {% note warning "Attention" %} The `max_data_weight_per_subquery` limit uses columnar statistics to account for column sampling from processed tables. Columnar statistics may be missing for old tables created before the statistics became available. For such tables, column orientation is not taken account in the limit. This means that when processing a very narrow slice of columns which forms 1% of the total table volume and is 1 GiB per core, CHYT will calculate that 100 GiB per core are processed. CHYT will not run such a query with the default settings. If this scenario is required or you need to process large amounts of data, configure this setting to a random large value. {% endnote %} ## Advanced configuration of the instance's ClickHouse part { #clickhouse_config } The `clickhouse_config` option is used to set up the configuration of the ClickHouse part. This option should be set so that it matches the standard ClickHouse XML configuration. The rules for converting a ClickHouse XML configuration into a CHYT YSON configuration can be described as follows: - Any non‑multiple configuration node in the ClickHouse configuration is a dict (`map`). - A multiple node is represented by a list (`list`). Below is an example of converting an artificial XML configuration into a YSON configuration: ```xml 42 qwe 3.14 hi! ``` ```json { foo = 42; bar = "qwe"; baz = [ {quux = 3.14}; {}; "hi!"; ]; } ``` Let's examine the `clickhouse_config` sub‑options that may be of use in CHYT: - `dictionaries` (`[]`): The configuration of external dicts. The value must be a list of dict configurations. Each dict is configured by the map with the following fields that retain the meaning of the original ClickHouse configuration: - `name`: The name of the external dict. - `source`: The [data source](https://clickhouse.com/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources/) for the external dict. - `layout`: [Representation](https://clickhouse.com/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout/) of the external dict in the instance memory. - `structure`: The [schema of data](https://clickhouse.com/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-structure/) stored in the dict. - `lifetime`: The dict [lifetime](https://clickhouse.com/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-lifetime/). ## External dicts { #external-dict } CHYT supports all `layout`, `structure`, and `lifetime` settings of regular ClickHouse. {% note info "Note" %} You can specify sources from original ClickHouse as the `source`, but they are not guaranteed to work. In particular, sources that require networking should technically work, but may face a lack of network access in practice. {% endnote %} There is also an additional data source type in which you can use static tables in YTsaurus. This type is called `yt` and has one parameter: - `path`: The path to the static table on the cluster which will serve as a data source for the dict. As an example of connecting a YTsaurus table as an external ClickHouse dict, you can look at: - The CHYT [configuration example](https://ytsaurus.tech/docs/en/user-guide/data-processing/chyt/reference/configuration.md#configuration_example) that connects this table as an external ClickHouse dict. - Example of using an external dict in queries: ```sql select dictGet('OS', 'OS', toUInt64(38)) as os_name, dictGetHierarchy('OS', toUInt64(38)) as hierarchy, dictGetChildren('OS', toUInt64(101)) as children ``` ## Configuration example { #configuration_example } Below is an example of a complete configuration that can be set by the `set-speclet` command. ```json { instance_count = 1; query_settings = { extremes = 1; }; yt_config = { subquery = { max_data_weight_per_subquery = 1000000000000; }; }; clickhouse_config = { dictionaries = [ { name = OS; layout = {flat = {}}; structure = { id = {name = Id}; attribute = [ { name = "OS"; type = "Nullable(String)"; null_value = "NULL"; }; { name = "ParentId"; type = "Nullable(UInt64)"; null_value = "NULL"; hierarchical = %true; }; { name = "RootId"; type = "Nullable(UInt64)"; null_value = "NULL"; }; ]; }; lifetime = 0; source = {yt = {path = "//sys/clickhouse/dictionaries/OS"}}; }; ]; }; } ```