--- metadata: - name: generator content: Diplodoc Platform v5.63.3 alternate: - https://ytsaurus.tech/docs/en/flow/connectors/servicelog.md - https://ytsaurus.tech/docs/ru/flow/connectors/servicelog.md - href: https://ytsaurus.tech/docs/en/flow/connectors/servicelog.md type: text/markdown title: Markdown version - href: https://ytsaurus.tech/docs/en/llms.txt?revision=181542d8c2d9f36be925894a517db8203cafae65 rel: describedby --- > **Documentation Index:** Fetch the complete configuration index at https://ytsaurus.tech/docs/en/llms.txt?revision=181542d8c2d9f36be925894a517db8203cafae65 # Service Log in YTsaurus Flow {% note warning "Deprecated" %} Service Log is deprecated. For new code that periodically scans keys in a computation's internal or external state, use [Key Visitor](https://ytsaurus.tech/docs/en/flow/concepts/key_visitor.md) feature. {% endnote %} The connector code is available [here](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/connectors/servicelog). Use the service log to regularly scan sorted tables. Unlike other connectors, it’s inherently infinite: after a table is fully read, it starts reading it again. You can also scan multiple tables in parallel, joining them by key columns. {% note info "Important" %} Currently, the first column in the key must be a `ui64` value, evenly distributed between `0` and `(1 << 64) - 1`. Typically, you can use a hash of the row ID for this value. {% endnote %} {% note info "Important" %} You can run the source in finite mode by setting the `finite` parameter to `true`. In this mode, the table is read exactly once, or (if reading has already started) each [partition](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#partition) reads its part of the table once. This mode is mainly intended to simplify test writing. {% endnote %} ## Working with replicated tables The service log supports replicated tables. To access them directly, you need to specify all replica clusters in the table’s `clusters` attribute. To access them via the meta-cluster RPC, it’s enough to specify the meta-cluster in the `cluster` attribute. {% note info "Important" %} Access via the meta-cluster RPC is only possible with `FetchType = EFetchType::SelectRows`. {% endnote %} ## Fullstates {#fullstates} You can specify multiple fetchers in the service log. This can be used for the “fullstate” scenario. A fullstate is a table external to the [pipeline](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#pipeline) that contains data for the same [keys](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#key) for which there is a [state](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#state) in the pipeline. This table is periodically loaded into the pipeline. You can do this efficiently: when generating the service log, join the pipeline’s state table and the fullstate table by keys right away. This way, events are generated for keys that exist in at least one of the fetchers. This approach gives you two useful properties: - During a single joined pass of the service log, each state is touched once. Instead of twice — once for the main service log event and once for loading the fullstate row. - The pipeline can react to events where a key exists in the pipeline’s state but not in the external fullstate. You can distinguish the case where “all non-key columns are null” from the case where “there is no key match” using the `ispresent` system column. ## How to write tests A common mistake is writing tests for the service log in finite mode (`finite = true`) while expecting it to see data generated by processing messages from other sources. With this approach, you can’t guarantee that the service log will see the expected rows during the scan: it might scan a certain key range before the rows appear there. Instead, use one of these approaches to write tests correctly: 1. Manually load the rows you want the service log to see before starting the test. With this approach, make sure the test result doesn’t depend on whether the service log sees the other rows. 2. Run two executions: in the first, load data from other sources; in the second, run only the service log source. 3. Make the test non-finite (`finite = false`). {% note info "Important" %} When using `FetchType = EFetchType::TableReader`, you must set `enable_dynamic_store_read = true` on the table. Otherwise, the service log might not immediately see the most recent data version. This is usually not critical for production, but it breaks tests. {% endnote %} ## Source settings The [source](https://ytsaurus.tech/docs/en/flow/concepts/glossary.md#source) class is `NYT::NFlow::TServiceLogSource`. ### Static spec ## NYT::NFlow::TUnitedParameters<NYT::NFlow::TServiceLogSource> {#NYT_NFlow_TUnitedParameters_NYT_NFlow_TServiceLogSource} Source: [yt/yt/flow/library/cpp/common/registry-inl.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/registry-inl.h) #| || **Parameter** | **Description** || || `finite` | **Type**: `bool` **Default value**: `false` Treat the source as finite: transition the stream to the completed state after reading all available messages. How the end of the data is determined depends on the connector. || || `table_joiner` | **Type**: `NYT::TIntrusivePtr<`[NYT::NFlow::TTableJoinerSpec](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_TTableJoinerSpec)`>` **Default value**: `{}` Joiner spec. || |# {% cut "**Additional parameters**" %} #| || `update_info_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration) **Default value**: `15s` Period for refreshing the source's auxiliary partition information. A connector may use this tick for status requests and read-session liveness checks. || || `byte_size_alpha` | **Type**: `double` **Default value**: `0.05` Exponential smoothing coefficient for the average byte total and message count per offset: larger values make the estimate react to new data faster. || |# {% endcut %} In the static spec of the service log, you define a list of table fetchers to scan, joining them by key columns. Most often, there’s just one fetcher, and the source simply scans a single table. ## NYT::NFlow::TTableJoinerSpec {#NYT_NFlow_TTableJoinerSpec} Source: [yt/yt/flow/library/cpp/connectors/servicelog/joiner.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/connectors/servicelog/joiner.h) #| || **Parameter** | **Description** || || `fetchers` | **Type**: `std::vector>` **Default value**: `[]` Fetcher specs. || |# ## NYT::NFlow::TFetcherInJoinerSpec {#NYT_NFlow_TFetcherInJoinerSpec} Source: [yt/yt/flow/library/cpp/connectors/servicelog/joiner.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/connectors/servicelog/joiner.h) #| || **Parameter** | **Description** || || `table_path` | **Type**: `NYT::NYPath::TRichYPath` **Default value**: `TString("")` Path to the table including the cluster (multiple clusters can be specified). || || `value_columns` | **Type**: `std::optional>` Non-key columns to read (by default, all columns are read). || || `attempts` | **Type**: `long` **Default value**: `1` || || `retry_timeout` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration) **Default value**: `5s` || || `fetch_type` | **Type**: [NYT::NFlow::EFetchType](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#NYT_NFlow_EFetchType) **Default value**: `table_reader` Important parameter that affects the type of load on YT. See the EFetchType description. || || `prefix` | **Type**: `std::string` **Default value**: `TString("")` Column name prefix to be added for non-key table columns. || |# In the fetcher settings, you specify which table to scan, which columns to take, and what prefix to use for these columns in the join result (if a join is performed). ##### Minimal spec example ``` "table_joiner" = { "fetchers" = [ { "table_path" = ; }; ]; }; ``` ## NYT::NFlow::EFetchType {#NYT_NFlow_EFetchType} #| || Possible values | **Description** || || `select_rows` | Supports dynamic (including replicated) tables. Does not support read-ahead; queries go directly to tablet nodes (recommended option). || || `table_reader` | Supports dynamic and static tables. Supports read-ahead; queries go through the master node. || |# ### Dynamic spec ## NYT::NFlow::TDynamicUnitedParameters<NYT::NFlow::TServiceLogSource> {#NYT_NFlow_TDynamicUnitedParameters_NYT_NFlow_TServiceLogSource} Source: [yt/yt/flow/library/cpp/common/registry-inl.h](https://github.com/ytsaurus/ytsaurus/tree/main/yt/yt/flow/library/cpp/common/registry-inl.h) #| || **Parameter** | **Description** || || `desired_partition_count` | **Type**: `long` **Default value**: `5` Desired number of partitions. Estimate as 1 partition per 1 MB/s of stream. || || `desired_cycle_time` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration) **Default value**: `12h` The source will try to maintain a message generation rate that traverses the entire table once every `desired_cycle_time`. || |# {% cut "**Additional parameters**" %} #| || `unavailable_threshold` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration) **Default value**: `5m` How long the source must be continuously unavailable for the partition to count as stably unavailable. Only time during which the job was running and seeing the failure counts: a gap between restarts is not charged, and any successful answer from the source resets the total. || || `throttler_period` | **Type**: [TDuration](https://ytsaurus.tech/docs/en/flow/generated_docs/all_yson_structs.md#TDuration) **Default value**: `10s` Throttling period. || |# {% endcut %} ## See also - [List of connectors](https://ytsaurus.tech/docs/en/flow/connectors/about.md) - [Stateful processing](https://ytsaurus.tech/docs/en/flow/concepts/stateful.md)