YTsaurus Flow Connectors
A connector is a component that links a pipeline to YTsaurus objects (a queue, a table, and so on). Each connector provides a source for reading messages and/or a sink for writing.
Note
The connectors used by a pipeline directly affect the message processing guarantees it provides. Before you choose connectors, review the Processing Guarantees section.
Integrations with external (non-YTsaurus) systems are described in the Extensions section.
List of connectors
|
Connector |
Has source |
Has sink |
Description |
|
✓ |
✓ |
Reading from and writing to an ordered dynamic table using the Queue API |
|
|
✓ |
✓ |
Reading from static tables: a fixed set or an unlimited sequence from a directory. Writing a continuous sequence of static tables in message arrival order |
|
|
Random |
✓ |
𐄂 |
Reading random data generated on the fly. Used for testing |
|
✓ |
𐄂 |
Generating a service log based on an external state table. A service log is the generation of all keys from a dynamic table at a specified frequency; essentially, it’s a way to re-scan all states once per the configured time |
|
|
𐄂 |
✓ |
Writing to a sorted dynamic table |
Changing the source
The source identifies its partitions by parameters that describe the physical source; the specific set of these parameters is listed in the documentation for the corresponding connector. These parameters are part of the partition key, so when you change them, the source describes a different set of partitions.
When you change the source, the partitions from the old source disappear from the set and are completed (Completing ⇒ Completed): the associated state, including stored offsets, is deleted and not saved with the expectation that the same partition will return (as during an interrupt during repartitioning). The partitions for the new source are created from scratch.
Warning
The parameters that identify the source are part of the static Spec, so you can change them only when the pipeline is stopped. The new source is read independently of what the old source has already processed: if it contains the same data, the data will be written to the output streams again.