Getting started with YTsaurus Flow

This section walks you through the steps to implement and run your own pipeline in Flow.

Choose a language

Flow supports several languages for implementing business logic:

  • C++ — native implementation, maximum performance, full control. Use this for high-load pipelines.
  • Java and Kotlin — run via the companion mechanism. They support Spring Boot. These are suitable for teams with a JVM stack.
  • Python — runs via the companion mechanism. This is the easiest way to prototype a pipeline or process a small data stream.
  • Go — runs via the companion mechanism. A single binary runs the pipeline and acts as a companion in the job. Suitable for teams with a Go stack.
  • YQL — declarative pipeline description as an SQL query. It has a low entry barrier and doesn’t require writing code in C++, Java, Kotlin, Go, or Python. It’s under active development, and not all planned features are available yet.

General plan

No matter which language you choose, creating a pipeline involves the following steps:

  1. Try the Quick start — run the C++ Word Count pipeline to get familiar with the Flow infrastructure.

  2. Review the basic concepts. Read the glossary to understand the Flow model: pipelines, streams, computations, and messages.

  3. Study the concepts. Get to know Computation, Watermarks and Timers, and Stateful processing, as well as the guarantees provided by the system.

  4. Explore examples in your chosen language:

  5. Check out the available connectors — queues, static tables, and others.

  6. Describe the pipeline spec in YSON format. In addition to the examples, the Spec & DynamicSpec section will help you.

  7. Implement your business logic in the language you’ve chosen, following the relevant quick start guide.

  8. Create the necessary objects in YTsaurus — tables, queues, and the pipeline.

  9. Write tests. Follow the instructions for your programming language:

  10. Run the pipeline and inspect its state with yt flow get-pipeline-state <pipeline_path>. For release rules, read Release and table change rules.

See also

Previous
Overview
Next
Quick start