SPYT Connect

SPYT Connect is a remote connection mechanism for a Spark driver, built on the Spark Connect protocol. It lets you run Spark SQL queries via Query Tracker in YTsaurus. You can also work with data directly from Python code via the Spark Connect API, without installing a JVM on the client side.

When query latency may occur

SPYT Connect starts the Spark driver on demand. If the driver is not active at the moment, you need to start it first — this takes time. The driver may be inactive in three cases:

• First access to SPYT Connect.
  You’re just getting started, and the driver launches from scratch.
• After idle time.
  To avoid wasting resources, the driver automatically stops after 10 minutes of inactivity. The next query will trigger its launch again. The idle timeout is configured via the spark.ytsaurus.connect.idle.timeout parameter in the SPYT configuration settings.
• After changing settings.
  If you change the resource configuration for a query (for example, the number of cores), the old driver stops, and a new one starts with the updated settings.

When you notice initialization delays depends on how you work with SPYT Connect:

• In Query Tracker (UI or API), session launch and sending a query to SPYT Connect happen within a single QT query. So when you launch it in the interface or send a query via the API, it seems that the query itself takes a long time to run. All subsequent queries will run quickly.
• In Python (Spark Connect API), you explicitly control the launch. The delay will occur exactly when you call the start_connect_server function (waiting for readiness). The actual computations and DataFrame operations will start without delays.

Launch modes

SPYT Connect operates in two modes — they determine how the Spark application is launched. The mode you choose affects the configuration and code in all connection methods.

Connection methods

Via Query Tracker UI

To run an SQL query using SPYT Connect:

1. Open the Queries tab in the YTsaurus interface.
2. Select SPYT from the engine list.
3. Enter an SQL query.
4. In the Settings field, specify the configuration in JSON format.
5. Click Run and wait for the result.

To work with an internal Spark cluster, add discovery_path — the path to the running cluster. The cluster must run on SPYT 2.9.0 or higher:

{
  "discovery_path": "//home/spark/my-cluster"
}

Via Query Tracker API

The example below shows how to send an SQL query via the API and read the result:

from yt.wrapper import YtClient, start_query, get_query_result, read_query_result

client = YtClient(proxy="<cluster-proxy>", token="<your-token>")

settings = {
    "cluster": "<cluster-name>",
    "spark_conf": {
        "spark.cores.max": "4"  # Spark-native parameter: maximum number of cores for the entire application
    }
}

# For an internal cluster, add discovery_path:
# settings["discovery_path"] = "//home/spark/my-cluster"

query_id = start_query(
    "spyt",
    "SELECT * FROM yt.`//home/my-table`",
    settings=settings,
    client=client
)

# Get metadata (for example, data schema)
result_meta = get_query_result(query_id=query_id, result_index=0, client=client)

# Iterate over the result
result = read_query_result(query_id=query_id, result_index=0, client=client)
for row in result:
    print(row)

For a description of configuration parameters, see the Configuration parameters section.

Via Spark Connect API

The main difference from classic Spark is only in how the session is created; the rest of the code remains the same.

The examples below show how to create a Spark session in each launch mode:

import spyt
from yt.wrapper import YtClient
from spyt.connect import start_connect_server, wait_for_spark_connect_endpoint
from pyspark.sql import SparkSession

client = YtClient(proxy="<cluster-proxy>", token="<your-token>")

operation = start_connect_server(client)
endpoint = wait_for_spark_connect_endpoint(client, operation.id)

try:
    spark = SparkSession.builder.remote(f"sc://{endpoint}").getOrCreate()
    df = spark.read.format("yt").load("yt:///home/my-table")
    df.show()
finally:
    if spark:
        spark.stop()

import spyt
from yt.wrapper import YtClient
from spyt.connect import start_connect_server_inner_cluster
from pyspark.sql import SparkSession

client = YtClient(proxy="<cluster-proxy>", token="<your-token>")

endpoint = start_connect_server_inner_cluster(client, discovery_path)

try:
    spark = SparkSession.builder.remote(f"sc://{endpoint}").getOrCreate()
    df = spark.read.format("yt").load("yt:///home/my-table")
    df.show()
finally:
    if spark:
        spark.stop()

QT configuration parameters

Parameters apply only when working via Query Tracker and are passed in JSON format — in the Settings field (UI) or in the settings dictionary (API).

Migrating from Livy

Starting from SPYT 2.10.0 and Query Tracker 0.4, integration via Livy is no longer supported. SPYT Connect replaces it.

The main difference from Livy: in Livy, all drivers ran on the same machine as the server, which limited the number of simultaneous sessions and didn’t allow flexible resource configuration. In SPYT Connect, each user launches a Spark driver as a separate YTsaurus operation with an individual configuration.

Versions and compatibility

The Spark Connect protocol appeared in Spark 3.4, so earlier Spark versions are not supported. The current SPYT Connect works starting from Spark 3.5.x.

What’s next

• Direct submit
• Internal SPYT cluster
• SPYT configuration parameters