Launching Spark tasks in YTsaurus

Client install

Install the ytsaurus-spyt package:

$ pip install ytsaurus-spyt

Submitting Spark applications directly to YTsaurus (available from SPYT version 1.76.0)

This method is applicable when there is no need for continuous cluster operation. It allows to use cluster resources only on demand. This method works a little bit longer comparing to inner Spark standalone cluster due to the need to launch a separate operation for every application, but it allows to release cluster resources immediately after application completes.

Direct submitting to YTsaurus is recommended in the following cases:

One-time calculations.
Low periodic tasks (less than once per hour).
Ad-hoc analytinc using spark-shell or pyspark console utilities (For analytics using Jupyter it is still needed to use inner standalone cluster).

To use this method follow these steps:

Activate SPYT configuration with source spyt-env command.
Upload an application executable file and it's dependencies to Cypress.
Submit the application with the following command:

$ spark-submit --master ytsaurus://<cluster-name> --deploy-mode cluster --num-executors 5 --queue research yt:/<path to .py file or .jar file>

Options:

--master — cluster proxy address;
--queue — the name of the scheduler pool where the task should be launched

The usage of other options corresponds to their description in spark-submit documentation (the full list is available with spark-submit --help command). Almost all of the options can be used except these:

--files, --archives — do not work with local files, you can only use those that have been pre-loaded to Cypress.

Selecting RPC proxy roles

For a regular launch, use the command above without additional role parameters. If no configuration source sets a role, discovery requests default. The cluster must have available RPC proxies with that role.

If you need other proxy groups, pass the role parameters via --conf. Ask your cluster administrator which roles are available. The connection model explains which clients use discovery and which can use the local job proxy.

In the templates below, replace yt.example.com with your cluster address and <other application parameters> with the launch parameters and application path, as in the command above. Replace spark-client and spark-cluster with roles accessible from the client host and cluster jobs, respectively.

Explicitly select the default role

spark-submit \
  --master ytsaurus://yt.example.com \
  --deploy-mode cluster \
  --conf spark.hadoop.yt.proxyRole=default \
  <other application parameters>

An explicit proxyRole=default overrides this parameter in the release and global configurations, such as custom in //home/spark/conf/global. If you omit it, the value from the next source by priority remains in effect. A separately configured clusterProxyRole is retained.

Different roles for the client and cluster

In this example, the application submission client uses spark-client. When connecting through discovery, the driver and executors inside the cluster use spark-cluster:

spark-submit \
  --master ytsaurus://yt.example.com \
  --deploy-mode cluster \
  --conf spark.hadoop.yt.proxyRole=spark-client \
  --conf spark.hadoop.yt.clusterProxyRole=spark-cluster \
  <other application parameters>

A separate executor role in client mode

If the executors need a separate role while the driver runs on the client host, pass both roles when launching:

spark-submit \
  --master ytsaurus://yt.example.com \
  --deploy-mode client \
  --conf spark.hadoop.yt.proxyRole=spark-client \
  --conf spark.hadoop.yt.clusterProxyRole=spark-cluster \
  <other application parameters>

In this mode, pass the separate executor role via --conf or SparkConf. For the reason, see Applying the cluster role.

Launching inner Spark Standalone cluster

This method is applicable for intensive cluster usage. In this mode YTsaurus allocates resources for inner Spark standalone cluster which is used to launch Spark applications. This mode is recommended in the following cases:

Launching high-frequent tasks (more than once per hour). The efficiency is achieved because the task startup time in standalone cluster is significantly less than YTsaurus operation startup time.
Ad-hoc analytics in Jupyter notebooks.
Ad-hoc analytics using Query Tracker and Livy in SPYT versions below 2.10.0. Starting with SPYT 2.10.0 and Query Tracker 0.4, use SPYT Connect.

For launching an inner standalone Spark cluster follow these steps:

Select an account you will use to start your cluster. You will need to upload any code that regularly runs on Spark to the YTsaurus system. The account used to start the cluster must have enough privilege to read the code.
Create a directory for Spark housekeeping data, such as my_discovery_path. The account used to start the cluster must have write privileges to the directory. Users that will run Spark jobs must have read access to the directory.
Start your cluster:
```
$ spark-launch-yt \
--proxy <cluster-name> \
--pool  my_pool \
--discovery-path my_discovery_path \
--worker-cores 16 \
--worker-num 5 \
--worker-memory 64G
```
Options:
- spark-launch-yt — Start the Vanilla YTsaurus transaction from a client host.
- --proxy — Cluster name.
- --pool — YTsaurus computational pool.
- --spyt-version — Spark housekeeping data directory.
- --worker-cores — Number of worker cores.
- --worker-num — Number of workers.
- --worker-memory — Amount of each worker's memory.
- --spark-cluster-version — Cluster version (optional).
Start a test job on your cluster:
```
$ spark-submit-yt \
--proxy <cluster-name> \
--discovery-path my_discovery_path \
--deploy-mode cluster \
yt:///sys/spark/examples/smoke_test.py
```
Options:
- spark-submit-yt — spark-submit wrapper that enables you to find out the Spark master address from the Vanilla transaction. The search uses proxy, id, and discovery-path as arguments.
- --proxy — Cluster name.
- --discovery-path — Spark housekeeping data directory.
- --deploy-mode (cluster or client) — Cluster startup mode.
- --spyt-version — SPYT version (optional).
- Address of the file with the code in YTsaurus.

Use

spark-launch-yt

$ spark-launch-yt \
--proxy <cluster-name> \
--pool my_pool \
--discovery-path my_discovery_path \
--worker-cores 16 \
--worker-num 5 \
--worker-memory 64G \
--spark-cluster-version 2.11.0

spark-discovery-yt

Retrieving links to the UI master, transaction, Spark History Server:
```
$ spark-discovery-yt \
--proxy <cluster-name> \
--discovery-path my_discovery_path
```

spark-submit-yt

$ spark-submit-yt \
--proxy <cluster-name> \
--discovery-path my_discovery_path \
--deploy-mode cluster \
--spyt-version 2.11.0 \
yt:///sys/spark/examples/smoke_test.py

Note

You can set environment variables to use instead of some of the command arguments, such as YT_PROXY instead of --proxy.

$ export YT_PROXY=<cluster-name>

$ spark-submit-yt \
--discovery-path my_discovery_path \
--deploy-mode cluster \
yt:///sys/spark/examples/smoke_test.py

Additional parameters

For additional cluster startup parameters, see Starting a Spark cluster.

Overview

Definitions in SPYT