Job file environment: layers and files

This section describes how to work with the job file environment in YTsaurus. You’ll learn what layers and individual files are, how to define them in the operation specification, how to prepare and upload them to Cypress, and how to speed up layer reads and diagnose issues.

What is the file environment

A job’s file environment is a set of directories and files where the user process runs. The environment is represented as a root file system: a tree of directories and files, similar to any Unix system.

To run code, you need dependencies: interpreters, libraries, binary files, configs, and data. If your code accesses a dependency — a directory or file that isn’t in the environment — an error occurs. That’s why you build the environment for a specific task.

The file environment consists of two parts:

• Root file system — built from layers listed in the layer_paths parameter. Suitable for a persistent environment that you reuse across operations.
• Individual files — artifacts in the /slot/sandbox subtree, listed in the file_paths parameter. Suitable for one‑time delivery of specific files to a job, such as binaries or configs.

The file environment is formed by the execution environment — the software environment where the job runs. The execution environment provides access to the file system and runs your code in it. For details on the available environments and how they differ, see the Execution environments section.

Layers

What is a layer

A layer is a file that contains a subset of the root file system. The subset is one or more directory subtrees with their nested structure preserved. Physically, a layer is stored as a single file that you upload to YTsaurus.

The simplest example of a layer is an archive. You prepare the environment on your machine, pack it into an archive, and upload it to YTsaurus. During operation execution, the system expands the archive and forms the job’s file environment from it.

How the environment is built from layers

You can build the root file system from a single layer or from multiple layers. Layers from layer_paths are merged into a single root file system using overlayfs — a Linux kernel mechanism that combines several file systems into one.

The root file system is built on the exec node during job preparation, before your code starts. Containerization — Porto or CRI — makes the assembled file system the root / of the container and runs your code in it. Without containerization, layers aren’t applied as a root file system.

How a layer maps to the tree

Each layer represents a part of the tree — from the top level down. The top level of a layer includes folders and files that sit directly in it, not nested inside other folders. After assembly, they appear directly at the root /.

Everything nested inside them ends up at lower levels — inside the corresponding root folders.

For example, if the archive’s top level contains the usr, bin, and home folders, they become /usr, /bin, /home after expansion. Their nested contents become /usr/lib, /bin/bash, and so on.

A single layer can contain multiple subtrees at once. For example, one archive can include both /usr and /var.

Base and delta layers

Most environments are built using the “base layer + delta” scheme. It separates the unchanging foundation from your custom files:

• Base layer — a layer with the core part of the environment: system utilities and libraries, such as /bin/bash and libc6. It’s often a Linux distribution image, for example Ubuntu, with packages and libraries installed.
• Delta layer — an addition to the base layer with your own libraries or binary artifacts.

A common configuration is one base layer and one or more delta layers: you use a ready‑made image as the base and put your custom files in the delta.

Layer formats

YTsaurus supports two layer formats: tar archive and SquashFS. For SquashFS, network access via NBD is also supported. The table below shows how the formats differ and when to use each:

Tar archive

A tar archive is a layer in the form of a .tar, .tar.gz, .tar.xz, or .tar.zstd archive with a subset of the file system.

The archive is convenient for several reasons:

• You can easily create it with the standard tar utility.
• You can easily inspect its contents: download the archive and unpack it.
• It works in any container environment.

During job preparation, the archive is downloaded to the exec node and unpacked to disk. Unpacking loads the CPU and Disk I/O, and the unpacked layer takes up disk space.

SquashFS

SquashFS is a file system image that is mounted on the exec node without unpacking. The mount point is then used as a layer in overlayfs. Mounting is much faster than unpacking an archive.

Compared to a tar archive, a SquashFS layer offers these advantages:

• Speeds up job preparation — you don’t need to unpack the image.
• Reduces disk load — a valuable resource on clusters.
• Saves disk space — the image isn’t stored in an unpacked form.

SquashFS over NBD

NBD (Network Block Device) is a way to use a SquashFS image over the network. Image data is read from data nodes as needed, without pre‑downloading to the exec node.

A local SquashFS layer is first fully downloaded to the exec node. NBD reads only the blocks that the job accesses from the data nodes. This way, NBD avoids the resources required by a local layer:

• CPU and Disk I/O for writing the image to disk.
• Disk space for storing the image.
• Network I/O for transferring image parts that the job doesn’t read.

Already‑read blocks are cached at multiple levels: in the kernel’s page cache, in the chunk reader caches, and in the caches on data nodes. Subsequent accesses to the same data are served from the cache without new network reads. This can make NBD layers significantly faster than regular layers.

NBD works on top of a SquashFS image. This format compresses data and is read‑only, which matches the purpose of a layer: the layer doesn’t change while the job runs.

Execution environments

The set of available layer formats depends on the execution environment configured on the cluster in the exec_node/slot_manager/job_environment/type parameter:

Individual files

What is an individual file

An individual file is an artifact from Cypress that is delivered to the job directly, without building a layer. Files are passed via the file_paths parameter and end up in the /slot/sandbox subtree of the root file system. In effect, the files supplement the root file system’s file environment.

Individual files are suitable for delivering configs, binaries, and models to a job. You place the file yourself and know which path to use in your code to access it.

Placing files in /slot/sandbox

Each file from file_paths ends up in the /slot/sandbox subtree of the root file system. By default, the file is available to the job at /slot/sandbox/<file name>, where the file name matches the original name from Cypress.

To place a file elsewhere in the root file system — for example, in /bin or /usr/lib — you must add it via a layer. For more details, see the Files vs. layers section.

Files vs. layers

Files and layers solve different tasks:

• Files via file_paths are suitable for delivering non‑system configs, models, and binaries. You can’t use files to deliver, for example, shared C/C++ libraries or system configs: files go into the /slot/sandbox subtree and can’t be placed anywhere else in the root file system.
• Layers via layer_paths are needed to build a system environment. This includes everything required to run your program: system libraries, the dynamic linker, system configs, and applications. A layer can contain files in any directory of the root file system — /usr, /bin, /lib, and others.

If you need to place a file outside /slot/sandbox — for example, a library in /usr/lib — build a delta layer and specify it in layer_paths. A library often can’t be in an arbitrary location: the system requires it to be at a specific path, or it won’t work.

How to set the job’s file environment in the specification

You set the job’s file environment in the operation specification using two parameters: layer_paths for root filesystem layers, and file_paths for individual files.

The layer_paths parameter

You pass layers to the operation via the layer_paths parameter—a list of paths to layers in Cypress. Here’s a minimal example with a single-layer environment:

import yt.wrapper as yt

spec = {
    "mapper": {
        "layer_paths": [
            "//path/to/my_layer",
        ]
    }
}

yt.run_map(mapper, source_table, destination_table, spec=spec)

Layer order

List the layers in layer_paths from top to bottom: delta layers first, the base layer last.

Layer order matters only when layers overlap in data—when they contain files or folders at the same path. In case of overlap, the top layer takes priority: the root filesystem uses the file from the top layer.

That’s why files from the delta override files from the base layer, even if a file with the same path exists in both.

If there’s no overlap, the order doesn’t matter. For example, when one layer contains /usr and another contains /var.

"layer_paths": [
    "//path/to/delta_layer",   # Top layer, delta
    "//path/to/base_layer",    # Bottom layer, base
]

If a file exists at the same path in both layers, the system uses the file from delta_layer. It takes not only the file’s contents from the top layer but also its metadata: access rights, owner and group, and timestamps.

Layer attributes

The layer format and access method are defined by the file attributes in Cypress:

For a SquashFS layer with NBD access, you can specify access_method directly in the layer path:

spec = {
    "mapper": {
        "layer_paths": [
            "<access_method=nbd>//path/to/my_layer.squashfs",  # SquashFS via NBD
            "//path/to/porto_layer",  # Base tar layer
        ]
    }
}

The access_method attribute in the specification is available starting from version 25.1. It’s handy when you need to use the same layer with different access methods or when you don’t have permissions to change the file’s attributes.

Default layer

If you don’t specify layer_paths, the cluster’s default image is used. Typically, it’s a minimal base Ubuntu image.

The file_paths parameter

You pass files to the operation via the file_paths parameter—a list of paths to files in Cypress with attributes. Each file goes into the /slot/sandbox subtree.

File name conflicts

If two files in file_paths have the same name and you don’t set the file_name attribute, the last file overwrites the previous one. Set a unique file_name for each file.

File attributes

You can set attributes for each file in file_paths:

Two attributes are key. The $value attribute specifies the path in Cypress where to get the file from. The file_name attribute specifies the path in the job’s file environment where to place it.

Here’s an example specification with file attributes:

spec = {
    "file_paths": [
        {
            "$value": "//path/to/nirvana-bundle.tar.gz",
            "$attributes": {
                "file_name": "nv_tmpfs/sys/static/nirvana-bundle.tar.gz",
                "executable": False,
                "bypass_artifact_cache": True,
            }
        },
        {
            "$value": "//path/to/job_launcher_native",
            "$attributes": {
                "file_name": "nv_tmpfs/sys/static/job_launcher_native",
                "executable": True,
                "bypass_artifact_cache": True,
            }
        },
    ],
    "mapper": {
        "layer_paths": [
            "//path/to/porto_layer",
        ]
    }
}

In this example, nirvana-bundle.tar.gz will be available at /slot/sandbox/nv_tmpfs/sys/static/nirvana-bundle.tar.gz. The job_launcher_native file will be at /slot/sandbox/nv_tmpfs/sys/static/job_launcher_native with execute permission.

Default file_paths

If you don’t specify file_paths, the /slot/sandbox subtree remains without user files.

Environment preparation order

The job’s environment is built in two steps:

1. Prepare the root filesystem. The system downloads the layers from layer_paths, mounts them using overlayfs, and builds the root filesystem.
2. Deliver the files. The system downloads the files from file_paths and places them in the /slot/sandbox subtree of the root filesystem.

Your custom code runs only after both steps are complete.

Specification examples

You can combine layers and files in a single specification. For example, use a base tar layer in layer_paths and additional artifacts in file_paths:

spec = {
    "file_paths": [
        {
            "$value": "//path/to/executor-config.yaml",
            "$attributes": {
                "file_name": "executor-config.yaml",
                "bypass_artifact_cache": True,
            }
        },
        {
            "$value": "//path/to/my_binary",
            "$attributes": {
                "file_name": "my_binary",
                "executable": True,
            }
        },
    ],
    "mapper": {
        "layer_paths": [
            "//path/to/porto_layer",
        ]
    }
}

The files will be available at /slot/sandbox/executor-config.yaml and /slot/sandbox/my_binary.

Prepare and upload layers and files to Cypress

Prepare layers

Create a tar archive

You build the archive, for example, with the tar command:

tar czf delta_layer.tar.gz configs data bins

Create a SquashFS image

There is no separate “NBD image”: the same SquashFS image is used for NBD access. The layer format and access method are defined not by the file extension, but by the attributes when you upload to Cypress. For more details, see the section Upload layers to Cypress.

Build the image from a folder with mksquashfs:

sudo apt install squashfs-tools
mkdir ~/mnt
# Populate ~/mnt with the required content
mksquashfs ~/mnt /tmp/my_layer.squashfs

If you already have a tar layer, you can convert it to SquashFS with tar2sqfs:

sudo apt install squashfs-tools-ng
tar2sqfs ~/dict.squashfs < ~/dict.tar.gz

When you build the image, you can set the SquashFS block size — a key parameter for performance when accessing via NBD. For more details, see the section Optimize NBD layers.

Upload layers to Cypress

For both tar layers and SquashFS images, you can set the storage medium ssd_blobs and the number of replicas — this speeds up layer downloading to exec nodes. For more details, see the section How to speed up layer reads.

Upload a tar layer

yt write-file //path/to/layer.tar.gz < ~/layer.tar.gz

If the filesystem format isn’t specified, the layer is treated as a tar archive.

Upload a SquashFS image

When you upload a SquashFS image, specify the filesystem format with the @filesystem attribute:

yt write-file //path/to/layer.squashfs < ~/layer.squashfs
yt set //path/to/layer.squashfs/@filesystem squashfs
yt set //path/to/layer.squashfs/@access_method local

To use a SquashFS image over the network, set the @access_method attribute to nbd:

yt write-file //home/user/layers/base.squashfs < ~/base.squashfs
yt set //home/user/layers/base.squashfs/@filesystem squashfs
yt set //home/user/layers/base.squashfs/@access_method nbd

After that, you can use the layer in operations:

map_spec = {"mapper": {"layer_paths": ["//home/user/layers/base.squashfs"]}}

yt.wrapper.run_map(
    Mapper(),
    source_table=args.input_table,
    destination_table=args.output_table,
    spec=map_spec,
)

Access methods:

• access_method=local — the image is downloaded to the exec node and mounted locally.
• access_method=nbd — the image is read over the network via NBD.

At any given time, the image is used in only one mode: local or nbd.

Prepare files

Before you upload files to Cypress, prepare them on your local machine:

• Gather the required files: scripts, binaries, configs, and models.
• Check file integrity: compare hashes or checksums if the files were downloaded from external sources.
• Check access rights: files that you plan to run must be executable.
• Organize the files into a logical directory structure, for example /scripts, /configs, /models.

Upload files to Cypress

Files for file_paths are uploaded to Cypress in the same way as layers. You can also configure storage parameters.

Choose a medium

You can increase file download speed by using the appropriate medium and setting the primary_medium attribute. Store frequently used files on SSD and rarely used files on HDD:

yt create --type file \
    --attributes '{primary_medium=ssd_blobs;account=sys;}' \
    --path //path/to/my_file
yt write-file //path/to/my_file < ~/my_file

For an existing file:

yt set //path/to/my_file/@primary_medium ssd_blobs
yt set //path/to/my_file/@account sys

Number of replicas

You can increase file download speed by setting the number of replicas with the replication_factor attribute. The default is 3, and the maximum is 20:

yt create --type file \
    --attributes '{replication_factor=10;}' \
    --path //path/to/my_file
yt write-file //path/to/my_file < ~/my_file

For an existing file:

yt set //path/to/my_file/@replication_factor 10

Naming and organization recommendations

To simplify file management in Cypress:

• Use folders based on purpose or version: //home/<user>/layers/, //home/<user>/configs/.
• Give files descriptive names so you can understand their content without opening them.
• When you update a file, create a new version in a separate folder instead of overwriting the existing one — this way, you can roll back if an error occurs.

Check availability

After uploading, check that the file is available:

yt list //path/to/
yt get //path/to/my_file/@size

Practical scenarios

Base layer and files via file_paths

Use this scenario when you need to add a few artifacts — configs, binaries, or models — to the standard environment without creating a new layer.

spec = {
    "file_paths": [
        {
            "$value": "//path/to/config.yaml",
            "$attributes": {
                "file_name": "config.yaml",
            }
        },
        {
            "$value": "//path/to/my_binary",
            "$attributes": {
                "file_name": "my_binary",
                "executable": True,
            }
        },
    ],
    "mapper": {
        "layer_paths": [
            "//path/to/porto_layer",
        ]
    }
}

The files will be available at the paths /slot/sandbox/config.yaml and /slot/sandbox/my_binary.

Delta layer instead of multiple file_paths

Use this scenario, for example, when you need to bring complex dependencies along with the files: pip, apt, conda, and shared libraries.

Build a delta layer from your files:

mkdir -p ~/delta/usr/lib ~/delta/usr/bin
cp ~/my_library.so ~/delta/usr/lib/
cp ~/my_binary ~/delta/usr/bin/
mksquashfs ~/delta /tmp/delta_layer.squashfs
yt write-file //path/to/delta_layer.squashfs < /tmp/delta_layer.squashfs
yt set //path/to/delta_layer.squashfs/@filesystem squashfs
yt set //path/to/delta_layer.squashfs/@access_method nbd

Use the delta layer in the operation together with the base layer:

spec = {
    "mapper": {
        "layer_paths": [
            "<access_method=nbd>//path/to/delta_layer.squashfs",  # Delta layer
            "//path/to/porto_layer",  # Base layer
        ]
    }
}

Files in the delta layer override files in the base layer when the paths match.

Optimization and diagnostics

Job environment preparation stages

Preparing a job's environment involves several stages. Each stage takes time and consumes resources. You can find preparation metrics in the YTsaurus web interface:

• On the operation page: go to the Jobs tab → Statistics column.
• On the job page: open the Job statistics section → Prepare stage.

Key job environment preparation stages:

Stage analysis

To find the bottleneck, compare the time spent on each stage and analyze outliers:

• Open the operation page and go to the Jobs tab.
• Find jobs with the longest preparation time — sort by the Prepare time column.
• Open the job and check the stage breakdown in the Job statistics section.
• Compare the stage times: the stage with the maximum time is the bottleneck.
• Apply the optimization methods from the table above to the identified stage.

Methods to optimize job environment preparation stages

If the Preparing root volume stage takes the most time, switching from tar to SquashFS over NBD gives the biggest performance gain. If it's Downloading artifacts, increase the number of replicas and check that the layer is stored on SSD. If it's Preparing artifacts, switch from tar to SquashFS to avoid unpacking.

Troubleshooting

RootVolumePreparationFailed

This error occurs when mounting a layer fails during root file system preparation. Possible causes:

• The layer image is corrupted.
• The file system format is incorrect — @filesystem.
• The NBD server is unavailable or not configured on the cluster.

NBD‑specific causes are listed in the Common NBD errors section.

NbdError

This error occurs when reading from an NBD device fails during job execution, for example, due to a connection drop with the data node. The job is automatically aborted and restarted. If errors keep recurring, the operation ends with an error.

NBD volume creation error

This error occurs when creating an NBD volume on the exec node fails. Possible causes:

• The NBD server isn't running or is unavailable on the exec node.
• The limit for NBD devices on the exec node is exceeded.
• There aren't enough resources to create the volume.

Contact your cluster administrator for diagnostics.

File permission error

If a file ends up in /slot/sandbox but your user code can't run it, check the executable attribute. By default, it's set to false. Set executable: true for executable files.

File name conflict in /slot/sandbox

If two files in file_paths have the same name and the file_name attribute isn't set, the last file overwrites the previous one. Set a unique file_name for each file.

Disk space shortage

Unpacked tar layers take up space on the exec node disk. Switch to SquashFS: the image is mounted without unpacking and isn't stored in an unpacked form.

Artifact download timeout

If the Downloading artifacts stage takes too long, increase replication_factor and check that the layer is stored on SSD. More replicas let you read data in parallel from different data nodes.

Quota and resource issues

If a job ends with a resource shortage error — CPU, memory, or disk — check the limits in the operation specification. Increase the cpu_limit, memory_limit, or disk_request parameters if needed. Also check that unpacked tar layers aren't taking up too much space: switch to SquashFS to avoid storing the image in an unpacked form.

How to speed up layer reads

Store layers on SSD

Reading from SSD is much faster than from HDD. This is especially important for NBD: data is read interactively from data nodes as the job accesses it, and HDD drives are too slow for this.

# When creating a file
yt create --type file \
    --attributes '{primary_medium=ssd_blobs;account=sys;}' \
    --path //path/to/layer.squashfs
yt write-file //path/to/layer.squashfs < ~/layer.squashfs

# For an existing file
yt set //path/to/layer.squashfs/@primary_medium ssd_blobs
yt set //path/to/layer.squashfs/@account sys

The migration to SSD happens in the background and may take time.

Increase the number of replicas

For NBD, three replicas can become a bottleneck when many jobs use the same layer simultaneously. More replicas let you read data in parallel from different data nodes. You pay for more replicas with extra storage space.

# When creating a file
yt create --type file \
    --attributes '{replication_factor=20;}' \
    --path //path/to/layer.squashfs
yt write-file //path/to/layer.squashfs < ~/layer.squashfs

# For an existing file
yt set //path/to/layer.squashfs/@replication_factor 20

By default, replication_factor=3; the maximum is 20.

Use the chunk cache

For tar and local SquashFS layers with access_method=local, the exec node caches downloaded layers in the chunk cache. When you rerun jobs with the same layers, they're taken from the cache without downloading from data nodes.

Two factors affect cache efficiency:

• Standard base layers are highly likely to be already cached on exec nodes.
• Changing the layer file invalidates the cache — the cached copy stops being used.

NBD layers

Optimizing NBD layers

SquashFS block size

The SquashFS file system works with blocks. By default, the block size is 128 KB. The larger the block, the more data the kernel reads at once and the fewer NBD requests are sent to data nodes.

Set the block size when building the image:

mksquashfs ~/mnt /tmp/my_layer.squashfs -b 1M

For example, for self‑driving systems, switching from a 128 KB block to a 1 MB block sped up md5sum calculation for a large binary from 250–350 seconds to 40 seconds.

Chunk block size

Don't confuse the SquashFS block size with the chunk block size. A file in Cypress consists of chunks, and a chunk is read from data nodes in blocks. By default, the chunk block size is 16 MB.

The kernel reads data from NBD devices in small chunks, usually no more than 8 KB. So, with NBD access, a smaller chunk block size — 512K, 1M, or 2M — reduces the volume of unnecessary reads.

Set the block size when uploading the file:

yt write-file \
    --file-writer '{block_size=2097152;}' \
    //path/to/layer.squashfs < ~/layer.squashfs

For example, for tasklets, reducing the chunk block size from 16 MB to 2 MB noticeably lowered the read volume.

NBD monitoring and diagnostics

Solomon sensors

The following NBD metrics are available on exec nodes.

Server metrics:

Device metrics. The file_path tag is the path to the layer file:

Volume metrics. The tag is type=nbd or type=squashfs:

Volume cache metrics:

Common NBD errors

Below are errors specific to NBD. General job environment preparation errors RootVolumePreparationFailed and NbdError are described in the Troubleshooting section.

NBD server is not present

This error occurs when you try to use NBD on a cluster where it isn't enabled, or in an environment other than the porto environment. Contact your cluster administrator to enable NBD.

Incorrect layer attributes

To access a layer over NBD, it must have the @filesystem=squashfs and @access_method=nbd attributes set. If the file system format is specified incorrectly, mounting the layer will fail with the RootVolumePreparationFailed error.