NBD on the YTsaurus cluster

This document describes how to configure and use NBD (Network Block Device) on the YTsaurus cluster. NBD lets you mount filesystem images from Cypress as layers of a job’s root filesystem. This speeds up environment setup, reduces disk load, and, under some conditions, lowers network load.

How NBD works

NBD (Network Block Device) is a Linux kernel mechanism that lets you mount block devices with data stored remotely. In YTsaurus, NBD is used to mount SquashFS filesystem images from Cypress as layers of a job’s root filesystem.

Architecture

Each exec-node runs an NBD server — a YTsaurus component that implements the NBD protocol over a Unix Domain Socket or TCP. Here’s how it works:

Sequence of events when preparing an NBD layer:

• The exec-node receives a task with layer_paths that includes an NBD layer.
• YT downloads the image chunk metadata, not the data itself.
• The NBD server registers an export for the image.
• The Linux kernel mounts /dev/nbdX to the export via a Unix Domain Socket.
• Porto mounts /dev/nbdX as a layer in overlayfs.
• When a job accesses a file, the kernel reads the needed blocks through /dev/nbdX → NBD server → data-nodes.

Block cache

The NBD server supports an in-memory LRU block cache to store compressed chunk data. The cache helps avoid repeated calls to data-nodes when different jobs read the same blocks. You configure the cache size with the block_cache_compressed_data_capacity parameter.

Volume cache

The exec-node caches read-only (RO) NBD volumes — mounted images. If several jobs use the same NBD layer on one exec-node, the system creates the volume once and reuses it. Cache metrics: exec_node/ronbd_volume_cache/missed_count, exec_node/ronbd_volume_cache/hit_count.

Installing packages

To work with NBD and SquashFS, install these packages:

sudo apt install nbd-client squashfs-tools

• nbd-client — a utility for manually mounting NBD devices. It’s used for diagnostics: the YTsaurus NBD server is built into the exec-node, and in normal operation the kernel connects to it directly.
• squashfs-tools — utilities mksquashfs and unsquashfs for building and checking SquashFS images.

To convert existing tar layers to SquashFS, also install squashfs-tools-ng with the tar2sqfs utility:

sudo apt install squashfs-tools-ng

Check the installation:

nbd-client --version
# This is nbd-client, from nbd 3.26.1

mksquashfs -version
# mksquashfs version 4.6.1 (2023/03/25)

NBD kernel module

To use NBD, you need to load the nbd kernel module. The nbds_max parameter defines how many NBD devices the kernel creates when loading the module. NBD devices can be created and removed dynamically.

Manually load the module:

modprobe nbd nbds_max=1024

For automatic loading after reboot (recommended):

Create the /etc/modules-load.d/nbd.conf file:

nbd

Create the /etc/modprobe.d/nbd.conf file:

options nbd nbds_max=1024

Check that the module is loaded:

lsmod | grep nbd
# nbd                    49152  0
cat /sys/module/nbd/parameters/nbds_max
# 128

Recommended nbds_max value: at least the number of job slots on the node multiplied by the maximum number of NBD layers in one job. For example, for 32 slots and 2 NBD layers per job: nbds_max=128. Devices can be created dynamically, so the value doesn’t limit operation, but having a pre-created buffer reduces overhead for device creation under load.

NBD configuration

You configure NBD via the exec-node’s dynamic config. All parameters are in the exec_node/nbd section.

Enabling NBD

exec_node:
  nbd:
    enabled: true

Full configuration example

exec_node:
  nbd:
    enabled: true
    block_cache_compressed_data_capacity: 536870912  # 512 MB
    client:
      io_timeout: 30000          # 30 seconds, in milliseconds
      reconnect_timeout: 5000    # 5 seconds, in milliseconds
      connection_count: 1
    server:
      thread_count: 2
      unix_domain_socket:
        path: /tmp/nbd.sock

Configuration parameters

Health check

Checking node status

After enabling NBD, make sure the exec-node is in the online state and has no alerts:

yt get //sys/exec_nodes/<node-address>/@state
# "online"

yt get //sys/exec_nodes/<node-address>/@alerts
# []

Checking via a test operation

Run a test operation with an NBD layer:

import yt.wrapper as yt

# Create a test SquashFS image and upload it to Cypress
# yt set //path/to/layer.squashfs/@filesystem squashfs
# yt set //path/to/layer.squashfs/@access_method nbd

yt.run_map(
    lambda row: row,
    source_table="//tmp/test_input",
    destination_table="//tmp/test_output",
    spec={
        "mapper": {
            "layer_paths": ["//path/to/layer.squashfs"],
        }
    }
)

Checking via logs

When the NBD server starts successfully, the exec-node logs (exec-node.info.log) include these entries:

NBD server started (UnixDomainSocket: /tmp/nbd.sock, ThreadCount: 2)

When an NBD device is created:

Creating NBD device (FilePath: //path/to/layer.squashfs, DeviceName: /dev/nbd0)
NBD device created (FilePath: //path/to/layer.squashfs, DeviceName: /dev/nbd0)

Monitoring

Solomon sensors

The system exports all NBD metrics to Solomon. Key sensors:

Server metrics:

Device metrics. The file_path tag is the path to the layer file in Cypress:

Volume metrics. Tags: type=nbd, file_path:

Volume cache metrics:

Key metrics for monitoring

Error handling

lsmod | grep nbd
dmesg | grep nbd

Diagnostics via Orchid

You can check the NBD server state via the exec node’s Orchid:

yt get //sys/exec_nodes/<node-address>/orchid/exec_node

Common issues and solutions

modprobe nbd nbds_max=256