NBD on the YTsaurus cluster

This document describes how to configure and use NBD (Network Block Device) on the YTsaurus cluster. NBD lets you mount filesystem images from Cypress as layers of a job’s root filesystem. This speeds up environment setup, reduces disk load, and, under some conditions, network load.

How NBD works

NBD (Network Block Device) is a Linux kernel mechanism that lets you mount block devices whose data is stored remotely. In YTsaurus, NBD is used to mount SquashFS filesystem images from Cypress as layers of a job’s root filesystem.

Architecture

An NBD server runs on each exec node. It’s a YTsaurus component that implements the NBD protocol over a Unix Domain Socket or TCP. Here’s how it works:

Sequence of events when preparing an NBD layer:

• The exec node receives a task with layer_paths that includes an NBD layer.
• YT downloads the image chunk metadata, not the data itself.
• The NBD server registers an export for the image.
• The Linux kernel mounts /dev/nbdX to the export via a Unix Domain Socket.
• Porto mounts /dev/nbdX as a layer in overlayfs.
• When the job accesses a file, the kernel reads the needed blocks through /dev/nbdX → NBD server → data nodes.

Block cache

The NBD server maintains an in-memory LRU block cache to store compressed chunk data. The cache helps avoid repeated requests to data nodes when different jobs read the same blocks. You configure the cache size with the block_cache_compressed_data_capacity parameter.

Volume cache

The exec node caches read-only (RO) NBD volumes, which are mounted images. If multiple jobs use the same NBD layer on one exec node, the system creates the volume once and reuses it. Cache metrics: exec_node/ronbd_volume_cache/missed_count, exec_node/ronbd_volume_cache/hit_count.

Installing packages

Install these packages to work with NBD and SquashFS:

sudo apt install nbd-client squashfs-tools

• nbd-client — a utility for manually mounting NBD devices. You use it for diagnostics: the YTsaurus NBD server is built into the exec node, and in normal operation the kernel connects to it directly.
• squashfs-tools — utilities mksquashfs and unsquashfs for building and checking SquashFS images.

To convert existing tar layers to SquashFS, also install squashfs-tools-ng with the tar2sqfs utility:

sudo apt install squashfs-tools-ng

Verify the installation:

nbd-client --version
# This is nbd-client, from nbd 3.26.1

mksquashfs -version
# mksquashfs version 4.6.1 (2023/03/25)

The nbd kernel module

You need to load the nbd kernel module for NBD to work. The nbds_max parameter defines how many NBD devices the kernel creates when loading the module. You can create and delete NBD devices dynamically.

Manually load the module:

modprobe nbd nbds_max=1024

For automatic loading after reboot, we recommend:

Create the /etc/modules-load.d/nbd.conf file:

nbd

Create the /etc/modprobe.d/nbd.conf file:

options nbd nbds_max=1024

Check that the module is loaded:

lsmod | grep nbd
# nbd                    49152  0
cat /sys/module/nbd/parameters/nbds_max
# 128

We recommend setting nbds_max to at least the number of job slots on the node multiplied by the maximum number of NBD layers in one job. For example, for 32 slots and 2 NBD layers per job: nbds_max=128. Devices can be created dynamically, so the value doesn’t limit operation. But having a pre-created reserve reduces overhead for device creation under load.

Configuring NBD

You configure NBD through the exec node’s dynamic config. All parameters are in the exec_node/nbd section.

Enabling NBD

exec_node:
  nbd:
    enabled: true

Full config example

exec_node:
  nbd:
    enabled: true
    block_cache_compressed_data_capacity: 536870912  # 512 MB
    client:
      io_timeout: 30000          # 30 seconds, in milliseconds
      reconnect_timeout: 5000    # 5 seconds, in milliseconds
      connection_count: 1
    server:
      thread_count: 2
      unix_domain_socket:
        path: /tmp/nbd.sock

Config parameters

Checking health

Checking node status

After enabling NBD, make sure the exec node is in the online state and has no alerts:

yt get //sys/exec_nodes/<node-address>/@state
# "online"

yt get //sys/exec_nodes/<node-address>/@alerts
# []

Checking with a test operation

Run a test operation with an NBD layer:

import yt.wrapper as yt

# Create a test SquashFS image and upload it to Cypress
# yt set //path/to/layer.squashfs/@filesystem squashfs
# yt set //path/to/layer.squashfs/@access_method nbd

yt.run_map(
    lambda row: row,
    source_table="//tmp/test_input",
    destination_table="//tmp/test_output",
    spec={
        "mapper": {
            "layer_paths": ["//path/to/layer.squashfs"],
        }
    }
)

Checking via logs

When the NBD server starts successfully, you’ll see these entries in the exec node logs (exec-node.info.log):

NBD server started (UnixDomainSocket: /tmp/nbd.sock, ThreadCount: 2)

When an NBD device is created:

Creating NBD device (FilePath: //path/to/layer.squashfs, DeviceName: /dev/nbd0)
NBD device created (FilePath: //path/to/layer.squashfs, DeviceName: /dev/nbd0)

Monitoring

Solomon sensors

The system exports all NBD metrics to Solomon. Key sensors:

Server metrics:

Device metrics. The file_path tag is the path to the layer file in Cypress:

Volume metrics. Tags: type=nbd, file_path:

Volume cache metrics:

Key metrics for monitoring

Error handling

lsmod | grep nbd
dmesg | grep nbd

Diagnostics via Orchid

You can check the NBD server status via the exec node Orchid:

yt get //sys/exec_nodes/<node-address>/orchid/exec_node

Common issues and solutions

modprobe nbd nbds_max=256