---
metadata:
  - name: generator
    content: Diplodoc Platform v5.50.6
  - property: og:title
    content: NBD on the YTsaurus cluster
  - property: og:description
    content: This document describes how to configure and use NBD on the YTsaurus cluster.
  - property: og:type
    content: article
  - property: og:url
    content: https://ytsaurus.tech/docs/en/admin-guide/nbd
  - property: article:tag
    content: admin-guide/nbd
  - property: article:modified_time
    content: '2026-08-11T17:00:00+03:00'
  - property: article:author
    content: Zakhar Telukh
alternate:
  - https://ytsaurus.tech/docs/en/admin-guide/nbd.md
  - https://ytsaurus.tech/docs/ru/admin-guide/nbd.md
---
> **Documentation Index:** Fetch the complete configuration index at https://ytsaurus.tech/docs/en/llms.txt?revision=cc1aee2ee307cf374f6a8e98ed868050ec82fbdc


<!-- source: en/_includes/admin-guide/nbd.md -->
# NBD on the YTsaurus cluster

This document describes how to configure and use NBD (Network Block Device) on the YTsaurus cluster. NBD lets you mount filesystem images from Cypress as layers of a job’s root filesystem. This speeds up environment setup, reduces disk load, and, under some conditions, lowers network load.

## How NBD works { #how-it-works }

NBD (Network Block Device) is a Linux kernel mechanism that lets you mount block devices with data stored remotely. In YTsaurus, NBD is used to mount SquashFS filesystem images from Cypress as layers of a job’s root filesystem.

### Architecture { #architecture }

Each exec-node runs an NBD server — a YTsaurus component that implements the NBD protocol over a Unix Domain Socket or TCP. Here’s how it works:

```mermaid
flowchart TB
    subgraph ExecNode["Exec-node"]
        Kernel["Linux kernel<br>/dev/nbdX"]
        NBDServer["NBD server (YT)<br>Block cache<br>(compressed data)"]
        OverlayFS["overlayfs<br>(job’s rootfs)"]
        
        Kernel <-->|"NBD protocol<br>(Unix socket)"| NBDServer
        Kernel -->|"mount"| OverlayFS
    end
    
    NBDServer -->|"RPC"| DataNodes["Data-nodes<br>(image chunks)"]
    
    style ExecNode fill:#f5f5f5
    style Kernel fill:#e1f5ff
    style NBDServer fill:#fff4e1
    style OverlayFS fill:#e8f5e9
    style DataNodes fill:#fce4ec
```

Sequence of events when preparing an NBD layer:

- The exec-node receives a task with `layer_paths` that includes an NBD layer.
- YT downloads the image chunk metadata, not the data itself.
- The NBD server registers an export for the image.
- The Linux kernel mounts `/dev/nbdX` to the export via a Unix Domain Socket.
- Porto mounts `/dev/nbdX` as a layer in overlayfs.
- When a job accesses a file, the kernel reads the needed blocks through `/dev/nbdX` → NBD server → data-nodes.

### Block cache { #block-cache }

The NBD server supports an in-memory LRU block cache to store compressed chunk data. The cache helps avoid repeated calls to data-nodes when different jobs read the same blocks. You configure the cache size with the `block_cache_compressed_data_capacity` parameter.

### Volume cache { #volume-cache }

The exec-node caches read-only (RO) NBD volumes — mounted images. If several jobs use the same NBD layer on one exec-node, the system creates the volume once and reuses it. Cache metrics: `exec_node/ronbd_volume_cache/missed_count`, `exec_node/ronbd_volume_cache/hit_count`.

### Installing packages { #packages }

To work with NBD and SquashFS, install these packages:

```bash
sudo apt install nbd-client squashfs-tools
```

- `nbd-client` — a utility for manually mounting NBD devices. It’s used for diagnostics: the YTsaurus NBD server is built into the exec-node, and in normal operation the kernel connects to it directly.
- `squashfs-tools` — utilities `mksquashfs` and `unsquashfs` for building and checking SquashFS images.

To convert existing tar layers to SquashFS, also install `squashfs-tools-ng` with the `tar2sqfs` utility:

```bash
sudo apt install squashfs-tools-ng
```

Check the installation:

```bash
nbd-client --version
# This is nbd-client, from nbd 3.26.1

mksquashfs -version
# mksquashfs version 4.6.1 (2023/03/25)
```

### NBD kernel module { #kernel-module }

To use NBD, you need to load the `nbd` kernel module. The `nbds_max` parameter defines how many NBD devices the kernel creates when loading the module. NBD devices can be created and removed dynamically.

Manually load the module:

```bash
modprobe nbd nbds_max=1024
```

For automatic loading after reboot (recommended):

Create the `/etc/modules-load.d/nbd.conf` file:

```ini
nbd
```

Create the `/etc/modprobe.d/nbd.conf` file:

```ini
options nbd nbds_max=1024
```

{% note warning %}

The `nbd` module must load automatically after the host reboots. Without this, the exec-node won’t be able to create NBD devices after a reboot.

{% endnote %}

Check that the module is loaded:

```bash
lsmod | grep nbd
# nbd                    49152  0
cat /sys/module/nbd/parameters/nbds_max
# 128
```

Recommended `nbds_max` value: at least the number of job slots on the node multiplied by the maximum number of NBD layers in one job. For example, for 32 slots and 2 NBD layers per job: `nbds_max=128`. Devices can be created dynamically, so the value doesn’t limit operation, but having a pre-created buffer reduces overhead for device creation under load.

## NBD configuration { #configuration }

You configure NBD via the exec-node’s dynamic config. All parameters are in the `exec_node/nbd` section.

### Enabling NBD { #enable }

```yaml
exec_node:
  nbd:
    enabled: true
```

{% note info %}

After enabling NBD, the exec-node starts the NBD server at boot. Changing `enabled` requires restarting the node.

{% endnote %}

### Full configuration example { #full-config }

```yaml
exec_node:
  nbd:
    enabled: true
    block_cache_compressed_data_capacity: 536870912  # 512 MB
    client:
      io_timeout: 30000          # 30 seconds, in milliseconds
      reconnect_timeout: 5000    # 5 seconds, in milliseconds
      connection_count: 1
    server:
      thread_count: 2
      unix_domain_socket:
        path: /tmp/nbd.sock
```

### Configuration parameters { #config-params }

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `exec_node/nbd/enabled` | `bool` | `false` | Enables or disables NBD on the exec-node. When `enabled: true`, the system starts the NBD server at node boot. |
| `exec_node/nbd/block_cache_compressed_data_capacity` | `int64`, bytes | `0` — cache disabled | Size of the compressed data block cache in bytes. The cache is stored in the exec-node’s memory and is used to cache chunk blocks read from data-nodes. Recommended value: from 512 MB to 4 GB, depending on available memory and load. |
| `exec_node/nbd/client/io_timeout` | `duration`, ms | `30000` — 30 seconds | Timeout for waiting for a response to an NBD read request. If the timeout is exceeded, the system aborts the job with `abort_reason=NbdError`. |
| `exec_node/nbd/client/reconnect_timeout` | `duration`, ms | `5000` — 5 seconds | Timeout for the NBD client to reconnect to the NBD server if the connection drops. |
| `exec_node/nbd/client/connection_count` | `int` | `1` | Number of connections the NBD client makes to the NBD server per device. |
| `exec_node/nbd/server/thread_count` | `int` | `2` | Number of NBD server threads. Recommended value: 2–4. |
| `exec_node/nbd/server/unix_domain_socket/path` | `string` | — | Path to the Unix Domain Socket that the Linux kernel uses to connect to the NBD server. Must be unique for each exec-node. |
| `exec_node/nbd/server/internet_domain_socket/port` | `int` | — | TCP socket port for the NBD server. The system uses this instead of a Unix Domain Socket if you need network access to the NBD server. |

## Health check { #health-check }

### Checking node status { #node-state }

After enabling NBD, make sure the exec-node is in the `online` state and has no alerts:

```bash
yt get //sys/exec_nodes/<node-address>/@state
# "online"

yt get //sys/exec_nodes/<node-address>/@alerts
# []
```

### Checking via a test operation { #test-operation }

Run a test operation with an NBD layer:

```python
import yt.wrapper as yt

# Create a test SquashFS image and upload it to Cypress
# yt set //path/to/layer.squashfs/@filesystem squashfs
# yt set //path/to/layer.squashfs/@access_method nbd

yt.run_map(
    lambda row: row,
    source_table="//tmp/test_input",
    destination_table="//tmp/test_output",
    spec={
        "mapper": {
            "layer_paths": ["//path/to/layer.squashfs"],
        }
    }
)
```

### Checking via logs { #logs }

When the NBD server starts successfully, the exec-node logs (`exec-node.info.log`) include these entries:

```text
NBD server started (UnixDomainSocket: /tmp/nbd.sock, ThreadCount: 2)
```

When an NBD device is created:

```text
Creating NBD device (FilePath: //path/to/layer.squashfs, DeviceName: /dev/nbd0)
NBD device created (FilePath: //path/to/layer.squashfs, DeviceName: /dev/nbd0)
```

## Monitoring { #monitoring }

### Solomon sensors { #sensors }

The system exports all NBD metrics to Solomon. Key sensors:

Server metrics:

#|
|| **Sensor** | **Description** ||
|| `nbd/server/count` | Shows the current number of NBD servers ||
|| `nbd/server/created` | Shows the number of created NBD servers ||
|#

Device metrics. The `file_path` tag is the path to the layer file in Cypress:

#|
|| **Sensor** | **Description** ||
|| `nbd/device/count` | Shows the current number of active NBD devices ||
|| `nbd/device/created` | Shows the number of created devices ||
|| `nbd/device/removed` | Shows the number of removed devices ||
|| `nbd/device/registered` | Shows the number of devices registered with the NBD server ||
|| `nbd/device/unregistered` | Shows the number of devices unregistered ||
|| `nbd/device/read_count` | Shows the number of read requests ||
|| `nbd/device/read_bytes` | Shows the number of bytes read ||
|| `nbd/device/read_time` | Shows the read time, histogram ||
|| `nbd/device/read_block_bytes_from_cache` | Shows the number of bytes read from the block cache ||
|| `nbd/device/read_block_bytes_from_disk` | Shows the number of bytes read from data nodes ||
|#

Volume metrics. Tags: `type=nbd`, `file_path`:

#|
|| **Sensor** | **Description** ||
|| `volumes/count` | Shows the current number of volumes ||
|| `volumes/created` | Shows the number of created volumes ||
|| `volumes/create_errors` | Shows the number of volume creation errors ||
|| `volumes/create_time` | Shows the volume creation time, histogram ||
|| `volumes/removed` | Shows the number of removed volumes ||
|| `volumes/remove_time` | Shows the volume removal time, histogram ||
|#

Volume cache metrics:

#|
|| **Sensor** | **Description** ||
|| `exec_node/ronbd_volume_cache/missed_count` | Shows the number of cache misses for RO NBD volumes ||
|| `exec_node/ronbd_volume_cache/hit_count` | Shows the number of cache hits. Tag: `hit_type=sync\|async` ||
|| `exec_node/squashfs_volume_cache/missed_count` | Shows the number of cache misses for SquashFS volumes ||
|| `exec_node/squashfs_volume_cache/hit_count` | Shows the number of cache hits for SquashFS volumes ||
|#

### Key metrics for monitoring { #key-metrics }

#|
|| **Metric** | **Description** ||
|| `nbd/device/read_block_bytes_from_cache` vs `nbd/device/read_block_bytes_from_disk` | Shows the block cache efficiency. If most data is read from disk, increase `block_cache_compressed_data_capacity` ||
|| `volumes/create_errors` | Shows issues with mounting NBD layers. A non‑zero value indicates errors ||
|| `exec_node/ronbd_volume_cache/missed_count` | Shows the volume cache efficiency. A high value when repeatedly launching the same layers may indicate volume cache issues ||
|#

## Error handling { #error-handling }

{% cut "NbdError" %}

Cause: a read error from an NBD device during job execution. The job is aborted with `abort_reason=NbdError`. Typical causes:

- A connection drop between the NBD server and the data node.
- Exceeding `io_timeout`.
- The data node storing the image chunks is unavailable.

Behavior: the job is automatically aborted and restarted. If errors repeat across several attempts, the operation ends with an error.

Diagnostics: in the exec node logs, look for entries with `NbdError` or `NBD read failed`. Check the availability of data nodes and the network state.

{% endcut %}

{% cut "RootVolumePreparationFailed" %}

Cause: an error mounting the layer during the job’s root filesystem preparation. Typical causes:

- A corrupted layer image.
- An incorrect filesystem type — `@filesystem`.
- The NBD server is not running or not configured.
- The `nbd` kernel module is not loaded.
- The kernel failed to prepare the NBD device. For more details, see the section [NBD device access errors](#troubleshooting).

Diagnostics: check the exec node logs and the kernel module state:

```bash
lsmod | grep nbd
dmesg | grep nbd
```

{% endcut %}

{% cut "NBD server is not present" %}

Cause: an attempt to use an NBD layer on an exec node where NBD is not enabled or the NBD server hasn’t started.

Solution: enable NBD in the dynamic config — `exec_node/nbd/enabled: true` — and make sure the NBD server started successfully.

{% endcut %}

### Diagnostics via Orchid { #orchid }

You can check the NBD server state via the exec node’s Orchid:

```bash
yt get //sys/exec_nodes/<node-address>/orchid/exec_node
```

## Common issues and solutions { #troubleshooting }

{% cut "NBD devices aren’t created after a reboot" %}

Symptom: after rebooting the host, jobs with NBD layers end with the `RootVolumePreparationFailed` error.

Cause: the `nbd` kernel module isn’t loaded automatically.

Solution: configure the module to auto‑load. For more details, see the section [NBD kernel module](#kernel-module).

{% endcut %}

{% cut "High latency on the first file access" %}

Symptom: the first file accesses in an NBD layer are slow.

Cause: data is read from data nodes, and the block cache is empty.

Solution:

- Increase `block_cache_compressed_data_capacity`.
- Store layers on SSD — use the `primary_medium=ssd_blobs` attribute.
- Increase the layer’s `replication_factor`.

{% endcut %}

{% cut "Frequent job aborts with NbdError" %}

Symptom: jobs are regularly aborted with `abort_reason=NbdError`.

Cause: unstable network or overloaded data nodes.

Solution:

- Increase `io_timeout`.
- Check the state of data nodes and the network.
- Make sure layers are stored on SSD with a sufficient `replication_factor`.

{% endcut %}

{% cut "NBD device access errors" %}

Symptom: errors like `No such device` or `Failed to open /dev/nbdX` appear in the logs.

Cause: the kernel failed to create an NBD device. This is usually due to an outdated kernel without support for dynamic device creation or a lack of system resources.

Solution: increase the number of devices created when loading the module:

```bash
modprobe nbd nbds_max=256
```

If the issue persists, check the kernel version and the output of `dmesg | grep nbd`.

{% endcut %}



<!-- endsource: en/_includes/admin-guide/nbd.md -->