Updating server components of YTsaurus

This article describes the process of fully updating the server components of a YTsaurus cluster via the Kubernetes operator: how to prepare for the update, update the operator, modify the Ytsaurus specification, monitor the update progress, and verify the result.

What you need to know before updating

Update only to the next major version.

Update the cluster sequentially by major versions, without skipping intermediate ones: first 25.1 → 25.2, then 25.2 → 25.3. Skipping a version may disrupt cluster operation. Within the target major version, you can select the latest minor version to get the most recent updates — for example, 25.2.0 → 25.3.2. See available versions in the release list.

The update cannot be undone.

Once the master servers exit read-only mode and apply irreversible changes, rolling back to the previous version is not supported.

Update the operator first, then the cluster.

Before updating the cluster, update the operator. Its version must be compatible with the target cluster version. Check compatibility in the compatibility table.

The cluster may be unavailable during the update.

By default, the cluster is fully unavailable during the update — all operations and requests are interrupted. Schedule the update during a period of minimal load. If the cluster must remain available, choose a different strategy — see Update strategies.

Cluster update

The update process consists of four steps:

1. Prepare the cluster
2. Update the operator
3. Start the update
4. Verify the result

1. Prepare the cluster for the update

Before changing the Ytsaurus specification, check the current state of the cluster and make sure the operator is ready to start a new update cycle.

1. Find out the current cluster version:

   $ kubectl get YTsaurus <cluster_name> -n <namespace> -o jsonpath='{.spec.coreImage}'
   
   ghcr.io/ytsaurus/ytsaurus:stable-25.1.0-relwithdebinfo
   

2. Check the cluster state:

   $ kubectl get YTsaurus <cluster_name> -n <namespace>
   
   NAME       CLUSTERSTATE   UPDATESTATE   UPDATINGCOMPONENTS   BLOCKEDCOMPONENTS
   YTsaurus   Running
   

   Before the update, the cluster must be in the Running state. If the cluster is in another state, fix the issues first — see the Troubleshooting section.

3. Make sure the operator is ready for the update. To do this, check its logs:

   kubectl logs -n <namespace> deployment/ytsaurus-ytop-chart-controller-manager --tail=100
   
   # If the deployment name is different, get the list
   
   kubectl get deployments -n <namespace>
   

   This is an important step: the operator must not be busy with pending actions. Proceed to the next step only when the logs contain the message INFO YTsaurus is running and happy. This means the operator considers the cluster consistent and is not performing any pending actions.

   Important

   If the operator does not report running and happy, do not start the update. This may lead to unpredictable cluster behavior.

4. Save a backup copy of the specification:

   The file is required to enable rollback before passing the point of no return:

   kubectl get YTsaurus <cluster_name> -n <namespace> -o yaml > ytsaurus-spec-backup.yaml
   

   After the master servers exit the read‑only mode and apply irreversible changes, rollback to the previous version is not supported.

If all checks are passed, you can proceed with updating the operator.

2. Update the operator

Update the operator first. New versions of server components may not work correctly with an older version of the operator.

Check the current operator version:

helm list -n <namespace>

See the list of releases and compatible versions on the releases page and in the compatibility table.

Update the operator:

helm upgrade YTsaurus --install oci://ghcr.io/ytsaurus/ytop-chart --version <new_version>

Verify that the operator has been updated:

kubectl get pods -n <namespace>

Right after the update, you may see both the old and new pods at the same time:

NAME                                                      READY   STATUS        RESTARTS   AGE
ytsaurus-ytop-chart-controller-manager-6f67fd5d5c-6bbws   2/2     Running       0          21s
ytsaurus-ytop-chart-controller-manager-7478f9b6cb-qr8wd   2/2     Terminating   0          23h

After some time, the old pod will be removed:

NAME                                                      READY   STATUS    RESTARTS   AGE
ytsaurus-ytop-chart-controller-manager-6f67fd5d5c-6bbws   2/2     Running   0          25s

3. Start the update

When you change the coreImage field in the Ytsaurus specification, the operator recreates pods for all server components from this specification, even if the image of a specific component hasn’t changed. This ensures version compatibility across all server components.

Components with their own release cycle — Query Tracker, YQL agent, and others — are not updated in this process. To learn how to update them, see the section Updating individual components. Before updating, check the compatibility of these components with the target cluster version in the compatibility table.

Save the current specification to a file for editing:

kubectl get YTsaurus <cluster_name> -n <namespace> -o yaml > ytsaurus-spec-new.yaml

Make the following changes to the ytsaurus-spec-new.yaml file:

• Set the updatePlan field with a list of components to update. For a full update, specify the Everything class — all server components will be updated.

• Change coreImage to the next version.

• Select the target version on the releases page. Open the required release and copy the image tag ghcr.io/ytsaurus/ytsaurus.

spec:
  coreImage: ghcr.io/ytsaurus/ytsaurus:stable-25.2.0-relwithdebinfo # Next version
  updatePlan:
    - class: Everything # Full update of all server components

By default, the operator uses the BulkUpdate strategy — the cluster will be completely unavailable during the update. If the cluster must stay available, configure the RollingUpdate strategy via the updatePlan field in the ytsaurus-spec-new.yaml file before applying the changes. See an example in the section Updating without cluster downtime using RollingUpdate.

Apply the modified specification:

kubectl apply -f ytsaurus-spec-new.yaml

Expected output:

ytsaurus.cluster.ytsaurus.tech/<cluster_name> configured

Monitor the status in the UPDATESTATE field of the Ytsaurus resource:

kubectl get YTsaurus <cluster_name> -n <namespace>

During the update, the output may look like this:

NAME       CLUSTERSTATE   UPDATESTATE            UPDATINGCOMPONENTS
ytsaurus   Updating       WaitingForPodsCreation   {ms hp ds dnd rp end tnd sch ca}

For a description of update stages and pod states, see the status reference below.

Check the pod statuses:

kubectl get pods -n <namespace>

If the update takes too long, check the events for the problematic pod:

kubectl describe pod <pod_name> -n <namespace> | grep -A 10 "Events:"

If the events show an image pull error ImagePullBackOff, verify that the image tag is correct and that this version exists in the registry. If the pod is in the Pending state, the issue is related to resources — check their availability in the cluster.

4. Verify the result

The update is complete when the following conditions are met:

• The cluster is in the Running state.
• All main pods are in the Running state.
• There are no pods in the cluster in the Init, Pending, ImagePullBackOff, and CrashLoopBackOff states.

Check the final cluster state:

kubectl get YTsaurus <cluster_name> -n <namespace>

Expected output:

NAME       CLUSTERSTATE   UPDATESTATE   UPDATINGCOMPONENTS   BLOCKEDCOMPONENTS
ytsaurus   Running

Check pod statuses:

kubectl get pods -n <namespace>

Example of expected output:

NAME                                      READY   STATUS     RESTARTS   AGE
ca-0                                      1/1     Running    0          5m
dnd-0                                     1/1     Running    0          5m
dnd-1                                     1/1     Running    0          5m
dnd-2                                     1/1     Running    0          5m
ds-0                                      1/1     Running    0          10m
end-0                                     2/2     Running    0          5m
hp-0                                      1/1     Running    0          5m
ms-0                                      1/1     Running    0          10m
qt-0                                      1/1     Running    15         75d
rp-0                                      1/1     Running    0          5m
sch-0                                   1/1     Running    0          5m
tnd-0                                     1/1     Running    0          10m

Verify that the new version is specified in the specification:

kubectl get YTsaurus <cluster_name> -n <namespace> -o jsonpath='{.spec.coreImage}'

Expected output:

ghcr.io/ytsaurus/ytsaurus:stable-25.2.0-relwithdebinfo

Make sure the operator is again logging Ytsaurus is running and happy:

kubectl logs -n <namespace> deployment/ytsaurus-ytop-chart-controller-manager --tail=100

After a successful update, clear the updatePlan field in the ytsaurus-spec-new.yaml file — leave it empty:

spec:
  updatePlan: []

You can achieve the same result with an explicit rule using the Nothing class — the operator won’t update any component:

spec:
  updatePlan:
    - class: Nothing

Apply the change:

kubectl apply -f ytsaurus-spec-new.yaml

The operator doesn’t clear updatePlan automatically — reset it yourself. An empty updatePlan means “don’t update anything” and protects against accidental component restarts during further specification changes. If you leave - class: Everything, the next change to coreImage will trigger a full update of all components again.

Updating to the next version: if you need to update to another version, repeat all steps starting from the Preparing for update section. Each time, save the specification anew via kubectl get ytsaurus — the operator might have changed it during the update, and the local file from last time is no longer up to date.

Troubleshooting updates

Use this section if the update doesn’t start, takes too long, fails, or behaves unpredictably.

Error from server (Conflict): error when applying patch: the object has been modified; please apply your changes to the latest version and try again

kubectl get YTsaurus <cluster_name> -n <namespace> -o yaml > ytsaurus-spec-new.yaml

Failed to pull image "ghcr.io/ytsaurus/ytsaurus:stable-X.X.X-relwithdebinfo": not found

kubectl describe pod <pod_name> -n <namespace> | grep -A 10 "Events:"

Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m10s  default-scheduler  Successfully assigned ytsaurus/ms-0 to docker-desktop
  Normal  Pulling    3m9s   kubelet            Pulling image "ghcr.io/ytsaurus/ytsaurus:stable-25.2.0-relwithdebinfo"

kubectl get YTsaurus <cluster_name> -n <namespace>

kubectl logs -n <namespace> deployment/ytsaurus-ytop-chart-controller-manager --tail=100

kubectl get YTsaurus -n <namespace>

NAME         CLUSTERSTATE   UPDATESTATE           UPDATINGCOMPONENTS
minisaurus   Updating       ImpossibleToStart

kubectl describe YTsaurus -n <namespace>

kubectl describe YTsaurus -n <namespace>
...
  Update Status:
    Conditions:
      Last Transition Time:  2023-09-26T09:18:11Z
      Message:               Tablet cell bundles ([sys default]) aren't in 'good' health
      Reason:                Update
      Status:                True
      Type:                  NoPossibility
    State:                   ImpossibleToStart

Found unrecognized options in dynamic cluster config

Advanced scenarios

Updating individual components

In the Ytsaurus specification, you can manage component images individually. How the operator handles such changes depends on the component type:

• Components with their own release cycle — Query Tracker, YQL agent, Strawberry, and others. For more information, see the section Components with their own release cycle.

  To update these components, change the image field in the Ytsaurus specification. This is the only way to update these components. When you change their image, the operator updates only their pods without stopping cluster operation.

  When fully updating the cluster via the coreImage field, these components’ pods will restart, but their image will remain the same — the one set in the image field.

  Example: updating Query Tracker to version 0.0.7:

  spec:
    queryTrackers:
      image: ghcr.io/ytsaurus/query-tracker:0.0.7
  

  kubectl apply -f ytsaurus-spec-new.yaml
  

  The operator will update only the qt pods, without affecting other components.

• Targeted update of server components

  You can override the image for a specific server component — for example, only for tabletNodes — by specifying a separate image for it instead of the common coreImage.

  Specify the image field for the required component in the ytsaurus-spec-new.yaml file and apply the change:

  kubectl apply -f ytsaurus-spec-new.yaml
  

Updating static configs

The operator automatically generates configuration files for each component — master servers, proxies, nodes — based on the Ytsaurus specification. Sometimes you need to modify parameters that go into these files — for example, add a new storage location locations or configure component-specific parameters via configOverrides.

In such cases, do not set the updatePlan field. It is sufficient to modify the specification and apply it — the operator will detect that the configuration has changed and restart only the pods of the affected components.

Make the necessary changes in ytsaurus-spec-new.yaml and apply them:

kubectl apply -f ytsaurus-spec-new.yaml

During the pod restart, the cluster will transition to the Reconfiguration state:

kubectl get ytsaurus <cluster_name> -n <namespace>

NAME       CLUSTERSTATE      UPDATESTATE
ytsaurus   Reconfiguration

Once the pods restart and become ready, the cluster will return to the Running state.

Automatic update after operator upgrade

When the operator is upgraded, its new version may automatically trigger a cluster update — without you changing the coreImage. This happens because different operator versions may generate different configuration files for the same components — for example, a new field may be added in the newer version. The operator detects the discrepancy between the current cluster state and the desired state and initiates reconciliation.

If the update turns out to be impossible, the cluster will remain in the Updating state, and UPDATESTATE will transition to ImpossibleToStart. In this case, there are two options:

• Roll back the operator to the previous version:

  helm rollback ytsaurus -n <namespace>
  

• Leave the updatePlan field empty in the Ytsaurus specification — the operator will cancel the update and will not run it automatically:

  spec:
    updatePlan: []
  

  kubectl apply -f ytsaurus-spec-new.yaml
  

  After that, bring the cluster to a working state and retry when you are ready, specifying the required components in updatePlan — for example, - class: Everything.

Updating the cluster without downtime via RollingUpdate

To keep the cluster available during the update, configure the RollingUpdate strategy via the updatePlan field in the Ytsaurus specification. The operator will update the pods one by one, maintaining the minimum number of available instances.

apiVersion: cluster.ytsaurus.tech/v1
kind: Ytsaurus
metadata:
  name: ytsaurus
spec:
  coreImage: ghcr.io/ytsaurus/ytsaurus:stable-25.2.0-relwithdebinfo
  updatePlan:
    - class: Everything
      strategy:
        rollingUpdate: {}

updatePlan:
  # HTTP Proxies — Rolling Update
  - component:
      type: HttpProxy
    strategy:
      rollingUpdate: {}
      runPreChecks: true

  # RPC Proxies — Rolling Update
  - component:
      type: RpcProxy
    strategy:
      rollingUpdate: {}
      runPreChecks: true

  # Data Nodes — Rolling Update
  - component:
      type: DataNode
    strategy:
      rollingUpdate: {}
      runPreChecks: true

  # Exec Nodes — Rolling Update with drain
  - component:
      type: ExecNode
    strategy:
      rollingUpdate: {}
      runPreChecks: true

  # Other components — BulkUpdate
  - class: Everything
    concurrency: 2  # no more than 2 instance groups simultaneously

apiVersion: cluster.ytsaurus.tech/v1
kind: Ytsaurus
metadata:
  name: ytsaurus
spec:
  coreImage: ghcr.io/ytsaurus/ytsaurus:stable-25.2.0-relwithdebinfo
  clusterFeatures:
    enableImageHeater: true
  updatePlan:
    - class: Everything
      strategy:
        rollingUpdate: {}

For a description of all updatePlan fields, see the section Configuring the updatePlan field.

Manual intervention

If the automatic update cannot complete and manual actions are required, temporarily disable the operator’s management of the cluster. To do this, set the isManaged: false flag in the Ytsaurus specification.

After this, the operator will stop making changes to the cluster, and you will be able to perform manual actions.

This scenario can be useful if the operator is stuck on a problematic component. It can also help if you need to manually check or restore the state of a part of the cluster. Or if you are troubleshooting an unusual error that cannot be resolved by simply reapplying the specification.

Status reference

• How to check the cluster status
• UPDATESTATE
• CLUSTERSTATE
• Pod states

How to check the cluster status

You can monitor the update via the Ytsaurus resource status:

kubectl get ytsaurus -n <namespace>

During the update, the cluster transitions to the Updating state, and the detailed stage is displayed in the UPDATESTATE field. After the update completes, the cluster returns to the Running state:

NAME         CLUSTERSTATE   UPDATESTATE   UPDATINGCOMPONENTS
minisaurus   Running        None

UPDATESTATE

Not all statuses appear during every update — the set depends on which components are affected. For example, the WaitingForTabletCells statuses appear only when updating tablet nodes, and WaitingForOpArchiveUpdate appears only when updating the scheduler.

CLUSTERSTATE

Pod states

Component reference

Components updated together with the cluster

The operator updates these components simultaneously with the cluster when the coreImage field in the Ytsaurus specification is changed. For each component, you can override the image separately via the image field.

Components with their own release cycle

These components have their own release cycles; the operator updates them independently via the image field of the specific component in the Ytsaurus specification.