Updating server components of YTsaurus

This article describes the process of fully updating the server components of a YTsaurus cluster via the Kubernetes operator: how to prepare for the update, update the operator, modify the Ytsaurus specification, monitor the update progress, and verify the result.

What you need to know before updating

Update only to the next major version.

Update the cluster sequentially by major versions, without skipping intermediate ones: first 25.1 → 25.2, then 25.2 → 25.3. Skipping a version may disrupt cluster operation. Within the target major version, you can select the latest minor version to get the most recent updates — for example, 25.2.0 → 25.3.2. See available versions in the release list.

The update cannot be undone.

Once the master servers exit read-only mode and apply irreversible changes, rolling back to the previous version is not supported.

Update the operator first, then the cluster.

Before updating the cluster, update the operator. Its version must be compatible with the target cluster version. Check compatibility in the compatibility table.

The cluster may be unavailable during the update.

By default, the cluster is fully unavailable during the update — all operations and requests are interrupted. Schedule the update during a period of minimal load. If the cluster must remain available, choose a different strategy — see Update strategies.

Cluster update

The update process consists of four steps:

1. Prepare the cluster
2. Update the operator
3. Start the update
4. Verify the result

1. Prepare the cluster for the update

Before changing the Ytsaurus specification, check the current state of the cluster and make sure the operator is ready to start a new update cycle.

1. Find out the current cluster version:

   $ kubectl get YTsaurus <cluster_name> -n <namespace> -o jsonpath='{.spec.coreImage}'
   
   ghcr.io/ytsaurus/ytsaurus:stable-25.1.0-relwithdebinfo
   

2. Check the cluster state:

   $ kubectl get YTsaurus <cluster_name> -n <namespace>
   
   NAME       CLUSTERSTATE   UPDATESTATE   UPDATINGCOMPONENTS   BLOCKEDCOMPONENTS
   YTsaurus   Running
   

   Before the update, the cluster must be in the Running state. If the cluster is in another state, fix the issues first — see the Troubleshooting section.

3. Make sure the operator is ready for the update. To do this, check its logs:

   kubectl logs -n <namespace> deployment/ytsaurus-ytop-chart-controller-manager --tail=100
   
   # If the deployment name is different, get the list
   
   kubectl get deployments -n <namespace>
   

   This is an important step: the operator must not be busy with pending actions. Proceed to the next step only when the logs contain the message INFO YTsaurus is running and happy. This means the operator considers the cluster consistent and is not performing any pending actions.

   Important

   If the operator does not report running and happy, do not start the update. This may lead to unpredictable cluster behavior.

4. Save a backup copy of the specification:

   The file is required to enable rollback before passing the point of no return:

   kubectl get YTsaurus <cluster_name> -n <namespace> -o yaml > ytsaurus-spec-backup.yaml
   

   After the master servers exit the read‑only mode and apply irreversible changes, rollback to the previous version is not supported.

If all checks are passed, you can proceed with updating the operator.

2. Update the operator

Update the operator first. New versions of server components may not work correctly with an older version of the operator.

Check the current operator version:

helm list -n <namespace>

See the list of releases and compatible versions on the releases page and in the compatibility table.

Update the operator:

helm upgrade YTsaurus --install oci://ghcr.io/ytsaurus/ytop-chart --version <new_version>

Verify that the operator has been updated:

kubectl get pods -n <namespace>

Right after the update, you may see both the old and new pods at the same time:

NAME                                                      READY   STATUS        RESTARTS   AGE
ytsaurus-ytop-chart-controller-manager-6f67fd5d5c-6bbws   2/2     Running       0          21s
ytsaurus-ytop-chart-controller-manager-7478f9b6cb-qr8wd   2/2     Terminating   0          23h

After some time, the old pod will be removed:

NAME                                                      READY   STATUS    RESTARTS   AGE
ytsaurus-ytop-chart-controller-manager-6f67fd5d5c-6bbws   2/2     Running   0          25s

3. Start the update

When you change the coreImage field in the Ytsaurus specification, the operator recreates pods for all server components from this specification, even if the image of a specific component hasn’t changed. This ensures version compatibility across all server components.

Components with their own release cycle — Query Tracker, YQL agent, and others — are not updated in this process. To learn how to update them, see the section Updating individual components. Before updating, check the compatibility of these components with the target cluster version in the compatibility table.

Save the current specification to a file for editing:

kubectl get YTsaurus <cluster_name> -n <namespace> -o yaml > ytsaurus-spec-new.yaml

Make the following changes to the ytsaurus-spec-new.yaml file:

• Set the updatePlan field with a list of components to update. For a full update, specify the Everything class — all server components will be updated.

• Change coreImage to the next version.

• Select the target version on the releases page. Open the required release and copy the image tag ghcr.io/ytsaurus/ytsaurus.

spec:
  coreImage: ghcr.io/ytsaurus/ytsaurus:stable-25.2.0-relwithdebinfo # Next version
  updatePlan:
    - class: Everything # Full update of all server components

By default, the operator uses the BulkUpdate strategy — the cluster will be completely unavailable during the update. If the cluster must stay available, configure the RollingUpdate strategy via the updatePlan field in the ytsaurus-spec-new.yaml file before applying the changes. See an example in the section Updating without cluster downtime using RollingUpdate.

Apply the modified specification:

kubectl apply -f ytsaurus-spec-new.yaml

Expected output:

ytsaurus.cluster.ytsaurus.tech/<cluster_name> configured

Monitor the status in the UPDATESTATE field of the Ytsaurus resource:

kubectl get YTsaurus <cluster_name> -n <namespace>

During the update, the output may look like this:

NAME       CLUSTERSTATE   UPDATESTATE            UPDATINGCOMPONENTS
ytsaurus   Updating       WaitingForPodsCreation   {ms hp ds dnd rp end tnd sch ca}

For a description of update stages and pod states, see the status reference below.

Check the pod statuses:

kubectl get pods -n <namespace>

If the update takes too long, check the events for the problematic pod:

kubectl describe pod <pod_name> -n <namespace> | grep -A 10 "Events:"

If the events show an image pull error ImagePullBackOff, verify that the image tag is correct and that this version exists in the registry. If the pod is in the Pending state, the issue is related to resources — check their availability in the cluster.

4. Verify the result

The update is complete when the following conditions are met:

• The cluster is in the Running state.
• All main pods are in the Running state.
• There are no pods in the cluster in the Init, Pending, ImagePullBackOff, and CrashLoopBackOff states.

Check the final cluster state:

kubectl get YTsaurus <cluster_name> -n <namespace>

Expected output:

NAME       CLUSTERSTATE   UPDATESTATE   UPDATINGCOMPONENTS   BLOCKEDCOMPONENTS
ytsaurus   Running

Check pod statuses:

kubectl get pods -n <namespace>

Example of expected output:

NAME                                      READY   STATUS     RESTARTS   AGE
ca-0                                      1/1     Running    0          5m
dnd-0                                     1/1     Running    0          5m
dnd-1                                     1/1     Running    0          5m
dnd-2                                     1/1     Running    0          5m
ds-0                                      1/1     Running    0          10m
end-0                                     2/2     Running    0          5m
hp-0                                      1/1     Running    0          5m
ms-0                                      1/1     Running    0          10m
qt-0                                      1/1     Running    15         75d
rp-0                                      1/1     Running    0          5m
sch-0                                   1/1     Running    0          5m
tnd-0                                     1/1     Running    0          10m

Verify that the new version is specified in the specification:

kubectl get YTsaurus <cluster_name> -n <namespace> -o jsonpath='{.spec.coreImage}'

Expected output:

ghcr.io/ytsaurus/ytsaurus:stable-25.2.0-relwithdebinfo

Make sure the operator is again logging Ytsaurus is running and happy:

kubectl logs -n <namespace> deployment/ytsaurus-ytop-chart-controller-manager --tail=100

After a successful update, clear the updatePlan field in the ytsaurus-spec-new.yaml file — leave it empty:

spec:
  updatePlan: []

You can achieve the same result with an explicit rule using the Nothing class — the operator won’t update any component:

spec:
  updatePlan:
    - class: Nothing

Apply the change:

kubectl apply -f ytsaurus-spec-new.yaml

The operator doesn’t clear updatePlan automatically — reset it yourself. An empty updatePlan means “don’t update anything” and protects against accidental component restarts during further specification changes. If you leave - class: Everything, the next change to coreImage will trigger a full update of all components again.

Updating to the next version: if you need to update to another version, repeat all steps starting from the Preparing for update section. Each time, save the specification anew via kubectl get ytsaurus — the operator might have changed it during the update, and the local file from last time is no longer up to date.

Troubleshooting updates

Use this section if the update doesn’t start, takes too long, fails, or behaves unpredictably.

Error from server (Conflict): error when applying patch: the object has been modified; please apply your changes to the latest version and try again

kubectl get YTsaurus <cluster_name> -n <namespace> -o yaml > ytsaurus-spec-new.yaml

Failed to pull image "ghcr.io/ytsaurus/ytsaurus:stable-X.X.X-relwithdebinfo": not found

kubectl describe pod <pod_name> -n <namespace> | grep -A 10 "Events:"

Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m10s  default-scheduler  Successfully assigned ytsaurus/ms-0 to docker-desktop
  Normal  Pulling    3m9s   kubelet            Pulling image "ghcr.io/ytsaurus/ytsaurus:stable-25.2.0-relwithdebinfo"

kubectl get YTsaurus <cluster_name> -n <namespace>

kubectl logs -n <namespace> deployment/ytsaurus-ytop-chart-controller-manager --tail=100

kubectl get YTsaurus -n <namespace>

NAME         CLUSTERSTATE   UPDATESTATE           UPDATINGCOMPONENTS
minisaurus   Updating       ImpossibleToStart

kubectl describe YTsaurus -n <namespace>

kubectl describe YTsaurus -n <namespace>
...
  Update Status:
    Conditions:
      Last Transition Time:  2023-09-26T09:18:11Z
      Message:               Tablet cell bundles ([sys default]) aren't in 'good' health
      Reason:                Update
      Status:                True
      Type:                  NoPossibility
    State:                   ImpossibleToStart

Found unrecognized options in dynamic cluster config

Advanced scenarios

Updating individual components

In the Ytsaurus specification, you can manage component images individually. How the operator handles such changes depends on the component type:

• Components with their own release cycle — Query Tracker, YQL agent, Strawberry, and others. For more information, see the section Components with their own release cycle.

  To update these components, change the image field in the Ytsaurus specification. This is the only way to update these components. When you change their image, the operator updates only their pods without stopping cluster operation.

  When fully updating the cluster via the coreImage field, these components’ pods will restart, but their image will remain the same — the one set in the image field.

  Example: updating Query Tracker to version 0.0.7:

  spec:
    queryTrackers:
      image: ghcr.io/ytsaurus/query-tracker:0.0.7
  

  kubectl apply -f y