Oops, I deleted my project on an Azure Red Hat OpenShift (ARO) cluster!

A customer ran into issues backing up and restoring their projects on the ARO cluster. So I figure it is time to run a quick test on this and see where the issues are. After testing for the customer, I decided to share my discovery from my test and hope to save time for others.

The use case is very simple here. We want to back up a project and restore the way things are if I delete the project.

My environment

• ARO 4.12.25
• Velero v1.11.1
• Azure CLI 2.50

Set up ARO for test

• Installing the ARO cluster is straightforward, and if you have the latest version of azure-cli, you can add ‘–version’ to specify the OpenShift version under the latest version supported in the ARO lifecycle [2].
• Install or update the azure-cli version as shown.

$ az version
{
  "azure-cli": "2.50.0",
  "azure-cli-core": "2.50.0",
  "azure-cli-telemetry": "1.0.8",
  "extensions": {}
}

• I followed the reference [1] to create an ARO cluster the 4.11.44 and upgrade to 4.12.25.
• Deploy a stateful application to the cluster that uses Persistent Volume Claim (PVC).

Set up to run Velero

• Install Velero CLI per [3]

$ velero version
Client:
Version: v1.11.1
Git commit: -
Server:
Version: v1.11.1

• I followed the reference [3] to set up the Azure account and blob container.
• Also, use the instruction to create a service principal for Velero to access the storage.

Now, we are ready to install Velero onto the ARO cluster

In the ARO backup and restore documentation [3], it uses “velero/velero-plugin-for-microsoft-azure:v1.1.0”. I learned I must use “velero/velero-plugin-for-microsoft-azure:v1.5.0” to restore my PVC data properly. My inspiration is from reference [5].

Here are the steps I did to backup & restore:

• Add data to my application for testing. My application pod uses a PVC

• Files are added to the directory /var/demo_files

• Now, I am ready to back up my application

$ velero backup create backup-2 --include-namespaces=ostoy --snapshot-volumes=true --include-cluster-resources=true
Backup request "backup-2" submitted successfully.
Run `velero backup describe backup-2` or `velero backup logs backup-2` for more details.

$ velero backup describe backup-2
Name:         backup-2
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.25.11+1485cc9
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=25

Phase:  Completed


Namespaces:
  Included:  ostoy
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  included

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  1h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-07-28 13:06:42 -0700 PDT
Completed:  2023-07-28 13:07:06 -0700 PDT

Expiration:  2023-08-27 13:06:42 -0700 PDT

Total items to be backed up:  2089
Items backed up:              2089

Velero-Native Snapshots:  1 of 1 snapshots completed successfully (specify --details for more information)

• Check if the backup is completed.

$ oc get backup backup-2 -n velero -o yaml

• And the status will report as completed

status:
  completionTimestamp: "2023-07-28T20:07:06Z"
  expiration: "2023-08-27T20:06:42Z"
  formatVersion: 1.1.0
  phase: Completed

• Now let’s delete the application

$ oc delete all --all -n ostoy; oc delete pvc ostoy-pvc; oc delete project ostoy

• Let’s restore it from the backup-2

$ velero restore create restore-2 --from-backup backup-2
Restore request "restore-2" submitted successfully.
Run `velero restore describe restore-2` or `velero restore logs restore-2` for more details.

$ velero restore describe restore-2
Name:         restore-2
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                                 InProgress
Estimated total items to be restored:  128
Items restored so far:                 100

Started:    2023-07-28 13:12:22 -0700 PDT
Completed:  <n/a>

Backup:  backup-2

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Existing Resource Policy:   <none>
ItemOperationTimeout:       1h0m0s

Preserve Service NodePorts:  auto

• Let’s check if the restore is completed

$ oc get restore restore-2 -n velero -o yaml

• Wait for the status shows as completed.
• Now, we can check the project and application data are restored.
• All pods are running, and the added data is restored below.

When using “velero/velero-plugin-for-microsoft-azure:v1.1.0” for Velero, I could not restore the data from the PVC. By using “velero/velero-plugin-for-microsoft-azure:v1.5.0” for Velero, I can now restore the application on the same ARO cluster and the application data.

Reference

• [1] ARO 4 installation: https://learn.microsoft.com/en-us/azure/openshift/tutorial-create-cluster
• [2] Support Lifecycle for ARO 4: https://learn.microsoft.com/en-us/azure/openshift/support-lifecycle
• [3] ARO backup and restore: https://learn.microsoft.com/en-us/azure/openshift/howto-create-a-backup
• [4] Velero docs: https://velero.io/docs/main/basic-install/
• [5] Inspiration reference: https://learn.microsoft.com/en-us/azure/aks/hybrid/backup-workload-cluster