A customer ran into issues backing up and restoring their projects on the ARO cluster. So I figure it is time to run a quick test on this and see where the issues are. After testing for the customer, I decided to share my discovery from my test and hope to save time for others.
The use case is very simple here. We want to back up a project and restore the way things are if I delete the project.
My environment
- ARO 4.12.25
- Velero v1.11.1
- Azure CLI 2.50
Set up ARO for test
- Installing the ARO cluster is straightforward, and if you have the latest version of azure-cli, you can add ‘–version’ to specify the OpenShift version under the latest version supported in the ARO lifecycle [2].
- Install or update the azure-cli version as shown.
$ az version { "azure-cli": "2.50.0", "azure-cli-core": "2.50.0", "azure-cli-telemetry": "1.0.8", "extensions": {} }
- I followed the reference [1] to create an ARO cluster the 4.11.44 and upgrade to 4.12.25.
- Deploy a stateful application to the cluster that uses Persistent Volume Claim (PVC).
Set up to run Velero
- Install Velero CLI per [3]
$ velero version Client: Version: v1.11.1 Git commit: - Server: Version: v1.11.1
- I followed the reference [3] to set up the Azure account and blob container.
- Also, use the instruction to create a service principal for Velero to access the storage.
Now, we are ready to install Velero onto the ARO cluster
In the ARO backup and restore documentation [3], it uses “velero/velero-plugin-for-microsoft-azure:v1.1.0”. I learned I must use “velero/velero-plugin-for-microsoft-azure:v1.5.0” to restore my PVC data properly. My inspiration is from reference [5].
Here are the steps I did to backup & restore:
- Add data to my application for testing. My application pod uses a PVC
- Files are added to the directory /var/demo_files
- Now, I am ready to back up my application
$ velero backup create backup-2 --include-namespaces=ostoy --snapshot-volumes=true --include-cluster-resources=true
Backup request "backup-2" submitted successfully.
Run `velero backup describe backup-2` or `velero backup logs backup-2` for more details.
$ velero backup describe backup-2
Name: backup-2
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.25.11+1485cc9
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=25
Phase: Completed
Namespaces:
Included: ostoy
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: included
Label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: true
TTL: 720h0m0s
CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 1h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2023-07-28 13:06:42 -0700 PDT
Completed: 2023-07-28 13:07:06 -0700 PDT
Expiration: 2023-08-27 13:06:42 -0700 PDT
Total items to be backed up: 2089
Items backed up: 2089
Velero-Native Snapshots: 1 of 1 snapshots completed successfully (specify --details for more information)
- Check if the backup is completed.
$ oc get backup backup-2 -n velero -o yaml
- And the status will report as completed
status: completionTimestamp: "2023-07-28T20:07:06Z" expiration: "2023-08-27T20:06:42Z" formatVersion: 1.1.0 phase: Completed
- Now let’s delete the application
$ oc delete all --all -n ostoy; oc delete pvc ostoy-pvc; oc delete project ostoy
- Let’s restore it from the backup-2
$ velero restore create restore-2 --from-backup backup-2 Restore request "restore-2" submitted successfully. Run `velero restore describe restore-2` or `velero restore logs restore-2` for more details. $ velero restore describe restore-2 Name: restore-2 Namespace: velero Labels: <none> Annotations: <none> Phase: InProgress Estimated total items to be restored: 128 Items restored so far: 100 Started: 2023-07-28 13:12:22 -0700 PDT Completed: <n/a> Backup: backup-2 Namespaces: Included: all namespaces found in the backup Excluded: <none> Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io Cluster-scoped: auto Namespace mappings: <none> Label selector: <none> Restore PVs: auto Existing Resource Policy: <none> ItemOperationTimeout: 1h0m0s Preserve Service NodePorts: auto
- Let’s check if the restore is completed
$ oc get restore restore-2 -n velero -o yaml
- Wait for the status shows as completed.
- Now, we can check the project and application data are restored.
- All pods are running, and the added data is restored below.
When using “velero/velero-plugin-for-microsoft-azure:v1.1.0” for Velero, I could not restore the data from the PVC. By using “velero/velero-plugin-for-microsoft-azure:v1.5.0” for Velero, I can now restore the application on the same ARO cluster and the application data.
Reference
- [1] ARO 4 installation: https://learn.microsoft.com/en-us/azure/openshift/tutorial-create-cluster
- [2] Support Lifecycle for ARO 4: https://learn.microsoft.com/en-us/azure/openshift/support-lifecycle
- [3] ARO backup and restore: https://learn.microsoft.com/en-us/azure/openshift/howto-create-a-backup
- [4] Velero docs: https://velero.io/docs/main/basic-install/
- [5] Inspiration reference: https://learn.microsoft.com/en-us/azure/aks/hybrid/backup-workload-cluster