Tag: dailyprompt-1892

Application Data Replication

My use case is to replicate the stateful Springboot application for disaster recovery. The application runs on OpenShift, and we want to leverage the existing toolsets to solve this problem. If it is just replicating the data from one data center to another, it should be super simple, right? In this blog, I share my journey of picking my solution.

The requirements are:

  • No code change
  • Cannot use ssh to copy the data
  • Cannot run the pod for replication using privileged containers
  • Must meet the security requirements

Solution 1: Writing the data to object storage

The simplest solution would be to have the application write the data to an object bucket, so we can mirror the object storage directly. However, it requires code changes for all the current applications.

Solution 2: Use rsync replication with VolSync Operator

We tested will the rsync-based replication using VolSync Operator. This will not be a good choice because it violates security policies on using SSH and UID 0 within containers.

Solution 3: Use rsync-tls replication with VolSync Operator

This is the one that meets all the requirements, and I am testing it out.

My test environment includes the following:

  • OpenShift (OCP) 4.11
  • OpenShift Data Foundation (ODF) 4.11
  • Advanced Cluster Security (ACS) 3.74.1
  • VolSync Operator 0.70

Setup

  • Install two OCP 4.11 clusters
  • Install and configure ODF on both OCP clusters
  • Install and configure ACS central on one of the cluster
  • Install and configure ACS secured cluster on both cluster
  • Install VolSync Operator on both clusters
  • Install a sample stateful application

Configure the rsync-tls replication CRs on the source and destination clusters

On the secondary cluster, under the namespace of the application

  • Click “Installed Operators” > VolSync
  • Click the “Replication Destination” tab
  • Click “Create ReplicationDestination” and select the “Current namespace only” option
  • On the Create ReplicationDestination screen, select YAML view
  • Replace the only “spec” section in the YAML with the below YAML 
spec:
 rsyncTLS:
   accessModes:
     - ReadWriteMany
   capacity: 1Gi
   copyMethod: Snapshot
   serviceType: LoadBalancer
   storageClassName: ocs-storagecluster-cephfs
   volumeSnapshotClassName: ocs-storagecluster-cephfsplugin-snapclass

Notes:
The serviceType is LoadBalancer. see Reference [2] for more details on picking the Service Type. Since I am using ODF, ocs-storagecluster-cephfs and ocs-storagecluster-cephfsplugin-snapclass are the storageClassName and volumeSnapshotClassName, respectively.

  • Check the status from the ReplicationDestination CR; the update should be similar, as shown below.
status:
 conditions:
   - lastTransitionTime: '2023-03-29T06:02:42Z'
     message: Synchronization in-progress
     reason: SyncInProgress
     status: 'True'
     type: Synchronizing
 lastSyncStartTime: '2023-03-29T06:02:07Z'
 latestMoverStatus: {}
 rsyncTLS:
    address: >-
      a5ac4da21394f4ef4b79b4178c8787ea-d67ec11e8f219710.elb.us-east-2.amazonaws.com
    keySecret: volsync-rsync-tls-ostoy-rep-dest

Notes:
We will need the value of the address and the keySecret under the “rsyncTLS” section to set up the source cluster for replication.

  • Copy the keySecret from the destination cluster to the source cluster
  • Log in to the destination cluster, and run the following command to create the psk.txt file.
oc extract secret/volsync-rsync-tls-ostoy-rep-dest --to=../ --keys=psk.txt -n
  • Log in to the source cluster, and execute the following command to create the keySecret.
oc create secret generic volsync-rsync-tls-ostoy-rep-dest --from-file=psk.txt -n
  • Now you are ready to create the ReplicationSource.
  • Log in to your source cluster from the UI
  • Click “Installed Operators” > VolSync
  • Click the “Replication Source” tab
  • Click “Create ReplicationSource” and select the “Current namespace only” option
  • On the Create ReplicationSource screen, select YAML view
  • Replace the only “spec” section in the YAML with the below YAML 
spec:
  rsyncTLS:
    address: >-
      a5ac4da21394f4ef4b79b4178c8787ea-d67ec11e8f219710.elb.us-east-2.amazonaws.com
    copyMethod: Clone
    keySecret: volsync-rsync-tls-ostoy-rep-dest
  sourcePVC: ostoy-pvc
  trigger:
    schedule: '*/5 * * * *'

I am using the address that was provided to me from the status of the ReplicationDestination CR and using the same keySecret that was from the destination.

  • On the destination OCP console, click “Storage” > VolumeSnapShots, and you will see a snapshot has been created.

  • Click “PersistentVolumeClaims”. There is a copy PVC from the source created under the namespace where you create your ReplicationDestination CR. Note the name of the PVC “volsync-ostoy-rep-dest-dst” here.
  • Let’s add some new content to the application on the source cluster.
  • Scale down the deployment for this application on the source
  • On the destination cluster, ensure the application uses “volsync-ostoy-rep-dest-dst” as the PVC in the deployment.
  • Deployment of the sample application on the Destination
  • Check the application and verify the new content was copied to the Destination.
  • The last task is verifying if the solution violates policies using SSH and UID 0.
  • Log in to the ACS console and enable the related policies.
  • Check if any related policies are violated under the application namespace and search by namespace from the violation menu.

References: