OpenShift Lessons Learned

High Level Steps:

• Create a ROSA classic 4.16.4 cluster
• Install AWS Controllers for Kubernetes – Amazon S3 operator
• Create a bucket via the ACK controller Operator

Step-by-Step guide:

• Create a ROSA classic cluster 4.16.4. I have recorded my command for my test below using the default option per the ROSA documentation.

$ rosa login --token="<my-token>"
$ rosa create ocm-role
$ rosa create user-role
$ rosa list account-roles
$ rosa create account-roles
$ rosa create oidc-config --mode=auto --yes
$ rosa create operator-roles --prefix demo --oidc-config-id <oidc-id>
$ rosa create cluster --sts --oidc-config-id <oidc-id> --operator-roles-prefix demo --sts --mode auto

Visit the link in the Reference section for details. In my test, I used a new AWS account and had to enable the ROSA service from the AWS management console. Also, I already have a Red Hat Hybrid cloud console (OCM) account.

• Create a cluster admin via OCM.
• Click on the Red Hat OpenShift tile from the OCM landing page

• Click on the newly created cluster

• Click Access Control tab and select htpasswd under the “Add identity provider”

• Add user and password information and click “Add”

• Click on the “Cluster Roles and Access” side tab –> click “Add user” –> select “cluster-admin” –> add the newly added admin username under the User ID.

• Click the blue “Open console” button to log in to OpenShift Console using the newly created user.

• I used the ROSA documentation to configure the ACK servicer controller for S3, and I made some minor modifications since I found some mistakes on the docs. I used CLI to install and configure the Operator and recorded the steps here.

$ oc login -u <admin-user> <api-url>
$ export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}" | sed 's/-[a-z0-9]{5}$//')
$ export REGION=$(rosa describe cluster -c ${CLUSTER_NAME} --output json | jq -r .region.id)
$ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed 's|^https://||')
$ export AWS_ACCOUNT_ID=aws sts get-caller-identity --query Account --output text
$ export ACK_SERVICE=s3
$ export ACK_SERVICE_ACCOUNT=ack-${ACK_SERVICE}-controller
$ export POLICY_ARN=arn:aws:iam::aws:policy/AmazonS3FullAccess
$ export AWS_PAGER=""
$ export SCRATCH="./tmp/${ROSA_CLUSTER_NAME}/ack"
$ mkdir -p ${SCRATCH}

Make sure you use the consistent variable names.

• Create a trust policy for ACK operator

$ cat <<EOF > "${SCRATCH}/trust-policy.json"
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Condition": {
   "StringEquals" : {
     "${OIDC_ENDPOINT}:sub": "system:serviceaccount:ack-system:${ACK_SERVICE_ACCOUNT}"
   }
 },
 "Principal": {
   "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
 },
 "Action": "sts:AssumeRoleWithWebIdentity"
 }
 ]
}
EOF

• Create AWS IAM ROLE for the ACK operator

$ ROLE_ARN=$(aws iam create-role --role-name "ack-${ACK_SERVICE}-controller" \
   --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
   --query Role.Arn --output text)
$ aws iam attach-role-policy --role-name "ack-${ACK_SERVICE}-controller" \
     --policy-arn ${POLICY_ARN}

• Configure OpenShift to install ACK operator

$ oc new-project ack-system

## note I added RECONCILE_DEFAULT_MAX_CONCURRENT_SYNCS to the configmap
$ cat <<EOF > "${SCRATCH}/config.txt"
ACK_ENABLE_DEVELOPMENT_LOGGING=true
ACK_LOG_LEVEL=debug
ACK_WATCH_NAMESPACE=
AWS_REGION=${REGION}
AWS_ENDPOINT_URL=
ACK_RESOURCE_TAGS=${CLUSTER_NAME}
ENABLE_LEADER_ELECTION=true
LEADER_ELECTION_NAMESPACE=
RECONCILE_DEFAULT_MAX_CONCURRENT_SYNCS='1'
EOF
$ oc -n ack-system create configmap \
  --from-env-file=${SCRATCH}/config.txt ack-${ACK_SERVICE}-user-config

• Install ACK S3 operator from OperatorHub

$ cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: ack-${ACK_SERVICE}-controller
  namespace: ack-system
spec:
  upgradeStrategy: Default
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ack-${ACK_SERVICE}-controller
  namespace: ack-system
spec:
  channel: alpha
  installPlanApproval: Automatic
  name: ack-${ACK_SERVICE}-controller
  source: community-operators
  sourceNamespace: openshift-marketplace
EOF

• Annotate the ACK S3 Operator service account with the AWS IAM role

$ oc create sa ack-s3-controller
$ oc -n ack-system annotate serviceaccount ${ACK_SERVICE_ACCOUNT} \
  eks.amazonaws.com/role-arn=${ROLE_ARN} && \
  oc -n ack-system rollout restart deployment ack-${ACK_SERVICE}-controller

• Validate the operator pod

$ oc -n ack-system get pods
NAME                                READY   STATUS    RESTARTS   AGE
ack-s3-controller-5785d5fbc-qv86g   1/1     Running   0          129

• Create a S3 bucket via ACK S3 operator
• Login to OpenShift console -> click Operators on the left nav -> Installed Operators

• Click Bucket link –> click Create Bucket
• Enter the CR’s and the bucket’s name –> click Create at the bottom of the page.

• Bucket created

• List it using AWS S3 CLI

$ aws s3 ls
2024-08-05 10:39:25 testme1
2024-08-05 12:49:13 testme2-bucket

Congratulation! You have created an S3 bucket via the ACK service controller.

Reference:

• ROSA documentation: https://docs.openshift.com/rosa/cloud_experts_tutorials/cloud-experts-using-aws-ack.html
• Community documentation: https://aws-controllers-k8s.github.io/community/docs/user-docs/openshift/
• Red Hat Hybrid Cloud Console: https://console.redhat.com/

A customer ran into issues backing up and restoring their projects on the ARO cluster. So I figure it is time to run a quick test on this and see where the issues are. After testing for the customer, I decided to share my discovery from my test and hope to save time for others.

The use case is very simple here. We want to back up a project and restore the way things are if I delete the project.

My environment

• ARO 4.12.25
• Velero v1.11.1
• Azure CLI 2.50

Set up ARO for test

• Installing the ARO cluster is straightforward, and if you have the latest version of azure-cli, you can add ‘–version’ to specify the OpenShift version under the latest version supported in the ARO lifecycle [2].
• Install or update the azure-cli version as shown.

$ az version
{
  "azure-cli": "2.50.0",
  "azure-cli-core": "2.50.0",
  "azure-cli-telemetry": "1.0.8",
  "extensions": {}
}

• I followed the reference [1] to create an ARO cluster the 4.11.44 and upgrade to 4.12.25.
• Deploy a stateful application to the cluster that uses Persistent Volume Claim (PVC).

Set up to run Velero

• Install Velero CLI per [3]

$ velero version
Client:
Version: v1.11.1
Git commit: -
Server:
Version: v1.11.1

• I followed the reference [3] to set up the Azure account and blob container.
• Also, use the instruction to create a service principal for Velero to access the storage.

Now, we are ready to install Velero onto the ARO cluster

In the ARO backup and restore documentation [3], it uses “velero/velero-plugin-for-microsoft-azure:v1.1.0”. I learned I must use “velero/velero-plugin-for-microsoft-azure:v1.5.0” to restore my PVC data properly. My inspiration is from reference [5].

Here are the steps I did to backup & restore:

• Add data to my application for testing. My application pod uses a PVC

• Files are added to the directory /var/demo_files

• Now, I am ready to back up my application

$ velero backup create backup-2 --include-namespaces=ostoy --snapshot-volumes=true --include-cluster-resources=true
Backup request "backup-2" submitted successfully.
Run `velero backup describe backup-2` or `velero backup logs backup-2` for more details.

$ velero backup describe backup-2
Name:         backup-2
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.25.11+1485cc9
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=25

Phase:  Completed


Namespaces:
  Included:  ostoy
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  included

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  1h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-07-28 13:06:42 -0700 PDT
Completed:  2023-07-28 13:07:06 -0700 PDT

Expiration:  2023-08-27 13:06:42 -0700 PDT

Total items to be backed up:  2089
Items backed up:              2089

Velero-Native Snapshots:  1 of 1 snapshots completed successfully (specify --details for more information)

• Check if the backup is completed.

$ oc get backup backup-2 -n velero -o yaml

• And the status will report as completed

status:
  completionTimestamp: "2023-07-28T20:07:06Z"
  expiration: "2023-08-27T20:06:42Z"
  formatVersion: 1.1.0
  phase: Completed

• Now let’s delete the application

$ oc delete all --all -n ostoy; oc delete pvc ostoy-pvc; oc delete project ostoy

• Let’s restore it from the backup-2

$ velero restore create restore-2 --from-backup backup-2
Restore request "restore-2" submitted successfully.
Run `velero restore describe restore-2` or `velero restore logs restore-2` for more details.

$ velero restore describe restore-2
Name:         restore-2
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                                 InProgress
Estimated total items to be restored:  128
Items restored so far:                 100

Started:    2023-07-28 13:12:22 -0700 PDT
Completed:  <n/a>

Backup:  backup-2

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Existing Resource Policy:   <none>
ItemOperationTimeout:       1h0m0s

Preserve Service NodePorts:  auto

• Let’s check if the restore is completed

$ oc get restore restore-2 -n velero -o yaml

• Wait for the status shows as completed.
• Now, we can check the project and application data are restored.
• All pods are running, and the added data is restored below.

When using “velero/velero-plugin-for-microsoft-azure:v1.1.0” for Velero, I could not restore the data from the PVC. By using “velero/velero-plugin-for-microsoft-azure:v1.5.0” for Velero, I can now restore the application on the same ARO cluster and the application data.

Reference

• [1] ARO 4 installation: https://learn.microsoft.com/en-us/azure/openshift/tutorial-create-cluster
• [2] Support Lifecycle for ARO 4: https://learn.microsoft.com/en-us/azure/openshift/support-lifecycle
• [3] ARO backup and restore: https://learn.microsoft.com/en-us/azure/openshift/howto-create-a-backup
• [4] Velero docs: https://velero.io/docs/main/basic-install/
• [5] Inspiration reference: https://learn.microsoft.com/en-us/azure/aks/hybrid/backup-workload-cluster

OpenShift provides ways to observe and monitor cluster health. When we have more clusters, we want to monitor all clusters from a centralized location. We can use Red Hat Advanced Cluster Management (RHACM) helps to manage and control all the clusters from a single console. Also, we can enable observability from ACM to observe all clusters from RHACM.

The biggest complaint I got is that we are getting so many alerts. How do we really know when and how to react to the alerts that my organization cares about?

This is my example of how I will start tackling the issue. I am going to share the steps here on how I set up my OpenShift environment to try to solve the problem.

Environment:

• OpenShift 4.11
• Red Hat Advanced Cluster Management Operator 2.7

My test environment

• Install OpenShift 4.11
• Install Red Hat Advanced Cluster Management Operator 2.7

Click the OperatorHub from the OpenShift console left menu -> click on “Advanced Cluster Management for Kubernetes” Tile -> click install

• Once RHACM is installed and create “MultiClusterHub” customer resource (CR)

Enable the Observability

• Prepare an S3 bucket. In my case, I used AWS for my object storage.

aws s3 mb s3://shchan-acm

• Create “open-cluster-management-observability” namespace

oc create namespace open-cluster-management-observability 

• Create “pull-secret” in the namespace

DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-`

oc create secret generic multiclusterhub-operator-pull-secret \
    -n open-cluster-management-observability \
    --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
    --type=kubernetes.io/dockerconfigjson

• Create a YAML file as below and name it “thanos-object-storage.yaml.” The credential will need to have proper permission to access the bucket. I am using an IAM user that has full access to the bucket. See the reference section for permission details.

apiVersion: v1
kind: Secret
metadata:
  name: thanos-object-storage
  namespace: open-cluster-management-observability
type: Opaque
stringData:
  thanos.yaml: |
    type: s3
    config:
      bucket: shchan-acm
      endpoint: s3.us-east-2.amazonaws.com
      insecure: true
      access_key: xxxxxxxxxxxxxxxxx
      secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

• Create the secret for object storage

oc create -f thanos-object-storage.yaml -n open-cluster-management-observability

• Create MultiClusterObservability customer resource in a YAML file, multiclusterobservability_cr.yaml.

apiVersion: observability.open-cluster-management.io/v1beta2
kind: MultiClusterObservability
metadata:
  name: observability
spec:
  observabilityAddonSpec: {}
  storageConfig:
    metricObjectStorage:
      name: thanos-object-storage
      key: thanos.yaml

• Run the following command to create the CR

oc apply -f multiclusterobservability_cr.yaml

Create custom rule

The use case here is to get a notification for a given issue when it happens. Since every alert is sent to the same notifier. It is not easy to react to the important alert.

• Create a “kube-node-health” group for alerting when any node is down for any reason. Create a Configmap “thanos-ruler-custom-rules” with the following rules in the open-cluster-management-observability namespace. Add “custom_rules.yaml” in the data section of the YAML file. Noted that I added “tag: kubenode” in the labels section. Note that it can be any label. This is just an example.

data:
 custom_rules.yaml: |
   groups:
     - name: kube-node-health
       rules:
       - alert: NodeNotReady
         annotations:
           summary: Notify when any node on a cluster is in NotReady state
           description: "One of the node of the cluster is down: Cluster {{ $labels.cluster }} {{ $labels.clusterID }}."
         expr: kube_node_status_condition{condition="Ready",job="kube-state-metrics",status="true"} != 1
         for: 5s
         labels:
           instance: "{{ $labels.instance }}"
           cluster: "{{ $labels.cluster }}"
           clusterID: "{{ $labels.clusterID }}"
           tag: kubenode
           severity: critical

• You can view the logs from one of the alert manager pods to monitor if the rules are applied correctly. You can check if the log has any errors.

To test the alert from the ACM console

• Shut down one worker node
• Go to ACM

• Click Infrastucture -> Cluster -> Grafana

• Login the grafana dashboard
• Click “Explore”

• Click “Metrics browser to expand
• Select “alertname” and all the and “NodeNotReady” alert shows up under the list

• Here the alert was fired because one of the nodes was down.

Let’s configure the alert manager

We want to send this “NodeNotReady” alert to a specific Slack channel.

• Extract the data from the “alertmanager-config secret.

oc -n open-cluster-management-observability get secret alertmanager-config --template='{{ index .data "alertmanager.yaml" }}' |base64 -d > alertmanager.yaml

• Edit the alert-manager.yaml file as the following example. Note that I have 2 Slack channels for 2 receivers, respectively. One for the specific “tag: kubenode” alert and the other one for all the rest alerts.

"global":
  "slack_api_url": "https://hooks.slack.com/services/TDxxxx3S6/B0xxxxZLE2D/BN35PToxxxxmRTRxxxxN6R4"

"route":
  "group_by":
  - "alertname"
  "group_interval": "5m"
  "group_wait": "30s"
  "repeat_interval": "12h"
  "receiver": "my-team"
  "routes":
  - "match":
      "tag": "kubenode"
    "receiver": "slack-notification"

"receivers":
- "name": "slack-notification"
  "slack_configs":
  - "api_url": "https://hooks.slack.com/services/TDxxxx3S6/B0xxxxUK7B7/vMtVpxxxx4kESxxxxeDSYu3"
    "channel": "#kubenode"
    "text": "{{ range .Alerts }}<!channel> {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"


- "name": "my-team"
  "slack_configs":
  - "api_url": "https://hooks.slack.com/services/TDxxxx3S6/B0xxxxZLE2D/BN35PToxxxxmRTRxxxxN6R4"
    "channel": "#critical-alert"
    "text": "{{ .GroupLabels.alertname }}"

• Save the alertmanager.yaml and replace the secret.

oc -n open-cluster-management-observability create secret generic alertmanager-config --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n open-cluster-management-observability replace secret --filename=-

• When the Node is shut down, a message should show in the slack channel like the one below.

• You will also see many alerts show up on the other channel, like the one below.

The idea here is to get meaningful alerts to the team and know what to do with the alerts.

The next is to continue to refine the customer rules and alert manager configuration per the needs.

Reference

• Observability documentation: https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.7/html/observability/index

Prerequisites

• An OpenShift 4 cluster
• Compliance Operator installed

Overview

Steps to Generate OpenSCAP Evaluation Report

• Log in OpenShift Cluster

# oc login -u <username> https://api.<clusterid>.<subdomain>

• Assuming you have already installed the OpenShift Compliance Operator. Next, check the access to the OpenShift Compliance project using the command shown below.

# oc project openshift-compliance

• The default compliance profiles will be available once the operator is installed. The command below lists out all compliance profiles Custom Resource Definition (CRD) profiles.compliance.openshift.io.

# oc get profiles.compliance.openshift.io

• To get custom resource ScanSetting via the below command. It shows two default scan settings.

# oc get ScanSetting
NAME                 AGE
default              2d10h
default-auto-apply   2d10h

• Check out the “default” ScanSetting

Name:         default
Namespace:    openshift-compliance
Labels:       <none>
Annotations:  <none>
API Version:  compliance.openshift.io/v1alpha1
Kind:         ScanSetting
Metadata:
  Creation Timestamp:  2021-10-19T16:22:18Z
  Generation:          1
  Managed Fields:
...
  Resource Version:  776981
  UID:               f453726d-665a-432e-88a9-a4ad60176ac7
Raw Result Storage:
  Pv Access Modes:
    ReadWriteOnce
  Rotation:  3
  Size:      1Gi
Roles:
  worker
  master
Scan Tolerations:
  Effect:    NoSchedule
  Key:       node-role.kubernetes.io/master
  Operator:  Exists
Schedule:    0 1 * * *
Events:      <none>

• Create ScanSettingBinding as shown in scan-setting-binding-example.yaml below.

# cat scan-setting-binding-example.yaml
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: cis-compliance
profiles:
  - name: ocp4-cis-node
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1

• Create the above sample ScanSettingBinding custom resource.

# oc create -f scan-setting-binding-example.yaml

• Verify the creation of the ScanSettingBinding

# oc get scansettingbinding

• Custom resource ComplianceSuites is to help tracking the state of the scans. The following command is to check the state of the scan you defined in your ScanSettingBinding.

# oc get compliancesuite
NAME             PHASE     RESULT
cis-compliance   RUNNING   NOT-AVAILABLE

• ComplianceScan custom resource needs all the parameters to run OpenSCAP, such as profile id, image to get the content from, and data stream file path. It also can constain operational parameter.

# oc get compliancescan
NAME                   PHASE   RESULT
ocp4-cis               DONE    NON-COMPLIANT

• While the custom resource ComplianceCheckResult shows the aggregate result of the scan, it is useful to review the raw result from the scanner. The raw results are produced in the ARF format and can be large. Therefore, Compliance Operator creates a persistent volume (PV) for the raw result from the scan. Let’s take a look if the PVC is created for the scan.

# oc get pvc
NAME                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ocp4-cis               Bound    pvc-5ee57b02-2f6b-4997-a45c-3c4df254099d   1Gi        RWO            gp2            27m
ocp4-cis-node-master   Bound    pvc-57c7c411-fc9f-4a4d-a713-de91c934af1a   1Gi        RWO            gp2            27m
ocp4-cis-node-worker   Bound    pvc-7266404a-6691-4f3d-9762-9e30e50fdadb   1Gi        RWO            gp2            28m

• Once we know the raw result is created, we need the oc-compliance tool to get the raw result XML file. You will need to login to the registry.redhat.io.

# podman login -u <user> registry.redhat.io

• Download the oc-compliance tool

podman run --rm --entrypoint /bin/cat registry.redhat.io/compliance/oc-compliance-rhel8 /usr/bin/oc-compliance > ~/usr/bin/oc-compliance

• Fetch the raw results to a temporary location (/tmp/cis-compliance)

# oc-compliance fetch-raw scansettingbindings cis-compliance -o /tmp/cis-compliance
Fetching results for cis-compliance scans: ocp4-cis-node-worker, ocp4-cis-node-master, ocp4-cis
Fetching raw compliance results for scan 'ocp4-cis-node-worker'.....
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis-node-worker
Fetching raw compliance results for scan 'ocp4-cis-node-master'.....
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis-node-master
Fetching raw compliance results for scan 'ocp4-cis'...........
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis

• Inspect the output filesystem and extract the *.bzip2 file

# cd /tmp/cis-compliance/ocp4-cis
# ls
ocp4-cis-api-checks-pod.xml.bzip2

# bunzip2 -c  ocp4-cis-api-checks-pod.xml.bzip2  > /tmp/cis-compliance/ocp4-cis/ocp4-cis-api-checks-pod.xml

# ls /tmp/cis-compliance/ocp4-cis/ocp4-cis-api-checks-pod.xml
/tmp/cis-compliance/ocp4-cis/ocp4-cis-api-checks-pod.xml

• Convert ARF XML to html

# oscap xccdf generate report ocp4-cis-api-checks-pod.xml > report.html

• View the HTML as shown below.

Reference

• OpenShift Compliance Operator
• Compliance as Code

Thank you Juan Antonio Osorio Robles for sharing the diagram!