Way too many alerts? Which alerts are important?

OpenShift provides ways to observe and monitor cluster health. When we have more clusters, we want to monitor all clusters from a centralized location. We can use Red Hat Advanced Cluster Management (RHACM) helps to manage and control all the clusters from a single console. Also, we can enable observability from ACM to observe all clusters from RHACM.

The biggest complaint I got is that we are getting so many alerts. How do we really know when and how to react to the alerts that my organization cares about?

This is my example of how I will start tackling the issue. I am going to share the steps here on how I set up my OpenShift environment to try to solve the problem.


  • OpenShift 4.11
  • Red Hat Advanced Cluster Management Operator 2.7

My test environment

  • Install OpenShift 4.11
  • Install Red Hat Advanced Cluster Management Operator 2.7

Click the OperatorHub from the OpenShift console left menu -> click on “Advanced Cluster Management for Kubernetes” Tile -> click install

  • Once RHACM is installed and create “MultiClusterHub” customer resource (CR)

Enable the Observability

  • Prepare an S3 bucket. In my case, I used AWS for my object storage.
aws s3 mb s3://shchan-acm
  • Create “open-cluster-management-observability” namespace
oc create namespace open-cluster-management-observability 
  • Create “pull-secret” in the namespace
DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-`

oc create secret generic multiclusterhub-operator-pull-secret \
    -n open-cluster-management-observability \
    --from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
  • Create a YAML file as below and name it “thanos-object-storage.yaml.” The credential will need to have proper permission to access the bucket. I am using an IAM user that has full access to the bucket. See the reference section for permission details.
apiVersion: v1
kind: Secret
  name: thanos-object-storage
  namespace: open-cluster-management-observability
type: Opaque
  thanos.yaml: |
    type: s3
      bucket: shchan-acm
      endpoint: s3.us-east-2.amazonaws.com
      insecure: true
      access_key: xxxxxxxxxxxxxxxxx
      secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  • Create the secret for object storage
oc create -f thanos-object-storage.yaml -n open-cluster-management-observability
  • Create MultiClusterObservability customer resource in a YAML file, multiclusterobservability_cr.yaml.
apiVersion: observability.open-cluster-management.io/v1beta2
kind: MultiClusterObservability
  name: observability
  observabilityAddonSpec: {}
      name: thanos-object-storage
      key: thanos.yaml
  • Run the following command to create the CR
oc apply -f multiclusterobservability_cr.yaml

Create custom rule

The use case here is to get a notification for a given issue when it happens. Since every alert is sent to the same notifier. It is not easy to react to the important alert.

  • Create a “kube-node-health” group for alerting when any node is down for any reason. Create a Configmap “thanos-ruler-custom-rules” with the following rules in the open-cluster-management-observability namespace. Add “custom_rules.yaml” in the data section of the YAML file. Noted that I added “tag: kubenode” in the labels section. Note that it can be any label. This is just an example.
 custom_rules.yaml: |
     - name: kube-node-health
       - alert: NodeNotReady
           summary: Notify when any node on a cluster is in NotReady state
           description: "One of the node of the cluster is down: Cluster {{ $labels.cluster }} {{ $labels.clusterID }}."
         expr: kube_node_status_condition{condition="Ready",job="kube-state-metrics",status="true"} != 1
         for: 5s
           instance: "{{ $labels.instance }}"
           cluster: "{{ $labels.cluster }}"
           clusterID: "{{ $labels.clusterID }}"
           tag: kubenode
           severity: critical
  • You can view the logs from one of the alert manager pods to monitor if the rules are applied correctly. You can check if the log has any errors.

To test the alert from the ACM console

  • Shut down one worker node
  • Go to ACM
  • Click Infrastucture -> Cluster -> Grafana
  • Login the grafana dashboard
  • Click “Explore”
  • Click “Metrics browser to expand
  • Select “alertname” and all the and “NodeNotReady” alert shows up under the list
  • Here the alert was fired because one of the nodes was down.

Let’s configure the alert manager

We want to send this “NodeNotReady” alert to a specific Slack channel.

  • Extract the data from the “alertmanager-config secret.
oc -n open-cluster-management-observability get secret alertmanager-config --template='{{ index .data "alertmanager.yaml" }}' |base64 -d > alertmanager.yaml
  • Edit the alert-manager.yaml file as the following example. Note that I have 2 Slack channels for 2 receivers, respectively. One for the specific “tag: kubenode” alert and the other one for all the rest alerts.
  "slack_api_url": "https://hooks.slack.com/services/TDxxxx3S6/B0xxxxZLE2D/BN35PToxxxxmRTRxxxxN6R4"

  - "alertname"
  "group_interval": "5m"
  "group_wait": "30s"
  "repeat_interval": "12h"
  "receiver": "my-team"
  - "match":
      "tag": "kubenode"
    "receiver": "slack-notification"

- "name": "slack-notification"
  - "api_url": "https://hooks.slack.com/services/TDxxxx3S6/B0xxxxUK7B7/vMtVpxxxx4kESxxxxeDSYu3"
    "channel": "#kubenode"
    "text": "{{ range .Alerts }}<!channel> {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"

- "name": "my-team"
  - "api_url": "https://hooks.slack.com/services/TDxxxx3S6/B0xxxxZLE2D/BN35PToxxxxmRTRxxxxN6R4"
    "channel": "#critical-alert"
    "text": "{{ .GroupLabels.alertname }}"
  • Save the alertmanager.yaml and replace the secret.
oc -n open-cluster-management-observability create secret generic alertmanager-config --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n open-cluster-management-observability replace secret --filename=-
  • When the Node is shut down, a message should show in the slack channel like the one below.
  • You will also see many alerts show up on the other channel, like the one below.

The idea here is to get meaningful alerts to the team and know what to do with the alerts.

The next is to continue to refine the customer rules and alert manager configuration per the needs.


OpenShift Extended Update Support (EUS)

Something I recently learned from testing the EUS to EUS process, and I like to share the steps here. Starting OpenShift Container Platform (OCP) 4.12, Red hat is adding an additional 6 months of Extended Update Support (EUS) on even-numbered OCP release for the x86_64 architecture.

We can upgrade from a EUS version to the next EUS version with only a single reboot of non-control plane hosts. There are caveats that are listed in the reference section below.

Steps to upgrade from 4.10.x to 4.12 using the EUS-to-EUS upgrade

  • Verify that all machine config pools display a status of Up to date and that no machine config pool displays a status of “UPDATING.”
  • Set your channel to eus-<4.y+2>
  • The channel is set to eus-4.12
  • Pause all worker machine pools except for the master machine pool
  • Update 4+1 only with Partial cluster update
  • Wait for the partial upgrade to complete
  • Make sure the partial update is completed
  • No reboot for workers as noted that workers are paused on the K8S v 1.23
  • Check the OCP console
  • Check the KCS ​​https://access.redhat.com/articles/6955381. Review the k8s API version update and ensure the workloads are working properly and will be using the newly updated K8S API version.
  • Execute the following command:
    oc -n openshift-config patch cm admin-acks --patch '{"data":{"ack-4.11-kube-1.25-api-removals-in-4.12":"true"}}' --type=merge
  • OCP console shows
  • Click resume all updates
  • Double-check all machine pools. If updating, please wait for it to complete.
  • Upgrade completed and now showing OCP 4.12 (K8S v1.25)


Application Data Replication

My use case is to replicate the stateful Springboot application for disaster recovery. The application runs on OpenShift, and we want to leverage the existing toolsets to solve this problem. If it is just replicating the data from one data center to another, it should be super simple, right? In this blog, I share my journey of picking my solution.

The requirements are:

  • No code change
  • Cannot use ssh to copy the data
  • Cannot run the pod for replication using privileged containers
  • Must meet the security requirements

Solution 1: Writing the data to object storage

The simplest solution would be to have the application write the data to an object bucket, so we can mirror the object storage directly. However, it requires code changes for all the current applications.

Solution 2: Use rsync replication with VolSync Operator

We tested will the rsync-based replication using VolSync Operator. This will not be a good choice because it violates security policies on using SSH and UID 0 within containers.

Solution 3: Use rsync-tls replication with VolSync Operator

This is the one that meets all the requirements, and I am testing it out.

My test environment includes the following:

  • OpenShift (OCP) 4.11
  • OpenShift Data Foundation (ODF) 4.11
  • Advanced Cluster Security (ACS) 3.74.1
  • VolSync Operator 0.70


  • Install two OCP 4.11 clusters
  • Install and configure ODF on both OCP clusters
  • Install and configure ACS central on one of the cluster
  • Install and configure ACS secured cluster on both cluster
  • Install VolSync Operator on both clusters
  • Install a sample stateful application

Configure the rsync-tls replication CRs on the source and destination clusters

On the secondary cluster, under the namespace of the application

  • Click “Installed Operators” > VolSync
  • Click the “Replication Destination” tab
  • Click “Create ReplicationDestination” and select the “Current namespace only” option
  • On the Create ReplicationDestination screen, select YAML view
  • Replace the only “spec” section in the YAML with the below YAML
     - ReadWriteMany
   capacity: 1Gi
   copyMethod: Snapshot
   serviceType: LoadBalancer
   storageClassName: ocs-storagecluster-cephfs
   volumeSnapshotClassName: ocs-storagecluster-cephfsplugin-snapclass

The serviceType is LoadBalancer. see Reference [2] for more details on picking the Service Type. Since I am using ODF, ocs-storagecluster-cephfs and ocs-storagecluster-cephfsplugin-snapclass are the storageClassName and volumeSnapshotClassName, respectively.

  • Check the status from the ReplicationDestination CR; the update should be similar, as shown below.
   - lastTransitionTime: '2023-03-29T06:02:42Z'
     message: Synchronization in-progress
     reason: SyncInProgress
     status: 'True'
     type: Synchronizing
 lastSyncStartTime: '2023-03-29T06:02:07Z'
 latestMoverStatus: {}
    address: >-
    keySecret: volsync-rsync-tls-ostoy-rep-dest

We will need the value of the address and the keySecret under the “rsyncTLS” section to set up the source cluster for replication.

  • Copy the keySecret from the destination cluster to the source cluster
  • Log in to the destination cluster, and run the following command to create the psk.txt file.
oc extract secret/volsync-rsync-tls-ostoy-rep-dest --to=../ --keys=psk.txt -n
  • Log in to the source cluster, and execute the following command to create the keySecret.
oc create secret generic volsync-rsync-tls-ostoy-rep-dest --from-file=psk.txt -n
  • Now you are ready to create the ReplicationSource.
  • Log in to your source cluster from the UI
  • Click “Installed Operators” > VolSync
  • Click the “Replication Source” tab
  • Click “Create ReplicationSource” and select the “Current namespace only” option
  • On the Create ReplicationSource screen, select YAML view
  • Replace the only “spec” section in the YAML with the below YAML
    address: >-
    copyMethod: Clone
    keySecret: volsync-rsync-tls-ostoy-rep-dest
  sourcePVC: ostoy-pvc
    schedule: '*/5 * * * *'

I am using the address that was provided to me from the status of the ReplicationDestination CR and using the same keySecret that was from the destination.

  • On the destination OCP console, click “Storage” > VolumeSnapShots, and you will see a snapshot has been created.

  • Click “PersistentVolumeClaims”. There is a copy PVC from the source created under the namespace where you create your ReplicationDestination CR. Note the name of the PVC “volsync-ostoy-rep-dest-dst” here.
  • Let’s add some new content to the application on the source cluster.
  • Scale down the deployment for this application on the source
  • On the destination cluster, ensure the application uses “volsync-ostoy-rep-dest-dst” as the PVC in the deployment.
  • Deployment of the sample application on the Destination
  • Check the application and verify the new content was copied to the Destination.
  • The last task is verifying if the solution violates policies using SSH and UID 0.
  • Log in to the ACS console and enable the related policies.
  • Check if any related policies are violated under the application namespace and search by namespace from the violation menu.


Getting Started on OpenShift Compliance Operator

There are many documents out there on the OpenShift Compliance Operator. I share this with customers who want to learn how to work with OpenShift Operator and helped them to get started on the OpenShift Compliance Operator.

In this blog, I will walk you through how to generate the OpenSCAP evaluation report using the OpenShift Compliance Operator.

OpenShift Compliance Operator can be easily installed on OpenShift 4 as a security feature with the OpenShift Container Platform. The Compliance Operator uses OpenSCAP, a NIST-certified tool, to scan and enforce security policies provided by the content.



The compliance operator uses many custom resources. The diagram below helps me to understand the relationship between all the resources. In addition, the OpenShift documentation has details about the Compliance Operator custom resources.

Steps to Generate OpenSCAP Evaluation Report

Some default custom resources come as part of the compliance operator installation, such as ProfileBunble, Profiles, and ScanSetting.

First, we need to create the ScanSettingBinding, which defines the Profiles and the ScanSetting. The ScanSettingBinding tells Compliance Operator to evaluate for Profile(s) A with the specific scan setting.

  • Log in OpenShift Cluster
# oc login -u <username> https://api.<clusterid>.<subdomain>
# oc project openshift-compliance
  • The default compliance profiles will be available once the operator is installed. The command below lists out all compliance profiles Custom Resource Definition (CRD) profiles.compliance.openshift.io.
# oc get profiles.compliance.openshift.io
  • To get custom resource ScanSetting via the below command. It shows two default scan settings.
# oc get ScanSetting
NAME                 AGE
default              2d10h
default-auto-apply   2d10h
  • Check out the “default” ScanSetting
Name:         default
Namespace:    openshift-compliance
Labels:       <none>
Annotations:  <none>
API Version:  compliance.openshift.io/v1alpha1
Kind:         ScanSetting
  Creation Timestamp:  2021-10-19T16:22:18Z
  Generation:          1
  Managed Fields:
  Resource Version:  776981
  UID:               f453726d-665a-432e-88a9-a4ad60176ac7
Raw Result Storage:
  Pv Access Modes:
  Rotation:  3
  Size:      1Gi
Scan Tolerations:
  Effect:    NoSchedule
  Key:       node-role.kubernetes.io/master
  Operator:  Exists
Schedule:    0 1 * * *
Events:      <none>
  • Create ScanSettingBinding as shown in scan-setting-binding-example.yaml below.
# cat scan-setting-binding-example.yaml
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
  name: cis-compliance
  - name: ocp4-cis-node
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  name: default
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1
  • Create the above sample ScanSettingBinding custom resource.
# oc create -f scan-setting-binding-example.yaml
  • Verify the creation of the ScanSettingBinding
# oc get scansettingbinding
  • Custom resource ComplianceSuites is to help tracking the state of the scans. The following command is to check the state of the scan you defined in your ScanSettingBinding.
# oc get compliancesuite
NAME             PHASE     RESULT
cis-compliance   RUNNING   NOT-AVAILABLE
  • ComplianceScan custom resource needs all the parameters to run OpenSCAP, such as profile id, image to get the content from, and data stream file path. It also can constain operational parameter.
# oc get compliancescan
NAME                   PHASE   RESULT
ocp4-cis               DONE    NON-COMPLIANT
  • While the custom resource ComplianceCheckResult shows the aggregate result of the scan, it is useful to review the raw result from the scanner. The raw results are produced in the ARF format and can be large. Therefore, Compliance Operator creates a persistent volume (PV) for the raw result from the scan. Let’s take a look if the PVC is created for the scan.
# oc get pvc
NAME                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ocp4-cis               Bound    pvc-5ee57b02-2f6b-4997-a45c-3c4df254099d   1Gi        RWO            gp2            27m
ocp4-cis-node-master   Bound    pvc-57c7c411-fc9f-4a4d-a713-de91c934af1a   1Gi        RWO            gp2            27m
ocp4-cis-node-worker   Bound    pvc-7266404a-6691-4f3d-9762-9e30e50fdadb   1Gi        RWO            gp2            28m
  • Once we know the raw result is created, we need the oc-compliance tool to get the raw result XML file. You will need to login to the registry.redhat.io.
# podman login -u <user> registry.redhat.io
  • Download the oc-compliance tool
podman run --rm --entrypoint /bin/cat registry.redhat.io/compliance/oc-compliance-rhel8 /usr/bin/oc-compliance > ~/usr/bin/oc-compliance
  • Fetch the raw results to a temporary location (/tmp/cis-compliance)
# oc-compliance fetch-raw scansettingbindings cis-compliance -o /tmp/cis-compliance
Fetching results for cis-compliance scans: ocp4-cis-node-worker, ocp4-cis-node-master, ocp4-cis
Fetching raw compliance results for scan 'ocp4-cis-node-worker'.....
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis-node-worker
Fetching raw compliance results for scan 'ocp4-cis-node-master'.....
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis-node-master
Fetching raw compliance results for scan 'ocp4-cis'...........
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis
  • Inspect the output filesystem and extract the *.bzip2 file
# cd /tmp/cis-compliance/ocp4-cis
# ls

# bunzip2 -c  ocp4-cis-api-checks-pod.xml.bzip2  > /tmp/cis-compliance/ocp4-cis/ocp4-cis-api-checks-pod.xml

# ls /tmp/cis-compliance/ocp4-cis/ocp4-cis-api-checks-pod.xml
  • Convert ARF XML to html
# oscap xccdf generate report ocp4-cis-api-checks-pod.xml > report.html
  • View the HTML as shown below.


Thank you Juan Antonio Osorio Robles for sharing the diagram!

Argocd SSO Set up

  • Go to Administrator Console
  • Create a new project called keycloak
  • Click Operator
  • Click OperatorHub
  • Click on the Red Hat Single Sign-On Operator
  • Click Install
  • Click Install
  • Click “Create instance” in the Keycloak tile
  • The Keycloak CR is shown below
apiVersion: keycloak.org/v1alpha1
kind: Keycloak
  name: keycloak-dev
    app: keycloak-dev
  namespace: keycloak
    enabled: true
  instances: 1
  • Click Create
  • Go to Workloads > Pods
  • Click the keycload-dev creation and return “true”
$ oc get keycloak keycloak-dev -n keycloak -o jsonpath='{.status.ready}'
  • Operators > Installed Operators > Red Hat Single Sign-On Operator
  • Click Create instance
  • Enter the KeycloakRealm as shown below
apiVersion: keycloak.org/v1alpha1
kind: KeycloakRealm
  name: keycloakrealm
    realm: keycloakrealm
  namespace: keycloak
      app: keycloak-dev
    enabled: true
    displayName: "Keycloak-dev Realm"
    realm: keycloakrealm
  • Click Create
  • Make sure it returns true
$ oc get keycloakrealm keycloakrealm -n keycloak -o jsonpath='{.status.ready}'
  • Get the Keycloak Admin user secret name
$ oc get keycloak keycloak-dev --output="jsonpath={.status.credentialSecret}"
  • Get the Admin username and password
$ oc get secret credential-keycloak-dev -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'
  • Run the following to find out the URLs of Keycloak:
KEYCLOAK_URL=https://$(oc get route keycloak --template='{{ .spec.host }}')/auth &&
echo "" &&
echo "Keycloak:                 $KEYCLOAK_URL" &&
echo "Keycloak Admin Console:   $KEYCLOAK_URL/admin" &&
echo "Keycloak Account Console: $KEYCLOAK_URL/realms/myrealm/account" &&
echo ""
  • Open a browser with the Admin URL
  • Login with the admin username and password
  • Click Client on the left nav
  • Click Create on the right top corner
  • Enter the Argocd URL and the name of the client as ‘argocd’
  • Click Save
  • Set Access Type to confidential
  • Set Valid Redirect URIs to <argocd-url>/auth/callback
  • Set Base URL to /applications
  • Click Save
  • Scroll up and click “Credential” tab
  • IMPORTANT: Copy the secret and you will need this later
  • Configure the Group claim
  • Click Client Scope on the left nav
  • Click Create on the right
  • Set Name as group
  • Set Protocol as openid-connecgt
  • Display On Content Scope: on
  • Include to Token Scope: on
  • Click save
  • Click “Mappers” tab
  • Click Create on the top right
  • Set name as groups
  • Set Mapper Type as Group Membership
  • Set Token Claim Name as groups`
  • Click Clients on the left nav
  • Click argocd
  • Click “Client Scopes” tab
  • Select groups > Add selected
  • Click Groups on left nav
  • Click Create
  • Set the name as ArgoCDAdmins
  • Click Save
  • Encode the argocd credential you saved before
echo -n '<argocd credential>' | base64
  • Edit the argocd-secret
oc edit secret argocd-secret -n openshift-gitops
  • add the “oidc.keycloak.clientSecret: <encoded credential> as shown below.
apiVersion: v1
kind: Secret
  name: argocd-secret
  oidc.keycloak.clientSecret: <encoded credential>
  • Edit argocd Custom Resource
oc edit argocd -n openshift-gitops
  • Add the following into the yaml. Make sure update the issuer to make your settings
oidcConfig: |
    name: OpenShift Single Sign-On
    issuer: https://keycloak-keycloak.apps.cluster-72c5r.72c5r.sandbox1784.opentlc.com/auth/realms/keycloakrealm
    clientID: argocd
    clientSecret: $oidc.keycloak.clientSecret
    requestedScopes: ["openid", "profile", "email", "groups"]
  • From OpenShift Console top right corner, click About
  • Copy the API URL from the following screen
  • Go back to Keycloak, click Identity Providers on left nav
  • Select OpenShift v4 from the dropdown list
  • Set Display Name: Login with Openshift
  • Set Client ID: keycload-broker
  • Set Client Secret: <anything that you can remember>
  • Set Base URL: API URL
  • Set Default Scopes: user:full
  • Click Save
  • Add an Oauth Client
oc create -f <(echo '
kind: OAuthClient
apiVersion: oauth.openshift.io/v1
 name: keycloak-broker 
secret: "12345" 
- "https://keycloak-keycloak.apps.cluster-72c5r.72c5r.sandbox1784.opentlc.com/auth/realms/keycloakrealm/broker/openshift-v4/endpoint" 
grantMethod: prompt 
  • Configure the RBAC
oc edit configmap argocd-rbac-cm -n openshift-gitops
  • Modify the data as shown below
apiVersion: v1
kind: ConfigMap
  name: argocd-rbac-cm
  policy.csv: |
    g, ArgoCDAdmins, role:admin
  • Go to the Argocd URL, you will see the SSO icon. Click “LOG IN VIA OPENSHIFT”
  • Click Log in Openshift

Installing OpenShift using Temporary Credentials

One of the most frequently asked questions recently is how to install OpenShift on AWS with temporary credentials. The default OpenShift provisioning using AWS key and secret, which requires the Administrator privileges. The temporary credential often refers to AWS Security Token Service (STS), which allows end-users to assume an IAM role resulting in short-lived credentials.

Developers or platform teams will require approval from their security team to access the company AWS account. It can be challenging in some organizations to get access to Administrator privileges.

OpenShift 4.7 support for AWS Secure Token Service in manual mode is in Tech Preview. I decided to explore a little deeper—the exercise based on the information both on the OpenShift documentation and the upstream repos. I am recording the notes from my test run. I hope you will find it helpful.

OpenShift 4 version

OCP 4.7.9

Build sts-preflight binary

git clone https://github.com/sjenning/sts-preflight.git
go get github.com/sjenning/sts-preflight
cd <sts-preflight directory>
go build .

Getting the AWS STS

As an AWS administrator, I found the sts-preflight tool helpful in this exercise. The documentation has the manual steps, but I choose to use the sts-preflight tool here.

  • Create STS infrastructure in AWS:
./sts-preflight  create --infra-name <sts infra name> --region <aws region>

# ./sts-preflight  create --infra-name sc-example --region us-west-1
2021/04/28 13:24:42 Generating RSA keypair
2021/04/28 13:24:56 Writing private key to _output/sa-signer
2021/04/28 13:24:56 Writing public key to _output/sa-signer.pub
2021/04/28 13:24:56 Copying signing key for use by installer
2021/04/28 13:24:56 Reading public key
2021/04/28 13:24:56 Writing JWKS to _output/keys.json
2021/04/28 13:24:57 Bucket sc-example-installer created
2021/04/28 13:24:57 OIDC discovery document at .well-known/openid-configuration updated
2021/04/28 13:24:57 JWKS at keys.json updated
2021/04/28 13:24:57 OIDC provider created arn:aws:iam::##########:oidc-provider/s3.us-west-1.amazonaws.com/sc-example-installer
2021/04/28 13:24:57 Role created arn:aws:iam::##########:role/sc-example-installer
2021/04/28 13:24:58 AdministratorAccess attached to Role sc-example-installer
  • Create an OIDC token:
# ./sts-preflight token
2021/04/28 13:27:06 Token written to _output/token
  • Get STS credential:
# ./sts-preflight assume
Run these commands to use the STS credentials
export AWS_ACCESS_KEY_ID=<temporary key>
export AWS_SECRET_ACCESS_KEY=<temporary secret>
export AWS_SESSION_TOKEN=<session token>
  • The above short-lived key, secret, and token can be given to the person who are installing OpenShift.
  • Export all the AWS environment variables before proceeding to installation.

Start the Installation

As a Developer or OpenShift Admin, you will get the temporary credentials information and export the AWS environment variables before installing the OCP cluster.

# oc adm release extract quay.io/openshift-release-dev/ocp-release:4.7.9-x86_64 --credentials-requests --cloud=aws --to=./credreqs ; cat ./credreqs/*.yaml > credreqs.yaml
  • Create install-config.yaml for installation:
# ./openshift-install create install-config --dir=./sc-sts
? SSH Public Key /root/.ssh/id_rsa.pub
? Platform aws
INFO Credentials loaded from default AWS environment variables
? Region us-east-1
? Base Domain sc.ocp4demo.live
? Cluster Name sc-sts
? Pull Secret [? for help] 
INFO Install-Config created in: sc-sts
  • Make sure that we install the cluster in Manual mode:
# cd sc-sts
# echo "credentialsMode: Manual" >> install-config.yaml
  • Create install manifests:
# cd ..
# ./openshift-install create manifests --dir=./sc-sts
  • Using the sts-preflight tool to create AWS resources. Make sure you are in the sts-preflight directory:
#./sts-preflight create --infra-name sc-example --region us-west-1 --credentials-requests-to-roles ./credreqs.yaml
2021/04/28 13:45:34 Generating RSA keypair
2021/04/28 13:45:42 Writing private key to _output/sa-signer
2021/04/28 13:45:42 Writing public key to _output/sa-signer.pub
2021/04/28 13:45:42 Copying signing key for use by installer
2021/04/28 13:45:42 Reading public key
2021/04/28 13:45:42 Writing JWKS to _output/keys.json
2021/04/28 13:45:42 Bucket sc-example-installer already exists and is owned by us
2021/04/28 13:45:42 OIDC discovery document at .well-known/openid-configuration updated
2021/04/28 13:45:42 JWKS at keys.json updated
2021/04/28 13:45:43 Existing OIDC provider found arn:aws:iam::000000000000:oidc-provider/s3.us-west-1.amazonaws.com/sc-example-installer
2021/04/28 13:45:43 Existing Role found arn:aws:iam::000000000000:role/sc-example-installer
2021/04/28 13:45:43 AdministratorAccess attached to Role sc-example-installer
2021/04/28 13:45:43 Role arn:aws:iam::000000000000:role/sc-example-openshift-machine-api-aws-cloud-credentials created
2021/04/28 13:45:43 Saved credentials configuration to: _output/manifests/openshift-machine-api-aws-cloud-credentials-credentials.yaml
2021/04/28 13:45:43 Role arn:aws:iam::000000000000:role/sc-example-openshift-cloud-credential-operator-cloud-credential- created
2021/04/28 13:45:44 Saved credentials configuration to: _output/manifests/openshift-cloud-credential-operator-cloud-credential-operator-iam-ro-creds-credentials.yaml
2021/04/28 13:45:44 Role arn:aws:iam::000000000000:role/sc-example-openshift-image-registry-installer-cloud-credentials created
2021/04/28 13:45:44 Saved credentials configuration to: _output/manifests/openshift-image-registry-installer-cloud-credentials-credentials.yaml
2021/04/28 13:45:44 Role arn:aws:iam::000000000000:role/sc-example-openshift-ingress-operator-cloud-credentials created
2021/04/28 13:45:44 Saved credentials configuration to: _output/manifests/openshift-ingress-operator-cloud-credentials-credentials.yaml
2021/04/28 13:45:45 Role arn:aws:iam::000000000000:role/sc-example-openshift-cluster-csi-drivers-ebs-cloud-credentials created
2021/04/28 13:45:45 Saved credentials configuration to: _output/manifests/openshift-cluster-csi-drivers-ebs-cloud-credentials-credentials.yaml
  • Copy the generated manifest files and tls directory from sts-preflight/_output directory to installation directory:
# cp sts-preflight/_output/manifests/* sc-scs/manifests/
# cp -a sts-preflight/_output/tls sc-scs/
  • I ran both ./sts-preflight token and ./sts-preflight assume again to make sure I have enough time to finish my installation
  • Export the AWS environment variables.
  • I did not further restrict the role in my test.
  • Start to provision a OCP cluster:
# ./openshift-install create cluster --log-level=debug --dir=./sc-sts
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/mufg-sts/sc-sts-test/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.sc-sts-test.xx.live
INFO Login to the console with user: "kubeadmin", and password: "xxxxxxxxxxx"
DEBUG Time elapsed per stage:
DEBUG     Infrastructure: 7m28s
DEBUG Bootstrap Complete: 11m6s
DEBUG  Bootstrap Destroy: 1m21s
DEBUG  Cluster Operators: 12m28s
INFO Time elapsed: 32m38s

#Cluster was created successfully.
  • Verify the components are assuming the IAM roles:
# oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode
role_arn = arn:aws:iam::000000000000:role/sc-sts-test-openshift-image-registry-installer-cloud-credentials
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
  • Adding and deleting worker node works as well:
Increase the count from one of the MachineSets from Administrator console, worker node was able to provisioned.
Decrease the count from one of the MachineSets from Administrator console, worker node was deleted.

Delete the Cluster

  • Obtain a new temporary credential:
cd <sts-preflight directory>
# ./sts-preflight token
2021/04/29 08:19:01 Token written to _output/token

# ./sts-preflight assume
Run these commands to use the STS credentials
export AWS_ACCESS_KEY_ID=<temporary key>
export AWS_SECRET_ACCESS_KEY=<temporary secret>
export AWS_SESSION_TOKEN=<session token>
  • Export all AWS environment variables using the result output from last step
  • Delete the cluster:
# ./openshift-install destroy cluster --log-level=debug --dir=./sc-sts-test
DEBUG OpenShift Installer 4.7.9
DEBUG Built from commit fae650e24e7036b333b2b2d9dfb5a08a29cd07b1
INFO Credentials loaded from default AWS environment variables
DEBUG search for matching resources by tag in us-east-1 matching aws.Filter{"kubernetes.io/cluster/sc-sts-rj4pw":"owned"}
INFO Deleted                                       id=vpc-0bbacb9858fe280f9
INFO Deleted                                       id=dopt-071e7bf4cfcc86ad6
DEBUG search for matching resources by tag in us-east-1 matching aws.Filter{"kubernetes.io/cluster/sc-sts-test-rj4pw":"owned"}
DEBUG search for matching resources by tag in us-east-1 matching aws.Filter{"openshiftClusterID":"ab9baacf-a44f-47e8-8096-25df62c3b1dc"}
DEBUG no deletions from us-east-1, removing client
DEBUG search for IAM roles
DEBUG search for IAM users
DEBUG search for IAM instance profiles
DEBUG Search for and remove tags in us-east-1 matching kubernetes.io/cluster/sc-sts-test-rj4pw: shared
DEBUG No matches in us-east-1 for kubernetes.io/cluster/sc-sts-test-rj4pw: shared, removing client
DEBUG Purging asset "Metadata" from disk
DEBUG Purging asset "Master Ignition Customization Check" from disk
DEBUG Purging asset "Worker Ignition Customization Check" from disk
DEBUG Purging asset "Terraform Variables" from disk
DEBUG Purging asset "Kubeconfig Admin Client" from disk
DEBUG Purging asset "Kubeadmin Password" from disk
DEBUG Purging asset "Certificate (journal-gatewayd)" from disk
DEBUG Purging asset "Cluster" from disk
INFO Time elapsed: 4m39s


Red Hat OpenShift on Amazon (ROSA) is GA!

I have previously blogged about the pre-GA ROSA, and now it is GA. I decided to write up my GA experience on ROSA.

Let’s get started here.

Enable ROSA on AWS

After logging into AWS, enter openshift in the search box on the top of the page.

Click on the “Red Hat OpenShift Service on AWS” Service listed.

It will then take you to a page as shown below and click to enable the OpenShift service.

Once it is complete, it will show Service enabled.

Click to download the CLI and click on the OS where you run your ROSA CLI. It will start downloading to your local drive.


Extract the downloaded CLI file and rosa add to your local path.

tar zxf rosa-macosx.tar.gz
mv rosa /usr/local/bin/rosa

Setting AWS Account

I have set up my AWS account as my IAM user account with proper access per the documentation. There is more information about the account access requirements for ROSA. It is available here.

I have configured my AWS key and secret in my .aws/credentials.

Create Cluster

Verify AWS account access.

rosa verify permissions


I: Validating SCP policies...
I: AWS SCP policies ok

Verify the quota for the AWS account.

rosa verify quota --region=us-west-2


I: Validating AWS quota...
I: AWS quota ok

Obtain Offline Access Token from the management portal cloud.redhat.com (if you don’t have one yet) by clicking Create One Now link

Go to https://cloud.redhat.com/openshift/token/rosa, and you will have to log in and prompt to accept terms as shown below.

Click View Terms and Conditions.

Check the box to agree the terms and click Submit.

Copy the token from cloud.redhat.com.

rosa login --token=<your cloud.redhat.com token>


I: Logged in as 'your_username' on 'https://api.openshift.com'

Verify the login

rosa whoami


AWS Account ID:               ############
AWS Default Region:           us-west-2
AWS ARN:                      arn:aws:iam::############:user/username
OCM API:                      https://api.openshift.com
OCM Account ID:               xxxxxyyyyyzzzzzwwwwwxxxxxx
OCM Account Name:             User Name
OCM Account Username:         User Name
OCM Account Email:            name@email.com
OCM Organization ID:          xxxxxyyyyyzzzzzwwwwwxxxxxx
OCM Organization Name:        company name
OCM Organization External ID: 11111111

Configure account and make sure everyone setup correctly

rosa init


I: Logged in as 'your_username' on 'https://api.openshift.com'
I: Validating AWS credentials...
I: AWS credentials are valid!
I: Validating SCP policies...
I: AWS SCP policies ok
I: Validating AWS quota...
I: AWS quota ok
I: Ensuring cluster administrator user 'osdCcsAdmin'...
I: Admin user 'osdCcsAdmin' already exists!
I: Validating SCP policies for 'osdCcsAdmin'...
I: AWS SCP policies ok
I: Validating cluster creation...
I: Cluster creation valid
I: Verifying whether OpenShift command-line tool is available...
I: Current OpenShift Client Version: 4.7.2

Creating cluster using interactive mode.

rosa create cluster -i
I: Interactive mode enabled.
Any optional fields can be left empty and a default will be selected.
? Cluster name: [? for help] 

Enter the name of the ROSA cluster.

? Multiple availability zones (optional): [? for help] (y/N) 

Enter y/N.

? AWS region:  [Use arrows to move, type to filter, ? for more help]
> us-west-2 

Select the AWS region and hit <enter>.

? OpenShift version:  [Use arrows to move, type to filter, ? for more help]
> 4.7.2

Select the version and hit <enter>.

? Install into an existing VPC (optional): [? for help] (y/N)

Enter y/N.

? Compute nodes instance type (optional):  [Use arrows to move, type to filter, ? for more help]
> r5.xlarge

Select the type and hit <enter>.

? Enable autoscaling (optional): [? for help] (y/N)

Enter y/N.

? Compute nodes: [? for help] (2)

Enter the numbers of workers to start.

? Machine CIDR: [? for help] (

Enter the machine CIDR or use default.

? Service CIDR: [? for help] (

Enter the service CIDR or use default.

? Pod CIDR: [? for help] (

Enter the pod CIDR or use default.

? Host prefix: [? for help] (23)

Enter the host prefix or use default

? Private cluster (optional): (y/N) 

Enter y/N.


Restrict master API endpoint and application routes to direct, private connectivity. You will not be able to access your cluster until you edit network settings in your cloud provider. I also learned that you would need one private subnet and one public subnet for each AZ for your existing private VPC for the GA version of ROSA. There will be more improvement to provide for the private cluster in the future release.


I: Creating cluster 'rosa-c1'
I: To create this cluster again in the future, you can run:
   rosa create cluster --cluster-name rosa-c1 --region us-west-2 --version 4.7.2 --compute-nodes 2 --machine-cidr --service-cidr --pod-cidr --host-prefix 23
I: To view a list of clusters and their status, run 'rosa list clusters'
I: Cluster 'rosa-c1' has been created.
I: Once the cluster is installed you will need to add an Identity Provider before you can login into the cluster. See 'rosa create idp --help' for more information.
I: To determine when your cluster is Ready, run 'rosa describe cluster -c rosa-c1'.
I: To watch your cluster installation logs, run 'rosa logs install -c rosa-c1 --watch'.
Name:                       rosa-c1
ID:                         xxxxxxxxxxyyyyyyyyyyyaxxxxxxxxx
External ID:
OpenShift Version:
Channel Group:              stable
DNS:                        rosa-c1.xxxx.p1.openshiftapps.com
AWS Account:                xxxxxxxxxxxx
Console URL:
Region:                     us-west-2
Multi-AZ:                   false
 - Master:                  3
 - Infra:                   2
 - Compute:                 2 (m5.xlarge)
 - Service CIDR:  
 - Machine CIDR:  
 - Pod CIDR:      
 - Host Prefix:             /23
State:                      pending (Preparing account)
Private:                    No
Created:                    Mar 30 2021 03:10:25 UTC
Details Page:               https://cloud.redhat.com/openshift/details/xxxxxxxxxxyyyyyyyyyyyaxxxxxxxxx

Copy the URL from the Details Page to the browser and click view logs to see the status of the installation.

When ROSA is completed, you will see the similar page as below.

You will need to access the OpenShift cluster.

Configure Quick Access

Add cluster-admin user

rosa create admin -c rosa-c1


I: Admin account has been added to cluster 'rosa-c1'.
I: Please securely store this generated password. If you lose this password you can delete and recreate the cluster admin user.
I: To login, run the following command:

   oc login https://api.rosa-c1.xxxx.p1.openshiftapps.com:6443 --username cluster-admin --password xxxxx-xxxxx-xxxxx-xxxxx

I: It may take up to a minute for the account to become active.

Test user access

$ oc login https://api.rosa-c1.xxxx.p1.openshiftapps.com:6443 --username cluster-admin --password xxxxx-xxxxx-xxxxx-xxxxx
Login successful.

You have access to 86 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".

Configure Identity Provider

There are options for identity providers. I am using Github in this example.

I am not going to explain how to get identity to provide set up here. I did that in last blog. I will walk through the step to configure ROSA using Github.

rosa create idp --cluster=rosa-c1 -i
I: Interactive mode enabled.
Any optional fields can be left empty and a default will be selected.
? Type of identity provider:  [Use arrows to move, type to filter]
> github

Select one IDP

? Identity provider name: [? for help] (github-1)

Enter the name of the IDP configured on the ROSA

? Restrict to members of:  [Use arrows to move, type to filter, ? for more help]
> organizations

Select organizations

? GitHub organizations:

Enter the name of the organization. My example is `sc-rosa-idp`

? To use GitHub as an identity provider, you must first register the application:
  - Open the following URL:
  - Click on 'Register application'

Open a browser and use the above URL to register the application and cope client ID

? Client ID: [? for help] 

Enter the copied Client ID

? Client Secret: [? for help]

Enter client secret from the registered application.

? GitHub Enterprise Hostname (optional): [? for help] 

Hit <enter>

? Mapping method:  [Use arrows to move, type to filter, ? for more help]
> claim

Select claim

I: Configuring IDP for cluster 'rosa-c1'
I: Identity Provider 'github-1' has been created.
   It will take up to 1 minute for this configuration to be enabled.
   To add cluster administrators, see 'rosa create user --help'.
   To login into the console, open https://console-openshift-console.apps.rosa-c1.xxxx.p1.openshiftapps.com and click on github-1.

Congratulation! IDP configuration is completed.

Login in with IDP account

Open a browser with the URL from the IDP configuration. Our example is: https://console-openshift-console.apps.rosa-c1.xxxx.p1.openshiftapps.com.

Click github-1

Click Authorize sc-rosa-idp

Overall, it is straightforward to get started on creating a ROSA cluster on AWS. I hope this will help you in some ways.


Red Hat OpenShift on Amazon Documentation

Running ROSA on an Existing Private VPC

Pre-GA ROSA test

Test Run Pre-GA Red Hat OpenShift on AWS (ROSA)

I have an opportunity to try out the pre-GA ROSA. ROSA is a fully managed Red Hat OpenShift Container Platform (OCP) as a service and sold by AWS. I am excited to share my experience on ROSA. It installs OCP 4 from soup to nuts without configuring hosted zone and domain sever. As a developer, you may want to get the cluster up and running, so you can start doing the real work :). There are customization options with ROSA, but I am going to leave it for later exploration.

I am going to show you the steps I took to create OCP via ROSA. There are more use cases to test. I hope this blog will give you a taste of ROSA.

Creating OpenShift Cluster using ROSA Command

Since it is a pre-GA version, I download the ROSA command line tool from the here and have aws-cli available where I run the ROSA installation.

  • I am testing from my MacBook. I just move the “rosa” command line tool to /usr/local/bin/.
  • Verify that your AWS account if it has the necessary permissions using rosa verify permissions:
  • Verify that your AWS account has the necessary quota to deploy an Red Hat OpenShift Service on AWS cluster via rosa verify quota --region=<region>:
  • Log in your Red Hat account with ROSA command using rosa login --token=<token from cloud.redhat.com>:
  • Verify the AWS setup using rosa whoami:
  • Initialize the AWS for the cluster deploy via rosa init:
  •  Since I have OpenShift Client command line installed, it shows the existing OpenShift Client version. If you don’t have it, you can download the OpenShift Client command line via rosa download oc and make it available from your PATH. 
  • Create ROSA via rosa create cluster command below: 

Note: rosa create cluster -i with the interactive option, it provides customization for ROSA installation, such as multiple AZ, existing VPC, subnets, etc…

  • Copy the URL from Details Page to a browser and you can view the status for your ROSA installation. 
  • If you click View logs, you can watch the log from here until the cluster is completed.
  • When you see this screen, it means the cluster is created:
  • Now, you need to have a way to log into the OCP cluster. I created an organization called sc-rosa-idp on Github via rosa create idp --cluster=sc-rosa-test --interactive as show below.
  • Log into the OCP console via the URL from the output from last step:
  • Click github-1 –> redirect to authorize to the organization from Github -> log in Github.
  • Once you log in with your Github credential, we will see the OCP developer console: 
  • Grant cluster-admin role to the github user using rosa grant user cluster-admin --user <github user in your organization> --cluster <name of your rosa cluster>.
  • Click Administrator on the top left and access the OCP admin console with Admin access as shown below:

Delete ROSA

  • Go to the cluster from cloud.redhat.com, from Action –> select Delete cluster:
  • Enter the name of the cluster and click Delete:
  • The cluster shows as Uninstalling


Although it is a pre-GA without AWS console integration, I found it very easy to get my cluster up and running. If you cannot wait for GA, you can always request the preview access from here. Get a head start with ROSA!


ARO 4 and AAD Integration Take 2

In my last post on ARO 4, I have already walk through the steps to set up the Azure environment for creating ARO 4. My 2nd round testing requires the following specific requirements:

  • Use only one app registration
  • Not to use pull secret
You will need to complete the session for setting up Azure environment in my previous blog for ARO 4.

Create ARO 4 Cluster with existing service principal

Create a service principal
From the previous test, I learned that the process of creating ARO 4 will create a service principal. I am going to create a service principal before creating cluster.
$ az ad sp create-for-rbac --role Contributor --name all-in-one-sp
This command will return the appId and password information that we will need for the ARO 4 create command later.
Adding API permission to the service principal
  1. Login to Azure Portal
  2. Go to Azure Active Directory
  3. Click App registrations
  4. Click “All applications”
  5. Search for “app-in-one-sp”
  6. Click “View API permission”
  7. Click “Add a permission”
  8. Click “Azure Active Directory Graph”
  9. Click “Delegated Permissions”
  10. Check “User.Read”
  11. Click “Add permission” button at the bottom.
  12. Click “Grant admin consent …”
  13. A green check mark is shown under Status as shown below
Create ARO with existing service principal without pull secret
az aro create \
--resource-group $RESOURCEGROUP \
--name $CLUSTER \
--client-id <service principal application id> \
--client-secret <service principal password> \
--vnet aro-vnet \
--master-subnet master-subnet \
--worker-subnet worker-subnet \
--domain aro.ocpdemo.online
When I opted out the pull secret option, I will get the following message from the output of the azure cli.
No --pull-secret provided: cluster will not include samples or operators from Red Hat or from certified partners.
Adding api and ingress A record to the DNS zone
Using the output from the ARO 4 creation, Use the IP from the “apiserverProfile” portion is for api servier. The IP from “ingressProfiles” is for ingress. The example is shown below.
Test out the ARO cluster
az aro list-credentials \ 
--name $CLUSTER \ 
--resource-group $RESOURCEGROUP
Open the following URL from the browser and login using the kubeadmin with password from the above command
https://console-openshift-console.apps.<DNS domain>/

Integrate Azure Active Directory

The following steps are for getting the OAuth call back URL.
$ oc login -u kubeadmin -p <password> https://api.<DNS domain>:6443/ 
$ oauthCallBack=`oc get route oauth-openshift -n openshift-authentication -o jsonpath='{.spec.host}'` 
$ oauthCallBackURL=https://$oauthCallBack/oauth2callback/AAD
$ echo $oauthCallBackURL
where AAD is the name of the identity provider for OAuth configuration on OpenShift
Add the OAuth call back URL to the same service principal
  • Go to Azure Active Directory
  • Click App registration
  • Click on “all-in-one-sp” under all applications
  • Under Overview, click right top corner link for “Add a Redirect URI”
  • Click “Add a platform”
  • Click Web Application from the list of Configure platforms
  • Enter the value of the oauthCallBackURL from the previous step to the “Redirect URIs”
  • Click configure
Create a manifest file
cat > manifest.json<< EOF 
[{ "name": "upn", 
"source": null, 
"essential": false, 
"additionalProperties": [] 
{ "name": "email", 
"source": null, 
"essential": false, 
"additionalProperties": [] 
Update service principal with the manifest
$ az ad app update \
 --set optionalClaims.idToken=@manifest.json \
 --id <Service Principal appId>
Create secret to store service principal’s password
oc create secret generic openid-client-secret-azuread \
--namespace openshift-config \
--from-literal=clientSecret=<service principal password>
Create OAuth configuration
apiVersion: config.openshift.io/v1
kind: OAuth
  name: cluster
  - name: AAD
    mappingMethod: claim
    type: OpenID
      clientID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        name: openid-client-secret-azuread
      - email
      - profile
        include_granted_scopes: "true"
        - email
        - upn
        - name
        - email
      issuer: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apply the OAuth YAML
oc apply -f openid.yaml
Login openshift console via AAD


Azure Red Hat OpenShift 4 (ARO 4) integrate with Azure Active Directory

I happened to test out ARO 4 with Azure Active Directory integration. The Azure documentation is good, but I had to change a few while testing the steps. I am sharing my experience here and hope someone will find it useful.

Setting the requirements

Install or update Azure CLI
brew update && brew install azure-cli
Make sure you have permission to create resources in the resource group. I logged in as a global administrator when I am testing this.

Setup the environment variables
$ cat aro-env
LOCATION=centralus. # the location of your cluster
RESOURCEGROUP=aro-rg # the name of the resource group where you want to create your cluster
CLUSTER=poc #cluster-id of the ARO 4 cluster
$ source aro-env
Log in Azure
az login
Create a Resource Group
az group create \
--location $LOCATION
Add DNS zone
If you don’t have a DNS zone already, you can use this step.
  1. Login Azure Portal
  2. Type: “DNS Zones” in the search box on the top and click on “DNS Zones”
  3. Click “+Add” on the top
  4. Select the newly created resource group
  5. Enter your domain
  6. Select the location
  7. Create “Review+Create”


  • I am using a domain name outside of the Azure. You will need to add the NS records from the overview page of the DNS zone to your domain.
  • Request increase of quota from Azure portal. ARO requires a minimum of 40 cores.
Register Resource Provider
az account set --subscription
az provider register -n Microsoft.RedHatOpenShift --wait
az provider register -n Microsoft.Compute --wait
az provider register -n Microsoft.Storage --wait
Create a Virtual Network
az network vnet create \
--resource-group $RESOURCEGROUP \
--name aro-vnet \
Create an empty subnet for master nodes
az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name master-subnet \
--address-prefixes \
--service-endpoints Microsoft.ContainerRegistry
Create an empty subnet for worker nodes
az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name worker-subnet \
--address-prefixes \
--service-endpoints Microsoft.ContainerRegistry
Disable private endpoint policy
az network vnet subnet update \
--name master-subnet \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--disable-private-link-service-network-policies true
Once the above steps are done. You don’t have to redo the steps if you are going to reuse the names and resources.

Create Cluster

Please make sure you log in to Azure and environment variables are set.

Information that we need for creating a cluster
  • Get a copy of the pull secret from cloud.redhat.com. If you don’t have a user name created, please just register as a user for free.
  • Create an ARO cluster using the following command. Please apply to appropriate values.
    Some values were used in the example are explained as shown below.
    • aro-vnet – the name of virtual network
    • master-subnet – the name of master subnet
    • worker subnet – the name of worker subnet
    • ./pull-secret.txt – the path and pull secret where is located
    • aro.ocpdemo.online – custom domain for the cluster
az aro create \
--resource-group $RESOURCEGROUP \
--name $CLUSTER \
--vnet aro-vnet \
--master-subnet master-subnet \
--worker-subnet worker-subnet \
--pull-secret @./pull-secret.txt \
--domain aro.ocpdemo.online

The information from the JSON output of the above command can be useful if you are not familiar with OpenShift 4. You can find your API server IP, API URL, OpenShift console URL and ingress IP. You will need the API, and ingress IP for the next step.

{- Finished ..
"apiserverProfile": {
"ip": "x.x.x.x",
"url": "https://api.aro.ocpdemo.online:6443/",
"visibility": "Public"
"consoleProfile": {
"url": "https://console-openshift-console.apps.aro.ocpdemo.online/"
"ingressProfiles": [
"ip": "x.x.x.x",
"name": "default",
"visibility": "Public"

Post ARO Installation

Adding two A records for api and *.apps in the DNS zone
  1. Login to Azure portal
  2. Go to DNS zone
  3. Click onto the domain for the ARO cluster
  4. Click “+ Record Set” on the top menu to create an A record and add values to Name and IP. You will need to repeat this step for both api and *.apps A records.
    • Name: api or *.apps
    • IP: the *.apps/ingress IP is from the output of the creation of the ARO
  5. The below screenshot shows the DNS zone configuration and adding 2 A records.

Test ARO Cluster

Getting Kubeadmin credential
az aro list-credentials \
--name $CLUSTER \
--resource-group $RESOURCEGROUP
The command will return the kubeadmin credential.
Log in OpenShift Console
Open a browser and go to the OpenShift console or look for “consoleProfile” from the JSON output from ARO creation
https://console-openshift-console.apps.<DNS domain>/
The login user is kubeadmin and the password is the credential from the last command. Congrats!! The ARO installation is completed!

Azure Active Directory Integration

Getting oauthCallBackURL
  • Download OpenShift command line tool from console.
Download the OpenShift Command Lind Interface (CLI) from there. Once you extract it and add to the PATH. You can move on to the next step.
  • Login to ARO via OC CLI
$ oc login -u kubeadmin -p <password> https://api.<DNS domain>:6443/

$ oauthCallBack=`oc get route oauth-openshift -n openshift-authentication -o jsonpath='{.spec.host}'`

$ oauthCallBackURL=https://$oauthCallBack/oauth2callback/AAD
Note: AAD is the name of the identity provider when configuring OAuth on OpenShift

Creating Application on Azure Active Directory
az ad app create \
  --query appId -o tsv \
  --display-name poc-aro-auth \
  --reply-urls $oauthCallBackURL \
  --password '<ClientSecret>'
Note: Please note that the above command returns the registered Application Id (AppId) which you will need it when configuring the OAuth on OpenShift.
Get tenant Id
az account show --query tenantId -o tsv
Note: Please note that you will need the tenant Id for the OAuth configuration on OpenShift
Create manifest file
cat > manifest.json<< EOF
"name": "upn",
"source": null,
"essential": false,
"additionalProperties": []
"name": "email",
"source": null,
"essential": false,
"additionalProperties": []
Update the Azure Active Directory with a manifest
az ad app update \
--set optionalClaims.idToken=@manifest.json \
--id <AppId>
Update Application permission scope
az ad app permission add \
--api 00000002-0000-0000-c000-000000000000 \
--api-permissions 311a71cc-e848-46a1-bdf8-97ff7156d8e6=Scope \ 
--id <AppId>
Grant admin consent
  1. login Azure portal
  2. Go to Azure Active Directory
  3. Click App Registrations
  4. Click “All Application” and search for newly create application name
  5. Click onto the display name of the application
  6. Click view API permissions
  7. Click on the “check” to grant admin consent for directory
Add service principal
$ az ad sp create-for-rbac --role Contributor --name poc-aro-sp
You will need the “appId” from the output of the above command and that is the appId for the service principal
$az role assignment create --role "User Access Administrator" \
--assignee-object-id $(az ad sp list --filter "appId eq '<service-principal-appid>'" \
| jq '.[0].objectId' -r)
$az ad app permission add --id <appId> \ 
--api 00000002-0000-0000-c000-000000000000 \ 
--api-permissions 824c81eb-e3f8-4ee6-8f6d-de7f50d565b7=Role
This will output the follow command as shown below.
$az ad app permission grant --id <appid> --api 00000002-0000-0000-c000-000000000000
I also grant the admin consent for the API permission for the service principal.
Create secret for identity provider on OpenShift
oc create secret generic openid-client-secret-azuread \
--namespace openshift-config \
--from-literal=clientSecret=<your password>
Create YAML for identity provider for AAD
apiVersion: config.openshift.io/v1
kind: OAuth
  name: cluster
  - name: AAD
    mappingMethod: claim
    type: OpenID
      clientID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        name: openid-client-secret-azuread
      - email
      - profile
        include_granted_scopes: "true"
        - email
        - upn
        - name
        - email
      issuer: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  • The clientID is the AppId of your registered application.
  • Issuer URL is https://login.microsoftonline.com/<tenant id>.
  • The clientSecret is using the secret (openid-client-secret-azuread) that you created from the previous step.
Alternatively, you can obtain the clientID and tenant id from Azure Portal.
  • Login Azure Portal
  • Click Home
  • Click Azure Active Directory
  • Click App registrations on the left menu
  • Click all applications tab
  • Type the application that you just created in the search area
  • Click onto the application (my application is poc-aro-auth)
  • Under Overview, the information is shown as “Application (client) ID” and Directory (tenant) ID” as in the image below.
Update OpenShift OAuth Configuration
oc apply -f openid.yaml
Login OpenShift console via AAD
It will redirect you to Azure login page


Tip #1: If you are getting error, you can login as kubeadmin and check the logs from oauth-openshift pods under openshift-authentication project.

Tip #2: if you are creating a new registered application to try, make sure you clean up the user and identity.


Azure OpenShift 4 documentation

ARO and Azure Active Directory integration