OpenShift Lessons Learned

Prerequisites

• An OpenShift 4 cluster
• Compliance Operator installed

Overview

Steps to Generate OpenSCAP Evaluation Report

• Log in OpenShift Cluster

# oc login -u <username> https://api.<clusterid>.<subdomain>

• Assuming you have already installed the OpenShift Compliance Operator. Next, check the access to the OpenShift Compliance project using the command shown below.

# oc project openshift-compliance

• The default compliance profiles will be available once the operator is installed. The command below lists out all compliance profiles Custom Resource Definition (CRD) profiles.compliance.openshift.io.

# oc get profiles.compliance.openshift.io

• To get custom resource ScanSetting via the below command. It shows two default scan settings.

# oc get ScanSetting
NAME                 AGE
default              2d10h
default-auto-apply   2d10h

• Check out the “default” ScanSetting

Name:         default
Namespace:    openshift-compliance
Labels:       <none>
Annotations:  <none>
API Version:  compliance.openshift.io/v1alpha1
Kind:         ScanSetting
Metadata:
  Creation Timestamp:  2021-10-19T16:22:18Z
  Generation:          1
  Managed Fields:
...
  Resource Version:  776981
  UID:               f453726d-665a-432e-88a9-a4ad60176ac7
Raw Result Storage:
  Pv Access Modes:
    ReadWriteOnce
  Rotation:  3
  Size:      1Gi
Roles:
  worker
  master
Scan Tolerations:
  Effect:    NoSchedule
  Key:       node-role.kubernetes.io/master
  Operator:  Exists
Schedule:    0 1 * * *
Events:      <none>

• Create ScanSettingBinding as shown in scan-setting-binding-example.yaml below.

# cat scan-setting-binding-example.yaml
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: cis-compliance
profiles:
  - name: ocp4-cis-node
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1

• Create the above sample ScanSettingBinding custom resource.

# oc create -f scan-setting-binding-example.yaml

• Verify the creation of the ScanSettingBinding

# oc get scansettingbinding

• Custom resource ComplianceSuites is to help tracking the state of the scans. The following command is to check the state of the scan you defined in your ScanSettingBinding.

# oc get compliancesuite
NAME             PHASE     RESULT
cis-compliance   RUNNING   NOT-AVAILABLE

• ComplianceScan custom resource needs all the parameters to run OpenSCAP, such as profile id, image to get the content from, and data stream file path. It also can constain operational parameter.

# oc get compliancescan
NAME                   PHASE   RESULT
ocp4-cis               DONE    NON-COMPLIANT

• While the custom resource ComplianceCheckResult shows the aggregate result of the scan, it is useful to review the raw result from the scanner. The raw results are produced in the ARF format and can be large. Therefore, Compliance Operator creates a persistent volume (PV) for the raw result from the scan. Let’s take a look if the PVC is created for the scan.

# oc get pvc
NAME                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ocp4-cis               Bound    pvc-5ee57b02-2f6b-4997-a45c-3c4df254099d   1Gi        RWO            gp2            27m
ocp4-cis-node-master   Bound    pvc-57c7c411-fc9f-4a4d-a713-de91c934af1a   1Gi        RWO            gp2            27m
ocp4-cis-node-worker   Bound    pvc-7266404a-6691-4f3d-9762-9e30e50fdadb   1Gi        RWO            gp2            28m

• Once we know the raw result is created, we need the oc-compliance tool to get the raw result XML file. You will need to login to the registry.redhat.io.

# podman login -u <user> registry.redhat.io

• Download the oc-compliance tool

podman run --rm --entrypoint /bin/cat registry.redhat.io/compliance/oc-compliance-rhel8 /usr/bin/oc-compliance > ~/usr/bin/oc-compliance

• Fetch the raw results to a temporary location (/tmp/cis-compliance)

# oc-compliance fetch-raw scansettingbindings cis-compliance -o /tmp/cis-compliance
Fetching results for cis-compliance scans: ocp4-cis-node-worker, ocp4-cis-node-master, ocp4-cis
Fetching raw compliance results for scan 'ocp4-cis-node-worker'.....
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis-node-worker
Fetching raw compliance results for scan 'ocp4-cis-node-master'.....
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis-node-master
Fetching raw compliance results for scan 'ocp4-cis'...........
The raw compliance results are available in the following directory: /tmp/cis-compliance/ocp4-cis

• Inspect the output filesystem and extract the *.bzip2 file

# cd /tmp/cis-compliance/ocp4-cis
# ls
ocp4-cis-api-checks-pod.xml.bzip2

# bunzip2 -c  ocp4-cis-api-checks-pod.xml.bzip2  > /tmp/cis-compliance/ocp4-cis/ocp4-cis-api-checks-pod.xml

# ls /tmp/cis-compliance/ocp4-cis/ocp4-cis-api-checks-pod.xml
/tmp/cis-compliance/ocp4-cis/ocp4-cis-api-checks-pod.xml

• Convert ARF XML to html

# oscap xccdf generate report ocp4-cis-api-checks-pod.xml > report.html

• View the HTML as shown below.

Reference

• OpenShift Compliance Operator
• Compliance as Code

Thank you Juan Antonio Osorio Robles for sharing the diagram!

• Go to Administrator Console
• Create a new project called keycloak

• Click Operator
• Click OperatorHub
• Click on the Red Hat Single Sign-On Operator

• Click Install

• Click Install

• Click “Create instance” in the Keycloak tile

• The Keycloak CR is shown below

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: keycloak-dev
  labels:
    app: keycloak-dev
  namespace: keycloak
spec:
  externalAccess:
    enabled: true
  instances: 1

• Click Create
• Go to Workloads > Pods

• Click the keycload-dev creation and return “true”

$ oc get keycloak keycloak-dev -n keycloak -o jsonpath='{.status.ready}'
true

• Operators > Installed Operators > Red Hat Single Sign-On Operator
• Click Create instance

• Enter the KeycloakRealm as shown below

apiVersion: keycloak.org/v1alpha1
kind: KeycloakRealm
metadata:
  name: keycloakrealm
  labels:
    realm: keycloakrealm
  namespace: keycloak
spec:
  instanceSelector:
    matchLabels:
      app: keycloak-dev
  realm:
    enabled: true
    displayName: "Keycloak-dev Realm"
    realm: keycloakrealm

• Click Create

• Make sure it returns true

$ oc get keycloakrealm keycloakrealm -n keycloak -o jsonpath='{.status.ready}'
true

• Get the Keycloak Admin user secret name

$ oc get keycloak keycloak-dev --output="jsonpath={.status.credentialSecret}"
credential-keycloak-dev

• Get the Admin username and password

$ oc get secret credential-keycloak-dev -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

• Run the following to find out the URLs of Keycloak:

KEYCLOAK_URL=https://$(oc get route keycloak --template='{{ .spec.host }}')/auth &&
echo "" &&
echo "Keycloak:                 $KEYCLOAK_URL" &&
echo "Keycloak Admin Console:   $KEYCLOAK_URL/admin" &&
echo "Keycloak Account Console: $KEYCLOAK_URL/realms/myrealm/account" &&
echo ""

• Open a browser with the Admin URL

• Login with the admin username and password
• Click Client on the left nav

• Click Create on the right top corner

• Enter the Argocd URL and the name of the client as ‘argocd’

• Click Save
• Set Access Type to confidential
• Set Valid Redirect URIs to <argocd-url>/auth/callback
• Set Base URL to /applications

• Click Save
• Scroll up and click “Credential” tab

• IMPORTANT: Copy the secret and you will need this later

• Configure the Group claim
• Click Client Scope on the left nav

• Click Create on the right

• Set Name as group
• Set Protocol as openid-connecgt
• Display On Content Scope: on
• Include to Token Scope: on

• Click save
• Click “Mappers” tab

• Click Create on the top right
• Set name as groups
• Set Mapper Type as Group Membership
• Set Token Claim Name as groups`

• Click Clients on the left nav
• Click argocd

• Click “Client Scopes” tab
• Select groups > Add selected

• Click Groups on left nav
• Click Create

• Set the name as ArgoCDAdmins
• Click Save

• Encode the argocd credential you saved before

echo -n '<argocd credential>' | base64

• Edit the argocd-secret

oc edit secret argocd-secret -n openshift-gitops

• add the “oidc.keycloak.clientSecret: <encoded credential> as shown below.

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
data:
  oidc.keycloak.clientSecret: <encoded credential>

• Edit argocd Custom Resource

oc edit argocd -n openshift-gitops

• Add the following into the yaml. Make sure update the issuer to make your settings

oidcConfig: |
    name: OpenShift Single Sign-On
    issuer: https://keycloak-keycloak.apps.cluster-72c5r.72c5r.sandbox1784.opentlc.com/auth/realms/keycloakrealm
    clientID: argocd
    clientSecret: $oidc.keycloak.clientSecret
    requestedScopes: ["openid", "profile", "email", "groups"]

• From OpenShift Console top right corner, click About

• Copy the API URL from the following screen

• Go back to Keycloak, click Identity Providers on left nav
• Select OpenShift v4 from the dropdown list

• Set Display Name: Login with Openshift
• Set Client ID: keycload-broker
• Set Client Secret: <anything that you can remember>
• Set Base URL: API URL
• Set Default Scopes: user:full

• Click Save
• Add an Oauth Client

oc create -f <(echo '
kind: OAuthClient
apiVersion: oauth.openshift.io/v1
metadata:
 name: keycloak-broker 
secret: "12345" 
redirectURIs:
- "https://keycloak-keycloak.apps.cluster-72c5r.72c5r.sandbox1784.opentlc.com/auth/realms/keycloakrealm/broker/openshift-v4/endpoint" 
grantMethod: prompt 
')

• Configure the RBAC

oc edit configmap argocd-rbac-cm -n openshift-gitops

• Modify the data as shown below

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
data:
  policy.csv: |
    g, ArgoCDAdmins, role:admin

• Go to the Argocd URL, you will see the SSO icon. Click “LOG IN VIA OPENSHIFT”

• Click Log in Openshift

I have an opportunity to try out the pre-GA ROSA. ROSA is a fully managed Red Hat OpenShift Container Platform (OCP) as a service and sold by AWS. I am excited to share my experience on ROSA. It installs OCP 4 from soup to nuts without configuring hosted zone and domain sever. As a developer, you may want to get the cluster up and running, so you can start doing the real work :). There are customization options with ROSA, but I am going to leave it for later exploration.

I am going to show you the steps I took to create OCP via ROSA. There are more use cases to test. I hope this blog will give you a taste of ROSA.

Creating OpenShift Cluster using ROSA Command

Since it is a pre-GA version, I download the ROSA command line tool from the here and have aws-cli available where I run the ROSA installation.

• I am testing from my MacBook. I just move the “rosa” command line tool to /usr/local/bin/.

• Verify that your AWS account if it has the necessary permissions using rosa verify permissions:

• Verify that your AWS account has the necessary quota to deploy an Red Hat OpenShift Service on AWS cluster via rosa verify quota --region=<region>:

• You will need an account to log in to cloud.redhat.com. If you don’t have a user name created, please register as a user for free.
• After login to cloud.redhat.com, go to https://cloud.redhat.com/openshift/token/rosa

• Log in your Red Hat account with ROSA command using rosa login --token=<token from cloud.redhat.com>:

• Verify the AWS setup using rosa whoami:

• Initialize the AWS for the cluster deploy via rosa init:

•  Since I have OpenShift Client command line installed, it shows the existing OpenShift Client version. If you don’t have it, you can download the OpenShift Client command line via rosa download oc and make it available from your PATH. 
• Create ROSA via rosa create cluster command below: 

Note: rosa create cluster -i with the interactive option, it provides customization for ROSA installation, such as multiple AZ, existing VPC, subnets, etc…

• Copy the URL from Details Page to a browser and you can view the status for your ROSA installation. 

• If you click View logs, you can watch the log from here until the cluster is completed.

• When you see this screen, it means the cluster is created:

• Now, you need to have a way to log into the OCP cluster. I created an organization called sc-rosa-idp on Github via rosa create idp --cluster=sc-rosa-test --interactive as show below.

• Log into the OCP console via the URL from the output from last step:

• Click github-1 –> redirect to authorize to the organization from Github -> log in Github.

• Once you log in with your Github credential, we will see the OCP developer console: 

• Grant cluster-admin role to the github user using rosa grant user cluster-admin --user <github user in your organization> --cluster <name of your rosa cluster>.
• Click Administrator on the top left and access the OCP admin console with Admin access as shown below:

Delete ROSA

• Go to the cluster from cloud.redhat.com, from Action –> select Delete cluster:

• Enter the name of the cluster and click Delete:

• The cluster shows as Uninstalling: 

Although it is a pre-GA without AWS console integration, I found it very easy to get my cluster up and running. If you cannot wait for GA, you can always request the preview access from here. Get a head start with ROSA!

References

• Official Red Hat OpenShift on AWS documentation
• ROSA upstream project
• Configure Customize Application Domains 
• https://aws.amazon.com/rosa/
• https://www.openshift.com/products/amazon-openshift

In my last post on ARO 4, I have already walk through the steps to set up the Azure environment for creating ARO 4. My 2nd round testing requires the following specific requirements:

• Use only one app registration
• Not to use pull secret

Create ARO 4 Cluster with existing service principal

Create a service principal

$ az ad sp create-for-rbac --role Contributor --name all-in-one-sp

Adding API permission to the service principal

1. Login to Azure Portal
2. Go to Azure Active Directory
3. Click App registrations
4. Click “All applications”
5. Search for “app-in-one-sp”
6. Click “View API permission”
7. Click “Add a permission”
8. Click “Azure Active Directory Graph”
9. Click “Delegated Permissions”
10. Check “User.Read”
11. Click “Add permission” button at the bottom.
12. Click “Grant admin consent …”
13. A green check mark is shown under Status as shown below

Create ARO with existing service principal without pull secret

az aro create \
--resource-group $RESOURCEGROUP \
--name $CLUSTER \
--client-id <service principal application id> \
--client-secret <service principal password> \
--vnet aro-vnet \
--master-subnet master-subnet \
--worker-subnet worker-subnet \
--domain aro.ocpdemo.online

No --pull-secret provided: cluster will not include samples or operators from Red Hat or from certified partners.

Adding api and ingress A record to the DNS zone

Test out the ARO cluster

az aro list-credentials \ 
--name $CLUSTER \ 
--resource-group $RESOURCEGROUP

https://console-openshift-console.apps.<DNS domain>/

Integrate Azure Active Directory

$ oc login -u kubeadmin -p <password> https://api.<DNS domain>:6443/ 
$ oauthCallBack=`oc get route oauth-openshift -n openshift-authentication -o jsonpath='{.spec.host}'` 
$ oauthCallBackURL=https://$oauthCallBack/oauth2callback/AAD
$ echo $oauthCallBackURL

Add the OAuth call back URL to the same service principal

• Go to Azure Active Directory
• Click App registration
• Click on “all-in-one-sp” under all applications
• Under Overview, click right top corner link for “Add a Redirect URI”
• Click “Add a platform”
• Click Web Application from the list of Configure platforms
• Enter the value of the oauthCallBackURL from the previous step to the “Redirect URIs”
• Click configure

Create a manifest file

cat > manifest.json<< EOF 
[{ "name": "upn", 
"source": null, 
"essential": false, 
"additionalProperties": [] 
}, 
{ "name": "email", 
"source": null, 
"essential": false, 
"additionalProperties": [] 
}] 
EOF

Update service principal with the manifest

$ az ad app update \
 --set optionalClaims.idToken=@manifest.json \
 --id <Service Principal appId>

Create secret to store service principal’s password

oc create secret generic openid-client-secret-azuread \
--namespace openshift-config \
--from-literal=clientSecret=<service principal password>

Create OAuth configuration

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - name: AAD
    mappingMethod: claim
    type: OpenID
    openID:
      clientID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      clientSecret:
        name: openid-client-secret-azuread
      extraScopes:
      - email
      - profile
      extraAuthorizeParameters:
        include_granted_scopes: "true"
      claims:
        preferredUsername:
        - email
        - upn
        name:
        - name
        email:
        - email
      issuer: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Apply the OAuth YAML

oc apply -f openid.yaml

Login openshift console via AAD

Reference

• My previous ARO 4 blog
• Azure ARO documentation

I happened to test out ARO 4 with Azure Active Directory integration. The Azure documentation is good, but I had to change a few while testing the steps. I am sharing my experience here and hope someone will find it useful.

Setting the requirements

Install or update Azure CLI

brew update && brew install azure-cli

Setup the environment variables

$ cat aro-env
LOCATION=centralus. # the location of your cluster
RESOURCEGROUP=aro-rg # the name of the resource group where you want to create your cluster
CLUSTER=poc #cluster-id of the ARO 4 cluster
$ source aro-env

Log in Azure

az login

Create a Resource Group

az group create \
--name $RESOURCEGROUP \
--location $LOCATION

Add DNS zone

1. Login Azure Portal
2. Type: “DNS Zones” in the search box on the top and click on “DNS Zones”
3. Click “+Add” on the top
4. Select the newly created resource group
5. Enter your domain
6. Select the location
7. Create “Review+Create”

Notes:

• I am using a domain name outside of the Azure. You will need to add the NS records from the overview page of the DNS zone to your domain.
• Request increase of quota from Azure portal. ARO requires a minimum of 40 cores.

Register Resource Provider

az account set --subscription
az provider register -n Microsoft.RedHatOpenShift --wait
az provider register -n Microsoft.Compute --wait
az provider register -n Microsoft.Storage --wait

Create a Virtual Network

az network vnet create \
--resource-group $RESOURCEGROUP \
--name aro-vnet \
--address-prefixes 10.0.0.0/22

Create an empty subnet for master nodes

az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name master-subnet \
--address-prefixes 10.0.0.0/23 \
--service-endpoints Microsoft.ContainerRegistry

Create an empty subnet for worker nodes

az network vnet subnet create \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--name worker-subnet \
--address-prefixes 10.0.2.0/23 \
--service-endpoints Microsoft.ContainerRegistry

Disable private endpoint policy

az network vnet subnet update \
--name master-subnet \
--resource-group $RESOURCEGROUP \
--vnet-name aro-vnet \
--disable-private-link-service-network-policies true

Create Cluster

Please make sure you log in to Azure and environment variables are set.

Information that we need for creating a cluster

• Get a copy of the pull secret from cloud.redhat.com. If you don’t have a user name created, please just register as a user for free.
• Create an ARO cluster using the following command. Please apply to appropriate values.
  Some values were used in the example are explained as shown below.
  • aro-vnet – the name of virtual network
  • master-subnet – the name of master subnet
  • worker subnet – the name of worker subnet
  • ./pull-secret.txt – the path and pull secret where is located
  • aro.ocpdemo.online – custom domain for the cluster

az aro create \
--resource-group $RESOURCEGROUP \
--name $CLUSTER \
--vnet aro-vnet \
--master-subnet master-subnet \
--worker-subnet worker-subnet \
--pull-secret @./pull-secret.txt \
--domain aro.ocpdemo.online

The information from the JSON output of the above command can be useful if you are not familiar with OpenShift 4. You can find your API server IP, API URL, OpenShift console URL and ingress IP. You will need the API, and ingress IP for the next step.

{- Finished ..
"apiserverProfile": {
"ip": "x.x.x.x",
"url": "https://api.aro.ocpdemo.online:6443/",
"visibility": "Public"
...
},
"consoleProfile": {
"url": "https://console-openshift-console.apps.aro.ocpdemo.online/"
},
....
"ingressProfiles": [
{
"ip": "x.x.x.x",
"name": "default",
"visibility": "Public"
}
....

Post ARO Installation

Adding two A records for api and *.apps in the DNS zone

1. Login to Azure portal
2. Go to DNS zone
3. Click onto the domain for the ARO cluster
4. Click “+ Record Set” on the top menu to create an A record and add values to Name and IP. You will need to repeat this step for both api and *.apps A records.
   • Name: api or *.apps
   • IP: the *.apps/ingress IP is from the output of the creation of the ARO
5. The below screenshot shows the DNS zone configuration and adding 2 A records.

Test ARO Cluster

Getting Kubeadmin credential

az aro list-credentials \
--name $CLUSTER \
--resource-group $RESOURCEGROUP

Log in OpenShift Console

https://console-openshift-console.apps.<DNS domain>/

Azure Active Directory Integration

Getting oauthCallBackURL

• Download OpenShift command line tool from console.

• Login to ARO via OC CLI

$ oc login -u kubeadmin -p <password> https://api.<DNS domain>:6443/

$ oauthCallBack=`oc get route oauth-openshift -n openshift-authentication -o jsonpath='{.spec.host}'`

$ oauthCallBackURL=https://$oauthCallBack/oauth2callback/AAD

Creating Application on Azure Active Directory

az ad app create \
  --query appId -o tsv \
  --display-name poc-aro-auth \
  --reply-urls $oauthCallBackURL \
  --password '<ClientSecret>'

Get tenant Id

az account show --query tenantId -o tsv

Create manifest file

cat > manifest.json<< EOF
[{
"name": "upn",
"source": null,
"essential": false,
"additionalProperties": []
},
{
"name": "email",
"source": null,
"essential": false,
"additionalProperties": []
}]
EOF

Update the Azure Active Directory with a manifest

az ad app update \
--set optionalClaims.idToken=@manifest.json \
--id <AppId>

Update Application permission scope

az ad app permission add \
--api 00000002-0000-0000-c000-000000000000 \
--api-permissions 311a71cc-e848-46a1-bdf8-97ff7156d8e6=Scope \ 
--id <AppId>

Grant admin consent

1. login Azure portal
2. Go to Azure Active Directory
3. Click App Registrations
4. Click “All Application” and search for newly create application name
5. Click onto the display name of the application
6. Click view API permissions
7. Click on the “check” to grant admin consent for directory

Add service principal

$ az ad sp create-for-rbac --role Contributor --name poc-aro-sp

$az role assignment create --role "User Access Administrator" \
--assignee-object-id $(az ad sp list --filter "appId eq '<service-principal-appid>'" \
| jq '.[0].objectId' -r)

$az ad app permission add --id <appId> \ 
--api 00000002-0000-0000-c000-000000000000 \ 
--api-permissions 824c81eb-e3f8-4ee6-8f6d-de7f50d565b7=Role

$az ad app permission grant --id <appid> --api 00000002-0000-0000-c000-000000000000

Create secret for identity provider on OpenShift

oc create secret generic openid-client-secret-azuread \
--namespace openshift-config \
--from-literal=clientSecret=<your password>

Create YAML for identity provider for AAD

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - name: AAD
    mappingMethod: claim
    type: OpenID
    openID:
      clientID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      clientSecret:
        name: openid-client-secret-azuread
      extraScopes:
      - email
      - profile
      extraAuthorizeParameters:
        include_granted_scopes: "true"
      claims:
        preferredUsername:
        - email
        - upn
        name:
        - name
        email:
        - email
      issuer: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

• The clientID is the AppId of your registered application.
• Issuer URL is https://login.microsoftonline.com/<tenant id>.
• The clientSecret is using the secret (openid-client-secret-azuread) that you created from the previous step.

• Login Azure Portal
• Click Home
• Click Azure Active Directory
• Click App registrations on the left menu
• Click all applications tab
• Type the application that you just created in the search area
• Click onto the application (my application is poc-aro-auth)
• Under Overview, the information is shown as “Application (client) ID” and Directory (tenant) ID” as in the image below.

Update OpenShift OAuth Configuration

oc apply -f openid.yaml

Login OpenShift console via AAD

Troubeshoot

Reference

Azure OpenShift 4 documentation

ARO and Azure Active Directory integration

My task is to figure out how to configure the Metering correlating AWS billing. The OpenShift documentation in the reference is where I started. I decided to record the end-to-end steps on how I set this up since there were some lessons learned in the process of it. I hope this helps you to set up the Metering with AWS billing much smoother.

Prerequisites:

Setting up AWS Report

1. Before creating anything, you need to have data in the Billing & Cost Management Dashboard already.
2. If you have a brand new account, you may have to wait until you get some data to show up before you proceed. You will have to have access to Cost & Usage Report​ under AWS Billing to set up the report.
3. Log in to AWS, go to My Billing Dashboard
4. Click Cost & Usage Reports
5. Click Create reports
6. Provide a name and check Include resource IDs
7. Click Next
8. Click Configure → add S3 bucket name and Region-> click Next
9. Provide `prefix` and select your options for your report → Click Next
10. Once you created a report similar to the following
11. Click onto the S3 bucket and validate reports are being created under the folder.
12. Click Permissions tab
13. Click Bucket Policy
14. Copy and save the bucket policy somewhere you can get back to

Setting up AWS user permission policy

1. Go to My Security Credentials
2. Click Users → Click the username name will be used for accessing the reports and for OpenShift metering.
3. Click Add Permissions→ Attach existing policies directly​ → Create policy → click JSON
4. Paste the buck policy from the Cost & Usage report from the s3 bucket step #14 in the last session.
5. Use the same step to add the following policy:

   { 
     "Version": "2012-10-17", 
     "Statement": [ 
     { 
       "Sid": "1", 
       "Effect": "Allow", 
       "Action": [ 
         "s3:AbortMultipartUpload", 
         "s3:DeleteObject", 
         "s3:GetObject", 
         "s3:HeadBucket", 
         "s3:ListBucket", 
         "s3:ListMultipartUploadParts", 
         "s3:PutObject" 
        ], 
        "Resource": [ 
           "arn:aws:s3:::<YOUR S3 BUCKET NAME FOR BILLING REPORT>/*",  
           "arn:aws:s3:::<YOUR S3 BUCKET NAME FOR BILLING REPORT>"  
         ] 
       } 
       ] 
   }

6. Since I am using s3 bucket for metering storage, I also added the following policy to the user:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "1",
               "Effect": "Allow",
               "Action": [
                   "s3:AbortMultipartUpload",
                   "s3:DeleteObject",
                   "s3:GetObject",
                   "s3:HeadBucket",
                   "s3:ListBucket",
                   "s3:CreateBucket",
                   "s3:DeleteBucket",
                   "s3:ListMultipartUploadParts",
                   "s3:PutObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<YOUR S3 BUCKET NAME FOR METERING STORAGE>/*",
                   "arn:aws:s3:::<YOUR S3 BUCKET NAME FOR METERING STORAGE>"
               ]
           }
       ]
   }

Configuration:

Install Metering Operator

1. Login OpenShift Container Platform web console as cluster-admin, click Administration → Namespaces → Create Namespace​
2. Enter openshift-metering
3. Add openshift.io/cluster-monitoring=true as label → click Create.
4. Click Compute→ Machine Sets
5. If you are like me, the cluster is using the default configuration on AWS. In my test, I increase 1 more worker per AZs.
6. I notice that one pod for Metering requires more resources, and the standard size may not be big enough. I create an m5.2xlarge machine set. I only need 1 replica for this machineset.
   1. Create a template machine-set YAML:

      oc project openshift-machine-api
      oc get machineset poc-p6czj-worker-us-west-2a -o yaml > m52xLms.yaml

   2. Modify the YAML file by updating the name of the machine set and instance type, removing the status, timestamp, id, selflink, etc… Here is my example of a machine set for m5.2xlarge.

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: poc-p6czj
        name: poc-p6czj-xl-worker-us-west-2a
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: poc-p6czj
            machine.openshift.io/cluster-api-machineset: poc-p6czj-xl-worker-us-west-2a
        template:
          metadata:
            creationTimestamp: null
            labels:
              machine.openshift.io/cluster-api-cluster: poc-p6czj
              machine.openshift.io/cluster-api-machine-role: worker
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: poc-p6czj-xl-worker-us-west-2a
          spec:
            metadata:
              creationTimestamp: null
            providerSpec:
              value:
                ami:
                  id: ami-0f0fac946d1d31e97
                apiVersion: awsproviderconfig.openshift.io/v1beta1
                blockDevices:
                - ebs:
                    iops: 0
                    volumeSize: 120
                    volumeType: gp2
                credentialsSecret:
                  name: aws-cloud-credentials
                deviceIndex: 0
                iamInstanceProfile:
                  id: poc-p6czj-worker-profile
                instanceType: m5.2xlarge
                kind: AWSMachineProviderConfig
                metadata:
                  creationTimestamp: null
                placement:
                  availabilityZone: us-west-2a
                  region: us-west-2
                publicIp: null
                securityGroups:
                - filters:
                  - name: tag:Name
                    values:
                    - poc-p6czj-worker-sg
                subnet:
                  filters:
                  - name: tag:Name
                    values:
                    - poc-p6czj-private-us-west-2a
                tags:
                - name: kubernetes.io/cluster/poc-p6czj
                  value: owned
                userDataSecret:
                  name: worker-user-data

   3. run:

      oc create -f m52xLms.yaml
      # wait for the new machine for m5.2xlarge created
      oc get machineset

7. Create a secret to access the AWS account and make sure you are cluster-admin and run the following commands:

   oc project openshift-metering
   oc create secret -n openshift-metering generic my-aws-secret --from-literal=aws-access-key-id=<YOUR AWS KEY> --from-literal=aws-secret-access-key=<YOUR AWS SECRET>

8. Back to Console, click Operators → OperatorHub and type ‘metering` in the filter to find the Metering Operator.
9. Click the Metering (provided by Red Hat), review the package description, and then click install.
10. Under Installation Mode, select openshift-metering as namespace. Specify your update channel and approval strategy, then click Subscribe to install Metering.
11. Click Installed Operators from the left menu, wait for Succeeded as status is shown next to the Metering Operator.
12. Click Workloads → Pods → metering operator pod is in Running state
13. Go back to your terminal, run:

    oc project openshift-metering

14. We are now ready to create the MeteringConfig Object. Create a file `metering-config.yaml` as shown below. See the reference for more details of the MeteringConfig object.

    apiVersion: metering.openshift.io/v1
    kind: MeteringConfig
    metadata:
      name: operator-metering
      namespace: openshift-metering
    spec:
      openshift-reporting:
        spec:
          awsBillingReportDataSource:
            enabled: true
            bucket: "logs4reports"
            prefix: "bubble/ocpreports/"
            region: "us-west-2"
      storage:
        type: hive
        hive:
          s3:
            bucket: shanna-meter/demo
            createBucket: true
            region: us-west-2
            secretName: my-aws-secret
          type: s3
      presto:
        spec:
          config:
            aws:
              secretName: my-aws-secret
      hive:
        spec:
          config:
            aws:
              secretName: my-aws-secret
      reporting-operator:
        spec:
          config:
            aws:
              secretName: my-aws-secret
          resources:
            limits:
              cpu: 1
              memory: 500Mi
            requests:
              cpu: 500m
              memory: 100Mi

15. Create MeteringConfig:

    oc create -f metering-config.yaml

16. To monitor the process:

    watch 'oc get pod'

17. Wait until you see all pods are up and running:

    $ oc get pods
    NAME                              READY STATUS   RESTARTS AGE
    hive-metastore-0                   2/2   Running 0        2m35s
    hive-server-0                      3/3   Running 0        2m36s
    metering-operator-69b664dc57-knd86 2/2   Running 0        31m
    presto-coordinator-0               2/2   Running 0        2m8s
    reporting-operator-674cb5d7b-zxwf4 1/2   Running 0        96s

18. Verify the AWS report data source:

    $ oc get reportdatasource |grep aws
    aws-billing                                                                                                                                                     3m41s
    aws-ec2-billing-data-raw

19. Verify the AWS report queries:

    $ oc get reportquery |grep aws
    aws-ec2-billing-data                         5m19s
    aws-ec2-billing-data-raw                     5m19s
    aws-ec2-cluster-cost                         5m19s
    pod-cpu-request-aws                          5m19s
    pod-cpu-usage-aws                            5m19s
    pod-memory-request-aws                       5m18s
    pod-memory-usage-aws                         5m18s

    For more information about the ReportDataSource and the ReportQuery​, please check out the GitHub link in the reference.

20. Create reports to get AWS billing from the following YAML:

    apiVersion: metering.openshift.io/v1
    kind: Report
    metadata:
      name: pod-cpu-request-billing-run-once
    spec:
      query: "pod-cpu-request-aws"
      reportingStart: '2020-04-12T00:00:00Z'
      reportingEnd: '2020-04-30T00:00:00Z'
      runImmediately: true
    ---
    apiVersion: metering.openshift.io/v1
    kind: Report
    metadata:
      name: pod-memory-request-billing-run-once
    spec:
      query: "pod-memory-request-aws"
      reportingStart: '2020-04-12T00:00:00Z'
      reportingEnd: '2020-04-30T00:00:00Z'
      runImmediately: true

21. Create reports (status as `RunImmediately`):

    $ oc create -f aws-reports.yaml
    $ oc get reports
    NAME                                  QUERY                    SCHEDULE   RUNNING          FAILED   LAST REPORT TIME   AGE
    pod-cpu-request-billing-run-once      pod-cpu-request-aws                 RunImmediately                               5s
    pod-memory-request-billing-run-once   pod-memory-request-aws              RunImmediately                               5s

22. Wait until reports are completed (status as `Finished`):

    $ oc get reports
    NAME                                  QUERY                    SCHEDULE   RUNNING    FAILED   LAST REPORT TIME       AGE
    pod-cpu-request-billing-run-once      pod-cpu-request-aws                 Finished            2020-04-30T00:00:00Z   79s
    pod-memory-request-billing-run-once   pod-memory-request-aws              Finished            2020-04-30T00:00:00Z   79s

23. I created a simple script (viewReport.sh) as shown below to view any report which requires $1 as the name of the report from oc get reports

    reportName=$1
    reportFormat=csv
    token="$(oc whoami -t)"
    meteringRoute="$(oc get routes metering -o jsonpath='{.spec.host}')"
    curl --insecure -H "Authorization: Bearer ${token}" "https://${meteringRoute}/api/v1/reports/get?name=${reportName}&namespace=openshift-metering&format=$reportFormat"

24. Before running the script, please make sure you get a validate token via oc whoami -t
25. View report by run the simple script in step #23:

    ./viewReport.sh pod-cpu-request-billing-run-once
    period_start,period_end,pod,namespace,node,pod_request_cpu_core_seconds,pod_cpu_usage_percent,pod_cost
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,alertmanager-main-0,openshift-monitoring,ip-10-0-174-47.us-west-2.compute.internal,792.000000,0.006587,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,alertmanager-main-1,openshift-monitoring,ip-10-0-138-24.us-west-2.compute.internal,792.000000,0.006587,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,alertmanager-main-2,openshift-monitoring,ip-10-0-148-172.us-west-2.compute.internal,792.000000,0.006587,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,apiserver-9dhcr,openshift-apiserver,ip-10-0-157-2.us-west-2.compute.internal,1080.000000,0.008982,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,apiserver-fr7w5,openshift-apiserver,ip-10-0-171-27.us-west-2.compute.internal,1080.000000,0.008982,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,apiserver-sdlsj,openshift-apiserver,ip-10-0-139-242.us-west-2.compute.internal,1080.000000,0.008982,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,apiservice-cabundle-injector-54ff756f6d-f4vl6,openshift-service-ca,ip-10-0-157-2.us-west-2.compute.internal,72.000000,0.000599,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,authentication-operator-6d865c4957-2jsql,openshift-authentication-operator,ip-10-0-171-27.us-west-2.compute.internal,72.000000,0.000599,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,catalog-operator-868fd6ddb5-rmfk7,openshift-operator-lifecycle-manager,ip-10-0-139-242.us-west-2.compute.internal,72.000000,0.000599,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,certified-operators-58874b4f86-rcbsl,openshift-marketplace,ip-10-0-148-172.us-west-2.compute.internal,20.400000,0.000170,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,certified-operators-5b86f97d6f-pcvqk,openshift-marketplace,ip-10-0-148-172.us-west-2.compute.internal,16.800000,0.000140,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,certified-operators-5fdf46bd6d-hhtqd,openshift-marketplace,ip-10-0-148-172.us-west-2.compute.internal,37.200000,0.000309,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,cloud-credential-operator-868c5f9f7f-tw5pn,openshift-cloud-credential-operator,ip-10-0-157-2.us-west-2.compute.internal,72.000000,0.000599,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,cluster-autoscaler-operator-74b5d8858b-bwtfc,openshift-machine-api,ip-10-0-139-242.us-west-2.compute.internal,144.000000,0.001198,
    2020-04-12 00:00:00 +0000 UTC,2020-04-30 00:00:00 +0000 UTC,cluster-image-registry-operator-9754995-cqm7v,openshift-image-registry,ip-10-0-139-242.us-west-2.compute.internal,144.000000,0.001198,
    ...

26. The output from the preview steps are not too readable. Instead, I downloaded the output from the previous step into a file.

    ./viewReport.sh pod-cpu-request-billing-run-once > aws-pod-cpu-billing.txt

27. Import the output file into a spreadsheet as shown below:

Troubleshoot:

The most useful log is the report operator log for debugging any report issues.

Reference:

OpenShift metering documentation: https://docs.openshift.com/container-platform/4.3/metering/metering-about-metering.html

Configure AWS Billing Correlation: https://docs.openshift.com/container-platform/4.6/metering/configuring_metering/metering-configure-aws-billing-correlation.html

Addition information: https://github.com/kube-reporting/metering-operator/blob/master/Documentation/metering-architecture.md

Lesson learned from the last test (https://shanna-chan.blog/2019/07/26/openshift4-vsphere-static-ip/), and I got questions around clarification on using static IP. My apologies for the confusion from my last test since it was my test without any real documentation. I want to record all my errors so I can help others to troubleshoot.

Anyway, I decided to retest the installation of OCP 4.3 using static IP. The goal to clarify the installation instructions my last note from the last blog if you are trying to install OCP4 on the VMware environment manually using static IP.

Environment:

• OCP 4.3.5
• vSphere 6.7

List of VMs:

• Bootstrap 192.168.1.110
• Master0 192.168.1.111
• Master1 192.168.1.112
• Master2 192.168.1.113
• Worker0 192.168.1.114
• Worker1 192.168.1.115

Prerequisites:

The following components are already running in my test environment.

DNS Server

1. Add Zone /etc/named.conf. An example can be found here https://github.com/christianh814/openshift-toolbox/blob/master/ocp4_upi/docs/0.prereqs.md#dns
2. Configures the zone files for all the DNS entries. An example configuration is shown below.

   ; The api points to the IP of your load balancer
   api.ocp43	IN	A	192.168.1.72
   api-int.ocp43	IN	A	192.168.1.72
   ;
   ; The wildcard also points to the load balancer
   *.apps.ocp43	IN	A	192.168.1.72
   ;
   ; Create entry for the bootstrap host
   bootstrap0.ocp43	IN	A	192.168.1.110
   ;
   ; Create entries for the master hosts
   master01.ocp43	IN	A	192.168.1.111
   master02.ocp43	IN	A	192.168.1.112
   master03.ocp43	IN	A	192.168.1.113
   ;
   ; Create entries for the worker hosts
   worker01.ocp43	IN	A	192.168.1.114
   worker02.ocp43	IN	A	192.168.1.115
   ;
   ; The ETCd cluster lives on the masters...so point these to the IP of the masters
   etcd-0.ocp43	IN	A	192.168.1.111
   etcd-1.ocp43	IN	A	192.168.1.112
   etcd-2.ocp43	IN	A	192.168.1.113
   ;
   ; The SRV records are IMPORTANT....make sure you get these right...note the trailing dot at the end...
   _etcd-server-ssl._tcp.ocp43	IN	SRV	0 10 2380 etcd-0.ocp43.example.com.
   _etcd-server-ssl._tcp.ocp43	IN	SRV	0 10 2380 etcd-1.ocp43.example.com.
   _etcd-server-ssl._tcp.ocp43	IN	SRV	0 10 2380 etcd-2.ocp43.example.com.

Load balancer

1. Update /etc/haproxy/haproxy.cfg with cluster information. An example is shown below.

   #---------------------------------------------------------------------
   
   listen stats
       bind *:9000
       mode http
       stats enable
       stats uri /
       monitor-uri /healthz
   
   #---------------------------------------------------------------------
   #Cluster ocp43 - static ip test
   frontend openshift-api-server
       bind *:6443
       default_backend openshift-api-server
       mode tcp
       option tcplog
   
   backend openshift-api-server
       balance source
       mode tcp
       #server bootstrap0.ocp43.example.com 192.168.1.110:6443 check
       server master01.ocp43.example.com 192.168.1.111:6443 check
       server master02.ocp43.example.com 192.168.1.112:6443 check
       server master03.ocp43.example.com 192.168.1.113:6443 check
   
   frontend machine-config-server
       bind *:22623
       default_backend machine-config-server
       mode tcp
       option tcplog
   
   backend machine-config-server
       balance source
       mode tcp
       # server bootstrap0.ocp43.example.com 192.168.1.110:22623 check
       server master01.ocp43.example.com 192.168.1.111:22623 check
       server master02.ocp43.example.com 192.168.1.112:22623 check
       server master03.ocp43.example.com 192.168.1.113:22623 check
   
   frontend ingress-http
       bind *:80
       default_backend ingress-http
       mode tcp
       option tcplog
   
   backend ingress-http
       balance source
       mode tcp
       server worker01.ocp43.example.com 192.168.1.114:80 check
       server worker02.ocp43.example.com 192.168.1.115:80 check
   
   frontend ingress-https
       bind *:443
       default_backend ingress-https
       mode tcp
       option tcplog
   
   backend ingress-https
       balance source
       mode tcp
       server worker01.ocp43.example.com 192.168.1.114:443 check
       server worker02.ocp43.example.com 192.168.1.115:443 check

Web Server

1. Configure a web server. In my example, I configure httpd on an RHEL VM.

yum -y install httpd
systemctl enable --now httpd
firewall-cmd --add-service=8080/tcp --permanent
firewall-cmd --reload

Installation downloads

• From https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.3.5/
  • openshift-install-mac-4.3.5.tar.gz
  • openshift-client-mac-4.3.5.tar.gz
• From https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.3/latest/
  • rhcos-4.3.0-x86_64-installer.iso
  • rhcos-4.3.0-x86_64-metal.raw.gz

Installation Using Static IP address

Prepare installation

1. Generate SSH key:

   $ ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/vsphere-ocp43

2. Start ssh-agent:

   $ eval "$(ssh-agent -s)"

3.  Add ssh private key to the ssh-agent:

   $ ssh-add ~/.ssh/vsphere-ocp43
   Identity added: /Users/shannachan/.ssh/vsphere-ocp43 (shannachan@MacBook-Pro)

4. Download & extract OpenShift Installer:

   wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.3.5/openshift-install-mac-4.3.5.tar.gz
   tar zxvf openshift-install-mac-4.3.5.tar.gz

5. Download & extract OpenShift CLI:

   wget wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.3.5/openshift-client-mac-4.3.5.tar.gz
   tar zxvf openshift-client-mac-4.3.5.tar.gz

6. Copy or download the pull secret from cloud.redhat.com
   1. Go to cloud.redhat.com
   2. Login with your credential (create an account if you don’t have one)
   3. Click “Create Cluster”
   4. Click OpenShift Container Platform
   5. Scroll down and click “VMware vSphere”
   6. Click on “Download Pull Secret” to download the secret

Create Installation manifests and ignition files

1. Create an installation directory:

   mkdir ocp43

2. Create `install-config.yaml` as shown below.

   apiVersion: v1
   baseDomain: example.com
   compute:
   - name: worker
     replicas: 0
   controlPlane:
     hyperthreading: Enabled
     name: master
     replicas: 3
   metadata:
     name: ocp43
   platform:
     vsphere:
       vcenter: 192.168.1.200
       username: vsphereadmin
       password: xxxx
       datacenter: Datacenter
       defaultDatastore: datastore3T
   pullSecret: '<copy your pull secret here>'
   sshKey: '<copy your public key here>'

3. Backup install-config.yaml  and copy it into the installation directory
4. Generate Kubernetes manifests for the cluster:

   $./openshift-install create manifests --dir=./ocp43
   INFO Consuming Install Config from target directory
   WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings

5. Modify <installation directory>/manifests/cluster-scheduler-02-config.yml
6. Update mastersSchedulable to false
7. Obtain Ignition files:

   $ ./openshift-install create ignition-configs --dir=./ocp43
   INFO Consuming Common Manifests from target directory
   INFO Consuming Worker Machines from target directory
   INFO Consuming Master Machines from target directory
   INFO Consuming OpenShift Install (Manifests) from target directory
   INFO Consuming Openshift Manifests from target directory

8. Files that were created:

   $ tree ocp43
   ocp43
   ├── auth
   │   ├── kubeadmin-password
   │   └── kubeconfig
   ├── bootstrap.ign
   ├── master.ign
   ├── metadata.json
   └── worker.ign

Upload files to the webserver

1. Upload the rhcos-4.3.0-x86_64-metal.raw.gz to web server location
2. Upload all the ignition files to the webserver location
3. Update the file permission on the *.ign files on the webserver:

   chmod 644 *.ign

Note: check and make sure that you can download the ignition files and gz file for the webserver.

Custom ISO

Create all custom ISO files with the parameters that you need for each VMs. This step can skip if you plan to type all the kernel parameters by hand when prompt.

1. Download rhcos-4.3.0-x86_64-installer.iso and rhcos-4.3.0-x86_64-metal.raw.gz
2. Extract ISO to a temporary location:

   sudo mount rhcos-410.8.20190425.1-installer.iso /mnt/ 
   mkdir /tmp/rhcos 
   rsync -a /mnt/* /tmp/rhcos/ 
   cd /tmp/rhcos 
   vi isolinux/isolinux.cfg

3. Modify the boot entry similar to this:

   label linux
     menu label ^Install RHEL CoreOS
     kernel /images/vmlinuz
     append initrd=/images/initramfs.img nomodeset rd.neednet=1 coreos.inst=yes ip=192.168.1.110::192.168.1.1:255.255.255.0:bootstrap0.ocp43.example.com:ens192:none nameserver=192.168.1.188 coreos.inst.install_dev=sda coreos.inst.image_url=http://192.168.1.230:8080/rhcos-4.3.0-x86_64-metal.raw.gz coreos.inst.ignition_url=http://192.168.1.230:8080/bootstrap.ign

   where:

   ip=<ip address of the VM>::<gateway>:<netmask>:<hostname of the VM>:<interface>:none

   nameserver=<DNS>

   coreos.inst.image_url=http://<webserver host:port>/rhcos-4.3.0-x86_64-metal.raw.gz

   coreos.inst.ignition_url=http://<webserver host:port>/<bootstrap, master or worker ignition>.ign

4. Create new ISO as /tmp/rhcos_install.iso:

   sudo mkisofs -U -A "RHCOS-x86_64" -V "RHCOS-x86_64" -volset "RHCOS-x86_64" -J -joliet-long -r -v -T -x ./lost+found -o /tmp/rhcos_install.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img -no-emul-boot .

5.  Upload all the custom ISOs to the datastore for VM creation via vCenter
6. You will repeat the steps for all VMs with the specific IP and ign file. You only need to create individual VM for the cluster if you don’t want to type the kernel parameters at the prompt when installing via the ISO. I would recommend that since it actually takes less time to do that than typing the kernel parameters each time.

Create VM using custom ISO

1. Create a resource folder
   • Action -> New folder -> New VM or Template folder
   • I normally give the name as the cluster id
2. Create VM with 4 CPU and 16 RAM
   • Action -> New Virtual Machine
   • Select Create New Virtual Machine -> click Next
   • Add name
   • Select the VM folder -> Next
   • Select datacenter -> Next
   • Select storage -> Next
   • Use ESXi 6.7 -> Next
   • Select Linux and RHEL 7 -> Next
   • Use these parameters:
     • CPU: 4
     • Memory: 16 (Reserve all guest memory)
     • 120 GB disk
     • Select the corresponding ISO from Datastore and check “connect”
     • VMOption -> advantage -> Edit configuration -> Add configuration Params -> Add “disk.EnableUUID”: Specify TRUE
     • Click OK
     • Click Next
     • Click Finish
3. Power on the bootstrap, masters and workers VMs as the steps below
4. Go the VM console:
5. Hit Enter
6. You should see the login screen once the VM boots successfully
7. repeat on all servers and make sure the specific ISO for the given VM is used.

Tips: you can clone the existing VM and just modify the ISO files for VM creation.

Creating Cluster

1. Monitor the cluster:

   ./openshift-install --dir=<installation_directory> wait-for bootstrap-complete --log-level=info
   INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp43.example.com:6443...
   INFO API v1.16.2 up
   INFO Waiting up to 30m0s for bootstrapping to complete...
   INFO It is now safe to remove the bootstrap resources

2.  From the bootstrap VM, similar log messages are shown:

   $ ssh -i ~/.ssh/vsphere-ocp43 core@bootstrap-vm
   $ journalctl -b -f -u bootkube.service
   ...
   Mar 16 20:03:57 bootstrap0.ocp43.example.com bootkube.sh[2816]: Tearing down temporary bootstrap control plane...
   Mar 16 20:03:57 bootstrap0.ocp43.example.com podman[18629]: 2020-03-16 20:03:57.232567868 +0000 UTC m=+726.128069883 container died 695412d7eece5a9bd099aac5b6bc6a8d412c8037b14391ff54ee33132ebce0e1 (image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:222fbfd3323ec347babbda1a66929019221fcee82cfc324a173b39b218cf6c4b, name=zen_lamarr)
   Mar 16 20:03:57 bootstrap0.ocp43.example.com podman[18629]: 2020-03-16 20:03:57.379721836 +0000 UTC m=+726.275223886 container remove 695412d7eece5a9bd099aac5b6bc6a8d412c8037b14391ff54ee33132ebce0e1 (image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:222fbfd3323ec347babbda1a66929019221fcee82cfc324a173b39b218cf6c4b, name=zen_lamarr)
   Mar 16 20:03:57 bootstrap0.ocp43.example.com bootkube.sh[2816]: bootkube.service complete

3. Load balancer status
4. Remove the bootstrap from the Load Balancer. You can check the status of LB from the status page

Logging in to the Cluster

1.  Export the kubeadmiin credentials:

   export KUBECONFIG=./ocp43/auth/kubeconfig

2.  Verify cluster role via oc CLI

   $ oc whoami
   system:admin

3. Approving the CSRs

   $ oc get nodes
   NAME                         STATUS   ROLES    AGE   VERSION
   master01.ocp43.example.com   Ready    master   60m   v1.16.2
   master02.ocp43.example.com   Ready    master   60m   v1.16.2
   master03.ocp43.example.com   Ready    master   60m   v1.16.2
   worker01.ocp43.example.com   Ready    worker   52m   v1.16.2
   worker02.ocp43.example.com   Ready    worker   51m   v1.16.2
   
   $ oc get csr
   NAME        AGE   REQUESTOR                                                                   CONDITION
   csr-66l6l   60m   system:node:master02.ocp43.example.com                                      Approved,Issued
   csr-8r2dc   52m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
   csr-hvt2d   51m   system:node:worker02.ocp43.example.com                                      Approved,Issued
   csr-k2ggg   60m   system:node:master03.ocp43.example.com                                      Approved,Issued
   csr-kg72s   52m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
   csr-qvbg2   60m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
   csr-rtncq   52m   system:node:worker01.ocp43.example.com                                      Approved,Issued
   csr-tsfxx   60m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
   csr-wn7rp   60m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
   csr-zl87q   60m   system:node:master01.ocp43.example.com                                      Approved,Issued

4. If there is pending CSR, approve the CSR via the command below.

   oc adm certificate approve <csr_name>

5.  Validate the cluster components all available:

   $ oc get co
   NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                             4.3.5     True        False         False      41m
   cloud-credential                           4.3.5     True        False         False      63m
   cluster-autoscaler                         4.3.5     True        False         False      47m
   console                                    4.3.5     True        False         False      43m
   dns                                        4.3.5     True        False         False      54m
   image-registry                             4.3.5     True        False         False      49m
   ingress                                    4.3.5     True        False         False      48m
   insights                                   4.3.5     True        False         False      58m
   kube-apiserver                             4.3.5     True        False         False      53m
   kube-controller-manager                    4.3.5     True        False         False      54m
   kube-scheduler                             4.3.5     True        False         False      54m
   machine-api                                4.3.5     True        False         False      55m
   machine-config                             4.3.5     True        False         False      55m
   marketplace                                4.3.5     True        False         False      48m
   monitoring                                 4.3.5     True        False         False      42m
   network                                    4.3.5     True        False         False      59m
   node-tuning                                4.3.5     True        False         False      50m
   openshift-apiserver                        4.3.5     True        False         False      51m
   openshift-controller-manager               4.3.5     True        False         False      55m
   openshift-samples                          4.3.5     True        False         False      46m
   operator-lifecycle-manager                 4.3.5     True        False         False      55m
   operator-lifecycle-manager-catalog         4.3.5     True        False         False      55m
   operator-lifecycle-manager-packageserver   4.3.5     True        False         False      51m
   service-ca                                 4.3.5     True        False         False      58m
   service-catalog-apiserver                  4.3.5     True        False         False      50m
   service-catalog-controller-manager         4.3.5     True        False         False      50m
   storage                                    4.3.5     True        False         False      49m

Configure the Image Registry to use ephemeral storage for now.

I will update the image registry in the other blog since I want to focus on the completion of the installation.

To set emptyDir for the image registry:

oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'

Completing the installation:

$ ./openshift-install --dir=./ocp43 wait-for install-complete
INFO Waiting up to 30m0s for the cluster at https://api.ocp43.example.com:6443 to initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/Users/shannachan/projects/ocp4.3/ocp43/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp43.example.com
INFO Login to the console with user: kubeadmin, password: xxxxxxxxxxxxxx

Congratulation Cluster is up!

Troubleshoot tips:

Access any server via the command below:

ssh -i ~/.ssh/vsphere-ocp43 core@vm-server

Reference:

https://docs.openshift.com/container-platform/4.3/installing/installing_bare_metal/installing-bare-metal.html

https://docs.openshift.com/container-platform/4.3/installing/installing_vsphere/installing-vsphere.html

https://shanna-chan.blog/2019/07/26/openshift4-vsphere-static-ip/