Month: May 2025

OpenShift Agent Installer on bare metal in a restricted environment

My goal for this post is to share my steps for installing OpenShift using Agent Installer in a restricted environment using a mirror registry.

My limitation is that my hardware is ancient 🙂 I used ESXi to simulate the bare metal hosts, but did not use vSphere as the provider for the installation.

My condition for this test:

  • I can only use static IP addresses (no DHCP).
  • RHEL 9 is the provision server with installed nmstatectl, oc CLI, oc mirror, and mirror registry.
  • I used Agent Installer to install 4.16.39, a three-node compact cluster.

High-level preparation steps:

  1. Set up the DNS
  2. Create a cert for the mirror registry
  3. Install mirror registry
  4. Update CA trust on the provision host
  5. Mirror the image from the source (quay.io)
  6. Create agent-config.yaml and install-config.yaml
    • agent-config.yaml must define the NTP servers (of your choice)
    • install-config.yaml’s pullSecret must include the mirror registry credential and the cert for the mirror registry.

Download links

My example DNS configuration

Install mirror registry

I used the Red Hat mirror registry (Quay). You can also mirror the images using Nexus, Jfrog, or Harbor. Please use the Reference [4] to generate certs.
Run the following command to install the mirror registry.

$ ./mirror-registry -v install --quayHostname bastion.example.com --quayRoot /opt/ocpmirror --initUser admin --initPassword admin123456 --quayStorage /opt/mirrorStorage --sslCert ssl.cert --sslKey ssl.key

Mirror the images

To mirror images using plugin v2, you must have downloaded the ‘oc’ and ‘oc mirror’ plug-ins.

Download pullSecret.txt and update the credentials for your environment.

Please use reference [2] to configure the pullSecret.json. The following commands can be used to validate the pullSecret file.

$podman login --authfile local.json -u $QUAY_USER -p $QUAY_PWD $QUAY_HOST_NAME:$QUAY_PORT --tls-verify=false

$jq -cM -s '{"auths": ( .[0].auths + .[1].auths ) }' local.json ~/pull-secret.txt > pull-secret.json

$podman login --authfile ./pull-secret.json quay.io
$podman login --authfile ./pull-secret.json registry.redhat.io
$podman login --authfile ./pull-secret.json $QUAY_HOST_NAME:$QUAY_PORT

My example imageSetConfiguration file

Run the following command to mirror the images to the mirror registry.

$ oc mirror --config imageSetConfiguration-v2-4.16.39.yaml --authfile /root/mirror-reg/pull-secret.json --workspace file:///opt/working-dir docker://bastion.example.com:8443/ocp4 --v2

Output from the ‘oc mirror’

Configuration files for the installation

  • agent-config.yaml
  • install-config.yaml

My example agent-config.yaml

apiVersion: v1beta1
kind: AgentConfig
metadata:
  name: demo 
additionalNTPSources:
- time1.google.com
- time2.google.com
rendezvousIP: 192.168.1.121
hosts:
  - hostname: max1.ocp.example.com
    rootDeviceHints:
      deviceName: /dev/sda
    interfaces:
      - name: ens160
        macAddress: 00:0c:29:5e:fe:f3
    networkConfig:
      interfaces:
        - name: ens160
          type: ethernet
          state: up
          mac-address: 00:0c:29:5e:fe:f3
          ipv4:
            enabled: true
            address:
              - ip: 192.168.1.121
                prefix-length: 23
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.1.188 
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.1.188
            next-hop-interface: ens160
            table-id: 254
  - hostname: max2.ocp.example.com
    rootDeviceHints:
      deviceName: /dev/sda
    interfaces:
      - name: ens160
        macAddress: 00:0c:29:a7:4d:e0
    networkConfig:
      interfaces:
        - name: ens160
          type: ethernet
          state: up
          mac-address: 00:0c:29:a7:4d:e0
          ipv4:
            enabled: true
            address:
              - ip: 192.168.1.122
                prefix-length: 23
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.1.188 
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.1.188
            next-hop-interface: ens160
            table-id: 254
  - hostname: max3.ocp.example.com
    rootDeviceHints:
      deviceName: /dev/sda
    interfaces:
      - name: ens160
        macAddress: 00:0c:29:59:2e:10
    networkConfig:
      interfaces:
        - name: ens160
          type: ethernet
          state: up
          mac-address: 00:0c:29:59:2e:10
          ipv4:
            enabled: true
            address:
              - ip: 192.168.1.123
                prefix-length: 23
            dhcp: false
      dns-resolver:
        config:
          server:
            - 192.168.1.188 
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: 192.168.1.188
            next-hop-interface: ens160
            table-id: 254

My example of install-config.yaml

apiVersion: v1
baseDomain: ocp.example.com
compute:
- name: worker
  replicas: 0
controlPlane:
  name: master
  replicas: 3
metadata:
  name: demo
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.0.0/23
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  baremetal:
    hosts:
      - name: max1.ocp.example.com
        role: master
        bootMACAddress: 00:0c:29:5e:fe:f3
      - name: max2.ocp.example.com
        role: master
        bootMACAddress: 00:0c:29:a7:4d:e0
      - name: max3.ocp.example.com
        role: master
        bootMACAddress: 00:0c:29:59:2e:10
    apiVIPs:
    - 192.168.1.126
    ingressVIPs:
    - 192.168.1.125
fips: false
pullSecret: '{"auths":{"..."}}}'
sshKey: 'ssh-rsa … root@bastion.example.com'
imageContentSources:
- mirrors:
  - bastion.example.com:8443/ocp4/openshift/release-images
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - bastion.example.com:8443/ocp4/openshift/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
 …
  -----END CERTIFICATE-----

Steps that I took before booting up the hosts

  1. I created VMs (bare metal hosts) on my ESXi host. Because I am using an ESXi host, I can get the MAC addresses from the UI.
  2. Add the MAC addresses to the agent-install.yaml
  3. Make a directory. I use ‘demo’ in my example here.
  4. Copy agent-config.yaml and install-config.yaml to the demo directory.
  5. Run the following command to create the ISO from the parent of the demo directory. The command will output agent.x86_64.iso to the demo directory.
$ openshift-install --dir demo agent create image

Now you have to ISO to boot all the hosts

  1. Upload the ISO to the ESXi datastore
  2. Configure all bare metal hosts (VM in my case) to boot with the agent.x86_64.iso
  3. Boot all three hosts in sequence and run the command below.
$./openshift-install --dir demo agent wait-for bootstrap-complete  --log-level=info

You will monitor the status of the bootstrap from the output. (It took a while to complete, as you can see)

[root@bastion ~]# ./openshift-install --dir demo agent create image
WARNING imageContentSources is deprecated, please use ImageDigestSources 
INFO Configuration has 3 master replicas and 0 worker replicas 
WARNING hosts from install-config.yaml are ignored   
WARNING The imageDigestSources configuration in install-config.yaml should have at least one source field matching the releaseImage value bastion.example.com:8443/ocp4/openshift/release-images@sha256:2754cd66072e633063b6bf26446978102f27dd19d4668b20df2c7553ef9ee4cf 
WARNING Certificate 2020B78FC3BA75A644FD58F757EFAE86C81FA384 from additionalTrustBundle is x509 v3 but not a certificate authority 
INFO The rendezvous host IP (node0 IP) is 192.168.1.121 
INFO Extracting base ISO from release payload     
INFO Verifying cached file                        
INFO Using cached Base ISO /root/.cache/agent/image_cache/coreos-x86_64.iso 
INFO Consuming Agent Config from target directory 
INFO Consuming Install Config from target directory 
INFO Generated ISO at demo/agent.x86_64.iso       
[root@bastion ~]# ./openshift-install --dir demo agent wait-for bootstrap-complete  --log-level=info
INFO Waiting for cluster install to initialize. Sleeping for 30 seconds 
INFO Cluster is not ready for install. Check validations 
…
INFO Host max2.ocp.example.com: calculated role is master 
INFO Cluster validation: api vips 192.168.1.126 belongs to the Machine CIDR and is not in use. 
INFO Cluster validation: ingress vips 192.168.1.125 belongs to the Machine CIDR and is not in use. 
INFO Cluster validation: The cluster has the exact amount of dedicated control plane nodes. 
INFO Host 946b4d56-fef7-9683-5b01-6405c8592e10: Successfully registered 
WARNING Host max1.ocp.example.com validation: No connectivity to the majority of hosts in the cluster 
WARNING Host max3.ocp.example.com validation: No connectivity to the majority of hosts in the cluster 
WARNING Host max3.ocp.example.com validation: Host couldn't synchronize with any NTP server 
WARNING Host max2.ocp.example.com validation: No connectivity to the majority of hosts in the cluster 
INFO Host max2.ocp.example.com: calculated role is master 
INFO Host max1.ocp.example.com validation: Host has connectivity to the majority of hosts in the cluster 
INFO Host max2.ocp.example.com validation: Host has connectivity to the majority of hosts in the cluster 
INFO Host max3.ocp.example.com validation: Host has connectivity to the majority of hosts in the cluster 
INFO Host max3.ocp.example.com: updated status from insufficient to known (Host is ready to be installed) 
INFO Preparing cluster for installation           
INFO Cluster validation: All hosts in the cluster are ready to install. 
INFO Host max3.ocp.example.com: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation) 
INFO Host max1.ocp.example.com: New image status quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a0fac1616598bda78643c7351837d412f822d49adc20b8f9940490f080310c92. result: success. time: 8.93 seconds; size: 411.27 Megabytes; download rate: 48.31 MBps 
INFO Host max1.ocp.example.com: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation) 
INFO Host max2.ocp.example.com: New image status quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a0fac1616598bda78643c7351837d412f822d49adc20b8f9940490f080310c92. result: success. time: 7.68 seconds; size: 411.27 Megabytes; download rate: 56.14 MBps 
INFO Host max2.ocp.example.com: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation) 
INFO Host max3.ocp.example.com: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation) 
INFO Cluster installation in progress             
INFO Host max3.ocp.example.com: updated status from preparing-successful to installing (Installation is in progress) 
INFO Host: max2.ocp.example.com, reached installation stage Starting installation: master 
INFO Host: max1.ocp.example.com, reached installation stage Installing: master 
INFO Host: max2.ocp.example.com, reached installation stage Writing image to disk: 5% 
…
INFO Host: max3.ocp.example.com, reached installation stage Writing image to disk: 100% 
INFO Bootstrap Kube API Initialized               
INFO Host: max1.ocp.example.com, reached installation stage Waiting for control plane: Waiting for masters to join bootstrap control plane 
INFO Host: max2.ocp.example.com, reached installation stage Rebooting 
INFO Host: max2.ocp.example.com, reached installation stage Configuring 
INFO Host: max3.ocp.example.com, reached installation stage Rebooting 
INFO Host: max3.ocp.example.com, reached installation stage Configuring 
INFO Host: max3.ocp.example.com, reached installation stage Joined 
INFO Host: max1.ocp.example.com, reached installation stage Waiting for bootkube 
INFO Host: max3.ocp.example.com, reached installation stage Done 
INFO Host: max1.ocp.example.com, reached installation stage Waiting for bootkube: waiting for ETCD bootstrap to be complete 
INFO Bootstrap configMap status is complete       
INFO Bootstrap is complete                        
INFO cluster bootstrap is complete

When bootstrap is completed …

Run the following command to wait for the installation to be completed.

$ ./openshift-install --dir demo agent wait-for install-complete

Now you can sit back and wait for it to complete.

[root@bastion ~]# ./openshift-install --dir demo agent wait-for install-complete
INFO Cluster installation in progress             
WARNING Host max1.ocp.example.com validation: Host couldn't synchronize with any NTP server 
INFO Host: max1.ocp.example.com, reached installation stage Waiting for controller: waiting for controller pod ready event 
INFO Bootstrap Kube API Initialized               
INFO Bootstrap configMap status is complete       
INFO Bootstrap is complete                        
INFO cluster bootstrap is complete                
INFO Cluster is installed                         
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 
INFO     export KUBECONFIG=/root/demo/auth/kubeconfig 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.demo.ocp.example.com 
INFO Login to the console with user: "kubeadmin", and password: "xxxxx-xxxxx-xxxxx-xxxxx"

Congratulations to me! I have successfully completed the installation.

Reference: