OpenShift4: vSphere + Static IP

There are many ways to install OCP4. One of the most common ask is how to install OCP4 with the static IP address on the vSphere environment. This is one of the use cases that I want to test out and hope I can share my lessons learned.

Environment:

vSphere 6.7 Update2
Run install from macOS Mojave 10.14.5

Requirements:

No DHCP server
Need to use static IP addresses

Problems I had:

Error #1: Dracut: FATAL: Sorry, ‘ip=dhcp’ does not make sense for multiple interface configurations.

Cause:

When I tried to overwrite the IP address by setting the kernel parameters using ip=<ip>::<gateway>:<net mask>:<FQDN>:<interface>:none with cloning from OVA.

Solution:

Setting the IP parameter before the initramfs is created from the rhcos-install.iso instead of from OVA.

Here are steps to create custom ISO with the parameters to simplify the process. You can use the downloaded ISO, but it will be a lot of typing, so the following steps are very useful when creating many VMs from the ISO.

Download the rhcos-4.1.0-x86_64-installer.iso and rhcos-4.1.0-x86_64-metal-bios.raw.gz from https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/
Extract the ISO to a temporary location:

sudo mount rhcos-410.8.20190425.1-installer.iso /mnt/
mkdir /tmp/rhcos
rsync -a /mnt/* /tmp/rhcos/
cd /tmp/rhcos
vi isolinux/isolinux.cfg

Modify the boot entry at the end of the file similar to this:

label linux
  menu label ^Install RHEL CoreOS
  kernel /images/vmlinuz
  append initrd=/images/initramfs.img nomodeset rd.neednet=1 coreos.inst=yes ip=192.168.1.124::192.168.1.1:255.255.255.0:bootstrap.ocp4.example.com:ens192:none nameserver=192.168.1.188 coreos.inst.install_dev=sda coreos.inst.image_url=http://192.168.1.231:8080/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.1.231:8080/static.ign

where:

ip=<ip address>::<gateway>:<netmask>:<hostname>:<interface>:none

nameserver=<DNS>

coreos.inst.image_url=http://<webserver host:port>/rhcos-4.1.0-x86_64-metal-bios.raw.gz

coreos.inst.ignition_url=http://<webserver host:port>/<master or worker ignition>.ign

Create new ISO as /tmp/rhcos_install.iso

sudo mkisofs -U -A "RHCOS-x86_64" -V "RHCOS-x86_64" -volset "RHCOS-x86_64" -J -joliet-long -r -v -T -x ./lost+found -o /tmp/rhcos_install.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img -no-emul-boot .

Update the custom ISO to the datastore for VM creation.

Error #2: No such host

no such host.png

Cause:

Most likely the network did not set up correctly when the master or worker start.

Solution:

In my case, this is an issue when creating masters/workers from an OVA and network configuration did not get setup when the RHCOS is booted.

Error #3: Getting EOF from LB

Cause:

Most likely the DNS and webserver configuration errors.

Solution:

Make sure all FQDN resolve to the correct IPs and restart related services.

Error #4: X509 cert error

Cause:

The reason in my case was the clocks on all servers were not synced and required to regenerate my SSH key.

Solution:

NTP was setup on DNS and webserver and make sure the clock is synced across. I also regenerate the SSH and update my install-config.yaml file.

Prerequisites:

DNS
Loadbalancer
Webserver
NTP
CoreOS Config Transpiler (filetranspiler): https://github.com/ashcrow/filetranspiler

The above components are required in my setup. I used the link [3] in the Reference section to setup DNS, load balancer, and webserver. I configured NTP on my DNS, webserver, load balancer and make sure I configure the time on my ESXi server as well. The filetranspiler is an awesome tool for manipulating the ignition files. I used it thought out the test here.

Preparing the infrastructure:

I started my installation with OCP 4 official documentation for vSphere (Reference [1] below).

SSH keygen

Captured my example steps here. Please use your own value.

ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/ocp4vsphere
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/ocp4vsphere

Download OpenShift 4 installer
- extract it
- chmod +x openshift-installer
- mv to /usr/local/bin directory

wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-install-mac-4.1.7.tar.gz

Download pull secret
1. Go to cloud.redhat.com
2. Login with your credential (create an account if you don’t have one)
3. Click “Create Cluster” –> “VMware vSphere”
4. Click on “Download Pull Secret” to download the secret
Create an installation directory
Create install-config.yaml
- https://docs.openshift.com/container-platform/4.1/installing/installing_vsphere/installing-vsphere.html#installation-vsphere-config-yaml_installing-vsphere
- backup your install-config.yaml
- copy install-config.yaml to the <installation directory>

apiVersion: v1
baseDomain: example.com 
compute:
- hyperthreading: Enabled   
  name: worker
  replicas: 0 
controlPlane:
  hyperthreading: Enabled   
  name: master
  replicas: 3 
metadata:
  name: ocp4
platform:
  vsphere:
    vcenter: <vCenter host>
    username: <administrator>
    password: <password>
    datacenter: dc
    defaultDatastore: datastore
pullSecret: '<your pull seceret>' 
sshKey: '<your public ssh key>'

Create ignition files

openshift-install create ignition-configs --dir=<installation_directory>

Prepare for creating bootstrap with hostname and the static IP

Download filetranspiler:

git clone https://github.com/ashcrow/filetranspiler

Copy <installation_directory>/bootstrap.ign to <filetranspile_directory>/

Create bootstrap hostname file:

echo "bootstrap.ocp4.example.com" > hostname

move hostname file to <filetranspile_directory>/bootstrap/etc/

Create ifcfg-ens192 file under

<filetranspile_directory>/bootstrap/etc/sysconfig/network-scripts with following content

NAME=ens192
DEVICE=ens192
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
IPADDR=<bootstrap IP address>
NETMASK=<netmask>
GATEWAY=<gateway>
DOMAIN=example.com
DNS1=<dns>
PREFIX=24
DEFROUTE=yes
IPV6INIT=no

Run this command to create new boostrap ignition file:

cd <filetranspile_directory>
./filetranspile -i bootstrap.ign -f bootstrap -o bootstrap-static.ign

Upload bootstrap-static.ign to the webserver:

scp bootstrap-static.ign user@<webserverip>:/var/www/html/bootstrap.ign

Create an append-bootstrap.ign. Example as shown below.

{
  "ignition": {
    "config": {
      "append": [
        {
          "source": "http://<webserverip:port>/bootstrap.ign", 
          "verification": {}
        }
      ]
    },
    "timeouts": {},
    "version": "2.1.0"
  },
  "networkd": {},
  "passwd": {},
  "storage": {},
  "systemd": {}
}

Encode the append-bootstrap.ign file.

openssl base64 -A -in append-bootstrap.ign -out append-bootstrap.64

Upload OVA to the vSphere datastore
- Download OVA from https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/rhcos-4.1.0-x86_64-vmware.ova
- Upload the OVA to datastore from vCenter
In the vSphere Client, create a template for the OVA image
- Use step #6 from https://docs.openshift.com/container-platform/4.1/installing/installing_vsphere/installing-vsphere.html#installation-vsphere-machines_installing-vsphere
Create a bootstrap VM using the steps here:
- Use step #7 from https://docs.openshift.com/container-platform/4.1/installing/installing_vsphere/installing-vsphere.html#installation-vsphere-machines_installing-vsphere
- Use content in append-bootstrap.64 for the value of guestinfo.ignition.config.data

Create custom ISO with master0 static IP by following Error#1 solution

Use adding the following parameters to create rhcos-install.iso for master0:

coreos.inst.install_dev=sda coreos.inst.image_url=http://192.168.1.231:8080/install/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.1.231:8080/master0.ign ip=192.168.1.120::192.168.1.1:255.255.255.0:master0.ocp4.example.com:ens192:none nameserver=192.168.1.188

Upload the custom rhcos-install.iso to webserver.

Create master0 ignition file using the same steps as creating bootstrap ignition file
- Create master0 hostname file under <filetranspile_directory>/bootstrap/etc/
- Create ifcfg-ens192 for master0 under <filetranspile_directory>/bootstrap/etc/sysconfig/network-scripts/ifcfg-ens192
- Create master0 ignition file with static ip info:
```
./filetranspile -i master.ign -f bootstrap -o master0-static.ign
```

- Upload master0-static.ign to the webserver:
```
scp master0-static.ign user@<webserverip>:/var/www/html/master0.ign
```
  - Note that master0.ign is used in the kernel parameter when installing the ISO.
- Create VM from the custom ISO
  - Create VM with 4 CPU and 16 RAM
  - Select the custom ISO
  - add “disk.EnableUUID”: Specify TRUE under VM Options and Edit Configuration.
  - Power on the VM
  - Go the VM console:
  - Hit <Tab>
  - you can modify the parameters per each server here.
  - Hit <enter>
  - The server will reboot after installation.
Repeat for all masters and workers.

Installation:

When you have all the VMs created, run the following command.

$ openshift-install --dir=ocp4 wait-for bootstrap-complete --log-level debug

DEBUG OpenShift Installer v4.1.7-201907171753-dirty 
DEBUG Built from commit 5175a461235612ac64d576aae09939764ac1845d 
INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp4.example.com:6443... 
INFO API v1.13.4+3a25c9b up                       
INFO Waiting up to 30m0s for bootstrapping to complete... 
DEBUG Bootstrap status: complete                  
INFO It is now safe to remove the bootstrap resources

Verification

$ export KUBECONFIG=ocp4/auth/kubeconfig
$ oc whoami

$ oc get nodes
NAME                       STATUS   ROLES    AGE     VERSION
master0.ocp4.example.com   Ready    master   35m     v1.13.4+205da2b4a
master1.ocp4.example.com   Ready    master   35m     v1.13.4+205da2b4a
master2.ocp4.example.com   Ready    master   35m     v1.13.4+205da2b4a
worker0.ocp4.example.com   Ready    worker   20m     v1.13.4+205da2b4a
worker1.ocp4.example.com   Ready    worker   11m     v1.13.4+205da2b4a
worker2.ocp4.example.com   Ready    worker   5m25s   v1.13.4+205da2b4a

Validate all CSR is approved

$ oc get csr

NAME        AGE     REQUESTOR                                                                   CONDITION
csr-6vqqn   35m     system:node:master1.ocp4.example.com                                        Approved,Issued
csr-7hlkk   20m     system:node:worker0.ocp4.example.com                                        Approved,Issued
csr-9p6sw   11m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-b4cst   35m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-gx4dz   5m33s   system:node:worker2.ocp4.example.com                                        Approved,Issued
csr-kqcfv   11m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-lh5zg   35m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-m2hvl   35m     system:node:master0.ocp4.example.com                                        Approved,Issued
csr-npb4l   35m     system:node:master2.ocp4.example.com                                        Approved,Issued
csr-rdpgm   20m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-s2d7z   11m     system:node:worker1.ocp4.example.com                                        Approved,Issued
csr-sx2r5   6m      system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-tvgbq   35m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-vvp2h   6m11s   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued

Patching the images registry for non-production environment

$oc project openshift-image-registry
$oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
config.imageregistry.operator.openshift.io/cluster patched

Next step?

To improve the process, we need to automate this.

Reference:

[1] OpenShift 4 Official Installation Documentation for vSphere

[2] http://Using Static IP for OCP4 Installation Guide

[3] Setting Up Pre-requisites Guide

2 thoughts on “OpenShift4: vSphere + Static IP”

shannachan says:

March 17, 2020 at 8:18 pm

Update: I cleaned up the steps and retested on OCP 4.3. The steps are recorded in my new post here: https://shanna-chan.blog/2020/03/16/openshift4-3-retest-static-ip-configuration-on-vsphere/.

LikeLike

Walid Shaari (@walidshaari) says:

June 27, 2020 at 7:30 pm

Hi, thank you for sharing, another cool way to create the coreos vms is using the coreos iso maker ansible playbooks found in https://github.com/chuckersjp/coreos-iso-maker

LikeLiked by 1 person

OpenShift4: vSphere + Static IP

Environment:

Requirements:

Problems I had:

Error #1: Dracut: FATAL: Sorry, ‘ip=dhcp’ does not make sense for multiple interface configurations.

Error #2: No such host

Error #3: Getting EOF from LB

Error #4: X509 cert error

Prerequisites:

Preparing the infrastructure:

Installation:

Verification

Next step?

Reference:

Published by

shannachan

2 thoughts on “OpenShift4: vSphere + Static IP”

Leave a comment Cancel reply

Environment:

Requirements:

Problems I had:

Error #1: Dracut: FATAL: Sorry, ‘ip=dhcp’ does not make sense for multiple interface configurations.

Error #2: No such host

Error #3: Getting EOF from LB

Error #4: X509 cert error

Prerequisites:

Preparing the infrastructure:

Installation:

Verification

Next step?

Reference:

Share this:

Related

Published by

shannachan

2 thoughts on “OpenShift4: vSphere + Static IP”

Leave a comment Cancel reply