Installing OpenShift behind proxy

I have been wanting to write this blog to summarize the challenges that I had with proxy when installing OpenShift. The truth is that I don’t have the complete list of how to solve every problem in a proxy environment. I will try to list out what I did in my past and help you to avoid or debug the proxy related issues as much as I can.

Environment Variable

  • Whitelisting hosts that the platform will be accessing, for example:

If you are using subscription manager on a RHEL, you will need to whitelist the following hosts.

For RHSM/RHN (rpms):

For RH’s docker registry:

Example of other access that you may need:

    • index.docker.io
    • github.com
    • maven.org (Maven Central)
    • docker.io (dockerhub connection)
    • npmjs.org (node js build)
  • Setup /etc/profile.d/proxy.sh on all the nodes for your platform
#cat /etc/profile.d/proxy.sh
export http_proxy=http://host.name:port/
export https_proxy=http://host.name:port/
export no_proxy=.example.com,.svc

Note: “.svc” is needed with you want to install Service Catalog

Add Proxy Information into Ansible Hosts File

Here is the list of parameters for proxy environment

openshift_http_proxy=http://IPADDR:PORT
openshift_https_proxy=https://IPADDR:PORT
openshift_no_proxy='.example.com,some-internal-hosts.com'

Update Dockerfile with Proxy Information

After installing, the internal docker registry service IP will need to add to the docker configuration in /etc/sysconfig/docker for NO_PROXY parameter.

Getting the service IP of docker-register on OpenShift

oc get svc docker-registry -n default

Append the service IP to the NO_PROXY list in the file.

Testing Build and Pushing Images to Registry

After the installation, it is always good to test out the build and make sure image can be pushed into the internal registry.

Here is my check list if the build failed or image cannot be pushed.

  • Check if the hosts that you are trying to access are on the whitelist for your proxy
  • Check if the gitNoProxy is configured correctly under the BuildDefaults plug-in in the /etc/origin/master/master-config.yaml. For example, if you are access to an internal git repo location, please make sure they repository server is on the gitNoProxy list.
  • In 3.7, you will also need to add the kubernetes service IP to the NO_PROXY environment variable and redeploy the docker-register. Otherwise, you will get error when trying to push images to the internal docker register. See this link for more details: https://bugzilla.redhat.com/show_bug.cgi?id=1511870.

Note: to get the service IP for kubernetes: oc get svc kubernetes -n default

Hopefully, the checklist will help you to avoid any proxy related issue during installation.

Install OpenShift on Atomic Host on AWS

This blog is to share my experience on installing OpenShift 3.7 on Atomic Host. Since I am using OpenShift Container Platform (supported by Red Hat), there are 2 options for installation. They are the RPM install which is on Red Hat Enterprise Linux (RHEL) and containerized install which is using Atomic Host (Container OS). In version 3.7, installation can be done via a container on Atomic Host, or from an Linux bastion host.

My test used a Linux host to install OpenShift on Atomic host using AWS which is one of the ways to get an Atomic instance provisioned. Atomic host are provisioned via private AMI image for cloud provider account, the AMI image is ami-e9494989 for my test. Here is where you need to register to get access for importing the the private AMI here https://www.redhat.com/en/technologies/cloud-computing/cloud-access.

There are many ways to automate the steps for installation which is not my blog is about. I want to test out how easy or hard to installation OpenShift on a container OS, so I want to test all the steps for the installation manually.

Setting Up on a Cloud Provider

There are few things we need to setup on AWS. A wildcard entry and a public master hostname are required prior to the installation. I used Route53 for adding the A records for both of the requirements.

Per my test, I also had to add a tag to all Atomic instances with key as KubernetesCluster and the value of the key can be anything. The value of the KubernetesCluster key, will be used for parameter openshift_clusterid in the ansible inventory file. Without this tag on the Atomic instances, I will not be able to register the OpenShift node with the cloud provider.

Setting Up Bastion host

Bastion host is a Linux host (RHEL) to run automation scripts to prepare and install OpenShift on all Atomic hosts. This is one of the option to install on Atomic host. I like this option because I can reuse the same bastion host to install more that one cluster.

The step to prepare the bastion host is straight forward.

subscription-manager register
subscription-manager attach --pool=
subscription-manager repos --disable="*"
subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.7-rpms" \
    --enable="rhel-7-fast-datapath-rpms"
yum install atomic-openshift-utils -y

Preparation before installation.

Preparation steps are available at https://docs.openshift.com/container-platform/latest/install_config/install/host_preparation.html.

1. Generate SSH key on Bastion host via 'ssh-keygen' as root

2. Distribute the SSH key too all hosts (master and node) using the following command from bastion host:
   ssh-copy-id -i ~/.ssh/id_rsa.pub 

3. Create a hosts.prepocp file which include all the hostnames for the cluster. 
   Example is shown below.
   [nodes]
   ip-172-31-7-15.us-west-2.compute.internal
   ip-172-31-5-243.us-west-2.compute.internal

4. Create a ansible-play (openshiftprep.yml) to automate the host preparation. 
   An Example is: here https://github.com/piggyvenus/examples/blob/master/installAnsibleSample/v3.7/atomic/openshiftprep.yml

5. Execute the ansible playbook which will register, update Atomic host and configure docker on to the added device (/dev/xvdb)
   ansible-playbook -i hosts.prepocp openshiftprep.yml

Create Ansible Hosts file for OpenShift Advance Installation

Since we will need to create an inventory file (often refer to ansible hosts file) for OpenShift installation, here is an example of OpenShift Advance Installation for Atomic host on AWS: https://raw.githubusercontent.com/piggyvenus/examples/master/installAnsibleSample/v3.7/atomic/hosts.atomic.template

Download this file and update the corresponding information for installation. Then, save this file as /etc/ansible/hosts.

There are a few lesson learned here as for creating the ansible hosts file. I added the following parameters to get successful installation:

openshift_release=v3.7.23
openshift_image_tag=v3.7.23
openshift_pkg_version=-3.7.23
openshift_clusterid=<value of key KubernetesCluster from AWS Atomic instance>

Setting up Atomic Host For OCP Installation

I learned that there are few extra steps which I had to add to prepare the Atomic installation before OpenShift Installation. These are the steps that I used in my test. I am not an Atomic expert. It does what I wanted it to do.

Besides adding disk for docker, I also need extra disk space for root partition. The OOTB Atomic instance from AWS has only 3GB root partition which is not enough for OpenShift installer. I have to do the following to get my docker and root partition configure to the way I wanted it. My goal is to extend my root partition to have extra disk space and configure docker using the added volume that I attached to the instance.

The preparation script did configure docker to use /dev/xvdb. After running the previous ansible playbook, the following steps were to use to extend my root partition.

ansible all -m shell -a "lvextend -L+50G /dev/mapper/atomicos-root"
ansible all -m shell -a "xfs_growfs /"
ansible all -m shell -a "df -h"

Next is to reboot all hosts via the following command.

systemctl reboot

The following step is to configure docker and startup docker after all hosts were rebooted from the bastion host.

1. Download this ansible playbook 
   https://raw.githubusercontent.com/piggyvenus/examples/master/installAnsibleSample/v3.7/atomic/openshiftprep2.yml
2. Run ansible playbook using the same hosts.prepocp file as shown below. 
   ansible-playbook -i hosts.prepocp openshiftprep2.yml

OpenShift Containerized Installation

Once the ansible host (/etc/ansible/hosts) is updated, installation can be started by executing the following command.

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

note: if the inventory file is /etc/ansible/hosts, no need to specify with "-i" option.

If there is no error after this step, you can access the OpenShift console via https://<your-public-master-hostname&gt;:8443/ and login as any username and password.

Setting up Persistence for Registry for Non-Production

There are many options to setup the persistence for Registry on AWS. Since I only have 1 registry for the single master cluster, I decided to use what gp2 which is configured as default storageclass after the installation. Here are the steps I setup storage for OpenShift insternal registry. I used ReadWriteOnce as access mode because AWSElasticBlockStore volume plugin only support ReadWriteOnce (https://kubernetes.io/docs/concepts/storage/persistent-volumes/)

1. ssh to the Atomic master host
2. /usr/local/bin/oc login -u system:admin
3. oc project default
4. run the following:
oc create -f - <<EOF
{
  "apiVersion": "v1",
  "kind": "PersistentVolumeClaim",
  "metadata": {
 "name": "registry-volume-claim",
 "labels": {
   "deploymentconfig": "docker-registry"
 }
  },
  "spec": {
 "accessModes": [ "ReadWriteOnce" ],
 "resources": {
   "requests": {
     "storage": "20Gi"
   }
        }
   }
}
EOF

2. oc volume deploymentconfigs/docker-registry --add --name=registry-storage -t pvc  --claim-name=registry-volume-claim --overwrite

Setting up Metrics with Dynamic storage

Since the dynamic provisioning is configured, I used the default gp2 storageclass to configure metics as well. Here are the steps.

1. Add following in /etc/ansible/hosts file on bastion host
openshift_metrics_install_metrics=true
openshift_metrics_hawkular_hostname=hawkular-metrics.<your wildcard suffix>
openshift_metrics_cassandra_storage_type=dynamic
openshift_metrics_image_version=v3.7.23

2. Run the metrics playbook to setup metrics from bastion host
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml 

If the playbook failed, simply uninstall metrics component by setting openshift_metrics_install_metrics=false and re-run the metric playbook.

Setting up Logging with Dynamic Storage

Logging can be configured via ansible playbook as well. I am using the default gp2 storageclass since it provides dynamic provision for the Persistence Volume.  Here are the steps.

1. Add following in /etc/ansible/hosts file on bastion host
openshift_logging_install_logging=true
openshift_logging_image_version=v3.7.23
openshift_logging_es_pvc_dynamic=true
openshift_logging_es_pvc_size=30Gi
openshift_logging_es_cluster_size=1
openshift_logging_es_memory_limit=1Gi

2.Run the logging playbook to setup logging from bastion host
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml

If the playbook failed, simply uninstall logging component by setting openshift_logging_install_logging=false and re-run the logging playbook.

Installing from a Container on an Atomic Host Option

Instead of using a bastion host to execute ansible playbook to install on an Atomic host. You can execute the following command to install OpenShift from a container. Here are the steps I tested.

1.  atomic install --system \
> --storage=ostree \
> --set INVENTORY_FILE=/root/hosts \
registry.access.redhat.com/openshift3/ose-ansible:v3.7
Getting image source signatures
Copying blob sha256:9cadd93b16ff2a0c51ac967ea2abfadfac50cfa3af8b5bf983d89b8f8647f3e4
 71.41 MB / ? [----------------------------------=-------------------------] 7s 
Copying blob sha256:4aa565ad8b7a87248163ce7dba1dd3894821aac97e846b932ff6b8ef9a8a508a
 1.21 KB / ? [=------------------------------------------------------------] 0s 
Copying blob sha256:7952714329657fa2bb63bbd6dddf27fcf717186a9613b7fab22aeb7f7831b08a
 146.93 MB / ? [---------------------------------------------=------------] 16s 
Copying config sha256:45abc081093b825a638ec53a19991af0612e96e099554bbdfa88b341cdfcd2e6
 4.23 KB / 4.23 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
Extracting to /var/lib/containers/atomic/ose-ansible-v3.7.0
systemctl daemon-reload
systemd-tmpfiles --create /etc/tmpfiles.d/ose-ansible-v3.7.conf
systemctl enable ose-ansible-v3.7

2. systemctl start ose-ansible-v3.7
3. journalctl -xfu ose-ansible-v3.7

Hope this will help someone to have a successful OpenShift containerized installation.

Installing OCP 3.7 on Azure

My goal of this blog is to share my experience on installing OCP 3.7 on Azure. With the information provided here, hope I can save you some time and have a successful installation on Azure. The exercise was to install OCP on Azure and enable dynamic provisioning as well. Here is the list that you may want to pay more attention when you plan your installation.

Use Azure instance name as the value of openshift_hostname in your inventory file

Assuming you are using OpenShift advanced installation on Azure, we will need to configure openshift_hostname in the inventory file to match the instance name in your Azure portal. The azure instance name is the name of your virtual machine.

Host Configuration

When creating virtual machines in your resource group, it will depends if you are using Azure domain or custom domain of yours. If you are not using the Azure domain, you can set your domain on you VM via the following command:

hostnamectl set-hostname <FQDN of the host>

Setup NetworkManager

Once the ansible inventory file was created, execute the following steps to make sure the Network Manager is working properly.

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/openshift-node/network_manager.yml

Configure resolv.conf on all hosts

export DOMAIN=`domainname -d`

ansible all -m shell -a "systemctl restart NetworkManager"

ansible all -m shell -a "nmcli con modify eth0 ipv4.dns-search $DOMAIN"

ansible all -m shell -a "systemctl restart NetworkManager"

Create service principal

This is a required step to setup Azure dynamic provision storage. There are few option for creating service principal. An example is shown below.  Please see https://github.com/microsoft/openshift-container-platform for more details around it.

az ad sp create-for-rbac -n <friendly name> --password <password> --role contributor --scopes /subscriptions/<subscription_id>

Location in azure.conf is not optional for managed and unmanaged disk

If you are using managed disk option, you would want to make sure location is also in your azure.conf.

Here is the example of /etc/azure/azure.conf.

tenantId: xxxxxxxxxx
subscriptionId: xxxxxxxxxxxxxx
aadClientId: xxxxxxxxxx
aadClientSecret: xxxxx
aadTenantId: xxxxxxxxxx
resourceGroup: xxxx
cloud: AzurePublicCloud
location: westus

Did not need to remove the node

The registration is automated. Once the installation completed, adding /etc/azure/azure.conf on all nodes

Update the master-config.yml with the following information:

kubernetesMasterConfig:
  apiServerArguments:
    cloud-config:
    - /etc/azure/azure.conf
    cloud-provider:
    - azure
    runtime-config:
    - apis/settings.k8s.io/v1alpha1=true
    storage-backend:
    - etcd3
    storage-media-type:
    - application/vnd.kubernetes.protobuf
  controllerArguments:
    cloud-provider:
    - azure
    cloud-config:
    - /etc/azure/azure.conf

restart atomic-openshift-master-api and atomic-openshift-master-controllers on the master via following commands:

systemctl restart atomic-openshift-master-api

systemctl restart atomic-openshift-master-controllers

Update all node-config.yaml as below

kubeletArguments:
  cloud-config:
  - /etc/azure/azure.conf
  cloud-provider:
  - azure

restart atomic-openshift-node on the nodes via following command

systemctl restart atomic-openshift-node

While you are starting up the atomic-openshift node, you can watch the log by running on the node

journalctl -u atomic-openshift-node -f

You should see the node get remove and re-register back with the cloud provider. There is no need to delete the node. If  the below command whiling restart the atomic-node-service, you will see the specific node get removed and added back to the cluster after sometime.

oc get nodes -w

Creating the correct storage type for your environment

You can validate your disk option before creating storageclass. If the type of storageclass does not match, the pod will throw error as it start to mount to the volume.

At the VM creation, the managed and unmanaged disk options are available.  I learned that we will need location string in the /etc/azure/azure.conf for managed disk option. It looks like RWX is not an option for Azure disk volume plug-in. There are the storageclass for unmanaged disk (top) and managed disk (bottom) I tried.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azure-storageclass
annotations:
  storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/azure-disk
parameters:
  storageAccount: xxxxxxxx

 

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow2
provisioner: kubernetes.io/azure-disk
parameters:
  storageaccounttype: Standard_LRS  
  kind: Managed
  location: westus

 To set the storageclass as default

oc annotate storageclass azure-storageclass storageclass.beta.kubernetes.io/is-default-class="true"

Now, you should be able to install an OpenShift cluster on Azure with Azure storage as default storageclass.