One of the most frequently asked questions recently is how to install OpenShift on AWS with temporary credentials. The default OpenShift provisioning using AWS key and secret, which requires the Administrator privileges. The temporary credential often refers to AWS Security Token Service (STS), which allows end-users to assume an IAM role resulting in short-lived credentials.
Developers or platform teams will require approval from their security team to access the company AWS account. It can be challenging in some organizations to get access to Administrator privileges.
OpenShift 4.7 support for AWS Secure Token Service in manual mode is in Tech Preview. I decided to explore a little deeper—the exercise based on the information both on the OpenShift documentation and the upstream repos. I am recording the notes from my test run. I hope you will find it helpful.
OpenShift 4 version
OCP 4.7.9
Build sts-preflight binary
git clone https://github.com/sjenning/sts-preflight.git go get github.com/sjenning/sts-preflight cd <sts-preflight directory> go build .
Getting the AWS STS
As an AWS administrator, I found the sts-preflight tool helpful in this exercise. The documentation has the manual steps, but I choose to use the sts-preflight tool here.
- Create STS infrastructure in AWS:
./sts-preflight create --infra-name <sts infra name> --region <aws region> # ./sts-preflight create --infra-name sc-example --region us-west-1 2021/04/28 13:24:42 Generating RSA keypair 2021/04/28 13:24:56 Writing private key to _output/sa-signer 2021/04/28 13:24:56 Writing public key to _output/sa-signer.pub 2021/04/28 13:24:56 Copying signing key for use by installer 2021/04/28 13:24:56 Reading public key 2021/04/28 13:24:56 Writing JWKS to _output/keys.json 2021/04/28 13:24:57 Bucket sc-example-installer created 2021/04/28 13:24:57 OIDC discovery document at .well-known/openid-configuration updated 2021/04/28 13:24:57 JWKS at keys.json updated 2021/04/28 13:24:57 OIDC provider created arn:aws:iam::##########:oidc-provider/s3.us-west-1.amazonaws.com/sc-example-installer 2021/04/28 13:24:57 Role created arn:aws:iam::##########:role/sc-example-installer 2021/04/28 13:24:58 AdministratorAccess attached to Role sc-example-installer
- Create an OIDC token:
# ./sts-preflight token 2021/04/28 13:27:06 Token written to _output/token
- Get STS credential:
# ./sts-preflight assume Run these commands to use the STS credentials export AWS_ACCESS_KEY_ID=<temporary key> export AWS_SECRET_ACCESS_KEY=<temporary secret> export AWS_SESSION_TOKEN=<session token>
- The above short-lived key, secret, and token can be given to the person who are installing OpenShift.
- Export all the AWS environment variables before proceeding to installation.
Start the Installation
As a Developer or OpenShift Admin, you will get the temporary credentials information and export the AWS environment variables before installing the OCP cluster.
- Download OpenShift CLI (oc) and OpenShift installer:
- Extract the AWS Credentials Request objects from the release image:
# oc adm release extract quay.io/openshift-release-dev/ocp-release:4.7.9-x86_64 --credentials-requests --cloud=aws --to=./credreqs ; cat ./credreqs/*.yaml > credreqs.yaml
- Create install-config.yaml for installation:
# ./openshift-install create install-config --dir=./sc-sts ? SSH Public Key /root/.ssh/id_rsa.pub ? Platform aws INFO Credentials loaded from default AWS environment variables ? Region us-east-1 ? Base Domain sc.ocp4demo.live ? Cluster Name sc-sts ? Pull Secret [? for help] INFO Install-Config created in: sc-sts
- Make sure that we install the cluster in Manual mode:
# cd sc-sts # echo "credentialsMode: Manual" >> install-config.yaml
- Create install manifests:
# cd .. # ./openshift-install create manifests --dir=./sc-sts
- Using the sts-preflight tool to create AWS resources. Make sure you are in the sts-preflight directory:
#./sts-preflight create --infra-name sc-example --region us-west-1 --credentials-requests-to-roles ./credreqs.yaml 2021/04/28 13:45:34 Generating RSA keypair 2021/04/28 13:45:42 Writing private key to _output/sa-signer 2021/04/28 13:45:42 Writing public key to _output/sa-signer.pub 2021/04/28 13:45:42 Copying signing key for use by installer 2021/04/28 13:45:42 Reading public key 2021/04/28 13:45:42 Writing JWKS to _output/keys.json 2021/04/28 13:45:42 Bucket sc-example-installer already exists and is owned by us 2021/04/28 13:45:42 OIDC discovery document at .well-known/openid-configuration updated 2021/04/28 13:45:42 JWKS at keys.json updated 2021/04/28 13:45:43 Existing OIDC provider found arn:aws:iam::000000000000:oidc-provider/s3.us-west-1.amazonaws.com/sc-example-installer 2021/04/28 13:45:43 Existing Role found arn:aws:iam::000000000000:role/sc-example-installer 2021/04/28 13:45:43 AdministratorAccess attached to Role sc-example-installer 2021/04/28 13:45:43 Role arn:aws:iam::000000000000:role/sc-example-openshift-machine-api-aws-cloud-credentials created 2021/04/28 13:45:43 Saved credentials configuration to: _output/manifests/openshift-machine-api-aws-cloud-credentials-credentials.yaml 2021/04/28 13:45:43 Role arn:aws:iam::000000000000:role/sc-example-openshift-cloud-credential-operator-cloud-credential- created 2021/04/28 13:45:44 Saved credentials configuration to: _output/manifests/openshift-cloud-credential-operator-cloud-credential-operator-iam-ro-creds-credentials.yaml 2021/04/28 13:45:44 Role arn:aws:iam::000000000000:role/sc-example-openshift-image-registry-installer-cloud-credentials created 2021/04/28 13:45:44 Saved credentials configuration to: _output/manifests/openshift-image-registry-installer-cloud-credentials-credentials.yaml 2021/04/28 13:45:44 Role arn:aws:iam::000000000000:role/sc-example-openshift-ingress-operator-cloud-credentials created 2021/04/28 13:45:44 Saved credentials configuration to: _output/manifests/openshift-ingress-operator-cloud-credentials-credentials.yaml 2021/04/28 13:45:45 Role arn:aws:iam::000000000000:role/sc-example-openshift-cluster-csi-drivers-ebs-cloud-credentials created 2021/04/28 13:45:45 Saved credentials configuration to: _output/manifests/openshift-cluster-csi-drivers-ebs-cloud-credentials-credentials.yaml
- Copy the generated manifest files and tls directory from sts-preflight/_output directory to installation directory:
# cp sts-preflight/_output/manifests/* sc-scs/manifests/ # cp -a sts-preflight/_output/tls sc-scs/
- I ran both
./sts-preflight tokenand./sts-preflight assumeagain to make sure I have enough time to finish my installation - Export the AWS environment variables.
- I did not further restrict the role in my test.
- Start to provision a OCP cluster:
# ./openshift-install create cluster --log-level=debug --dir=./sc-sts ... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/mufg-sts/sc-sts-test/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.sc-sts-test.xx.live INFO Login to the console with user: "kubeadmin", and password: "xxxxxxxxxxx" DEBUG Time elapsed per stage: DEBUG Infrastructure: 7m28s DEBUG Bootstrap Complete: 11m6s DEBUG Bootstrap Destroy: 1m21s DEBUG Cluster Operators: 12m28s INFO Time elapsed: 32m38s #Cluster was created successfully.
- Verify the components are assuming the IAM roles:
# oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode [default] role_arn = arn:aws:iam::000000000000:role/sc-sts-test-openshift-image-registry-installer-cloud-credentials web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
- Adding and deleting worker node works as well:
Delete the Cluster
- Obtain a new temporary credential:
cd <sts-preflight directory> # ./sts-preflight token 2021/04/29 08:19:01 Token written to _output/token # ./sts-preflight assume Run these commands to use the STS credentials export AWS_ACCESS_KEY_ID=<temporary key> export AWS_SECRET_ACCESS_KEY=<temporary secret> export AWS_SESSION_TOKEN=<session token>
- Export all AWS environment variables using the result output from last step
- Delete the cluster:
# ./openshift-install destroy cluster --log-level=debug --dir=./sc-sts-test
DEBUG OpenShift Installer 4.7.9
DEBUG Built from commit fae650e24e7036b333b2b2d9dfb5a08a29cd07b1
INFO Credentials loaded from default AWS environment variables
DEBUG search for matching resources by tag in us-east-1 matching aws.Filter{"kubernetes.io/cluster/sc-sts-rj4pw":"owned"}
...
INFO Deleted id=vpc-0bbacb9858fe280f9
INFO Deleted id=dopt-071e7bf4cfcc86ad6
DEBUG search for matching resources by tag in us-east-1 matching aws.Filter{"kubernetes.io/cluster/sc-sts-test-rj4pw":"owned"}
DEBUG search for matching resources by tag in us-east-1 matching aws.Filter{"openshiftClusterID":"ab9baacf-a44f-47e8-8096-25df62c3b1dc"}
DEBUG no deletions from us-east-1, removing client
DEBUG search for IAM roles
DEBUG search for IAM users
DEBUG search for IAM instance profiles
DEBUG Search for and remove tags in us-east-1 matching kubernetes.io/cluster/sc-sts-test-rj4pw: shared
DEBUG No matches in us-east-1 for kubernetes.io/cluster/sc-sts-test-rj4pw: shared, removing client
DEBUG Purging asset "Metadata" from disk
DEBUG Purging asset "Master Ignition Customization Check" from disk
DEBUG Purging asset "Worker Ignition Customization Check" from disk
DEBUG Purging asset "Terraform Variables" from disk
DEBUG Purging asset "Kubeconfig Admin Client" from disk
DEBUG Purging asset "Kubeadmin Password" from disk
DEBUG Purging asset "Certificate (journal-gatewayd)" from disk
DEBUG Purging asset "Cluster" from disk
INFO Time elapsed: 4m39s
OpenShift Lessons Learned 