1. Activate the ROSA with HCP product at the AWS console page by clicking the Get started button:

   

   If you have activated ROSA before but did not complete the process, you can click the button and complete the account linking as described in the following steps.

2. Confirm that you want your contact information to be shared with Red Hat and enable the service:

   
   • You will not be charged by enabling the service in this step. The connection is made for billing and metering that will take place only after you deploy your first cluster. This could take a few minutes.

3. After the process is completed, you will see a confirmation:

   

4. Other sections on this verification page show the status of additional prerequisites. In case any of these prerequisites are not met, a respective message is shown. Here is an example of insufficient quotas in the selected region:

   
   1. Click the Increase service quotas button or use the Learn more link to get more information about the about how to manage service quotas. In the case of insufficient quotas, note that quotas are region-specific. You can use the region switcher in the upper right corner of the web console to re-run the quota check for any region you are interested in and then submit service quota increase requests as needed.

5. If all the prerequisites are met, the page will look like this:

   

   The ELB service-linked role is created for you automatically. You can click any of the small Info blue links to get contextual help and resources.

1. Click the orange Continue to Red Hat button to proceed with account linking:

   

2. If you are not already logged in to your Red Hat account in your current browser’s session, you will be asked to log in to your account:

   
   • You can also register for a new Red Hat account or reset your password on this page.
   • Make sure to log into the Red Hat account that you plan to associate with the AWS account where you have activated ROSA with HCP in previous steps.
   • Only a single AWS account that will be used for service billing can be associated with a Red Hat account. Typically an organizational AWS account that has other AWS accounts, such as developer accounts, linked would be the one that is to be billed, rather than individual AWS end user accounts.
   • Red Hat accounts belonging to the same Red Hat organization will be linked with the same AWS account. Therefore, you can manage who has access to creating ROSA with HCP clusters on the Red Hat organization account level.

3. Complete the Red Hat account linking after reviewing the terms and conditions:

   Note

   This step is available only if the logged-in Red Hat account, or the Red Hat organization managing the Red Hat account, was not linked to an AWS account before.

   

   Both the Red Hat and AWS account numbers are shown on this screen.

4. Click the Connect accounts button if you agree with the service terms.

   If this is the first time you are using the Red Hat Hybrid Cloud Console, you will be asked to agree with the general managed services terms and conditions before being able to create the first ROSA cluster:

   

   Additional terms that need to be reviewed and accepted will be shown after clicking the View Terms and Conditions button:

   

   Submit your agreement once you have reviewed any additional terms when prompted at this time.

5. The Hybrid Cloud Console provides a confirmation that AWS prerequisites were completed and lists the first steps needed for cluster deployment:

   

6. The following steps pertain to technical deployment of the cluster:

   
   • It is possible that these steps will be performed on a different machine than where the service enablement and account linking were completed.

   • As mentioned previously, any Red Hat account belonging to the Red Hat organization that was linked with the AWS account that activated the ROSA service will have access to creating a cluster and will be able to select the billing AWS account that was linked under this Red Hat organization previously.

     The last section of this page shows cluster deployment options, either using the rosa CLI or through the web console:

     
   • The following steps describe cluster deployment using the rosa CLI.
   • If you are interested in deployment using the web console only, you can skip to the ROSA with HCP cluster deployment using the web console section. However, note that the rosa CLI is required for certain tasks, such as creating the account roles. If you are deploying ROSA for the first time, follow this the CLI steps until running the rosa whoami command, before skipping to the web console deployment steps.

1. Click the Download the ROSA CLI button to download the ROSA command line interface (CLI) for your operating system and set it up as described in the Help with ROSA CLI setup.

   Important

   Make sure that you have the most recent AWS CLI installed. See Instructions to install the AWS CLI for more information.

2. After the previous steps are completed, you can verify that both CLI are available by running the rosa version. This command shows an update notification if you are using an older version and aws –version commands in your terminal.

3. The prerequisite for creating a ROSA with HCP cluster is to log in using the rosa cli by the personalized command with your unique token shown under step 2.1. To authenticate, run this command on the web console. Use the copy button for easy copy and pasting of the command with full token into your terminal:

   

   Do not share your unique token.

4. The final prerequisite before your first cluster deployment is making sure the necessary account-wide roles and policies are created. The rosa CLI can help with that by using the command shown under step 2.2. To create the necessary account-wide roles and policies quickly… on the web console. The alternative to that is manual creation of these roles and policies.

5. After logging in, creating the account roles, and verifying your identity using the rosa whoami command, your terminal will look similar to this:

   

6. Initiate the cluster deployment using the presented command. You can click the copy button again and paste the command in your terminal:

   
7. To use a custom AWS profile, one of the non-default profiles specified in your ~/.aws/credentials, you can add the –profile <profile_name> selector to the rosa create cluster command so that the command looks like rosa create cluster –profile stage. If no AWS CLI profile is specified using this option, the default AWS CLI profile will determine the AWS infrastructure profile into which the cluster is deployed. The billing AWS profile is selected in one of the following steps.

8. After entering a cluster name, you will be asked whether to use the hosted control plane. Select yes:

   

9. When deploying a ROSA with HCP cluster, the billing AWS account needs to be specified:

   
   • Only AWS accounts that were linked to the Red Hat organization that is currently used will be shown.
   • The specified AWS account will be charged for using the ROSA service, regardless of whether the infrastructure AWS account is linked to it in the same AWS organization.
   • You can see an indicator of whether the ROSA contract is enabled for a given AWS billing account or not.
   • To select an AWS account that does not have the contract enabled, refer to the first few steps in this tutorial to enable the contract and allow the service charging, which is required for a successful cluster deployment.

10. In the following steps, you will specify technical details of the cluster that is to be deployed:

    
    • These steps are beyond the scope of this tutorial. See Creating ROSA with HCP clusters using the default options for more details about how to complete the ROSA with HCP cluster deployment using the CLI.

1. A cluster can be created using the web console by selecting the second option in the bottom section of the introductory Set up ROSA page:

   

2. The first step when creating a ROSA cluster using the web console is the control plane selection. Make sure the Hosted option is selected before clicking the Next button:

   

3. The next step Accounts and roles allows you specifying the infrastructure AWS account, into which the the ROSA cluster will be deployed and where the resources will be consumed and managed:

   
   • Click the How to associate a new AWS account, if you don not see the account into which you want to deploy the ROSA cluster for detailed information on how to create or link account roles for this association.
   • The rosa CLI is used for this.
   • If you are using multiple AWS accounts and have their profiles configured for the AWS CLI, you can use the --profile selector to specify the AWS profile when working with the rosa CLI commands.

4. The billing AWS account is selected in the immediately following section:

   
   • Only AWS accounts that were linked to the Red Hat organization that is currently used will be shown.
   • The specified AWS account will be charged for using the ROSA service, regardless of whether the infrastructure AWS account is linked to it in the same AWS organization.
   • You can see an indicator whether the ROSA contract is enabled for a given AWS billing account or not.
   • In case you would like to use an AWS account that does not have a contract enabled yet, you can either use the Connect ROSA to a new AWS billing account to reach the ROSA AWS console page, where you can activate it after logging in using the respective AWS account by following steps described earlier in this tutorial, or ask the administrator of the AWS account to do that for you.

1. To use the script, run the following commands in a bash terminal (the -p option defines a prefix for the roles):

   $ mkdir scratch
   $ cd scratch
   $ cat << 'EOF' > verify-permissions.sh
   #!/bin/bash
   while getopts 'p:' OPTION; do
     case "$OPTION" in
       p)
         PREFIX="$OPTARG"
         ;;
       ?)
         echo "script usage: $(basename \$0) [-p PREFIX]" >&2
         exit 1
         ;;
     esac
   done
   shift "$(($OPTIND -1))"
   rosa create account-roles --mode manual --prefix $PREFIX
   INSTALLER_POLICY=$(cat sts_installer_permission_policy.json | jq )
   CONTROL_PLANE_POLICY=$(cat sts_instance_controlplane_permission_policy.json | jq)
   WORKER_POLICY=$(cat sts_instance_worker_permission_policy.json | jq)
   SUPPORT_POLICY=$(cat sts_support_permission_policy.json | jq)
   simulatePolicy () {
       outputFile="${2}.results"
       echo $2
       aws iam simulate-custom-policy --policy-input-list "$1" --action-names $(jq '.Statement | map(select(.Effect == "Allow"))[].Action | if type == "string" then . else .[] end' "$2" -r) --output text > $outputFile
   }
   simulatePolicy "$INSTALLER_POLICY" "sts_installer_permission_policy.json"
   simulatePolicy "$CONTROL_PLANE_POLICY" "sts_instance_controlplane_permission_policy.json"
   simulatePolicy "$WORKER_POLICY" "sts_instance_worker_permission_policy.json"
   simulatePolicy "$SUPPORT_POLICY" "sts_support_permission_policy.json"
   EOF
   $ chmod +x verify-permissions.sh
   $ ./verify-permissions.sh -p SimPolTest

2. After the script completes, review each results file to ensure that none of the required API calls are blocked:

   $ for file in $(ls *.results); do echo $file; cat $file; done

   The output will look similar to the following:

   sts_installer_permission_policy.json.results
   EVALUATIONRESULTS       autoscaling:DescribeAutoScalingGroups   allowed *
   MATCHEDSTATEMENTS       PolicyInputList.1       IAM Policy
   ENDPOSITION     6       195
   STARTPOSITION   17      3
   EVALUATIONRESULTS       ec2:AllocateAddress     allowed *
   MATCHEDSTATEMENTS       PolicyInputList.1       IAM Policy
   ENDPOSITION     6       195
   STARTPOSITION   17      3
   EVALUATIONRESULTS       ec2:AssociateAddress    allowed *
   MATCHEDSTATEMENTS       PolicyInputList.1       IAM Policy
   ...

   Note

   If any actions are blocked, review the error provided by AWS and consult with your Administrator to determine if SCPs are blocking the required API calls.

1. Configure the following environment variables, changing the cluster name to suit your cluster:

   Note

   You must be logged in as an administrator.

   $ export ROSA_CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
   $ export REGION=$(rosa describe cluster -c ${ROSA_CLUSTER_NAME} --output json | jq -r .region.id)
   $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
   $ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
   $ export AWS_PAGER=""
   $ export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/clf-cloudwatch-sts"
   $ mkdir -p ${SCRATCH}

2. Ensure all fields output correctly before moving to the next section:

   $ echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

1. Create an Identity Access Management (IAM) policy for logging:

   $ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaCloudWatch'].{ARN:Arn}" --output text)
   $ if [[ -z "${POLICY_ARN}" ]]; then
   cat << EOF > ${SCRATCH}/policy.json
   {
     "Version": "2012-10-17",
     "Statement": [
        {
              "Effect": "Allow",
              "Action": [
                 "logs:CreateLogGroup",
                 "logs:CreateLogStream",
                 "logs:DescribeLogGroups",
                 "logs:DescribeLogStreams",
                 "logs:PutLogEvents",
                 "logs:PutRetentionPolicy"
              ],
              "Resource": "arn:aws:logs:*:*:*"
        }
   ]
   }
   EOF
   POLICY_ARN=$(aws iam create-policy --policy-name "RosaCloudWatch" \
   --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn --output text)
   fi
   $ echo ${POLICY_ARN}

2. Create an IAM role trust policy for the cluster:

   $ cat <<EOF > ${SCRATCH}/trust-policy.json
   {
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "${OIDC_ENDPOINT}:sub": "system:serviceaccount:openshift-logging:logcollector"
          }
        }
      }]
   }
   EOF
   $ ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
         --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
         --query Role.Arn --output text)
   $ echo ${ROLE_ARN}

3. Attach the IAM policy to the IAM role:

   $ aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
      --policy-arn ${POLICY_ARN}

1. Deploy the Red Hat OpenShift Logging Operator:

   $  cat << EOF | oc apply -f -
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        labels:
         operators.coreos.com/cluster-logging.openshift-logging: ""
        name: cluster-logging
        namespace: openshift-logging
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: cluster-logging
        source: redhat-operators
        sourceNamespace: openshift-marketplace
   EOF

2. Create a secret:

   $  cat << EOF | oc apply -f -
     apiVersion: v1
     kind: Secret
     metadata:
       name: cloudwatch-credentials
       namespace: openshift-logging
     stringData:
       role_arn: $ROLE_ARN
   EOF

1. Create a ClusterLogForwarder custom resource (CR):

   $ cat << EOF | oc apply -f -
     apiVersion: "logging.openshift.io/v1"
     kind: ClusterLogForwarder
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       outputs:
         - name: cw
           type: cloudwatch
           cloudwatch:
             groupBy: namespaceName
             groupPrefix: rosa-${ROSA_CLUSTER_NAME}
             region: ${REGION}
           secret:
             name: cloudwatch-credentials
       pipelines:
         - name: to-cloudwatch
           inputRefs:
             - infrastructure
             - audit
             - application
           outputRefs:
             - cw
   EOF

2. Create a ClusterLogging CR:

   $ cat << EOF | oc apply -f -
     apiVersion: logging.openshift.io/v1
     kind: ClusterLogging
     metadata:
       name: instance
       namespace: openshift-logging
     spec:
       collection:
         logs:
            type: vector
       managementState: Managed
   EOF

1. Delete the ClusterLogForwarder CR:

   $ oc delete -n openshift-logging clusterlogforwarder instance

2. Delete the ClusterLogging CR:

   $ oc delete -n openshift-logging clusterlogging instance

3. Detach the IAM policy to the IAM role:

   $ aws iam detach-role-policy --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch" \
      --policy-arn "${POLICY_ARN}"

4. Delete the IAM role:

   $ aws iam delete-role --role-name "${ROSA_CLUSTER_NAME}-RosaCloudWatch"

5. Delete the IAM policy:

   Important

   Only delete the IAM policy if there are no other resources using the policy.

   $ aws iam delete-policy --policy-arn "${POLICY_ARN}"

6. Delete the CloudWatch log groups:

   $ aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.audit"
   $ aws logs delete-log-group --log-group-name "rosa-${ROSA_CLUSTER_NAME}.infrastructure"

Procedure

1. Create a new project

   $ oc new-project waf-demo

2. Create a new TLS secret from a private key and a public certificate, where fullchain.pem is your full wildcard certificate chain (including any intermediaries) and privkey.pem is your wildcard certificate’s private key.

   Example

   $ oc -n waf-demo create secret tls waf-tls --cert=fullchain.pem --key=privkey.pem

3. Create a new CustomDomain custom resource (CR):

   Example waf-custom-domain.yaml

   apiVersion: managed.openshift.io/v1alpha1
   kind: CustomDomain
   metadata:
     name: cloudfront-waf
   spec:
     domain: apps.<company_name>.io 1
     scope: External
     loadBalancerType: NLB
     certificate:
       name: waf-tls
       namespace: waf-demo
     routeSelector: 2
       matchLabels:
        route: waf

   1

   The custom domain.

   2

   Filters the set of routes serviced by the CustomDomain ingress. In this tutorial, we will use the waf route selector, but if no value was to be provided, no filtering would occur.

4. Apply the CR:

   Example

   $ oc apply -f waf-custom-domain.yaml

5. Verify that your custom domain ingress controller has been deployed and is Ready:

   $ oc get customdomains

   Example output

   NAME               ENDPOINT                                                    DOMAIN                       STATUS
   cloudfront-waf     xxrywp.<company_name>.cluster-01.opln.s1.openshiftapps.com  *.apps.<company_name>.io     Ready

1. Retrieve the newly created custom domain ingress controller’s NLB hostname:

   $ NLB=$(oc -n openshift-ingress get service router-cloudfront-waf \
     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   $ echo "Origin domain: ${NLB}"

2. Import your certificate into Amazon Certificate Manager, where cert.pem is your wildcard certificate, fullchain.pem is your wildcard certificate’s chain and privkey.pem is your wildcard certificate’s private key.

   Note

   Regardless of what region your cluster is deployed, you must import this certificate to us-east-1 as Amazon CloudFront is a global AWS service.

   Example

   $ aws acm import-certificate --certificate file://cert.pem \
     --certificate-chain file://fullchain.pem \
     --private-key file://privkey.pem \
     --region us-east-1

3. Log into the AWS console to create a CloudFront distribution.

4. Configure the CloudFront distribution by using the following information:

   Note

   If an option is not specified in the table below, leave them the default (which may be blank).

   Option	Value
   Origin domain	Output from the command above [1]
   Name	rosa-waf-ingress [2]
   Viewer protocol policy	Redirect HTTP to HTTPS
   Allowed HTTP methods	GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
   Cache policy	CachingDisabled
   Origin request policy	AllViewer
   Web Application Firewall (WAF)	Enable security protections
   Use existing WAF configuration	true
   Choose a web ACL	cloudfront-waf
   Alternate domain name (CNAME)	*.apps.<company_name>.io [3]
   Custom SSL certificate	Select the certificate you imported from the step above [4]

   1. Run echo ${NLB} to get the origin domain.
   2. If you have multiple clusters, ensure the origin name is unique.
   3. This should match the wildcard domain you used to create the custom domain ingress controller.
   4. This should match the alternate domain name entered above.

5. Retrieve the Amazon CloudFront Distribution endpoint:

   $ aws cloudfront list-distributions --query "DistributionList.Items[?Origins.Items[?DomainName=='${NLB}']].DomainName" --output text

6. Update the DNS of your custom wildcard domain with a CNAME to the Amazon CloudFront Distribution endpoint from the step above.

   Example

   *.apps.<company_name>.io CNAME d1b2c3d4e5f6g7.cloudfront.net

1. Deploy a hello world application:

   $ oc -n waf-demo new-app --image=docker.io/openshift/hello-openshift

2. Create a route for the application specifying your custom domain name:

   Example

   $ oc -n waf-demo create route edge --service=hello-openshift hello-openshift-tls \
   --hostname hello-openshift.apps.<company_name>.io

3. Label the route to admit it to your custom domain ingress controller:

   $ oc -n waf-demo label route.route.openshift.io/hello-openshift-tls route=waf

1. Test that the app is accessible behind Amazon CloudFront:

   Example

   $ curl "https://hello-openshift.apps.<company_name>.io"

   Example output

   Hello OpenShift!

2. Test that the WAF denies a bad request:

   Example

   $ curl -X POST "https://hello-openshift.apps.<company_name>.io" \
     -F "user='<script><alert>Hello></alert></script>'"

   Example output

   <html>
   <head><title>403 Forbidden</title></head>
   <body>
   <center><h1>403 Forbidden</h1></center>
   </body>
   </html

   The expected result is a 403 Forbidden error, which means the AWS WAF is protecting your application.

1. Create an AWS IAM policy for the AWS Load Balancer Controller:

   Note

   The policy is sourced from the upstream AWS Load Balancer Controller policy plus permission to create tags on subnets. This is required by the operator to function.

   $ oc new-project aws-load-balancer-operator
   $ POLICY_ARN=$(aws iam list-policies --query \
        "Policies[?PolicyName=='aws-load-balancer-operator-policy'].{ARN:Arn}" \
        --output text)
   $ if [[ -z "${POLICY_ARN}" ]]; then
       wget -O "${SCRATCH}/load-balancer-operator-policy.json" \
          https://raw.githubusercontent.com/rh-mobb/documentation/main/content/docs/rosa/aws-load-balancer-operator/load-balancer-operator-policy.json
        POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
        --output text iam create-policy \
        --policy-name aws-load-balancer-operator-policy \
        --policy-document "file://${SCRATCH}/load-balancer-operator-policy.json")
   fi
   $ echo $POLICY_ARN

2. Create an AWS IAM trust policy for AWS Load Balancer Operator:

   $ cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Condition": {
      "StringEquals" : {
        "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager", "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"]
      }
    },
    "Principal": {
      "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
   }
   EOF

3. Create an AWS IAM role for the AWS Load Balancer Operator:

   $ ROLE_ARN=$(aws iam create-role --role-name "${CLUSTER_NAME}-alb-operator" \
      --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
      --query Role.Arn --output text)
   $ echo $ROLE_ARN
   
   $ aws iam attach-role-policy --role-name "${CLUSTER_NAME}-alb-operator" \
        --policy-arn $POLICY_ARN

4. Create a secret for the AWS Load Balancer Operator to assume our newly created AWS IAM role:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   stringData:
     credentials: |
       [default]
       role_arn = $ROLE_ARN
       web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
   EOF

5. Install the Red Hat AWS Load Balancer Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     channel: stable-v1.0
     installPlanApproval: Automatic
     name: aws-load-balancer-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: aws-load-balancer-operator.v1.0.0
   EOF

6. Deploy an instance of the AWS Load Balancer Controller using the operator:

   Note

   If you get an error here wait a minute and try again, it means the Operator has not completed installing yet.

   $ cat << EOF | oc apply -f -
   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController
   metadata:
     name: cluster
   spec:
     credentials:
       name: aws-load-balancer-operator
     enabledAddons:
       - AWSWAFv2
   EOF

7. Check the that the operator and controller pods are both running:

   $ oc -n aws-load-balancer-operator get pods

   You should see the following, if not wait a moment and retry:

   NAME                                                             READY   STATUS    RESTARTS   AGE
   aws-load-balancer-controller-cluster-6ddf658785-pdp5d            1/1     Running   0          99s
   aws-load-balancer-operator-controller-manager-577d9ffcb9-w6zqn   2/2     Running   0          2m4s

1. Create a new project for our sample application:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

3. Convert the pre-created service resource to a NodePort service type:

   $ oc -n hello-world patch service hello-openshift -p '{"spec":{"type":"NodePort"}}'

4. Deploy an AWS ALB using the AWS Load Balancer Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hello-openshift-alb
     namespace: hello-world
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
   spec:
     ingressClassName: alb
     rules:
       - http:
           paths:
             - path: /
               pathType: Exact
               backend:
                 service:
                   name: hello-openshift
                   port:
                     number: 8080
   EOF

5. Curl the AWS ALB Ingress endpoint to verify the hello world application is accessible:

   Note

   AWS ALB provisioning takes a few minutes. If you receive an error that says curl: (6) Could not resolve host, please wait and try again.

   $ INGRESS=$(oc -n hello-world get ingress hello-openshift-alb -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   $ curl "http://${INGRESS}"

   Example output

   Hello OpenShift!

1. Create an IAM Policy to allow for S3 Access:

   $ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaOadpVer1'].{ARN:Arn}" --output text)
   if [[ -z "${POLICY_ARN}" ]]; then
   $ cat << EOF > ${SCRATCH}/policy.json
   {
   "Version": "2012-10-17",
   "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:DeleteBucket",
        "s3:PutBucketTagging",
        "s3:GetBucketTagging",
        "s3:PutEncryptionConfiguration",
        "s3:GetEncryptionConfiguration",
        "s3:PutLifecycleConfiguration",
        "s3:GetLifecycleConfiguration",
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucketMultipartUploads",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts",
        "ec2:DescribeSnapshots",
        "ec2:DescribeVolumes",
        "ec2:DescribeVolumeAttribute",
        "ec2:DescribeVolumesModifications",
        "ec2:DescribeVolumeStatus",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:CreateSnapshot",
        "ec2:DeleteSnapshot"
      ],
      "Resource": "*"
    }
   ]}
   EOF
   $ POLICY_ARN=$(aws iam create-policy --policy-name "RosaOadpVer1" \
   --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn \
   --tags Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-oadp Key=operator_name,Value=openshift-oadp \
   --output text)
   fi
   $ echo ${POLICY_ARN}

2. Create an IAM Role trust policy for the cluster:

   $ cat <<EOF > ${SCRATCH}/trust-policy.json
   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Principal": {
         "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
       },
       "Action": "sts:AssumeRoleWithWebIdentity",
       "Condition": {
         "StringEquals": {
            "${OIDC_ENDPOINT}:sub": [
              "system:serviceaccount:openshift-adp:openshift-adp-controller-manager",
              "system:serviceaccount:openshift-adp:velero"]
         }
       }
     }]
   }
   EOF
   $ ROLE_ARN=$(aws iam create-role --role-name \
    "${ROLE_NAME}" \
     --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
     --tags Key=rosa_cluster_id,Value=${ROSA_CLUSTER_ID} Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-adp Key=operator_name,Value=openshift-oadp \
     --query Role.Arn --output text)
   
   $ echo ${ROLE_ARN}

3. Attach the IAM Policy to the IAM Role:

   $ aws iam attach-role-policy --role-name "${ROLE_NAME}" \
    --policy-arn ${POLICY_ARN}

1. Create a namespace for OADP:

   $ oc create namespace openshift-adp

2. Create a credentials secret:

   $ cat <<EOF > ${SCRATCH}/credentials
   [default]
   role_arn = ${ROLE_ARN}
   web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
   EOF
   $ oc -n openshift-adp create secret generic cloud-credentials \
    --from-file=${SCRATCH}/credentials

3. Deploy the OADP Operator:

   Note

   There is currently an issue with version 1.1 of the Operator with backups that have a PartiallyFailed status. This does not seem to affect the backup and restore process, but it should be noted as there are issues with it.

   $ cat << EOF | oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
    generateName: openshift-adp-
    namespace: openshift-adp
    name: oadp
   spec:
    targetNamespaces:
    - openshift-adp
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
    name: redhat-oadp-operator
    namespace: openshift-adp
   spec:
    channel: stable-1.2
    installPlanApproval: Automatic
    name: redhat-oadp-operator
    source: redhat-operators
    sourceNamespace: openshift-marketplace
   EOF

4. Wait for the Operator to be ready:

   $ watch oc -n openshift-adp get pods

   Example output

   NAME                                                READY   STATUS    RESTARTS   AGE
   openshift-adp-controller-manager-546684844f-qqjhn   1/1     Running   0          22s

5. Create Cloud Storage:

   $ cat << EOF | oc create -f -
   apiVersion: oadp.openshift.io/v1alpha1
   kind: CloudStorage
   metadata:
    name: ${CLUSTER_NAME}-oadp
    namespace: openshift-adp
   spec:
    creationSecret:
      key: credentials
      name: cloud-credentials
    enableSharedConfig: true
    name: ${CLUSTER_NAME}-oadp
    provider: aws
    region: $REGION
   EOF

6. Check your application’s storage default storage class:

   $ oc get pvc -n <namespace> 1

   1

   Enter your application’s namespace.

   Example output

   NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   applog   Bound    pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8   1Gi        RWO            gp3-csi        4d19h
   mysql    Bound    pvc-16b8e009-a20a-4379-accc-bc81fedd0621   1Gi        RWO            gp3-csi        4d19h

   $ oc get storageclass

   Example output

   NAME                PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   gp2                 kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   4d21h
   gp2-csi             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
   gp3                 ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
   gp3-csi (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h

   Using either gp3-csi, gp2-csi, gp3 or gp2 will work. If the application(s) that are being backed up are all using PV’s with CSI, include the CSI plugin in the OADP DPA configuration.

7. CSI only: Deploy a Data Protection Application:

   $ cat << EOF | oc create -f -
   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
    name: ${CLUSTER_NAME}-dpa
    namespace: openshift-adp
   spec:
    backupImages: true
    features:
      dataMover:
        enable: false
    backupLocations:
    - bucket:
        cloudStorageRef:
          name: ${CLUSTER_NAME}-oadp
        credential:
          key: credentials
          name: cloud-credentials
        prefix: velero
        default: true
        config:
          region: ${REGION}
    configuration:
      velero:
        defaultPlugins:
        - openshift
        - aws
        - csi
      restic:
        enable: false
   EOF

   Note

   If you run this command for CSI volumes, you can skip the next step.

8. Non-CSI volumes: Deploy a Data Protection Application:

   $ cat << EOF | oc create -f -
   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
    name: ${CLUSTER_NAME}-dpa
    namespace: openshift-adp
   spec:
    backupImages: true
    features:
      dataMover:
        enable: false
    backupLocations:
    - bucket:
        cloudStorageRef:
          name: ${CLUSTER_NAME}-oadp
        credential:
          key: credentials
          name: cloud-credentials
        prefix: velero
        default: true
        config:
          region: ${REGION}
    configuration:
      velero:
        defaultPlugins:
        - openshift
        - aws
      restic:
        enable: false
    snapshotLocations:
      - velero:
          config:
            credentialsFile: /tmp/credentials/openshift-adp/cloud-credentials-credentials
            enableSharedConfig: 'true'
            profile: default
            region: ${REGION}
          provider: aws
   EOF

1. Create a workload to back up:

   $ oc create namespace hello-world
   $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

2. Expose the route:

   $ oc expose service/hello-openshift -n hello-world

3. Check that the application is working:

   $ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`

   Example output

   Hello OpenShift!

4. Back up the workload:

   $ cat << EOF | oc create -f -
   apiVersion: velero.io/v1
   kind: Backup
   metadata:
    name: hello-world
    namespace: openshift-adp
   spec:
    includedNamespaces:
    - hello-world
    storageLocation: ${CLUSTER_NAME}-dpa-1
    ttl: 720h0m0s
   EOF

5. Wait until the backup is done:

   $ watch "oc -n openshift-adp get backup hello-world -o json | jq .status"

   Example output

   {
    "completionTimestamp": "2022-09-07T22:20:44Z",
    "expiration": "2022-10-07T22:20:22Z",
    "formatVersion": "1.1.0",
    "phase": "Completed",
    "progress": {
      "itemsBackedUp": 58,
      "totalItems": 58
    },
    "startTimestamp": "2022-09-07T22:20:22Z",
    "version": 1
   }

6. Delete the demo workload:

   $ oc delete ns hello-world

7. Restore from the backup:

   $ cat << EOF | oc create -f -
   apiVersion: velero.io/v1
   kind: Restore
   metadata:
    name: hello-world
    namespace: openshift-adp
   spec:
    backupName: hello-world
   EOF

8. Wait for the Restore to finish:

   $ watch "oc -n openshift-adp get restore hello-world -o json | jq .status"

   Example output

   {
    "completionTimestamp": "2022-09-07T22:25:47Z",
    "phase": "Completed",
    "progress": {
      "itemsRestored": 38,
      "totalItems": 38
    },
    "startTimestamp": "2022-09-07T22:25:28Z",
    "warnings": 9
   }

9. Check that the workload is restored:

   $ oc -n hello-world get pods

   Example output

   NAME                              READY   STATUS    RESTARTS   AGE
   hello-openshift-9f885f7c6-kdjpj   1/1     Running   0          90s

   $ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`

   Example output

   Hello OpenShift!

10. For troubleshooting tips please refer to the OADP team’s troubleshooting documentation
11. Additional sample applications can be found in the OADP team’s sample applications directory

1. Delete the workload:

   $ oc delete ns hello-world

2. Remove the backup and restore resources from the cluster if they are no longer required:

   $ oc delete backup hello-world
   $ oc delete restore hello-world

3. To delete the backup/restore and remote objects in s3:

   $ velero backup delete hello-world
   $ velero restore delete hello-world

4. Delete the Data Protection Application:

   $ oc -n openshift-adp delete dpa ${CLUSTER_NAME}-dpa

5. Delete the Cloud Storage:

   $ oc -n openshift-adp delete cloudstorage ${CLUSTER_NAME}-oadp

   Warning

   If this command hangs, you might need to delete the finalizer:

   $ oc -n openshift-adp patch cloudstorage ${CLUSTER_NAME}-oadp -p '{"metadata":{"finalizers":null}}' --type=merge

6. Remove the Operator if it is no longer required:

   $ oc -n openshift-adp delete subscription oadp-operator

7. Remove the namespace for the Operator:

   $ oc delete ns redhat-openshift-adp

8. Remove the Custom Resource Definitions from the cluster if you no longer wish to have them:

   $ for CRD in `oc get crds | grep velero | awk '{print $1}'`; do oc delete crd $CRD; done
   $ for CRD in `oc get crds | grep -i oadp | awk '{print $1}'`; do oc delete crd $CRD; done

9. Delete the AWS S3 Bucket:

   $ aws s3 rm s3://${CLUSTER_NAME}-oadp --recursive
   $ aws s3api delete-bucket --bucket ${CLUSTER_NAME}-oadp

10. Detach the Policy from the role:

    $ aws iam detach-role-policy --role-name "${ROLE_NAME}" \
     --policy-arn "${POLICY_ARN}"

11. Delete the role:

    $ aws iam delete-role --role-name "${ROLE_NAME}"

1. Create an AWS IAM policy for the AWS Load Balancer Controller:

   Note

   The policy is sourced from the upstream AWS Load Balancer Controller policy plus permission to create tags on subnets. This is required by the operator to function.

   $ oc new-project aws-load-balancer-operator
   $ POLICY_ARN=$(aws iam list-policies --query \
        "Policies[?PolicyName=='aws-load-balancer-operator-policy'].{ARN:Arn}" \
        --output text)
   $ if [[ -z "${POLICY_ARN}" ]]; then
       wget -O "${SCRATCH}/load-balancer-operator-policy.json" \
          https://raw.githubusercontent.com/rh-mobb/documentation/main/content/docs/rosa/aws-load-balancer-operator/load-balancer-operator-policy.json
        POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
        --output text iam create-policy \
        --policy-name aws-load-balancer-operator-policy \
        --policy-document "file://${SCRATCH}/load-balancer-operator-policy.json")
   fi
   $ echo $POLICY_ARN

2. Create an AWS IAM trust policy for AWS Load Balancer Operator:

   $ cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Condition": {
      "StringEquals" : {
        "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager", "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"]
      }
    },
    "Principal": {
      "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
   }
   EOF

3. Create an AWS IAM role for the AWS Load Balancer Operator:

   $ ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
      --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
      --query Role.Arn --output text)
   $ echo $ROLE_ARN
   
   $ aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
        --policy-arn $POLICY_ARN

4. Create a secret for the AWS Load Balancer Operator to assume our newly created AWS IAM role:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   stringData:
     credentials: |
       [default]
       role_arn = $ROLE_ARN
       web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
   EOF

5. Install the Red Hat AWS Load Balancer Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     channel: stable-v1.0
     installPlanApproval: Automatic
     name: aws-load-balancer-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: aws-load-balancer-operator.v1.0.0
   EOF

6. Deploy an instance of the AWS Load Balancer Controller using the operator:

   Note

   If you get an error here wait a minute and try again, it means the Operator has not completed installing yet.

   $ cat << EOF | oc apply -f -
   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController
   metadata:
     name: cluster
   spec:
     credentials:
       name: aws-load-balancer-operator
   EOF

7. Check the that the operator and controller pods are both running:

   $ oc -n aws-load-balancer-operator get pods

   You should see the following, if not wait a moment and retry:

   NAME                                                             READY   STATUS    RESTARTS   AGE
   aws-load-balancer-controller-cluster-6ddf658785-pdp5d            1/1     Running   0          99s
   aws-load-balancer-operator-controller-manager-577d9ffcb9-w6zqn   2/2     Running   0          2m4s

1. Create a new project:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

3. Configure a NodePort service for the AWS ALB to connect to:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     name: hello-openshift-nodeport
     namespace: hello-world
   spec:
     ports:
       - port: 80
         targetPort: 8080
         protocol: TCP
     type: NodePort
     selector:
       deployment: hello-openshift
   EOF

4. Deploy an AWS ALB using the AWS Load Balancer Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hello-openshift-alb
     namespace: hello-world
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
   spec:
     ingressClassName: alb
     rules:
       - http:
           paths:
             - path: /
               pathType: Exact
               backend:
                 service:
                   name: hello-openshift-nodeport
                   port:
                     number: 80
   EOF

5. Curl the AWS ALB Ingress endpoint to verify the hello world application is accessible:

   Note

   AWS ALB provisioning takes a few minutes. If you receive an error that says curl: (6) Could not resolve host, please wait and try again.

   $ INGRESS=$(oc -n hello-world get ingress hello-openshift-alb \
       -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   $ curl "http://${INGRESS}"

   Example output

   Hello OpenShift!

6. Deploy an AWS NLB for your hello world application:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     name: hello-openshift-nlb
     namespace: hello-world
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-type: external
       service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
       service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
   spec:
     ports:
       - port: 80
         targetPort: 8080
         protocol: TCP
     type: LoadBalancer
     selector:
       deployment: hello-openshift
   EOF

7. Test the AWS NLB endpoint:

   Note

   NLB provisioning takes a few minutes. If you receive an error that says curl: (6) Could not resolve host, please wait and try again.

   $ NLB=$(oc -n hello-world get service hello-openshift-nlb \
     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   $ curl "http://${NLB}"

   Example output

   Hello OpenShift!

1. Delete the hello world application namespace (and all the resources in the namespace):

   $ oc delete project hello-world

2. Delete the AWS Load Balancer Operator and the AWS IAM roles:

   $ oc delete subscription aws-load-balancer-operator -n aws-load-balancer-operator
   $ aws iam detach-role-policy \
     --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
     --policy-arn $POLICY_ARN
   $ aws iam delete-role \
     --role-name "${ROSA_CLUSTER_NAME}-alb-operator"

3. Delete the AWS IAM policy:

   $ aws iam delete-policy --policy-arn $POLICY_ARN

Procedure

1. Create the cluster’s OAuth callback URL by changing the specified variables and running the following command:

   Note

   Remember to save this callback URL; it will be required later in the process.

   $ domain=$(rosa describe cluster -c <cluster_name> | grep "DNS" | grep -oE '\S+.openshiftapps.com')
   $ echo "OAuth callback URL: https://oauth-openshift.apps.$domain/oauth2callback/AAD"

   The "AAD" directory at the end of the OAuth callback URL must match the OAuth identity provider name that you will set up later in this process.

2. Create the Entra ID application by logging in to the Azure portal, and select the App registrations blade. Then, select New registration to create a new application.

   

3. Name the application, for example openshift-auth.
4. Select Web from the Redirect URI dropdown and enter the value of the OAuth callback URL you retrieved in the previous step.

5. After providing the required information, click Register to create the application.

   

6. Select the Certificates & secrets sub-blade and select New client secret.

   

7. Complete the requested details and store the generated client secret value. This secret is required later in this process.

   Important

   After initial setup, you cannot see the client secret. If you did not record the client secret, you must generate a new one.

   

8. Select the Overview sub-blade and note the Application (client) ID and Directory (tenant) ID. You will need these values in a future step.

   

1. Click the Token configuration sub-blade and select the Add optional claim button.

   

2. Select the ID radio button.

   

3. Select the email claim checkbox.

   

4. Select the preferred_username claim checkbox. Then, click Add to configure the email and preferred_username claims your Entra ID application.

   

5. A dialog box appears at the top of the page. Follow the prompt to enable the necessary Microsoft Graph permissions.

   

Procedure

1. From the Token configuration sub-blade, click Add groups claim.

   

2. To configure group claims for your Entra ID application, select Security groups and then click the Add.

   Note

   In this example, the group claim includes all of the security groups that a user is a member of. In a real production environment, ensure that the groups that the group claim only includes groups that apply to Red Hat OpenShift Service on AWS.

   

Procedure

1. Create the variables by running the following command:

   $ CLUSTER_NAME=example-cluster 1
   $ IDP_NAME=AAD 2
   $ APP_ID=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy 3
   $ CLIENT_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx 4
   $ TENANT_ID=zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz 5

   1

   Replace this with the name of your ROSA cluster.

   2

   Replace this value with the name you used in the OAuth callback URL that you generated earlier in this process.

   3

   Replace this with the Application (client) ID.

   4

   Replace this with the Client Secret.

   5

   Replace this with the Directory (tenant) ID.

2. Configure the cluster’s OAuth provider by running the following command. If you enabled group claims, ensure that you use the --group-claims groups argument.

   • If you enabled group claims, run the following command:

     $ rosa create idp \
     --cluster ${CLUSTER_NAME} \
     --type openid \
     --name ${IDP_NAME} \
     --client-id ${APP_ID} \
     --client-secret ${CLIENT_SECRET} \
     --issuer-url https://login.microsoftonline.com/${TENANT_ID}/v2.0 \
     --email-claims email \
     --name-claims name \
     --username-claims preferred_username \
     --extra-scopes email,profile \
     --groups-claims groups

   • If you did not enable group claims, run the following command:

     $ rosa create idp \
     --cluster ${CLUSTER_NAME} \
     --type openid \
     --name ${IDP_NAME} \
     --client-id ${APP_ID} \
     --client-secret ${CLIENT_SECRET} \
     --issuer-url https://login.microsoftonline.com/${TENANT_ID}/v2.0 \
     --email-claims email \
     --name-claims name \
     --username-claims preferred_username \
     --extra-scopes email,profile

1. Log in to your ROSA cluster by running the following command:

   $ oc login --token=<your-token> --server=<your-server-url>

   You can find your login token by accessing your cluster in pull secret from Red Hat OpenShift Cluster Manager.

2. Validate that your cluster has STS by running the following command:

   $ oc get authentication.config.openshift.io cluster -o json \
     | jq .spec.serviceAccountIssuer

   Example output

   "https://xxxxx.cloudfront.net/xxxxx"

   If your output is different, do not proceed. See Red Hat documentation on creating an STS cluster before continuing this process.

3. Set the SecurityContextConstraints permission to allow the CSI driver to run by running the following command:

   $ oc new-project csi-secrets-store
   $ oc adm policy add-scc-to-user privileged \
       system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
   $ oc adm policy add-scc-to-user privileged \
       system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws

4. Create environment variables to use later in this process by running the following command:

   $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
   $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster \
      -o jsonpath='{.spec.serviceAccountIssuer}' | sed  's|^https://||')
   $ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
   $ export AWS_PAGER=""

1. Use Helm to register the secrets store CSI driver by running the following command:

   $ helm repo add secrets-store-csi-driver \
       https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

2. Update your Helm repositories by running the following command:

   $ helm repo update

3. Install the secrets store CSI driver by running the following command:

   $ helm upgrade --install -n csi-secrets-store \
       csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver

4. Deploy the AWS provider by running the following command:

   $ oc -n csi-secrets-store apply -f \
       https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml

5. Check that both Daemonsets are running by running the following command:

   $ oc -n csi-secrets-store get ds \
       csi-secrets-store-provider-aws \
       csi-secrets-store-driver-secrets-store-csi-driver

6. Label the Secrets Store CSI Driver to allow use with the restricted pod security profile by running the following command:

   $ oc label csidriver.storage.k8s.io/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted

1. Create a secret in Secrets Manager by running the following command:

   $ SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret \
       --name MySecret --secret-string \
       '{"username":"shadowman", "password":"hunter2"}' \
       --query ARN --output text)
   $ echo $SECRET_ARN

2. Create an IAM Access Policy document by running the following command:

   $ cat << EOF > policy.json
   {
      "Version": "2012-10-17",
      "Statement": [{
         "Effect": "Allow",
         "Action": [
           "secretsmanager:GetSecretValue",
           "secretsmanager:DescribeSecret"
         ],
         "Resource": ["$SECRET_ARN"]
         }]
   }
   EOF

3. Create an IAM Access Policy by running the following command:

   $ POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
   --output text iam create-policy \
   --policy-name openshift-access-to-mysecret-policy \
   --policy-document file://policy.json)
   $ echo $POLICY_ARN

4. Create an IAM Role trust policy document by running the following command:

   Note

   The trust policy is locked down to the default service account of a namespace you create later in this process.

   $ cat <<EOF > trust-policy.json
   {
      "Version": "2012-10-17",
      "Statement": [
      {
      "Effect": "Allow",
      "Condition": {
        "StringEquals" : {
          "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"]
         }
       },
       "Principal": {
          "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
       },
       "Action": "sts:AssumeRoleWithWebIdentity"
       }
       ]
   }
   EOF

5. Create an IAM role by running the following command:

   $ ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \
   --assume-role-policy-document file://trust-policy.json \
   --query Role.Arn --output text)
   $ echo $ROLE_ARN

6. Attach the role to the policy by running the following command:

   $ aws iam attach-role-policy --role-name openshift-access-to-mysecret \
       --policy-arn $POLICY_ARN

1. Create an OpenShift project by running the following command:

   $ oc new-project my-application

2. Annotate the default service account to use the STS Role by running the following command:

   $ oc annotate -n my-application serviceaccount default \
       eks.amazonaws.com/role-arn=$ROLE_ARN

3. Create a secret provider class to access our secret by running the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: my-application-aws-secrets
   spec:
     provider: aws
     parameters:
       objects: |
         - objectName: "MySecret"
           objectType: "secretsmanager"
   EOF

4. Create a Deployment by using our secret in the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: my-application
     labels:
       app: my-application
   spec:
     volumes:
     - name: secrets-store-inline
       csi:
         driver: secrets-store.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: "my-application-aws-secrets"
     containers:
     - name: my-application-deployment
       image: k8s.gcr.io/e2e-test-images/busybox:1.29
       command:
         - "/bin/sleep"
         - "10000"
       volumeMounts:
       - name: secrets-store-inline
         mountPath: "/mnt/secrets-store"
         readOnly: true
   EOF

5. Verify the Pod has the secret mounted by running the following commandv:

   $ oc exec -it my-application -- cat /mnt/secrets-store/MySecret

1. Delete the application by running the following command:

   $ oc delete project my-application

2. Delete the secrets store csi driver by running the following command:

   $ helm delete -n csi-secrets-store csi-secrets-store-driver

3. Delete Security Context Constraints by running the following command:

   $ oc adm policy remove-scc-from-user privileged \
       system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
   $ oc adm policy remove-scc-from-user privileged \
       system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws

4. Delete the AWS provider by running the following command:

   $ oc -n csi-secrets-store delete -f \
   https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml

5. Delete AWS Roles and Policies by running the following command:

   $ aws iam detach-role-policy --role-name openshift-access-to-mysecret \
       --policy-arn $POLICY_ARN
   $ aws iam delete-role --role-name openshift-access-to-mysecret
   $ aws iam delete-policy --policy-arn $POLICY_ARN

6. Delete the Secrets Manager secret by running the following command:

   $ aws secretsmanager --region $REGION delete-secret --secret-id $SECRET_ARN

1. Configure the following environment variables, changing the cluster name to suit your cluster:

   $ export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
   $ export REGION=$(rosa describe cluster -c ${ROSA_CLUSTER_NAME} --output json | jq -r .region.id)
   $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
   $ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
   $ export ACK_SERVICE=s3
   $ export ACK_SERVICE_ACCOUNT=ack-${ACK_SERVICE}-controller
   $ export POLICY_ARN=arn:aws:iam::aws:policy/AmazonS3FullAccess
   $ export AWS_PAGER=""
   $ export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/ack"
   $ mkdir -p ${SCRATCH}

2. Ensure all fields output correctly before moving to the next section:

   $ echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

1. Create an AWS Identity Access Management (IAM) trust policy for the ACK Operator:

   $ cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Condition": {
      "StringEquals" : {
        "${OIDC_ENDPOINT}:sub": "system:serviceaccount:ack-system:${ACK_SERVICE_ACCOUNT}"
      }
    },
    "Principal": {
      "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
   }
   EOF

2. Create an AWS IAM role for the ACK Operator to assume with the AmazonS3FullAccess policy attached:

   Note

   You can find the recommended policy in each project’s GitHub repository, for example https://github.com/aws-controllers-k8s/s3-controller/blob/main/config/iam/recommended-policy-arn.

   $ ROLE_ARN=$(aws iam create-role --role-name "ack-${ACK_SERVICE}-controller" \
      --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
      --query Role.Arn --output text)
   $ echo $ROLE_ARN
   
   $ aws iam attach-role-policy --role-name "ack-${ACK_SERVICE}-controller" \
        --policy-arn ${POLICY_ARN}

1. Create a project to install the ACK S3 Operator into:

   $ oc new-project ack-system

2. Create a file with the ACK S3 Operator configuration:

   Note

   ACK_WATCH_NAMESPACE is purposefully left blank so the controller can properly watch all namespaces in the cluster.

   $ cat <<EOF > "${SCRATCH}/config.txt"
   ACK_ENABLE_DEVELOPMENT_LOGGING=true
   ACK_LOG_LEVEL=debug
   ACK_WATCH_NAMESPACE=
   AWS_REGION=${REGION}
   AWS_ENDPOINT_URL=
   ACK_RESOURCE_TAGS=${CLUSTER_NAME}
   ENABLE_LEADER_ELECTION=true
   LEADER_ELECTION_NAMESPACE=
   EOF

3. Use the file from the previous step to create a ConfigMap:

   $ oc -n ack-system create configmap \
     --from-env-file=${SCRATCH}/config.txt ack-${ACK_SERVICE}-user-config

4. Install the ACK S3 Operator from OperatorHub:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: ack-${ACK_SERVICE}-controller
     namespace: ack-system
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: ack-${ACK_SERVICE}-controller
     namespace: ack-system
   spec:
     channel: alpha
     installPlanApproval: Automatic
     name: ack-${ACK_SERVICE}-controller
     source: community-operators
     sourceNamespace: openshift-marketplace
   EOF

5. Annotate the ACK S3 Operator service account with the AWS IAM role to assume and restart the deployment:

   $ oc -n ack-system annotate serviceaccount ${ACK_SERVICE_ACCOUNT} \
     eks.amazonaws.com/role-arn=${ROLE_ARN} && \
     oc -n ack-system rollout restart deployment ack-${ACK_SERVICE}-controller

6. Verify that the ACK S3 Operator is running:

   $ oc -n ack-system get pods

   Example output

   NAME                                 READY   STATUS    RESTARTS   AGE
   ack-s3-controller-585f6775db-s4lfz   1/1     Running   0          51s

1. Deploy an S3 bucket resource:

   $ cat << EOF | oc apply -f -
   apiVersion: s3.services.k8s.aws/v1alpha1
   kind: Bucket
   metadata:
      name: ${CLUSTER-NAME}-bucket
      namespace: ack-system
   spec:
      name: ${CLUSTER-NAME}-bucket
   EOF

2. Verify the S3 bucket was created in AWS:

   $ aws s3 ls | grep ${CLUSTER_NAME}-bucket

   Example output

   2023-10-04 14:51:45 mrmc-test-maz-bucket

1. Delete the S3 bucket resource:

   $ oc -n ack-system delete bucket.s3.services.k8s.aws/${CLUSTER-NAME}-bucket

2. Delete the ACK S3 Operator and the AWS IAM roles:

   $ oc -n ack-system delete subscription ack-${ACK_SERVICE}-controller
   $ aws iam detach-role-policy \
     --role-name "ack-${ACK_SERVICE}-controller" \
     --policy-arn ${POLICY_ARN}
   $ aws iam delete-role \
     --role-name "ack-${ACK_SERVICE}-controller"

3. Delete the ack-system project:

   $ oc delete project ack-system

1. Configure the following environment variables, replacing CLUSTER_NAME with the name of your cluster:

   $ export DOMAIN=apps.<company_name>.io 1
   $ export AWS_PAGER=""
   $ export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
   $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
   $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
   $ export SCRATCH="/tmp/${CLUSTER_NAME}/external-dns"
   $ mkdir -p ${SCRATCH}

   1

   The custom domain.

2. Ensure all fields output correctly before moving to the next section:

   $ echo "Cluster: ${CLUSTER_NAME}, Region: ${REGION}, AWS Account ID: ${AWS_ACCOUNT_ID}"

Procedure

1. Create a new project:

   $ oc new-project external-dns-operator

2. Create a new TLS secret from a private key and a public certificate, where fullchain.pem is your full wildcard certificate chain (including any intermediaries) and privkey.pem is your wildcard certificate’s private key:

   $ oc -n external-dns-operator create secret tls external-dns-tls --cert=fullchain.pem --key=privkey.pem

3. Create a new CustomDomain custom resource (CR):

   Example external-dns-custom-domain.yaml

   apiVersion: managed.openshift.io/v1alpha1
   kind: CustomDomain
   metadata:
     name: external-dns
   spec:
     domain: apps.<company_name>.io 1
     scope: External
     loadBalancerType: NLB
     certificate:
       name: external-dns-tls
       namespace: external-dns-operator

   1

   The custom domain.

4. Apply the CR:

   $ oc apply -f external-dns-custom-domain.yaml

5. Verify that your custom domain Ingress Controller has been deployed and has a Ready status:

   $ oc get customdomains

   Example output

   NAME               ENDPOINT                                                    DOMAIN                       STATUS
   external-dns       xxrywp.<company_name>.cluster-01.opln.s1.openshiftapps.com  *.apps.<company_name>.io     Ready

1. Retrieve the Amazon Route 53 public hosted zone ID:

   $ export ZONE_ID=$(aws route53 list-hosted-zones-by-name --output json \
     --dns-name "${DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')

2. Create an AWS IAM Policy document that allows the External DNS Operator to update only the custom domain public hosted zone:

   $ cat << EOF > "${SCRATCH}/external-dns-policy.json"
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "route53:ChangeResourceRecordSets"
         ],
         "Resource": [
           "arn:aws:route53:::hostedzone/${ZONE_ID}"
         ]
       },
       {
         "Effect": "Allow",
         "Action": [
           "route53:ListHostedZones",
           "route53:ListResourceRecordSets"
         ],
         "Resource": [
           "*"
         ]
       }
     ]
   }
   EOF

3. Create an AWS IAM policy:

   $ export POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER_NAME}-AllowExternalDNSUpdates" \
     --policy-document file://${SCRATCH}/external-dns-policy.json \
     --query 'Policy.Arn' --output text)

4. Create an AWS IAM user:

   $ aws iam create-user --user-name "${CLUSTER_NAME}-external-dns-operator"

5. Attach the policy:

   $ aws iam attach-user-policy --user-name "${CLUSTER_NAME}-external-dns-operator" --policy-arn $POLICY_ARN

   Note

   This will be changed to STS using IRSA in the future.

6. Create AWS keys for the IAM user:

   $ SECRET_ACCESS_KEY=$(aws iam create-access-key --user-name "${CLUSTER_NAME}-external-dns-operator")

7. Create static credentials:

   $ cat << EOF > "${SCRATCH}/credentials"
   [default]
   aws_access_key_id = $(echo $SECRET_ACCESS_KEY | jq -r '.AccessKey.AccessKeyId')
   aws_secret_access_key = $(echo $SECRET_ACCESS_KEY | jq -r '.AccessKey.SecretAccessKey')
   EOF

1. Install the External DNS Operator from OperatorHub:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: external-dns-group
     namespace: external-dns-operator
   spec:
     targetNamespaces:
     - external-dns-operator
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: external-dns-operator
     namespace: external-dns-operator
   spec:
     channel: stable-v1.1
     installPlanApproval: Automatic
     name: external-dns-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

2. Wait until the External DNS Operator is running:

   $ oc rollout status deploy external-dns-operator --timeout=300s

3. Create a secret from the AWS IAM user credentials:

   $ oc -n external-dns-operator create secret generic external-dns \
     --from-file "${SCRATCH}/credentials"

4. Deploy the ExternalDNS controller:

   $ cat << EOF | oc apply -f -
   apiVersion: externaldns.olm.openshift.io/v1beta1
   kind: ExternalDNS
   metadata:
     name: ${DOMAIN}
   spec:
     domains:
       - filterType: Include
         matchType: Exact
         name: ${DOMAIN}
     provider:
       aws:
         credentials:
           name: external-dns
       type: AWS
     source:
       openshiftRouteOptions:
         routerName: external-dns
       type: OpenShiftRoute
     zones:
       - ${ZONE_ID}
   EOF

5. Wait until the controller is running:

   $ oc rollout status deploy external-dns-${DOMAIN} --timeout=300s

1. Create a new project for your sample application:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

3. Create a route for the application specifying your custom domain name:

   $ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls \
   --hostname hello-openshift.${DOMAIN}

4. Check if the DNS record was created automatically by ExternalDNS:

   Note

   It can take a few minutes for the record to appear in Amazon Route 53.

   $ aws route53 list-resource-record-sets --hosted-zone-id ${ZONE_ID} \
      --query "ResourceRecordSets[?Type == 'CNAME']" | grep hello-openshift

5. Optional: You can also view the TXT records that indicate they were created by ExternalDNS:

   $ aws route53 list-resource-record-sets --hosted-zone-id ${ZONE_ID} \
      --query "ResourceRecordSets[?Type == 'TXT']" | grep ${DOMAIN}

6. Navigate to your custom console domain in the browser where you see the OpenShift login:

   $ echo console.${DOMAIN}

1. Configure the following environment variables:

   $ export DOMAIN=apps.<company_name>.io 1
   $ export EMAIL=<youremail@company_name.io> 2
   $ export AWS_PAGER=""
   $ export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
   $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
   $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
   $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
   $ export SCRATCH="/tmp/${CLUSTER_NAME}/dynamic-certs"
   $ mkdir -p ${SCRATCH}

   1

   The custom domain.

   2

   The e-mail Let’s Encrypt will use to send notifications about your certificates.

2. Ensure all fields output correctly before moving to the next section:

   $ echo "Cluster: ${CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

1. Retrieve the Amazon Route 53 public hosted zone ID:

   Note

   This command looks for a public hosted zone that matches the custom domain you specified earlier as the DOMAIN environment variable. You can manually specify the Amazon Route 53 public hosted zone by running export ZONE_ID=<zone_ID>, replacing <zone_ID> with your specific Amazon Route 53 public hosted zone ID.

   $ export ZONE_ID=$(aws route53 list-hosted-zones-by-name --output json \
     --dns-name "${DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')

2. Create an AWS IAM policy document for the cert-manager Operator that provides the ability to update only the specified public hosted zone:

   $ cat <<EOF > "${SCRATCH}/cert-manager-policy.json"
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "route53:GetChange",
         "Resource": "arn:aws:route53:::change/*"
       },
       {
         "Effect": "Allow",
         "Action": [
           "route53:ChangeResourceRecordSets",
           "route53:ListResourceRecordSets"
         ],
         "Resource": "arn:aws:route53:::hostedzone/${ZONE_ID}"
       },
       {
         "Effect": "Allow",
         "Action": "route53:ListHostedZonesByName",
         "Resource": "*"
       }
     ]
   }
   EOF

3. Create the IAM policy using the file you created in the previous step:

   $ POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER_NAME}-cert-manager-policy" \
     --policy-document file://${SCRATCH}/cert-manager-policy.json \
     --query 'Policy.Arn' --output text)

4. Create an AWS IAM trust policy for the cert-manager Operator:

   $ cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Condition": {
      "StringEquals" : {
        "${OIDC_ENDPOINT}:sub": "system:serviceaccount:cert-manager:cert-manager"
      }
    },
    "Principal": {
      "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
   }
   EOF

5. Create an IAM role for the cert-manager Operator using the trust policy you created in the previous step:

   $ ROLE_ARN=$(aws iam create-role --role-name "${CLUSTER_NAME}-cert-manager-operator" \
      --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
      --query Role.Arn --output text)

6. Attach the permissions policy to the role:

   $ aws iam attach-role-policy --role-name "${CLUSTER_NAME}-cert-manager-operator" \
     --policy-arn ${POLICY_ARN}

1. Create a project to install the cert-manager Operator into:

   $ oc new-project cert-manager-operator

   Important

   Do not attempt to use more than one cert-manager Operator in your cluster. If you have a community cert-manager Operator installed in your cluster, you must uninstall it before installing the cert-manager Operator for Red Hat OpenShift.

2. Install the cert-manager Operator for Red Hat OpenShift:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-cert-manager-operator-group
     namespace: cert-manager-operator
   spec:
     targetNamespaces:
     - cert-manager-operator
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openshift-cert-manager-operator
     namespace: cert-manager-operator
   spec:
     channel: stable-v1
     installPlanApproval: Automatic
     name: openshift-cert-manager-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

   Note

   It takes a few minutes for this Operator to install and complete its set up.

3. Verify that the cert-manager Operator is running:

   $ oc -n cert-manager-operator get pods

   Example output

   NAME                                                        READY   STATUS    RESTARTS   AGE
   cert-manager-operator-controller-manager-84b8799db5-gv8mx   2/2     Running   0          12s

4. Annotate the service account used by the cert-manager pods with the AWS IAM role you created earlier:

   $ oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn=${ROLE_ARN}

5. Restart the existing cert-manager controller pod by running the following command:

   $ oc -n cert-manager delete pods -l app.kubernetes.io/name=cert-manager

6. Patch the Operator’s configuration to use external nameservers to prevent DNS-01 challenge resolution issues:

   $ oc patch certmanager.operator.openshift.io/cluster --type merge \
     -p '{"spec":{"controllerConfig":{"overrideArgs":["--dns01-recursive-nameservers-only","--dns01-recursive-nameservers=1.1.1.1:53"]}}}'

7. Create a ClusterIssuer resource to use Let’s Encrypt by running the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-production
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       email: ${EMAIL}
       # This key doesn't exist, cert-manager creates it
       privateKeySecretRef:
         name: prod-letsencrypt-issuer-account-key
       solvers:
       - dns01:
           route53:
            hostedZoneID: ${ZONE_ID}
            region: ${REGION}
            secretAccessKeySecretRef:
              name: ''
   EOF

8. Verify the ClusterIssuer resource is ready:

   $ oc get clusterissuer.cert-manager.io/letsencrypt-production

   Example output

   NAME                     READY   AGE
   letsencrypt-production   True    47s

1. Create a new project:

   $ oc new-project custom-domain-ingress

2. Create and configure a certificate resource to provision a certificate for the custom domain Ingress Controller:

   Note

   The following example uses a single domain certificate. SAN and wildcard certificates are also supported.

   $ cat << EOF | oc apply -f -
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: custom-domain-ingress-cert
     namespace: custom-domain-ingress
   spec:
     secretName: custom-domain-ingress-cert-tls
     issuerRef:
        name: letsencrypt-production
        kind: ClusterIssuer
     commonName: "${DOMAIN}"
     dnsNames:
     - "${DOMAIN}"
   EOF

3. Verify the certificate has been issued:

   Note

   It takes a few minutes for this certificate to be issued by Let’s Encrypt. If it takes longer than 5 minutes, run oc -n custom-domain-ingress describe certificate.cert-manager.io/custom-domain-ingress-cert to see any issues reported by cert-manager.

   $ oc -n custom-domain-ingress get certificate.cert-manager.io/custom-domain-ingress-cert

   Example output

   NAME                         READY   SECRET                           AGE
   custom-domain-ingress-cert   True    custom-domain-ingress-cert-tls   9m53s

4. Create a new CustomDomain custom resource (CR):

   $ cat << EOF | oc apply -f -
   apiVersion: managed.openshift.io/v1alpha1
   kind: CustomDomain
   metadata:
     name: custom-domain-ingress
   spec:
     domain: ${DOMAIN}
     scope: External
     loadBalancerType: NLB
     certificate:
       name: custom-domain-ingress-cert-tls
       namespace: custom-domain-ingress
   EOF

5. Verify that your custom domain Ingress Controller has been deployed and has a Ready status:

   $ oc get customdomains

   Example output

   NAME                    ENDPOINT                                                               DOMAIN            STATUS
   custom-domain-ingress   tfoxdx.custom-domain-ingress.cluster.1234.p1.openshiftapps.com         example.com       Ready

6. Prepare a document with the necessary DNS changes to enable DNS resolution for your custom domain Ingress Controller:

   $ INGRESS=$(oc get customdomain.managed.openshift.io/custom-domain-ingress --template={{.status.endpoint}})
   $ cat << EOF > "${SCRATCH}/create-cname.json"
   {
     "Comment":"Add CNAME to custom domain endpoint",
     "Changes":[{
         "Action":"CREATE",
         "ResourceRecordSet":{
           "Name": "*.${DOMAIN}",
         "Type":"CNAME",
         "TTL":30,
         "ResourceRecords":[{
           "Value": "${INGRESS}"
         }]
       }
     }]
   }
   EOF

7. Submit your changes to Amazon Route 53 for propagation:

   $ aws route53 change-resource-record-sets \
     --hosted-zone-id ${ZONE_ID} \
     --change-batch file://${SCRATCH}/create-cname.json

   Note

   While the wildcard CNAME record avoids the need to create a new record for every new application you deploy using the custom domain Ingress Controller, the certificate that each of these applications use is not a wildcard certificate.

1. Create the necessary OpenShift resources cert-manager requires to manage certificates for OpenShift routes.

   This step creates a new deployment (and therefore a pod) that specifically monitors annotated routes in the cluster. If the issuer-kind and issuer-name annotations are found in a new route, it requests the Issuer (ClusterIssuer in this case) for a new certificate that is unique to this route and which will honor the hostname that was specified while creating the route.

   Note

   If the cluster does not have access to GitHub, you can save the raw contents locally and run oc apply -f localfilename.yaml -n cert-manager.

   $ oc -n cert-manager apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml

   The following additional OpenShift resources are also created in this step:

   • ClusterRole - grants permissions to watch and update the routes across the cluster
   • ServiceAccount - uses permissions to run the newly created pod
   • ClusterRoleBinding - binds these two resources

2. Ensure that the new cert-manager-openshift-routes pod is running successfully:

   $ oc -n cert-manager get pods

   Example result

   NAME                                             READY   STATUS    RESTARTS   AGE
   cert-manager-866d8f788c-9kspc                    1/1     Running   0          4h21m
   cert-manager-cainjector-6885c585bd-znws8         1/1     Running   0          4h41m
   cert-manager-openshift-routes-75b6bb44cd-f8kd5   1/1     Running   0          6s
   cert-manager-webhook-8498785dd9-bvfdf            1/1     Running   0          4h41m

1. Create a new project for your sample application:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc -n hello-world new-app --image=docker.io/openshift/hello-openshift

3. Create a route to expose the application from outside the cluster:

   $ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls --hostname hello.${DOMAIN}

4. Verify the certificate for the route is untrusted:

   $ curl -I https://hello.${DOMAIN}

   Example output

   curl: (60) SSL: no alternative certificate subject name matches target host name 'hello.example.com'
   More details here: https://curl.se/docs/sslcerts.html
   
   curl failed to verify the legitimacy of the server and therefore could not
   establish a secure connection to it. To learn more about this situation and
   how to fix it, please visit the web page mentioned above.

5. Annotate the route to trigger cert-manager to provision a certificate for the custom domain:

   $ oc -n hello-world annotate route hello-openshift-tls cert-manager.io/issuer-kind=ClusterIssuer cert-manager.io/issuer-name=letsencrypt-production

   Note

   It takes 2-3 minutes for the certificate to be created. The renewal of the certificate will automatically be managed by the cert-manager Operator as it approaches expiration.

6. Verify the certificate for the route is now trusted:

   $ curl -I https://hello.${DOMAIN}

   Example output

   HTTP/2 200
   date: Thu, 05 Oct 2023 23:45:33 GMT
   content-length: 17
   content-type: text/plain; charset=utf-8
   set-cookie: 52e4465485b6fb4f8a1b1bed128d0f3b=68676068bb32d24f0f558f094ed8e4d7; path=/; HttpOnly; Secure; SameSite=None
   cache-control: private

1. Run the following command to create the admin user:

   rosa create admin --cluster=<cluster-name>

   Example output

   W: It is recommended to add an identity provider to login to this cluster. See 'rosa create idp --help' for more information.
   I: Admin account has been added to cluster 'my-rosa-cluster'. It may take up to a minute for the account to become active.
   I: To login, run the following command:
   oc login https://api.my-rosa-cluster.abcd.p1.openshiftapps.com:6443 \
   --username cluster-admin \
   --password FWGYL-2mkJI-00000-00000

2. Copy the log in command returned to you in the previous step and paste it into your terminal. This will log you in to the cluster using the CLI so you can start using the cluster.

   $ oc login https://api.my-rosa-cluster.abcd.p1.openshiftapps.com:6443 \
   >    --username cluster-admin \
   >    --password FWGYL-2mkJI-00000-00000

   Example output

   Login successful.
   
   You have access to 79 projects, the list has been suppressed. You can list all projects with ' projects'
   
   Using project "default".

3. To check that you are logged in as the admin user, run one of the following commands:

   • Option 1:

     $ oc whoami

     Example output

     cluster-admin

   • Option 2:

     oc get all -n openshift-apiserver

     Only an admin user can run this command without errors.

4. You can now use the cluster as an admin user, which will suffice for this tutorial. For actual deployment, it is highly recommended to set up an identity provider, which is explained in the next tutorial.