Cloud Based Auto-Registration

Background:

As part of our work to Improve the Public Cloud RHEL Experience, Red Hat built a fleet/account wide method for getting RHEL connected.

In particular, we wanted to overhaul the customer cloud onboarding experience. Previously, we asked customers who wanted to use RHEL in the public cloud using previously purchased subscriptions to:

• Go to https://access.redhat.com/management/cloud and select which cloud provider they were deploying their workloads on.
• Inform us of the number of entitlements they would be using on that public cloud
• Keep this number up to date as they deployed and retired workloads.

By doing this, the customer got (where applicable) access to Gold Images (cloud images which the cloud provider does not charge them hourly for the RHEL usage, just the infrastructure costs), and arguably they were able to govern their usage. However, this workflow had a number of flaws:

• Keeping that number of entitlements properly up to date is difficult and ultimately wasn't used by any customer facing reporting, so:
• Other than the gold image enablement (which only needs to happen once per product line, per cloud account) it was of little value.
• Asking customers to manage entitlements is inconsistent with the subscription experience we have built elsewhere (if Red Hat doesn't require you to attach subscriptions to systems see Simple Content Access, it shouldn't require you to attach subscriptions at the account level either).
• Accuracy of this reporting was suspect at best, because the accuracy of it never impacted any operational workflow. We never denied support based on these numbers, nor blocked your access to content based on them.
• This workflow violates our core subscription experience principle of "Don’t afflict the customer with operational burden solely for subscription management".

However, there are some real problems that are worth addressing. Particularly, exactly how does a customer, who is deploying workloads on the public cloud, track their usage of Red Hat software and ensure that they don't deploy more than they expected? We could say "Mr(s). Customer, figure it out yourself". But this violates our core principle of:

"In all cases, avoid the temptation to fall back upon “this is the customer’s burden to report.”`.

We have a better solution:

Just use the reporting tool that already exists already: Subscription Watch

Content Delivery in the Public Cloud

For a user of RHEL on the public cloud, regardless of how you are paying for it, we want to ensure that you have reliable access to content out-of-box. (Read: we need yum to 'just work'™).

Red Hat Update Infrastructure (RHUI) provides this capability.

RHUI is a product that we offer to our cloud providers (itself built on the Pulp Project) to deliver highly available, in-cloud updates. Generally the cloud provider runs the RHUI infrastructure to provide this content.

Access to RHUI is delivered via a special RPM that gets built into the cloud image. That RPM contains yum repository definitions, and a certificate that allows you to access that cloud's RHUI infrastructure. That is what you get when you spin up workloads regardless of possessing a Red Hat account, and if all you cared about was spinning up a RHEL system and installing the most common RHEL bits, this is sufficient.

Note : in some gold images, we did not previously include access to RHUI. We have retrofitted gold images to include access to RHUI.

This content delivery model is the bedrock on which the rest of this experience is built upon, so getting this standardized and consistent is key to both a simplified subscription experience and will help in your understanding of the design decisions made later.

We could ask the user to run subscription-manager on all of their cloud instances to connect them to us. And you'd be correct in calling us hypocrites if we asked users to do this as it violates the aforementioned core principle of "Doesn’t afflict the customer with operational burden solely for subscription management".

Additionally, our core principles compel us to automate this:

"Work required for customers to support their subscriptions should be minimal and automated to the greatest possible extent. Subscription management is a distraction for customers and adds real, measurable overhead to the cost of Red Hat software. Manual workflows should be avoided"

ok, theory done, let's cover how it works

Auto-registration.

Autoregistration consists of three major parts:

• A version of the subscription-manager package which knows how to (and has been explicitly configured to) perform the autoregistration process
• A service hosted by Red Hat which maintains a mapping of Red Hat accounts and cloud provider accounts.
• An interface to allow the user to associate their Red Hat account with their cloud provider account.

We give a user a way to say "I own both this Red Hat account and this cloud provider account" and since I own both of them, "I'd like to hook them together".

Let's show how this works end to end:

Firstly, we head over to console.redhat.com and open Sources. This is where we link external entities to our Red Hat accounts. We are going to select Amazon Web Services (AWS) as our provider. Similar workflows exists for Azure or Google.

Next, we give our source a name

Next, we select how we want to configure authorization for the account. We have two different models

• Account Authorization - Give us root and we'll do the needful
• Manual Authorization - You configure the credential yourself and provide it to us.

For this example, I am going to use Manual Authorization.

Next I have to select which application do I want to configure, Cost Management or RHEL Management. The RHEL Management application is the one we want and includes ALL of the following:

• Gold Images - These gold images do not have the hourly RHEL surcharge associated with them.
• High precision subscription watch data - We give the user the ability to configure advanced metering capabilities on AWS. From the (Subscription Watch documentation): "Although the subscriptions service currently has the ability to identify RHEL based instances for multiple cloud providers, it is not able to identify and track the activities of individual instances as they start and stop, sometimes multiple times * per day. The public cloud metering tool adds that capability for AWS instances, resulting in more accurate monitoring of usage for those instances by the subscriptions service."
• Autoregistration - We setup the mapping between the user's Red Hat account (which we know because they are signed in) and their cloud provider account (which they are giving us in this workflow) to allow subscription-manager, when it checks in, to be authorized to register into the user's account.

Lastly, I input my Amazon Resource Number (ARN), or equivalent on other clouds. This unique authentication token allows the user to attest that they own the account and allows us to verify it. This is obviously important as we have to ensure proper multi-tenancy.

Lastly, we give the user a chance to review and confirm their selections:

So what have we accomplished:

• We've gotten access to those gold images. Those will show up in my Amazon account shortly.
• We now have setup cloud metering, allowing us to query the users Amazon account via the API to identify and count Red Hat workloads
• We've enabled Subscription Watch for the account to show this usage over time.
• We've recorded a mapping of Red Hat account <-> cloud provider account, which is used when subscription-manager calls home to authorize the system to be registered.

From this point forward, EVERY SINGLE INSTANCE that is instantiated from an image in this cloud provider account that is configured to auto-register will connect to Red Hat and register. No further user action is required. They just need to spin up instances and use them however they'd like and view the usage of the Red Hat products they contain in Subscription Watch

Now let's spin up some workloads:

Autoregistration at the client level

Since we setup an AWS source, let's spin up an instance in their console. For a Gold Image, make sure you are selecting "My AMIs" -> "Shared With Me" to ensure you aren't getting that hourly uplift for RHEL applied.

Auto-registration was introduced in RHEL 8.3.1, so any image newer than that will include it. RHEL9 based images will include it when RHEL9 goes GA, and at the time of this post auto-registration is still being backported to the RHEL7 family.

I am going to spin up a RHEL 8.5 instance. but first, let's recap subscription-manager's role in all of this:

We historically think of subscription-manager as being 'the tool to attach entitlements', and while it is that, the nature of what it does means that it has built a set of core competencies that are worth using in a connected customer experience, even if not in the traditional 'let's attach subs' use-cases. Let's talk about those core competencies

Subscription-manager's core competencies allow a system to:

• Establish an identity with Red Hat. (Register; get uniquely identified and establish a data pipeline between host and us)
• Gather facts & submit them.
  • system facts like number of socket or cores
  • business facts like "How am I using this system" (System Purpose)
  • custom(er) facts
• Configure the system’s authorization to protected content provided via a Satellite or the CDN
  • Via entitlement attachment or
  • Simple Content Access.
• Configure yum/dnf so that you can install protected content provided via a Satellite or the CDN (like RPM and RPM-OStree content)

Historically, we've used ALL of these capabilities simultaneously, but there are valid use-cases to use some, but not all of them. autoregistration is one such use case. We are using it here to establish an identity and send facts (so that we can power usage reporting in Subscription Watch), and reusing that identity establishment to make it easier to adopt Insights. But we aren't using the 'manage your content parts'. (We don't need to. We made sure that the instance has access to the most common RHEL bits via RHUI).

Now that my system is up and running, I am going to log in and run subscription-manager config to see the configuration,

[server]
   hostname = [subscription.rhsm.redhat.com]
   insecure = [0]
   no_proxy = []
   port = [443]
   prefix = [/subscription]
   proxy_hostname = []
   proxy_password = []
   proxy_port = []
   proxy_scheme = [http]
   proxy_user = []
   server_timeout = [180]
   ssl_verify_depth = [3]

[rhsm]
   auto_enable_yum_plugins = [1]
   baseurl = [https://cdn.redhat.com]
   ca_cert_dir = [/etc/rhsm/ca/]
   consumercertdir = [/etc/pki/consumer]
   entitlementcertdir = [/etc/pki/entitlement]
   full_refresh_on_yum = [0]
   inotify = [1]
   manage_repos = 0
   package_profile_on_trans = [0]
   pluginconfdir = [/etc/rhsm/pluginconf.d]
   plugindir = [/usr/share/rhsm-plugins]
   productcertdir = [/etc/pki/product]
   repo_ca_cert = /etc/rhsm/ca/redhat-uep.pem
   repomd_gpg_url = []
   report_package_profile = [1]

[rhsmcertd]
   auto_registration = 1
   auto_registration_interval = [60]
   autoattachinterval = [1440]
   certcheckinterval = [240]
   disable = [0]
   splay = [1]

[logging]
   default_log_level = [INFO]

[] - Default value in use

Of particular note are these four settings, which are now the defaults in Red Hat's cloud images:

• auto_registration = 1 - This is the setting that tells subscription-manager to actually attempt to auto-registration. The default value for this is 0. We explicitly set this value during the production of cloud images. On clouds where the provider creates the Gold Image [Azure, GCP], we instruct Microsoft & Google (respectively) to do the same.
• auto_registration_interval = 60 - This setting defines the interval which autoregistration is attempted. autoregistration is attempted three times per invocation of the rhsmcertd service at this interval. Example: with this value set to 60, a system will attempt to auto-register three times, 60 minutes apart. If autoreg is unsuccessful after three tries, the rhsmcertd will attempt no further registration attempts until the service is restarted. (Note: in our cloud images, rhsmcertd is configured to run at boot time, so a restart of the instance restarts this process too)
• manage_repos = 0 - This tells subscription-manager to not manage CDN provided content in /etc/yum.repos.d/redhat.repo. (The default is 1; for systems not connected to RHUI, such as those registered to RHSM directly or Satellite, we want those repos managed by subscription-manager). We made a decision that when auto-registration is enabled we'd not enable CDN content by default, so we told subscription-manager to not do that. A hybrid approach using both RHUI and CDN is fairly uncommon, but we can change that configuration later if needed. However, if you do want to enable a CDN based repo. (like the Optional/CRB repo needed for EPEL), you can (by setting this value back to 1 and enabling the repo), but you absolutely should not have to attach an entitlement to do such. And this is the case if you have Simple Content Access enabled. Note: We will begin defaulting net-new Red Hat accounts to be SCA enabled in the near future (mid-2022).
• splay = 1 - This setting applies a random offset during registration to prevent 'thundering herds'. This distributes load so that a large number of systems started up at roughly the same time don't all check in at the same time.

Additionally, to support auto-registration, the rhsmcertd daemon AND the subscription-manager yum plugins are now enabled in Red Hat's cloud images.

Let's go spelunking into the logs (/var/log/rhsm/rhsmcertd.log)

tail /var/log/rhsm/rhsmcertd.log
Fri Apr  1 10:42:35 2022 [INFO] Starting rhsmcertd...
Fri Apr  1 10:42:35 2022 [INFO] Auto-registration interval: 60.0 minutes [3600 seconds]
Fri Apr  1 10:42:35 2022 [INFO] Auto-attach interval: 1440.0 minutes [86400 seconds]
Fri Apr  1 10:42:35 2022 [INFO] Cert check interval: 240.0 minutes [14400 seconds]
Fri Apr  1 10:42:35 2022 [INFO] Waiting 2.0 minutes plus 3287 splay seconds [3407 seconds total] before performing first auto-register
Fri Apr  1 10:42:35 2022 [INFO] Waiting 2.0 minutes plus 43071 splay seconds [43191 seconds total] before performing first auto-attach.
Fri Apr  1 10:42:35 2022 [INFO] Waiting 2.0 minutes plus 1723 splay seconds [1843 seconds total] before performing first cert check.

We can see that the system is up and running, and will attempt auto-registration within 3407 seconds. (just a little over an hour)

Fri Apr  1 11:44:23 2022 [INFO] (Auto-registration) performed successfully.

subscription-manager grabs a copy of the Instance Metadata document, which is digitally signed by the cloud provider, which contains a lot of information about the instance, including (important to autoregistration) the cloud provider account which this instance is in, presents that to the trust service (our service that maintains the mapping of Red Hat account <-> cloud provider account) to request registration. And remember, we explicitly allowed this by creating a source in our previous stage. Now, let's check to make sure that we properly registered. We can do this with the subscription-manager identity command.

subscription-manager identity
system identity: ca03e0d1-7898-40bf-8cb7-b00faf2b2189
name: ip-172-30-0-68.ec2.internal
org name: 7843216
org ID: 7843216

Let's recap what we've done:

We spun up an image, with a proper version of subscription-manager, that was configured to auto-register
It attempted auto-registration, as directed
And because we configured a source, this registration is allowed.

Great, what should I do next?

Now that have a properly registered host, You can use any of our hosted capabilities:

I want to use Insights.

Simply run insights-client --register to connect the system to Insights. On OS versions where rhc is available, install rhc and run rhc connect. In either case, they will connect with no further prompt for authentication. (and in particular, they reuse the identity certificate that sub-man gets issued).

I want to use content provided by my subscriptions which isn't provided via RHUI.

RHUI provides a subset of the content which is available in the Red Hat CDN. If you need to enable CDN repositories, that's totally supported. You just need to tell subscription-manager "hey, you are back in the business of managing repos". This is done by editing /etc/rhsm/rhsm.conf or (preferred) via subscription-manager config --rhsm.manage_repos=1 (which does the same thing, but more elegantly ) and then issuing the respective command to enable a repo (e.g. subscription-manager repos --enable codeready-builder-for-rhel-8-$(arch)-rpms)

I want to see my usage of RHEL

Visit Subscription Watch. We'll show you your usage of RHEL, broken out by Physical, Virtual, and which Public Cloud your system is running on. Note: Subscription Watch tallies up its inventory once a day, so there may be a delay of ~ 24 hours before a system shows up in subscription reporting.

Frequently Asked Questions

Q: It would be nice to automate the registration to Insights so that the user doesn't have to take an explicit step. Is that on the roadmap?

A: Yes. This is a feature we have on our backlog and will deliver in the near future.

Q: On which clouds do we support autoregistration?

A: at the time of this post (01-Apr-2022), AWS & Azure. Google is inflight and will likely be completed in mid-2022.

Q: What if I don't want to use auto-registration?

A: Don't setup a cloud source.

Q: Why does auto-registration take ~ an hour to happen?

A: it's a balancing act between consistency and load. Remember that auto-registration was originally built to 'establish a data pipeline for reporting & analytics' and not something 'critical path' like access to content. Because we've ensured that yum works out-of-box due to the presence of RHUI, we can handle registration asynchronously. If that paradigm changes, we can revise that model.

Q: Auto-registration sounds awesome. Can I use it with images that are purchased via the cloud provider (such as PAYG or Marketplace)?

A: Yes. PAYG images and Marketplace images are (and will be) configured to use auto-registration.

Q: But I thought registering a system via subscription-manager consumes a subscription? Wouldn't I get 'double-billed'?

A: This is a two-part answer. It is important to separate 'connect a system' from 'use a subscription'. They are no longer synonymous. Simple Content Access made this a reality. We expect customers to adopt SCA in general, and in particular when cloud instances are in use. Secondarily, when the insights-client is in use, we know which systems are marketplace or not and can (not) count them appropriately in Subscription Watch.

Additional Reading

• Subscription Watch
• Simple Content Access