WHAT IS HAPPENING?

Some Red Hat OpenShift Container Platform clusters will see their reported size double in the Subscriptions service. This change does not indicate a scaling event on the customer side, it is a correction in the data pipeline that feeds the Subscriptions service.

WHEN IS THIS HAPPENING?
Changes were implemented on Thursday, December 14th. The initial Subscriptions service tallies that include these changes will be reflected on Friday, December 15th.

WHO IS AFFECTED BY THIS CHANGE?

Customers running the OpenShift Container Platform in a self-managed, virtualized environment on x86 architectures may be impacted.
-Clusters running on hypervisors that expose multithreading CPU behavior to hosted virtual machines SHOULD see cluster sizes increase.
-Observed behavior on public clouds is empirically variable, based on a variety of factors including cloud/region combinations.

WHY IS THIS HAPPENING?

Default hypervisor settings on several major virt engines do NOT expose multithreading information to the Linux kernel when hardware is virtualized. In response to this, Red Hat chose to assume a customer-favored stance that assumed a multithreading factor of "2" for virtualized, x86 deployments (something that is predominantly true). This adjustment was applied by the Subscriptions service beginning in November of 2021.

In November of 2022, the OpenShift metric used by the Subscriptions service began to interpret multithreading information when it was present, rather than reporting by rote all observed vCPUs as individual logical handlers. At this point, multithreading was being applied TWICE in scenarios where it was previously applied only by the Subscriptions service. Analysis of erroneous cluster sizing began as it was reported in support cases in internal, QE testing scenarios.

WHAT SOLUTION IS BEING IMPLEMENTED

The OpenShift Cluster Manager is reporting forward the threads_per_core interpretation applied by OpenShift, informing the Subscriptions service when multithreading size corrections are already in place. When the Subscriptions service sees a threads_per_core metric >1 for virtualized, x86 deployments, it will apply no further multithreading adjustments.

Net effect:
-Virtualized, x86 OpenShift Container Platform clusters with full multithreading information exposed by the hypervisor will be sized accurately within OCM and passed forward "as-is" to the Subscriptions service
-Virtualized, x86 OpenShift Container Platform clusters with no or insufficient multithreading information present will be sized using "vCPUs" in OCM, and will have assumed multithreading applied by the Subscriptions service.

In all cases, assumed multithreading for x86 chips will be applied. In no case will it be applied twice.

HOW ARE WE PREVENTING THIS FROM HAPPENING AGAIN?
By way of longer range solutions, cluster size metrics aimed specifically at the "subscribed portion of a cluster" are being introduced and kept distinct from more generalized cluster size metrics. "subscription_cpu_total" and "subscription_socket_total" metrics will be maintained, governed, and tested specifically for transacting OpenShift subscriptions.

WILL THIS BE REFLECTED IN OCM AND IN OpenShift DOCS?
It will! The docs work is captured in Red Hat's public-facing Jira instance here: BIZ-655 . You can also find more information on how the OCM and Subscriptions service interfaces are improving their precision in language and presentation here: BIZ-654

WHAT IF I HAVE FURTHER QUESTIONS?
Your account team can help you understand the right subscription profile for your OpenShift clusters, and will work with you to get that profile in place. If you do not have an account representative, Customer Service can help you out.

UPDATE FOR 18/MAR/2024

The December 14th update to the Subscription Usage service’s counting logic exposed some gaps in our data pipeline, wherein certain metrics available at the cluster level were being inadvertently impacted by node-level attributes intended to help influence ‘Is the cluster multithreading?’ logic. To mitigate this, we have created a new, service-level metric that specifically tracks “subscription obligation in cores” and is isolated from other cluster size metrics more focused on operational status. We have also identified and closed a gap in QE testing that allowed the December 14th flaws to creep in.

When calculating the subscription obligation, there are challenges in identifying - at a node level - our ability to know whether the cores are multithreading or not. Future versions of OpenShift may receive enhancements to address this inability, but in an effort to rectify inaccuracies in the December 14th release we have opted to apply an assumption that is by and large accurate, with inaccuracies that are customer-favored. With changes released on 18/Mar/2024, current behavior is:

Treat all nodes showing "hyperthreaded=false" as "needs adjustment," apply an assumption of SMT = 2, and divide the observed CPU count by 2.

The simplest interpretation of this assumption is:
“Any core that is in fact NOT multithreading will still be counted as .5 cores.”

SOME NUMBERS CONNECTED TO OUR ASSUMPTION
-There are 31,141 cluster under observation, including managed service clusters (ROSA, OSD, ARO)
-Of these 20,145 (65.5%) have a config that makes determining multithreading impossible.
-For entirely physical clusters where we have STRONG multithreading detection, 10.2% of schedulable cores are not multithreading. Mostly physical deployments represent about 15% of observed clusters, so ~10% of ~15% (or ~1.5%) of cores will be incorrect.
-Based on best practices, we believe that virtualized environments are more likely to use multithreading, with an overall “miss” rate smaller than 10.2%

SUMMARY OF CURRENT-STATE COUNTING LOGIC
On March 19th 2024, all customers should see Subscription Usage service tallies that reflect a universally applied, single assumption of multithreading at a factor of 2. This is very close to what was intended with our December 14th release, with the main deviation being “about 10% of physical cores that are highly likely to be single-threading will be counted as multithreading x2."

The Subscription Management program appreciates everyone’s patience as we designed, developed, tested, and deployed a new metric that did not require updates to OpenShift, and integrated cleanly with our managed OpenShift offerings.