Procedure

1. Install the Red Hat build of OpenTelemetry Operator:

   1. Go to Operators → OperatorHub and search for Red Hat build of OpenTelemetry Operator.

   2. Select the Red Hat build of OpenTelemetry Operator that is provided by Red Hat → Install → Install → View Operator.

      Important

      This installs the Operator with the default presets:

      • Update channel → stable
      • Installation mode → All namespaces on the cluster
      • Installed Namespace → openshift-operators
      • Update approval → Automatic
   3. In the Details tab of the installed Operator page, under ClusterServiceVersion details, verify that the installation Status is Succeeded.
2. Create a project of your choice for the OpenTelemetry Collector instance that you will create in the next step by going to Home → Projects → Create Project.

3. Create an OpenTelemetry Collector instance.

   1. Go to Operators → Installed Operators.
   2. Select OpenTelemetry Collector → Create OpenTelemetry Collector → YAML view.

   3. In the YAML view, customize the OpenTelemetryCollector custom resource (CR) with the OTLP, Jaeger, Zipkin receivers and the debug exporter.

      apiVersion: opentelemetry.io/v1alpha1
      kind: OpenTelemetryCollector
      metadata:
        name: otel
        namespace: <project_of_opentelemetry_collector_instance>
      spec:
        mode: deployment
        config: |
          receivers:
            otlp:
              protocols:
                grpc:
                http:
            jaeger:
              protocols:
                grpc:
                thrift_binary:
                thrift_compact:
                thrift_http:
            zipkin:
          processors:
            batch:
            memory_limiter:
              check_interval: 1s
              limit_percentage: 50
              spike_limit_percentage: 30
          exporters:
            debug:
          service:
            pipelines:
              traces:
                receivers: [otlp,jaeger,zipkin]
                processors: [memory_limiter,batch]
                exporters: [debug]

   4. Select Create.

Verification

1. Use the Project: dropdown list to select the project of the OpenTelemetry Collector instance.
2. Go to Operators → Installed Operators to verify that the Status of the OpenTelemetry Collector instance is Condition: Ready.
3. Go to Workloads → Pods to verify that all the component pods of the OpenTelemetry Collector instance are running.

Procedure

1. Install the Red Hat build of OpenTelemetry Operator:

   1. Create a project for the Red Hat build of OpenTelemetry Operator by running the following command:

      $ oc apply -f - << EOF
      apiVersion: project.openshift.io/v1
      kind: Project
      metadata:
        labels:
          kubernetes.io/metadata.name: openshift-opentelemetry-operator
          openshift.io/cluster-monitoring: "true"
        name: openshift-opentelemetry-operator
      EOF

   2. Create an Operator group by running the following command:

      $ oc apply -f - << EOF
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-opentelemetry-operator
        namespace: openshift-opentelemetry-operator
      spec:
        upgradeStrategy: Default
      EOF

   3. Create a subscription by running the following command:

      $ oc apply -f - << EOF
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: opentelemetry-product
        namespace: openshift-opentelemetry-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: opentelemetry-product
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      EOF

   4. Check the Operator status by running the following command:

      $ oc get csv -n openshift-opentelemetry-operator

2. Create a project of your choice for the OpenTelemetry Collector instance that you will create in a subsequent step:

   • To create a project without metadata, run the following command:

     $ oc new-project <project_of_opentelemetry_collector_instance>

   • To create a project with metadata, run the following command:

     $ oc apply -f - << EOF
     apiVersion: project.openshift.io/v1
     kind: Project
     metadata:
       name: <project_of_opentelemetry_collector_instance>
     EOF

3. Create an OpenTelemetry Collector instance in the project that you created for it.

   Note

   You can create multiple OpenTelemetry Collector instances in separate projects on the same cluster.

   1. Customize the OpenTelemetry Collector custom resource (CR) with the OTLP, Jaeger, and Zipkin receivers and the debug exporter:

      apiVersion: opentelemetry.io/v1alpha1
      kind: OpenTelemetryCollector
      metadata:
        name: otel
        namespace: <project_of_opentelemetry_collector_instance>
      spec:
        mode: deployment
        config: |
          receivers:
            otlp:
              protocols:
                grpc:
                http:
            jaeger:
              protocols:
                grpc:
                thrift_binary:
                thrift_compact:
                thrift_http:
            zipkin:
          processors:
            batch:
            memory_limiter:
              check_interval: 1s
              limit_percentage: 50
              spike_limit_percentage: 30
          exporters:
            debug:
          service:
            pipelines:
              traces:
                receivers: [otlp,jaeger,zipkin]
                processors: [memory_limiter,batch]
                exporters: [debug]

   2. Apply the customized CR by running the following command:

      $ oc apply -f - << EOF
      <OpenTelemetryCollector_custom_resource>
      EOF

Verification

1. Verify that the status.phase of the OpenTelemetry Collector pod is Running and the conditions are type: Ready by running the following command:

   $ oc get pod -l app.kubernetes.io/managed-by=opentelemetry-operator,app.kubernetes.io/instance=<namespace>.<instance_name> -o yaml

2. Get the OpenTelemetry Collector service by running the following command:

   $ oc get service -l app.kubernetes.io/managed-by=opentelemetry-operator,app.kubernetes.io/instance=<namespace>.<instance_name>

Procedure

1. Create a project for an OpenTelemetry Collector instance.

   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: observability

2. Create a service account.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-sidecar
     namespace: observability

3. Grant the permissions to the service account for the k8sattributes and resourcedetection processors.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-sidecar
     namespace: observability
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

4. Deploy the OpenTelemetry Collector as a sidecar.

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: observability
   spec:
     serviceAccount: otel-collector-sidecar
     mode: sidecar
     config: |
       serviceAccount: otel-collector-sidecar
       receivers:
         otlp:
           protocols:
             grpc:
             http:
       processors:
         batch:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
           timeout: 2s
       exporters:
         otlp:
           endpoint: "tempo-<example>-gateway:8090" 1
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger]
             processors: [memory_limiter, resourcedetection, batch]
             exporters: [otlp]

   1

   This points to the Gateway of the TempoStack instance deployed by using the <example> Tempo Operator.

5. Create your deployment using the otel-collector-sidecar service account.
6. Add the sidecar.opentelemetry.io/inject: "true" annotation to your Deployment object. This will inject all the needed environment variables to send data from your workloads to the OpenTelemetry Collector instance.

Procedure

1. Create a project for an OpenTelemetry Collector instance.

   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: observability

2. Create a service account.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment
     namespace: observability

3. Grant the permissions to the service account for the k8sattributes and resourcedetection processors.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector
     namespace: observability
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

4. Deploy the OpenTelemetry Collector instance with the OpenTelemetryCollector custom resource.

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: observability
   spec:
     mode: deployment
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
         opencensus:
         otlp:
           protocols:
             grpc:
             http:
         zipkin:
       processors:
         batch:
         k8sattributes:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlp:
           endpoint: "tempo-<example>-distributor:4317" 1
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger, opencensus, otlp, zipkin]
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]

   1

   This points to the Gateway of the TempoStack instance deployed by using the <example> Tempo Operator.

5. Set the environment variables in the container with your instrumented application.

   Name	Description	Default value
   OTEL_SERVICE_NAME	Sets the value of the service.name resource attribute.	""
   OTEL_EXPORTER_OTLP_ENDPOINT	Base endpoint URL for any signal type with an optionally specified port number.	https://localhost:4317
   OTEL_EXPORTER_OTLP_CERTIFICATE	Path to the certificate file for the TLS credentials of the gRPC client.	https://localhost:4317
   OTEL_TRACES_SAMPLER	Sampler to be used for traces.	parentbased_always_on
   OTEL_EXPORTER_OTLP_PROTOCOL	Transport protocol for the OTLP exporter.	grpc
   OTEL_EXPORTER_OTLP_TIMEOUT	Maximum time interval for the OTLP exporter to wait for each batch export.	10s
   OTEL_EXPORTER_OTLP_INSECURE	Disables client transport security for gRPC requests. An HTTPS schema overrides it.	False

Procedure

1. Set the relevant log level in the OpenTelemetryCollector custom resource (CR):

     config: |
       service:
         telemetry:
           logs:
             level: debug 1

   1

   Collector’s log level. Supported values include info, warn, error, or debug. Defaults to info.

2. Use the oc logs command or the web console to retrieve the logs.

Procedure

1. Enable the telemetry service by adding the following lines in the OpenTelemetryCollector custom resource:

     config: |
       service:
         telemetry:
           metrics:
             address: ":8888" 1

   1

   The address at which the internal collector metrics are exposed. Defaults to :8888.

1. Retrieve the metrics by running the following command, which uses the port-forwarding Collector pod:

   $ oc port-forward <collector_pod>

2. Access the metrics endpoint at http://localhost:8888/metrics.

Procedure

1. Configure the OpenTelemetryCollector custom resource as follows:

     config: |
       exporters:
         debug:
           verbosity: detailed
       service:
         pipelines:
           traces:
             exporters: [debug]
           metrics:
             exporters: [debug]
           logs:
             exporters: [debug]

2. Use the oc logs command or the web console to export the logs to the standard output.

Procedure

1. Configure the OpenTelemetry Collector as a sidecar.

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: <otel-collector-namespace>
   spec:
     mode: sidecar
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
       processors:
         batch:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
           timeout: 2s
       exporters:
         otlp:
           endpoint: "tempo-<example>-gateway:8090" 1
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger]
             processors: [memory_limiter, resourcedetection, batch]
             exporters: [otlp]

   1

   This endpoint points to the Gateway of a TempoStack instance deployed by using the <example> Tempo Operator.

2. Create a service account for running your application.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-sidecar

3. Create a cluster role for the permissions needed by some processors.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector-sidecar
   rules:
     1
   - apiGroups: ["config.openshift.io"]
     resources: ["infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   The resourcedetectionprocessor requires permissions for infrastructures and infrastructures/status.

4. Create a ClusterRoleBinding to set the permissions for the service account.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector-sidecar
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: otel-collector-example
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

5. Deploy the OpenTelemetry Collector as a sidecar.
6. Remove the injected Jaeger Agent from your application by removing the "sidecar.jaegertracing.io/inject": "true" annotation from your Deployment object.
7. Enable automatic injection of the OpenTelemetry sidecar by adding the sidecar.opentelemetry.io/inject: "true" annotation to the .spec.template.metadata.annotations field of your Deployment object.
8. Use the created service account for the deployment of your application to allow the processors to get the correct information and add it to your traces.

Procedure

1. Configure OpenTelemetry Collector deployment.

2. Create the project where the OpenTelemetry Collector will be deployed.

   apiVersion: project.openshift.io/v1
   kind: Project
   metadata:
     name: observability

3. Create a service account for running the OpenTelemetry Collector instance.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: otel-collector-deployment
     namespace: observability

4. Create a cluster role for setting the required permissions for the processors.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: otel-collector
   rules:
     1
     2
   - apiGroups: ["", "config.openshift.io"]
     resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
     verbs: ["get", "watch", "list"]

   1

   Permissions for the pods and namespaces resources are required for the k8sattributesprocessor.

   2

   Permissions for infrastructures and infrastructures/status are required for resourcedetectionprocessor.

5. Create a ClusterRoleBinding to set the permissions for the service account.

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: otel-collector
   subjects:
   - kind: ServiceAccount
     name: otel-collector-deployment
     namespace: observability
   roleRef:
     kind: ClusterRole
     name: otel-collector
     apiGroup: rbac.authorization.k8s.io

6. Create the OpenTelemetry Collector instance.

   Note

   This collector will export traces to a TempoStack instance. You must create your TempoStack instance by using the Red Hat Tempo Operator and place here the correct endpoint.

   apiVersion: opentelemetry.io/v1alpha1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
     namespace: observability
   spec:
     mode: deployment
     serviceAccount: otel-collector-deployment
     config: |
       receivers:
         jaeger:
           protocols:
             grpc:
             thrift_binary:
             thrift_compact:
             thrift_http:
       processors:
         batch:
         k8sattributes:
         memory_limiter:
           check_interval: 1s
           limit_percentage: 50
           spike_limit_percentage: 30
         resourcedetection:
           detectors: [openshift]
       exporters:
         otlp:
           endpoint: "tempo-example-gateway:8090"
           tls:
             insecure: true
       service:
         pipelines:
           traces:
             receivers: [jaeger]
             processors: [memory_limiter, k8sattributes, resourcedetection, batch]
             exporters: [otlp]

7. Point your tracing endpoint to the OpenTelemetry Operator.

8. If you are exporting your traces directly from your application to Jaeger, change the API endpoint from the Jaeger endpoint to the OpenTelemetry Collector endpoint.

   Example of exporting traces by using the jaegerexporter with Golang

   exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(url))) 1

   1

   The URL points to the OpenTelemetry Collector API endpoint.

Procedure

1. Go to Operators → Installed Operators → Red Hat build of OpenTelemetry Operator → OpenTelemetryInstrumentation or OpenTelemetryCollector.
2. To remove the relevant instance, select → Delete …​ → Delete.
3. Optional: Remove the Red Hat build of OpenTelemetry Operator.

Procedure

1. Get the name of the OpenTelemetry Collector instance by running the following command:

   $ oc get deployments -n <project_of_opentelemetry_instance>

2. Remove the OpenTelemetry Collector instance by running the following command:

   $ oc delete opentelemetrycollectors <opentelemetry_instance_name> -n <project_of_opentelemetry_instance>

3. Optional: Remove the Red Hat build of OpenTelemetry Operator.