Skip to content

feat: forward cluster-autoscaler and capi provider logs to shoebox#5161

Open
ashishmax31 wants to merge 1 commit intomainfrom
new-categories-for-shoebox
Open

feat: forward cluster-autoscaler and capi provider logs to shoebox#5161
ashishmax31 wants to merge 1 commit intomainfrom
new-categories-for-shoebox

Conversation

@ashishmax31
Copy link
Copy Markdown
Collaborator

https://redhat.atlassian.net/browse/ARO-26491

What

This PR allows us to forward two new log categories to from the ocm namespace, capi provider and cluster autoscaler to shoebox.

Why

Customer request.

Testing

Tested in personal dev env. Emitted logs in the comments.

[] Need to update the shoebox manifest(with the new manifest) in all regions before we can promote this to stage or prod.

Copilot AI review requested due to automatic review settings May 7, 2026 06:19
@openshift-ci openshift-ci Bot requested review from raelga and roivaz May 7, 2026 06:19
@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented May 7, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ashishmax31

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Alias filter.shoebox_router
Match ocm.logs
Rule $kubernetes['container_name'] ^(audit-logs|kube-apiserver|kube-controller-manager|kube-scheduler|cloud-controller-manager|csi-driver|snapshot-controller)$ shoeboxlogs true
Rule $kubernetes['container_name'] ^(audit-logs|kube-apiserver|kube-controller-manager|kube-scheduler|cloud-controller-manager|csi-driver|snapshot-controller|manager|cluster-autoscaler)$ shoeboxlogs true
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The capi provider pods's main container is called "manager" which is very generic. But i have added a check to check the pod name to match "capi-provider" so that we don't send logs from arbitrary containers with name "manager".

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Shoebox forwarding support for additional control plane log categories emitted in ocm-* namespaces, specifically cluster-autoscaler and CAPI provider logs, by expanding Fluent Bit routing and Shoebox transformation logic.

Changes:

  • Extend the Shoebox routing rule to include cluster-autoscaler and manager (for CAPI provider) container logs.
  • Add Shoebox category mapping for cluster-autoscaler and infer capi-provider category based on manager container + capi-provider* pod name.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread observability/arobit/deploy/templates/forwarder-configmap.yaml Outdated
Comment thread observability/arobit/deploy/templates/forwarder-configmap.yaml
Comment thread observability/arobit/deploy/templates/forwarder-configmap.yaml
@ashishmax31
Copy link
Copy Markdown
Collaborator Author

Sample logs from testing:

{"date":1778133797.641207,"category":"capi-provider","resourceId":"/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/ARO-RG/PROVIDERS/MICROSOFT.REDHATOPENSHIFT/OPENSHIFTCLUSTERS/PERS-USW3ASNA-MGMT-1/TEST","properties":{"raw":"I0507 06:03:17.640968       1 azuremachine_controller.go:240] \"Reconciling AzureMachine\" logger=\"controllers.AzureMachineReconciler.reconcileNormal\" controller=\"azuremachine\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AzureMachine\" AzureMachine=\"ocm-arohcppers-2q0alucdj30s4cal4h2pkdkk27ue2vuf-asnaraya/asnaraya-np-1-gqv2l-zsdvm\" namespace=\"ocm-arohcppers-2q0alucdj30s4cal4h2pkdkk27ue2vuf-asnaraya\" name=\"asnaraya-np-1-gqv2l-zsdvm\" reconcileID=\"54545cb0-540b-4942-bcb1-f01c7ea97e3b\" x-ms-correlation-request-id=\"cb8ec5c9-ebbc-4611-b051-219ce46de8e2\""},"location":"westus3","time":"2026-05-07T06:03:17.641207571Z","operationName":"Microsoft.RedHatOpenshift/hcpOpenShiftClusters/capi-provider","level":"Informational"}
{"date":1778133797.641237,"category":"capi-provider","resourceId":"/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/ARO-RG/PROVIDERS/MICROSOFT.REDHATOPENSHIFT/OPENSHIFTCLUSTERS/PERS-USW3ASNA-MGMT-1/TEST","properties":{"raw":"I0507 06:03:17.641033       1 azuremachine_controller.go:240] \"Reconciling AzureMachine\" logger=\"controllers.AzureMachineReconciler.reconcileNormal\" controller=\"azuremachine\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AzureMachine\" AzureMachine=\"ocm-arohcppers-2q0alucdj30s4cal4h2pkdkk27ue2vuf-asnaraya/asnaraya-np-1-gqv2l-bhb86\" namespace=\"ocm-arohcppers-2q0alucdj30s4cal4h2pkdkk27ue2vuf-asnaraya\" name=\"asnaraya-np-1-gqv2l-bhb86\" reconcileID=\"2e9b7d59-9207-456f-b7be-846ef0d01ac0\" x-ms-correlation-request-id=\"11544e80-d1a8-4f6c-85e0-dcc25c58fef7\""},"location":"westus3","time":"2026-05-07T06:03:17.64123647Z","operationName":"Microsoft.RedHatOpenshift/hcpOpenShiftClusters/capi-provider","level":"Informational"}
{"date":1778133764.970622,"category":"cluster-autoscaler","resourceId":"/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/ARO-RG/PROVIDERS/MICROSOFT.REDHATOPENSHIFT/OPENSHIFTCLUSTERS/PERS-USW3ASNA-MGMT-1/TEST","properties":{"raw":"I0507 06:02:44.970504       1 static_autoscaler.go:275] Starting main loop"},"location":"westus3","time":"2026-05-07T06:02:44.970621512Z","operationName":"Microsoft.RedHatOpenshift/hcpOpenShiftClusters/cluster-autoscaler","level":"Informational"}
{"date":1778133764.979377,"category":"cluster-autoscaler","resourceId":"/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/ARO-RG/PROVIDERS/MICROSOFT.REDHATOPENSHIFT/OPENSHIFTCLUSTERS/PERS-USW3ASNA-MGMT-1/TEST","properties":{"raw":"I0507 06:02:44.979277       1 clusterapi_controller.go:770] discovered node group: MachineDeployment/ocm-arohcppers-2q0alucdj30s4cal4h2pkdkk27ue2vuf-asnaraya/asnaraya-np-1 (min: 2, max: 4, replicas: 2)"},"location":"westus3","time":"2026-05-07T06:02:44.979377163Z","operationName":"Microsoft.RedHatOpenshift/hcpOpenShiftClusters/cluster-autoscaler","level":"Informational"}

Signed-off-by: Ashish <asnaraya@redhat.com>
@ashishmax31 ashishmax31 force-pushed the new-categories-for-shoebox branch from 9847c88 to b4c12fc Compare May 7, 2026 06:34
@ashishmax31
Copy link
Copy Markdown
Collaborator Author

/hold

@ashishmax31
Copy link
Copy Markdown
Collaborator Author

Do not merge until the shoebox manifest is updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants