Debugging the RabbitMQ Kubernetes Operators

This information describes how to debug a running instance of the RabbitMQ Kubernetes Operators.

Retrieve Information about CPU/Memory Usage for the Kubernetes Operator Pods

Important: Do not complete the following steps on a production system.

By using the pprof tool, you can expose CPU and memory profiling data for the Kubernetes Operator Pods. Profiling is a debugging technique used to generate data about how a piece of software is running by exposing information about the software's consumption of memory, CPU, and asynchronicity.

You might want to do this if you are seeing high resource consumption on one of your Operator Pods for example. To use the pprof tool, enable it by completing the following steps:

1. Enable the ENABLE_DEBUG_PPROF variable on the operator that you want to retrieve debugging information from by running the following command. For example, for the Cluster Operator, run:

$ kubectl -n rabbitmq-system set env deployment/rabbitmq-cluster-operator ENABLE_DEBUG_PPROF=True
deployment.apps/rabbitmq-cluster-operator env updated

2. Using kubectl, complete a port-forward operation so that metrics can be collected on your machine from the correct port on the Operator Pod. For the RabbitMQ Cluster Operator, the default port is 9782 and for all other operators, the port is 8080. For example, to complete the port-forward operation on the RabbitMQ Cluster Operator Pod, run:

$ kubectl -n rabbitmq-system port-forward deployment/rabbitmq-cluster-operator 9782
Forwarding from 127.0.0.1:9782 -> 9782
Forwarding from [::1]:9782 -> 9782

3. In a separate terminal, you can now use the go tool pprof to profile the Operator Pod. For example, to analyse memory allocations in the Pod, run:

$ go tool pprof "localhost:9782/debug/pprof/heap"
Fetching profile over HTTP from http://localhost:9782/debug/pprof/heap
Saved profile in /home/pprof/pprof.manager.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz

This opens a browser window to visualise the memory allocations in the profile. For more information on how to use pprof, see here.

The following tables lists the profiles that are exposed by the Operators.

Path	Profile Description
/debug/pprof	A list of all the profiles that are available on the system.
/debug/pprof/allocs	A sample of all past memory allocations.
/debug/pprof/block	Stack traces that led to blocking on synchronization primitives.
/debug/pprof/cmdline	The command line invocation of the current program.
/debug/pprof/goroutine	Stack traces of all current goroutines.
/debug/pprof/heap	A sample of memory allocations of live objects. You can specify the gcGET parameter to run GC before taking the heap sample. For example: /debug/pprof/heap?gc=1
/debug/pprof/mutex	Stack traces of holders of contended mutexes.
/debug/pprof/profile	CPU profile. You can specify the duration in the seconds GET parameter. For example: /debug/pprof/profile?seconds=5
/debug/pprof/threadcreate	Stack traces that lead to new OS threads being created.
/debug/pprof/trace	A trace of the execution of the current program. You can specify the duration in the seconds GET parameter. For example: /debug/pprof/trace?seconds=5