Troubleshooting the Contrast Agent Operator

  • Updated

Issue

All of the documented steps for installing and configuring the Contrast Agent Operator have been followed but the application has not been onboarded.

Cause

There are several possible reasons that an application is not onboarded when using the Contrast Agent Operator.  The first of these to rule out is whether the operator has been correctly installed and configured.  The following steps should help with that:

Are all the necessary resources deployed and in the correct namespaces?

Run the following command:

kubectl get secrets,clusteragentconnections,agentinjectors --all-namespaces

and verify that output looks something like this:

 

These three resources are required at a minimum for the operator to function correctly:

  • The Secret exists and is deployed to the contrast-agent-operator namespace.
  • The ClusterAgentConnection exists and is deployed to the contrast-agent-operator namespace.
  • The AgentInjector exists and is deployed to the same namespace as the application to be instrumented.

Is the application deployment correctly tagged?

The AgentInjector should contain a label definition under spec.selector.labels that indicates to the Agent Operator which deployments should be injected with the agent.

 For example, this kubectl command will display the AgentInjector manifest:

kubectl get agentinjector webgoatdotnetcore -o yaml --namespace default

Note the label definition.  This tells the operator that any deployment tagged with contrast=monitored should be injected.  The name and value here are arbitrary and can be anything you choose.  Glob patterns are also supported.

Now, verify that the application deployment correctly specifies the same label. Note that there are several labels that can be defined in an application deployment - but the important one here is metadata.labels.  You can see the corresponding label here:

kubectl get deployment webgoatdotnetcore -o yaml

The following kubectl command will return the corresponding label unambiguously:

kubectl get deployments --show-labels

Note that, in general, you can review any of the resources currently deployed to K8s, using this command structure:

kubectl get [resource type] [resource name] -o yaml --namespace [namespace]

For example:

kubectl get clusteragentconnection default-agent-connection -o yaml --namespace contrast-agent-operator

Logs

The Agent Operator logs

This deployment runs as a pod in the contrast-agent-operator namespace. You can view or tail logs from this pod to look for problems with agent injectors, configurations, and connections.

This kubectl command will display the logs for the deployment:

kubectl logs -f deployment/contrast-agent-operator --namespace contrast-agent-operator

Here's an example showing the operator startup followed by checking available pods for patching and then a successful injection on a pod:

[2023-06-23 19:48:36.0774 INFO Program] Starting the Contrast Security Agent Operator 1.0.0.0.
[2023-06-23 19:48:37.1566 INFO OptionsLogger] Option 'install-source' was changed from 'unknown' (default) -> 'kustomize'.
[2023-06-23 19:48:37.9095 INFO ApplicationStartup] Registered mutation webhook "contrast.k8s.agentoperator.controllers.v1pod.podmutationwebhook" under "/v1/pods/podmutationwebhook/mutate".
[2023-06-23 19:48:38.2095 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/contrast-agent-operator/contrast-agent-operator-5545877df8-8kjg7' was reconciled.
[2023-06-23 19:48:38.2150 INFO MergingStateProvider] Merging state modified events until '06/23/2023 19:48:48 +00:00'.
[2023-06-23 19:48:38.2222 INFO MatchInjectorsHandler] Reactions are disabled, cluster state is settling or instance is not leading.
[2023-06-23 19:48:38.2398 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/contrast-agent-operator/contrast-agent-operator-6547f5c6d8-x2qxg' was reconciled.
[2023-06-23 19:48:38.2398 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/default/webgoatdotnetcore-6f745d4b6b-7ndsj' was reconciled.
[2023-06-23 19:48:38.2398 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/kube-system/aws-node-2cglk' was reconciled.
[2023-06-23 19:48:38.2398 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/kube-system/aws-node-s8zlp' was reconciled.
[2023-06-23 19:48:38.2398 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/kube-system/coredns-5c5677bc78-m7krb' was reconciled.
[2023-06-23 19:48:38.2398 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/kube-system/coredns-5c5677bc78-szqr7' was reconciled.
[2023-06-23 19:48:38.2398 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/kube-system/kube-proxy-79zl5' was reconciled.
[2023-06-23 19:48:38.2398 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/kube-system/kube-proxy-7zq7k' was reconciled.
[2023-06-23 19:48:38.2575 TRACE ClusterIdHandler] Internal cluster id was updated. (Generated: 2023-02-23T15:49:04.6346830+00:00)
[2023-06-23 19:48:38.2635 DEBUG BaseApplier`2:SecretResource] Resource 'SecretResource/contrast-agent-operator/contrast-cluster-id' was reconciled.
[2023-06-23 19:48:38.2651 DEBUG BaseApplier`2:ClusterAgentConnectionResource] Resource 'ClusterAgentConnectionResource/contrast-agent-operator/default-agent-connection' was reconciled.
.....
.....
[2023-06-23 19:50:26.4766 TRACE BaseSyncingHandler`3:ClusterAgentConnectionResource] Checking for cluster 'AgentConnectionSecret' eligible for generation across 1 templates in 3 namespaces.
[2023-06-23 19:50:26.4766 TRACE BaseSyncingHandler`3:ClusterAgentConnectionResource] Completed checking for entity generation after 5ms.
[2023-06-23 19:50:26.4766 TRACE BaseSyncingHandler`3:ClusterAgentConnectionResource] Checking for cluster 'AgentConnection' eligible for generation across 1 templates in 3 namespaces.
[2023-06-23 19:50:26.4766 TRACE BaseSyncingHandler`3:ClusterAgentConnectionResource] Completed checking for entity generation after 0ms.
[2023-06-23 19:50:26.5361 DEBUG PodMutationWebhook] Admission with method "CREATE".
[2023-06-23 19:50:26.5451 TRACE PodPatcher] Selected agent injector 'DotNetCore'.
[2023-06-23 19:50:26.5543 TRACE GlobMatcher] Compiling glob pattern '*'.
[2023-06-23 19:50:26.5596 INFO PodInjectionHandler] Patching pod from 'default/webgoatdotnetcore' using injector 'default/webgoatdotnetcore'.
[2023-06-23 19:50:26.6298 DEBUG PodMutationWebhook] AdmissionHook "contrast.k8s.agentoperator.controllers.v1pod.podmutationwebhook" did return "True" for "CREATE".
[2023-06-23 19:50:26.6585 DEBUG PodMutationWebhook] Admission with method "CREATE".
[2023-06-23 19:50:26.6585 TRACE PodPatcher] Selected agent injector 'DotNetCore'.
[2023-06-23 19:50:26.6585 INFO PodInjectionHandler] Patching pod from 'default/webgoatdotnetcore' using injector 'default/webgoatdotnetcore'.
[2023-06-23 19:50:26.6643 DEBUG PodMutationWebhook] AdmissionHook "contrast.k8s.agentoperator.controllers.v1pod.podmutationwebhook" did return "True" for "CREATE".
[2023-06-23 19:50:26.6800 DEBUG BaseApplier`2:PodResource] Resource 'PodResource/default/webgoatdotnetcore-569f49b79d-swzmn' was reconciled.
[2023-06-23 19:50:35.9086 DEBUG BaseApplier`2:PodResource] Resource 'default/webgoatdotnetcore-6f745d4b6b-7ndsj' of type 'PodResource' was deleted.
[2023-06-23 19:50:36.5137 TRACE MergingStateProvider] Flushing state modified, 2 events were merged.
[2023-06-23 19:50:36.5137 INFO MatchInjectorsHandler] Cluster state changed, re-calculating injection points (2 changes merged).
[2023-06-23 19:50:36.5137 TRACE MatchInjectorsHandler] Calculating changes needed for 'DeploymentResource/contrast-agent-operator/contrast-agent-operator'...
[2023-06-23 19:50:36.5137 TRACE MatchInjectorsHandler] Calculating changes needed for 'DeploymentResource/default/webgoatdotnetcore'...
[2023-06-23 19:50:36.5137 INFO PodTemplateStatusHandler] Pod 'default/webgoatdotnetcore-569f49b79d-swzmn' status was updated 'None' -> 'InjectionComplete'.
[2023-06-23 19:50:36.5477 TRACE ResourcePatcher] Preparing to patch status 'default/webgoatdotnetcore-569f49b79d-swzmn' ('Pod/v1') with '{"lastTransitionTime":"2023-06-23T19:50:36.516960\u002B00:00","message":"The pod is eligible for agent injection and is currently injected.","reason":"InjectionComplete","status":"True","type":"agents.contrastsecurity.com/injection-converged"}'.
[2023-06-23 19:50:36.6026 TRACE ResourcePatcher] Patch complete after 67ms.

If the operator logs indicate status is still in InjectionPending this article should help in tracking down the issue: Contrast-agent-operator stuck in InjectionPending.

Enabling more verbose logging for the Agent Operator

The log snippet above shows output at the TRACE logging level.  The default is INFO. See How to get logs from the Agent Operator for detail on configuring more detailed operator logging.

Getting cluster event logs

These may provide additional insight into problems when injectors are not working.

kubectl get events

The application deployment logs

You can utilize the regular STDOUT on pods to get an indication of whether the agent was successfully injected.

For example - to show the logs for a given deployment:

kubectl logs -f deployment/webgoatdotnetcore

Or to use the pod name - first, fetch the name:

kubectl get pods

kubectl logs pods/webgoatdotnetcore-569f49b79d-swzmn

An injected pod has two containers - one is the contrast-init container.  You can view its logs like so:

kubectl logs pods/webgoatdotnetcore-569f49b79d-swzmn -c contrast-init

Control plane logging

This may be more difficult to get and requires a cluster administrator. This is logging on the entire cluster, Kubernetes APIs, controllers, schedulers and auditors.

If access is available, you are looking to get recent API server and Control manager logging.

Some things to look for:

  • Kubernetes is unable to contact the operator
    • The issue could be due to non-standard security policies or configurations.
  • Focus on webhook-related errors
  • If there are no webhook-related errors, the issue could, again, be related to security policies or configurations.

Agent Logs

If all of the above checks out and it appears that the agent is being injected successfully, but the application is still not showing up in the Contrast UI, the next place to look would be the agent logs themselves.  Connect a terminal to the running pod - for example:

kubectl exec --stdin --tty [pod name] -- /bin/sh

and the agent logs can be found in /contrast/data/logs.

Related Articles

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request