How To enable logging to monitor queue sizes and gather metrics in Contrast On-Premises

Objective

Monitor the performance of an On-Premises Contrast UI (EOP) installation.

Queue sizes

EOP uses message queues and background processing for many tasks. Monitoring the sizes of the message queues may yield useful information on activity and warn of an influx of data or the need for more resources.

To add detailed logging of queue size and performance, $CONTRAST_HOME/data/conf/servo.properties needs to be modified. This is an encrypted properties file, so to edit it, you’ll need to use the Encrypted Properties Editor

Steps:

1. Run the editor from the $CONTRAST_HOME directory:

    bin/edit-properties -e data/esapi -f data/conf/servo.properties

2. Add the following three new properties with these values:

   servo.activemq.observers: FILE
   servo.activemq.file.metrics: QueueSize
   servo.activemq.file.zerovaluemetrics: true

3. Save the changes

After a restart, queue status should be logged once a minute (but only when the value is nonzero, because of the servo.activemq.file.zerovaluemetrics setting) to $CONTRAST_HOME/data/logs/servo_activemq_<date>.log

Example format:

2019-05-15 12:27:32,082,Active,{Jmx.destinationName=queue.library.update, hostname=Contrast-MBP, ip=192.168.0.30},1

The final number in the log entry is the size of the corresponding queue (in the example above, the library.update queue has one entry).

To change the logging interval, you can set the property servo.refresh to a positive integer, representing the number of minutes between each output.

Recommendation: Collect and graph this data, in order to observe trends in your environment. Consider alerting when queue size is above some factor of normal.

Metric Logging

EOP can output metrics about a number of operational details; including, but not limited to database connection pool statistics, response times for API calls & methods, and JVM memory usage.

To make changes to how this data is logged, edit $CONTRAST_HOME/conf/metrics.properties which may not exist so you will need to create it.

Steps:

1. Create $CONTRAST_HOME/conf/metrics.properties
2. The contents of that file should be as follows:

   metrics.enabled=true
   metrics.collect.garbage=true
   metrics.collect.memory=true
   metrics.collect.threads=true
   metrics.slf4j.enabled=true
   metrics.slf4j.pollingPeriod=1

After the contrast-server process has been restarted, there will be new log entries from Slf4jReporter in $CONTRAST_HOME/data/logs/contrast.log with key-value pairs.

Some of these are GUAGE metrics - those detailing memory usage; some are HISTOGRAM metrics - those detailing database connection pool statistics; and some are TIMER metrics - detailing how long it took for a certain task to complete.

(More details are available on the types of metrics available here.)

As with the queue statistics detailed above, our recommendation is to collect and graph this data in order to gather statistics on the normal operation of your instances. 

Just as an FYI: In our SaaS environment, this data is sent off to Graphite with the following additional configuration in metrics.properties: 

metrics.graphite.enabled=true
metrics.graphite.host=localhost
metrics.graphite.port=2003
metrics.graphite.server=UDP
metrics.graphite.protocol=UDP
metrics.hostname=app