Monitoring & Metrics Gathering in Contrast On-Premises

Objective

Monitor the performance of an On-Premises Contrast UI (EOP) installation.

Queue sizes

EOP uses message queues and background processing for many tasks. Monitoring the sizes of the message queues may yield useful information on activity and warn of an influx of data or the need for more resources.

To add detailed logging of queue size and performance, $CONTRAST_HOME/data/conf/servo.properties needs to be modified. This is an encrypted properties file, so to edit it, you’ll need to use the Encrypted Properties Editor

Steps:

1. Run the editor:

    $CONTRAST_HOME/bin/edit-properties -e $CONTRAST_HOME/esapi/ -f $CONTRAST_HOME/conf/servo.properties

2. Enter the name of the property to edit:

   servo.activemq.observers

3. Enter a new value for servo.activemq.observers:

    FILE

4. Save the changes

After a restart, queue status should be logged once a minute to $CONTRAST_HOME/data/logs/servo_activemq_<date>.log

Example format:

2019-05-15 12:27:32,082,Active,{Jmx.destinationName=queue.library.update, hostname=Contrast-MBP, ip=192.168.0.30},1

To change the logging interval, you can set the property servo.refresh to a positive integer, representing the number of minutes between each output.

Recommendation: Collect and graph this data, in order to observe trends in your environment. Consider alerting when queue size is above some factor of normal.

Metric Logging

EOP can output metrics about a number of operational details; including, but not limited to database connection pool statistics, response times for API calls & methods, and JVM memory usage.

To make changes to how this data is logged, edit $CONTRAST_HOME/conf/metrics.properties which may not exist so you will need to create it.

Steps:

1. Create $CONTRAST_HOME/conf/metrics.properties
2. The contents of that file should be as follows:

   metrics.enabled=true
   metrics.collect.garbage=true
   metrics.collect.memory=true
   metrics.collect.threads=true
   metrics.slf4j.enabled=true
   metrics.slf4j.pollingPeriod=1

After the contrast-server process has been restarted, there will be new log entries from Slf4jReporter in $CONTRAST_HOME/data/logs/contrast.log with key-value pairs.

Some of these are GUAGE metrics - those detailing memory usage; some are HISTOGRAM metrics - those detailing database connection pool statistics; and some are TIMER metrics - detailing how long it took for a certain task to complete.

(More details are available on the types of metrics available here.)

As with the queue statistics detailed above, our recommendation is to collect and graph this data in order to gather statistics on the normal operation of your instances. 

Just as an FYI: In our SaaS environment, this data is sent off to Graphite with the following additional configuration in metrics.properties: 

metrics.graphite.enabled=true
metrics.graphite.host=localhost
metrics.graphite.port=2003
metrics.graphite.server=UDP
metrics.graphite.protocol=UDP
metrics.hostname=app