Queue sizes
TeamServer uses message queues and background processing for many tasks. Monitoring the sizes of the message queues may yield useful information on activity and warn of an influx of data or the need for more resources.
Outputting info about this is controlled by editing $CONTRAST_HOME/conf/servo.properties, which is an encrypted properties file, so to edit it, you’ll need to use the Encrypted Editor
Steps:
- Run the encrypted editor:
$CONTRAST_HOME/bin/edit-properties -e $CONTRAST_HOME/esapi/ -f $CONTRAST_HOME/conf/servo.properties
- Enter the name of the property to edit:
servo.activemq.observers
- Enter a new value for servo.activemq.observers:
FILE
- Save the changes (to all TeamServer nodes if using a distributed setup)
After a restart, queue status should be logged once a minute to $CONTRAST_HOME/data/logs/servo_activemq_<date>.log
Example format:
2019-05-15 12:27:32,082,Active,{Jmx.destinationName=queue.library.update, hostname=Contrast-MBP, ip=192.168.0.30},1
To change the logging interval, you can set the property servo.refresh to a positive integer, representing how many minutes between each output.
Recommendation: Collect and graph this data, in order to observe trends in your environment. Consider alerting when queue size is above some factor of normal.
Metric Logging
TeamServer can output metrics about a number of operational details; including, but not limited to database connection pool statistics, response times for API calls & methods, and JVM memory usage.
Outputting this info is controlled by editing $CONTRAST_HOME/conf/metrics.properties which may not exist in your current TeamServer setup.
Steps:
- Create $CONTRAST_HOME/conf/metrics.properties (on each TeamServer if using a distributed setup)
- The contents of that file should be as follows:metrics.enabled=true
metrics.collect.garbage=true
metrics.collect.memory=true
metrics.collect.threads=true
metrics.slf4j.enabled=true
metrics.slf4j.pollingPeriod=1
After the TeamServer process has been restarted, there will be new log entries from Slf4jReporter in $CONTRAST_HOME/data/logs/contrast.log with key value pairs.
Some of these are GUAGE metrics, e.g. those detailing memory usage; some are HISTOGRAM metrics, e.g. those detailing database connection pool statistics; and some are TIMER metrics, detailing how long it took for a certain task to complete.
(More details are available on the types of metrics available here.)
As with the queue statistics detailed above, our recommendation is to collect and graph this data in order to gather statistics about normal operation of your TeamServer(s).
In our SaaS environment, these are sent off to Graphite with the following additional configuration in metrics.properties:
metrics.graphite.enabled=true
metrics.graphite.host=localhost
metrics.graphite.port=2003
metrics.graphite.server=UDP
metrics.graphite.protocol=UDP
metrics.hostname=app