Overview
This article aims to provide guidance for users of Contrast On-Premises who are encountering performance issues.
Performance issues may manifest in several ways, including a slow UI, connectivity timeouts in agent logs, CPU or memory exhaustion on the system running the Contrast UI or, in extreme cases, crashes.
Background
The Contrast UI consists of two main components: the Server and the Database.
The Contrast Server is a highly transactional system that presents calculated and real-time data sets back to consumers of the data. Consumers of data include both users of the UI (including API end-users) and Contrast Agents. The more consumers that interface with the system, the greater the demand for computing and memory.
All persistent data is stored in the Database, and requested updates to this data (from both end-users and Agents) are predominantly queued (using ActiveMQ) by the Contrast Server and serviced on a first-in-first-out basis as resources become available.
As should be clear, resource usage by the Contrast Server and Database (whether it be memory, CPU or concurrent threading) can become a delicate balancing act when things get busy.
There are several options for tuning both the Contrast UI Server and Database as detailed below:
Contrast UI Server considerations
Allocating Memory
By default, the Contrast UI Server (which runs on Java in an embedded Tomcat-based application server) will allocate 50% of available system memory to the Java heap.
If system memory is plentiful, it may be worthwhile to increase this allocation to up to 75% of system memory.
You can increase this by editing $CONTRAST_HOME/bin/contrast-server.vmoptions to set -XX:MaxRAMPercentage to 75.0. (Restart the server for this change to take effect.)
Tuning the ActiveMQ Queues
As mentioned earlier, the vast majority of requests made to the Contrast UI are queued and then processed by dedicated consumers as resources become available.
There are several queues to be managed. To enable additional logging which provides visibility into how large and how quickly the various queues are being serviced, see this article: How To enable logging to monitor queue sizes and gather metrics in Contrast On-Premises.
Once this logging is enabled, you will be able to see which - if any - of the queues are problematic in that they build up quicker than they can be depleted.
If performance of a particular queue is a concern, then a possible remediation is to increase the number of consumers of that queue by adding a JVM option to $CONTRAST_HOME/bin/contrast-server.vmoptions.
Some common queues and the corresponding JVM option to modify the consumers are shown in the table below:
| Queue | JVM Option | Default Value | Suggested Value |
|---|---|---|---|
| queue.routecoverage | -Djms.channel.concurrency.routeCoverage | 2 | 4 or more |
| queue.route-coverage.vulnerability.auto-verification | -Djms.channel.concurrency.routeCoverageVulns | 1 | 2 or more |
| queue.ng.update.app | -Djms.channel.concurrency.applicationUpdate | 2 | 4 or more |
Redis Cache
To reduce load on the database, it's possible to set up a cache which will offset common read-only queries by pulling the result from the cache rather than making a query to the database. Some guidance on configuring that option is provided in our documentation here: Use Redis as a shared cache.
Database Considerations
Embedded vs. Distributed
The simplest Contrast On-Premises installation consists of both the Contrast Server and Database residing on the same physical system.
This configuration is ideal for a Proof of Concept or for a system where it is expected that very few users and Contrast Agents will be using the Contrast UI. More detail is included in our documentation here, but in general if either of the following is true:
- 100 or more Contrast Agents have been deployed or continuous integration of agents with large automated regression suites is in use
- You are using the Contrast REST API architecture for automation or data extraction purposes
it is highly recommended that a distributed installation of Contrast should be used.
At a minimum, hosting the database on a separate server allows for much more control over resource allocation and tuning of both the Server and the Database. Additionally, configuring two or more Contrast UI Server nodes and balancing load between them provides for the ability to, for example, direct end-user traffic to a dedicated system and use another to service Agent requests.
Memory and CPU Usage
For the most efficient use of memory and CPU, it is recommended that the following three configuration options (found in the MySQL configuration file my.cnf) are set as described:
innodb_buffer_pool_size - set to 80% of available memory
innodb_buffer_pool_instances - set to same as number of CPUs
innodb_sync_array_size - set to same as number of CPUs
Connection Pool Size
This article: How to modify the size of the database connections pool details how to tune the size of the connections pools used by the Contrast UI Server.
Swap Space Issues
If the system hosting the database uses excessive amounts of swap space see this article: Why is my On-Premises Contrast Distributed Database Server using over 99% of its swap space?
Additional Tuning Options
In addition to the above, it is possible to split message queue (ActiveMQ) processing to a dedicated server to free up resources on the Contrast UI Server node(s), and to configure a replica database to service all read-only requests to further take load from the database.
For further details please contact support@contrastsecurity.com or open a ticket directly in our portal.