Node.js - Runtime Performance Troubleshooting

Prerequisite

• You've gone through the Assessing your Resource Usage and Request Processing Time sections in FAQ: Does the Node.js agent affect my application's performance?

Objective

Running the Contrast Security agent along with a Node.js application (and any other technology actually) will always provide a little bit of an overhead resulting in increased Memory and CPU usage and request latency. As we know this is inevitable because we are running additional code - the agent’s code along with the code of the given application. We at Contrast Security are constantly working to minimize this footprint of our agent and we’ve gone to great lengths in this task, especially in the last year. Nevertheless, the increase in resource usage can prove to be troublesome for some testing and/or production environments, so here are steps to gather performance logs for Contrast Support.

Table of Contents

1. Identify the Troublesome "Phase"
2. Latency
   1. Route-metrics Logs
   2. Performance Logs
3. Resources (CPU/Memory)
   1. Flame Graphs
   2. Memory Heap Dumps

Process

Identify the Troublesome "Phase"

We can divide the application life into two phases - application start-up and handling requests.
At the start of each new application, the agent is performing a rewriting of some of the code provided so it can be instrumented. And during the handling requests phase, the agent is tracking each request made. There is a possibility of performance issues in both of the phases. Here, we will talk only about performance issues in the second phase - when your application is handling requests. First, you can identify at what “phase” of your application your issue occurs. If it is in the start-up phase, please visit Node.js - Startup Troubleshooting. If it is in the handling requests phase, continue reading below.

Our default configuration is made such as to be suitable for a common type of customer and if you have a more specific case it might need a little tweaking. To do this we suggest relying on the experience of our support team and providing us with collected performance data that we will use to tailor a configuration that will answer your needs. If you are using some process monitoring tool - be sure to send us the logs from it. Other than that here are a few steps that you can take that will help us solve the issue you are facing:

Latency

If your performance issues are related to latency, please gather the following route-metrics logs and performance logs and open a ticket with customer support for help.

Route-metrics Logs:

1. Install the library to your project with npm install @contrast/route-metrics

2. Start your application, but requiring @contrast/route-metrics first. Make sure you require the @contrast/route-metrics library before any other. If, for example, your usual application start script with the agent is node -r @contrast/agent index.js then to start it with @contrast/route-metrics you should change it to:

   node -r @contrast/route-metrics -r @contrast/agent index.js

3. If no other configuration to @contrast/route-metrics is done a file named route-metrics.log should appear in your application’s working directory.

4. Run your test suite or exercise the routes that you are having an issue with.

5. Data about the requests routes, methods, and latency will be written to the route-metrics.log.

   Rename or move the file into another directory and repeat steps 2-4 but this time run your application without the agent so we can have a baseline of your routes performance

6. Send the two log files to our team for analysis.

Performance Logs:

1. Enable DEBUG level of logging for the agent. This can be done either by changing the agent.logger.level to DEBUG in the agent’s configuration .yaml file or by setting an environment variable CONTRAST__AGENT__LOGGER__LEVEL=DEBUG.

2. Enable request performance logging for the agent. This can be done either by changing the agent.node.req_perf_logging to true in the agent’s configuration .yaml file or by setting an environment variable CONTRAST__AGENT__NODE__REQ_PERF_LOGGING=true.

3. Run your application with the agent and run your test suite or just exercise the routes that you are having issues with.

4. Send the agent log from this run to our team for analysis.

Example YAML:

agent:
  logger:
    level: DEBUG
  node:
    req_perf_logging: true

debug: [timestamp] contrast:req-perf-logger - {"path":"xxx","method":"xxx", 2"instrumentationTime":xxx,"totalTime":xxx,"events":[...]}

{ 
"name": "xxx",
"count": xxx,
"times": { "pre": xxx, "post": xxx, "orig": xxx }
}

1. Make sure the agent log level is set to ERROR. This can be done either by changing the agent.logger.level to ERROR in the agent’s configuration .yaml file or by setting an environment variable CONTRAST__AGENT__LOGGER__LEVEL=ERROR

2. Start your application with the agent and the --prof flag, for example:
   

   node --prof -r @contrast/agent index.js

3. Exercise the routes that you are having issues with and then quit the application.

   It’s important to go through the parts/routes that have problems so the eventual culprits for the bad performance can be identified. It’s not advisable to run long test suites for this log, because it will create huge files.

4. Node.js will store the collected data in a file named isolate-xxxxxxxx-v8.log Repeat the first two steps but this time run your application without the agent like:

   node --prof index.js

5. Send the two isolate-xxxxxxxx-v8.log files to our team for analysis

Memory Heap Dumps:

1. Make sure the agent log level is set to ERROR. This can be done either by changing the agent.logger.level to ERROR in the agent’s configuration .yaml file or by setting an environment variable CONTRAST__AGENT__LOGGER__LEVEL=ERROR

2. Enable heap dump generation for the agent. This can be done either by changing the agent.heap_dump.enable to true in the agent’s configuration .yaml file or by setting an environment variable CONTRAST__AGENT__HEAP_DUMP__ENABLE=true

3. Exercise the routes that you are having issues with and then quit the application.

   It’s important to go through the parts/routes that have problems so the eventual culprits for the bad performance can be identified. It’s not advisable to run long test suites for these logs, because it will create huge files.

4. By default, the agent will create 5 heap dump files in a folder named contrast_heap_dumps under the root directory for the app, each with a name ending with -contrast.heapsnapshot. These dumps will have data about the memory usage for the first two minutes of your application. If you think that the issues are occurring at a later stage you can contact our team to provide you with correct settings so we can observe the period in time that we are interested in. If you want to do that on your own you can play around with these agent properties:

   1. agent.heap_dump.delay_ms - sets an initial delay - from the time the agent starts to the time we start to create heap dumps. The time is in ms and the default here is 10000.

   2. agent.heap_dump.window_ms - sets the window between the creation of two heap dumps in ms. The default is 30000.

   3. agent.heap_dump.count - sets the number of heap dumps we want to create. After this number is reached the feature is disabled. The default is 5.

5. Keep in mind that depending on your application the heap dump size may vary and sometimes be greater than 1 GB. After creating the necessary heap dumps send them to our team for analysis

Example YAML:

agent:
  logger:
    level: ERROR
  heap_dump:
    enable: true

Conclusion

The more data we have, the easier and faster our investigation of your issues will be. Even if you are not able to provide our team with all of the aforementioned logs and files, you can send some of them or contact our team to talk through your problems or concerns about gathering and providing this data to us.