Question
What is the best place for Contrast Ruby Assess agent in our testing pipeline?
Answer
Contrast finds vulnerabilities from HTTP-based testing. So end-to-end test suites are the best fit for Contrast agents. Testing the flow of the application from start to finish. This is the only way we’ll see data flow as that’s the only environment in which data flow exists.
Testing tools such as Rspec are designed to test individual functions of the application in isolation and are not a good fit for Contrast agents. These tests, like all unit tests, represent, but are not actual data flow of the application. Because of the nature of RSpec, there is a heavy penalty to re-instrument the suite on each test with relatively little benefit. Some static and trigger vulnerabilities may be seen, but not those from data flow.
Contrast is forced to fully instrument the app for EACH of the few hundred test cases that cause a forked process that runs in RSpec. This can cause performance degradation.
Unit tests are also not a good fit for Contrast agents. There is no request to follow, so there are no sources and most tests will not have triggers. Those that do, most likely have the methods will be stubbed out, which we purposely ignore so as not to break those tests when we’re accidentally added.
To eliminate the performance degradation Contrast should be removed from environments in which forked processes occur repeatedly.
- Deactivate Contrast in environments where testing frameworks fork repeatedly (like RSpec) or do not perform HTTP-based tests to eliminate the latency that Contrast can introduce.
- Activate Contrast in environments/situations where interactive testing by humans or automation using HTTP is occurring (without unit/mock tests that fork).
The recommended way to do this is as follows:
group :staging do
gem 'contrast-agent'
end
Another way to do this is as follows in the contrast_security.yaml.
Only set this property if you want to turn off Contrast. Set to true to turn the agent on, set to false to turn the agent off.
enable: true
Contrast is looking into ways to detect RSpec and other methods of testing that perform large amounts of process forking to eliminate the need to make these configuration updates. In the meantime, please activate Contrast in environments where end-to-end HTTP-based testing is happening, which can be automated or manual. In CI environments, QA environments, UAT environments, or even developer smoke-testing environments.