|License Type||SaaS & On-Premise|
|Agent Mode||Assess & Protect|
|Main Product Category||.NET Framework Agent|
The .NET Framework agent logs only INFO level messages by default, but when troubleshooting specific issues it is often necessary to change that level to obtain additional detail on what the agent is seeing and doing.
In certain are scenarios, bad instrumentation can cause a web server process to crash or a specific page to error out. Deeper logging and perhaps a dump of the worker process(es) will often be needed for successful troubleshooting of such issues.
The article details the steps needed to gather additional logs and process dumps as needed.
TRACE Level Logs
The .NET Framework agent logs information to the logs folder within
C:\ProgramData\Contrast\dotnet\, the Windows 2008/2012/2016 ProgramData folder. Depending on the setup of the Windows profile and folder view settings, the folders may be hidden. If so, paste the paths into the Windows Explorer location; you may need to replace the drive letter C with D.
You can change which information is logged by setting the logging level in your contrast_security.yaml file to TRACE as follows:
log_discovered_routesis needed only if troubleshooting route coverage related issues.
There are two primary types of issue for which Contrast needs to gather process dumps:
- Process Crash
- Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage
Determine whether the worker process crashed
Check your scenario against the following indicators to determine whether the worker process crashed.
The web application is unresponsive after installing the .NET Framework agent.
The Windows Event Log (Event Viewer > Windows Logs > Application) has Error entries for the
- The ".NET Runtime" error has details such as:
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP XXXXXXXXX with exit code YYYYYYY
The "Application Error" entry has details similar to:
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96 Faulting module name: clr.dll, version: 4.7.2114.0, time stamp: 0x59a63e48 Exception code: 0xc0000005 Fault offset: 0x00000000002ff61c Faulting process id: 0x3724 Faulting application start time: 0x01d337d711f21e68 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll Report Id: 4fc99650-a3ca-11e7-80e8-005056bd4248
- The ".NET Runtime" error has details such as:
Once you confirm that the observed bug is a process crash, you're ready to gather information to file a bug.
Gather information on the process crash
Complete the following steps to gather information to send to Contrast Support.
- Set up the ProcDump utility to capture crash dumps automatically.
- Download the current version of ProcDump from the Microsoft documentation site to the Windows server with the agent.
- Create a folder to store the dumps and then set ProcDump as the default debugger on occurrence of a crash by running the following commands from an administrator command prompt - for example:
md c:\dumps procdump.exe -ma -i c:\dumps
- Install the latest .NET Framework agent.
- Stop the .NET Framework agent service.
- Enable additional logging.
- Start > Notepad > Right click > Run as Administrator
- File > Open >
- Add the following configuration to the yaml file to enable verbose logging and logging of every method JIT-compiled by the CLR:
- Start the .NET Framework agent service.
- Exercise the application to reproduce the crash.
Once you've reproduced the crash, gather the following items and include them in your bug report:
- Agent Logs: All files in the agent log directory,
C:\ProgramData\Contrast\dotnet\logs; right click on the LOGS folder > Send To > Compressed (zip) folder.
- Windows Event Log: Event Viewer > Windows Logs > Application > Save All Events As > "MyEvents.evtx"
- Crash Dumps: Create a zip file of each w3wp process dump file in
w3wp.exe_171002_151601.dmp). Dump files can be quite large.
You can then "uninstall" ProcDump (revert the default debugger to the Windows default) by running
Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage
Determine whether there was an unhandled exception
The above process also helps the .NET Framework engineering team resolve issues such as application errors caused by the .NET agent. Use the following indicators to determine if the .NET Framework agent is causing an application error:
- You've observed the application working normally without the agent.
- You've observed a page of the application "crashing" (returning a 500 error) with the agent.
- There are no errors for ."NET Runtime" and "Application Error" in the Windows Event Log.
- There may be warnings for "ASP.NET" in the Windows Event Log, for example:
Source: ASP.NET 4.0.30319.0 Date: 10/9/2017 9:22:46 AM Event ID: 1309 Task Category: Web Event Level: Warning Keywords: Classic User: N/A Computer: FOO.COMPUTER.COM Description: Event code: 3005 Event message: An unhandled exception has occurred. Event time: 09/10/2017 9:22:46 AM Event time (UTC): 09/10/2017 2:22:46 PM Event ID: f706787c1f1247e6a87b777a90413c3d Event sequence: 9 Event occurrence: 1 Event detail code: 0 Application information: Application domain: /LM/W3SVC/1/ROOT/FOO-1-131520325424796488 Trust level: Full Application Virtual Path: /Foo Application Path: E:\MCMSFiles\inetpub\wwwroot\Foo\ Machine name: FOO Process information: Process ID: 176840 Process name: w3wp.exe Account name: System Exception information: Exception type: ArgumentOutOfRangeException Exception message: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index at System.Collections.ArrayList.get_Item(Int32 index) at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index) at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686 at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468 at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549 at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) Request information: Request URL: https://www.foo-staging.com/Foo/Fabrikam.aspx Request path: /Foo/Fabrikam.aspx User host address: 18.104.22.168 User: msteeber Is authenticated: True Authentication Type: Thread account name: System Thread information: Thread ID: 19 Thread account name: System Is impersonating: False Stack trace: at System.Collections.ArrayList.get_Item(Int32 index) at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index) at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686 at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468 at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549 at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
As the process hasn't crashed, ProcDump won't automatically capture process dumps. Instead, you must gather the process dump manually as follows:
Find the Process ID of the worker process that you need.
IIS Manager > Worker Processes: Find the "Application Pool Name" you need, and take note of the value in the "Process Id".
If the exception type is know (e.g. in the above example, a
ArgumentOutOfRangeExceptionwas thrown) you can configure ProcDump to automatically capture a dump on when it encounters that exception. Full details of the available options are here, but for example, to capture a dump when an exception occurs in the process with process ID
NNNNN, where that exception contains the term "Argument":
procdump -ma -e -f Argument NNNNN
If the exception type is not known, or you wish to capture a dump to troubleshoot excessive resource usage, for example: from an administrator command prompt, replace
NNNNNwith the process ID from the previous step and run the following command:
procdump -ma NNNNN
-ma option specifies that ProcDump should write a dump file with all process memory. The default dump format only includes thread and handle information.)
It may also be useful to generate a System Information report using the .NET Agent Diagnostics utility as detailed here.
Other useful information may be available in the Windows Event Viewer.
If you encountered a bug other than a process crash or unhandled exception - maybe the .NET Tray has an inaccurate state, or the agent found a false positive - please contact us via the Submit a Request link at the bottom of this page. Contrast doesn't usually need process dumps, but trace-level logs and a detailed description of the problem are very helpful when it's time to fix these bugs.