|License Type||SaaS & On-Premise|
|Agent Mode||Assess & Protect|
|Main Product Category||.NET Core Agent|
The .NET Core agent logs only INFO level messages by default, but when troubleshooting specific issues it is often necessary to change that level to obtain additional detail on what the agent is seeing and doing.
In certain are scenarios, bad instrumentation can cause a web server process to crash or a specific page to error out. Deeper logging and perhaps a dump of the worker process(es) will often be needed for successful troubleshooting of such issues.
The article details the steps needed to gather additional logs and process dumps as needed.
TRACE Level and Debug Logs
The .NET Core agent logs information to the logs folder within
C:\ProgramData\Contrast\dotnet-core\ on Windows or
/var/tmp/contrast/dotnet-core/ on Linux, by default.
- Depending on the setup of the Windows profile and folder view settings, the
ProgramDatafolder may be hidden.
- You can change the default logging location by setting the environment variable CONTRAST_CORECLR_LOGS_DIRECTORY='/path/to/logs'
You can change which information is logged by setting the logging level in your contrast_security.yaml file to TRACE, together with optional additional .NET specific debug options as follows (or use the corresponding environment variable options that you can determine by converting the yaml below using our Agent Configuration Editor):
log_discovered_routesis needed only if troubleshooting route coverage related issues.
It is also very helpful to provide detailed information on the .NET Core environment. The following command produces most of what support will need.
There are two primary types of issue for which Contrast needs to gather process dumps:
- Process Crash
- Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage
The method for collecting a process dump varies by platform:
For Linux Based Operating Systems
If the process is crashing while the Contrast agent is attached or trying to attach, these environment variables can be set to create a dump on crash:
Exercise the application to reproduce the crash, and any resulting dumps will be located at
Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage
If the application runs but a page throws an error or fails to load, a dump can be created manually. The location of the `createdump` command may differ based on which version of .NET Core is installed but is generally located in
createdump -h [pid]
The dump will be written to
bob@ubuntudesktop:~$ ps -ax | grep dotnet
9186 pts/0 SLl+ 0:00 dotnet run
9241 pts/0 SLl+ 0:01 /usr/share/dotnet/dotnet /home/bob/.nuget/packages/microsoft.aspnetcore.razor.design/2.1.1/tools/rzc.dll server -p razor-issues-2406
9258 pts/0 SLl+ 0:06 dotnet exec /home/bob/Desktop/SampleApp/bin/Debug/netcoreapp2.2/SampleApp.dll
In this case we want the `dotnet exec` process because it is the process running the application:
bob@ubuntudesktop:~$ /usr/share/dotnet/shared/Microsoft.NETCore.App/5.0.6/createdump -h 9258
Writing full dump to file /tmp/coredump.9258
Written 12285857792 bytes (2999477 pages) to core file
Agent is failing to load
On Linux, if the agent fails to load then there will be no logs created. The preferred method for troubleshooting this issue is to use the diagnostics utility that is packaged with the agent - details are documented here - but if that yields insufficient detail, then the
strace utility (system call tracer) on Linux can help us diagnose why the agent is not loading into their process.
As an example, this command runs `dotnet SampleApp.dll` and outputs all kernel calls to the
strace -o strace_output.txt dotnet SampleApp.dll
For Windows Based Operating Systems:
First identify how the dotnet core app is running to determine which process to inspect. The possible options are:
- IIS In-Process.
- IIS Out-of-process
- Standalone process.
If the application is hosted in IIS or Azure App Services, check the
web.config file. The hostingModel element of the aspNetCore tag will indicate whether it’s in-process or out-of-process.
If the application is running in
InProcess mode, you will need the
w3wp.exe process. If running OutOfProcess or Standalone then you will need the
If you are unsure, identifying this process is most easily achieved with SysInternals Process Explorer on Windows or using the "Advanced Tools" blade (Kudu) in Azure App Services then the "Process Explorer" tab. Look for the
w3wp.exe processes (in Azure App Services - note that you can ignore the
w3wp.exe process labeled SCM as that is the process running the Kudu console itself), then check the loaded DLLs. Look for
Contrast.SensorsNetCore.dll as a loaded module,
If the application process crashes with the Contrast .NET Core agent, complete the following steps to gather information to send to Contrast Support. Note that these steps only apply to Windows and not Azure App Services.
- Set up the ProcDump utility to capture crash dumps automatically.
- Download the current version of ProcDump from the Microsoft documentation site to the Windows server with the agent.
- Create a folder to store the dumps and then set ProcDump as the default debugger on occurrence of a crash by running the following commands from an administrator command prompt - for example:
md c:\dumps procdump.exe -accepteula -ma -i c:\dumps
- Install the latest .NET Core agent.
- Stop the application running with Contrast
- Enable additional logging as detailed above.
- Start the application running with Contrast
- Exercise the application to reproduce the crash.
Once you've reproduced the crash, gather the following items and include them in your bug report:
- Agent Logs: All files in the agent log directory,
C:\ProgramData\Contrast\dotnet-core\logs; right click on the LOGS folder > Send To > Compressed (zip) folder.
- Windows Event Log: Event Viewer > Windows Logs > Application > Save All Events As > "MyEvents.evtx"
- Crash Dumps: Create a zip file of each w3wp process dump file in
w3wp.exe_171002_151601.dmp). Dump files can be quite large.
You can then "uninstall" ProcDump (revert the default debugger to the Windows default) by running
Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage
Determine whether there was an unhandled exception
Use the following indicators to determine if the .NET Framework agent is causing an application error:
- You've observed the application working normally without the agent.
- You've observed a page of the application "crashing" (returning a 500 error) with the agent.
- There are no errors for ."NET Runtime" and "Application Error" in the Windows Event Log.
- There may be warnings for "ASP.NET" in the Windows Event Log, for example:
Source: ASP.NET 4.0.30319.0 Date: 10/9/2017 9:22:46 AM Event ID: 1309 Task Category: Web Event Level: Warning Keywords: Classic User: N/A Computer: FOO.COMPUTER.COM Description: Event code: 3005 Event message: An unhandled exception has occurred. Event time: 09/10/2017 9:22:46 AM Event time (UTC): 09/10/2017 2:22:46 PM Event ID: f706787c1f1247e6a87b777a90413c3d Event sequence: 9 Event occurrence: 1 Event detail code: 0 Application information: Application domain: /LM/W3SVC/1/ROOT/FOO-1-131520325424796488 Trust level: Full Application Virtual Path: /Foo Application Path: E:\MCMSFiles\inetpub\wwwroot\Foo\ Machine name: FOO Process information: Process ID: 176840 Process name: w3wp.exe Account name: System Exception information: Exception type: ArgumentOutOfRangeException Exception message: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index at System.Collections.ArrayList.get_Item(Int32 index) at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index) at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686 at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468 at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549 at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) Request information: Request URL: https://www.foo-staging.com/Foo/Fabrikam.aspx Request path: /Foo/Fabrikam.aspx User host address: 220.127.116.11 User: msteeber Is authenticated: True Authentication Type: Thread account name: System Thread information: Thread ID: 19 Thread account name: System Is impersonating: False Stack trace: at System.Collections.ArrayList.get_Item(Int32 index) at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index) at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686 at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468 at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549 at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
As the process hasn't crashed, ProcDump won't automatically capture process dumps. Instead, you must gather the process dump manually as follows:
Find the Process ID of the process running the application (see here)
If the exception type is know (e.g. in the above example, a
ArgumentOutOfRangeExceptionwas thrown) you can configure ProcDump to automatically capture a dump on when it encounters that exception. Full details of the available options are here, but for example, to capture a dump when an exception occurs in the process with process ID
pid, where that exception contains the term "Argument":
procdump -ma -e -w -f Argument [pid]
(The -w option will wait for the specified process to start if it is not running)
If the exception type is not known, or you wish to capture a dump to troubleshoot excessive resource usage or some other issue that does not result in an exception, for example, then from an administrator command prompt, run the following command:
procdump -accepteula -ma [pid]
If only a single process is running with a given process name, you can also use that name instead of the process ID as in the following examples:
procdump -accepteula -ma w3wp.exe
procdump -accepteula -ma dotnet.exe
-ma option specifies that ProcDump should write a dump file with all process memory. The default dump format only includes thread and handle information.)
It may also be useful to generate a System Information report using the .NET Agent Diagnostics utility as detailed here.
Other useful information may be available in the Windows Event Viewer.
These verbose logs will allow our support team to diagnose any issues you may encounter. To contact the Contrast Support Team, please submit a ticket to our online support portal.