Objective
The .NET Core agent logs only INFO level messages by default, but when troubleshooting specific issues it is often necessary to change that level to obtain additional detail on what the agent is seeing and doing.
In certain scenarios, bad instrumentation can cause a web server process to crash or a specific page to error out. Deeper logging and perhaps a dump of the worker process(es) will often be needed for successful troubleshooting of such issues.
The article details the steps needed to gather additional logs and process dumps as needed.
Process
Changing the logging level
C:\ProgramData\Contrast\dotnet-core\
on Windows or /var/tmp/contrast/dotnet-core/
on Linux, by default.
- Depending on the setup of the Windows profile and folder view settings, the
ProgramData
folder may be hidden. - You can change the default logging location by setting the environment variable CONTRAST_CORECLR_LOGS_DIRECTORY='/path/to/logs'
You can change which information is logged by setting the logging level in your contrast_security.yaml file to TRACE, together with optional additional .NET specific debug options as follows (or use the corresponding environment variable options by converting the yaml below using our Agent Configuration Editor):
agent:
logger:
level: TRACE
dotnet:
debug:
log_method_sigs: true
log_modified_il: true
log_discovered_routes: true
exception_logging:
mode: ALL
capture_stack_traces: true
To troubleshoot issues specific to route discovery, add the following to the above options:
agent:
dotnet:
debug:
log_discovered_routes: true
To troubleshoot issue specific to application performance, set the logger level to INFO and enable performance timing logging:
agent:
logger:
level: INFO
dotnet:
debug:
performance_timings:
enable: true
It is also very helpful to provide detailed information on the .NET Core environment. The following command produces most of what support will need.
dotnet --info
Process Dumps
There are two primary types of issue for which Contrast needs to gather process dumps:
- Process Crash
- Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage
The method for collecting a process dump varies by platform:
For Linux Based Operating Systems
Process Crash
If the process is crashing while the Contrast agent is attached or trying to attach, these environment variables can be set to create a dump on crash:
export COMPlus_DbgEnableMiniDump=1
export COMPlus_DbgMiniDumpType=2
Exercise the application to reproduce the crash, and any resulting dumps will be located at /tmp/coredump.[pid]
Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage
If the application runs but a page throws an error or fails to load, a dump can be created manually using either the dotnet createdump utility, or the Contrast agent diagnostics utility. IN either case, you will first need to determine the pid of the running application like so:
$ ps -ax | grep dotnet
9186 pts/0 SLl+ 0:00 dotnet run
9241 pts/0 SLl+ 0:01 /usr/share/dotnet/dotnet /home/bob/.nuget/packages/microsoft.aspnetcore.razor.design/2.1.1/tools/rzc.dll server -p razor-issues-2406
9258 pts/0 SLl+ 0:06 dotnet exec /home/bob/Desktop/SampleApp/bin/Debug/netcoreapp2.2/SampleApp.dll
In this case we want the `dotnet exec` process (with pid 9258) because it is the process running the application.
Using createdump
The location of the `createdump` command may differ based on which version of .NET Core is installed but is generally located in /usr/share/dotnet/shared/Microsoft.NETCore.App/<SDK version>/
createdump -h [pid]
The dump will be written to /tmp/coredump.[pid]
For Example:
$ /usr/share/dotnet/shared/Microsoft.NETCore.App/5.0.6/createdump -h 9258
Writing full dump to file /tmp/coredump.9258
Written 12285857792 bytes (2999477 pages) to core file
Using the agent diagnostics
The .NET Core agent comes bundled with a diagnostics utility that includes an option to generate a dump file of any dotnet process. To make use of that utility, use thecreate-dump
option to generate a dump of type "Full" as detailed here.
For Example:
$ ./contrast-dotnet-diagnostics create-dump --type Full --pid 9258
2023-05-01 18:04:23.2944|INFO|NLogManager|Applying new log level 'warn' (Warn).
Diagnostics running as '.NET Core' on Linux (x64), Non-Azure.
Preparing to create Full dump file for process 15517.
The directory was not specified. Falling back to the current directory of '/home/bob/Contrast.NET.Core/Contrast.NET.Core_2.1.5.0/diagnostics/linux-x64'.
Detected .NET Core app running on Linux.
Dump file successfully written to '/home/bob/Contrast.NET.Core/Contrast.NET.Core_2.1.5.0/diagnostics/linux-x64/15517-20230501-180423-358.dmp'.
Agent is failing to load
On Linux, if the agent fails to load then there will be no logs created. The preferred method for troubleshooting this issue is to use the diagnostics utility that is packaged with the agent - details are documented here - but if that yields insufficient detail, then the strace
utility (system call tracer) on Linux can help us diagnose why the agent is not loading into their process.
As an example, this command runs `dotnet SampleApp.dll` and outputs all kernel calls to the strace_output.txt
file.
strace -o strace_output.txt dotnet SampleApp.dll
For Windows Based Operating Systems:
First identify how the dotnet core app is running to determine which process to inspect. The possible options are:
- IIS In-Process.
- IIS Out-of-process
- Standalone process.
If the application is hosted in IIS or Azure App Services, check the web.config
file. The hostingModel element of the aspNetCore tag will indicate whether it’s in-process or out-of-process.
If the application is running in InProcess
mode, you will need the w3wp.exe
process. If running OutOfProcess or Standalone then you will need the dotnet.exe
process.
If you are unsure, identifying this process is most easily achieved with SysInternals Process Explorer on Windows or using the "Advanced Tools" blade (Kudu) in Azure App Services then the "Process Explorer" tab. Look for the dotnet.exe
or w3wp.exe
processes (in Azure App Services - note that you can ignore the w3wp.exe
process labeled SCM as that is the process running the Kudu console itself), then check the loaded DLLs. Look for Contrast.SensorsNetCore.dll
as a loaded module,
Process Crash
If the application process crashes with the Contrast .NET Core agent, complete the following steps to gather information to send to Contrast Support. Note that these steps only apply to Windows and not Azure App Services.
Check whether process dumps have already been generated
If Windows Error Reporting (WER) is enabled, by default IIS will put its crash dumps in
“C:\ProgramData\Microsoft\Windows\WER\ReportQueue” or “C:\ProgramData\Microsoft\Windows\WER\ReportArchive” directory.
A message to that effect can be seen in the normal Event Viewer (Windows Logs->Application) log. There will be an Error entry for the crash itself, followed by an Information message that says the crash dump was saved. If this information message is missing, WER is likely disabled, so you should use the ProcDump method instead (see below).
Configure ProcDump to generate dump files
- Set up the ProcDump utility to capture crash dumps automatically.
- Download the current version of ProcDump from the Microsoft documentation site to the Windows server with the agent.
- Create a folder to store the dumps and then set ProcDump as the default debugger on occurrence of a crash by running the following commands from an administrator command prompt - for example:
md c:\dumps procdump.exe -accepteula -ma -i c:\dumps
- Install the latest .NET Core agent.
- Stop the application running with Contrast
- Enable additional logging as detailed above.
- Start the application running with Contrast
- Exercise the application to reproduce the crash.
Once you've reproduced the crash, gather the following items and include them in your bug report:
- Agent Logs: All files in the agent log directory,
C:\ProgramData\Contrast\dotnet-core\logs
; right click on the LOGS folder > Send To > Compressed (zip) folder. - Windows Event Log: Event Viewer > Windows Logs > Application > Save All Events As > "MyEvents.evtx"
- Crash Dumps: Create a zip file of each w3wp process dump file in
C:\dumps
(e.g.,w3wp.exe_171002_151601.dmp
). Dump files can be quite large.
You can then "uninstall" ProcDump (revert the default debugger to the Windows default) by running procdump.exe -u
.
Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage
Determine whether there was an unhandled exception
Use the following indicators to determine if the .NET Framework agent is causing an application error:
- You've observed the application working normally without the agent.
- You've observed a page of the application "crashing" (returning a 500 error) with the agent.
- There are no errors for ."NET Runtime" and "Application Error" in the Windows Event Log.
- There may be warnings for "ASP.NET" in the Windows Event Log, for example:
Source: ASP.NET 4.0.30319.0 Date: 10/9/2017 9:22:46 AM Event ID: 1309 Task Category: Web Event Level: Warning Keywords: Classic User: N/A Computer: FOO.COMPUTER.COM Description: Event code: 3005 Event message: An unhandled exception has occurred. Event time: 09/10/2017 9:22:46 AM Event time (UTC): 09/10/2017 2:22:46 PM Event ID: f706787c1f1247e6a87b777a90413c3d Event sequence: 9 Event occurrence: 1 Event detail code: 0 Application information: Application domain: /LM/W3SVC/1/ROOT/FOO-1-131520325424796488 Trust level: Full Application Virtual Path: /Foo Application Path: E:\MCMSFiles\inetpub\wwwroot\Foo\ Machine name: FOO Process information: Process ID: 176840 Process name: w3wp.exe Account name: System Exception information: Exception type: ArgumentOutOfRangeException Exception message: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index at System.Collections.ArrayList.get_Item(Int32 index) at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index) at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686 at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468 at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549 at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) Request information: Request URL: https://www.foo-staging.com/Foo/Fabrikam.aspx Request path: /Foo/Fabrikam.aspx User host address: 1.2.3.4 User: msteeber Is authenticated: True Authentication Type: Thread account name: System Thread information: Thread ID: 19 Thread account name: System Is impersonating: False Stack trace: at System.Collections.ArrayList.get_Item(Int32 index) at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index) at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686 at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468 at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549 at System.Web.UI.Control.PreRenderRecursiveInternal() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
As the process hasn't crashed, ProcDump won't automatically capture process dumps. Instead, you must gather the process dump manually as follows:
Using the agent diagnostics
The .NET Core agent comes bundled with a diagnostics utility that includes an option to generate a dump file of any .NET Core process. To make use of that utility:
- Find the Process ID of the process running the application (see here)
- Use the .NET Agent Diagnostics Utility
create-dump
option to generate a dump of type "Full" as detailed here.
Using SysInternals ProcDump
-
-
Find the Process ID of the process running the application (see here)
- Download the current version of ProcDump from the Microsoft documentation site to the Windows server with the agent.
-
If the exception type is know (e.g. in the above example, a
ArgumentOutOfRangeException
was thrown) you can configure ProcDump to automatically capture a dump on when it encounters that exception. Full details of the available options are here, but for example, to capture a dump when an exception occurs in the process with process IDpid
, where that exception contains the term "Argument":procdump -ma -e -w -f Argument [pid]
-
(The -w option will wait for the specified process to start if it is not running)
-
-
If the exception type is not known, or you wish to capture a dump to troubleshoot excessive resource usage or some other issue that does not result in an exception, for example, then from an administrator command prompt, run the following command:
procdump -accepteula -ma [pid]
-
If only a single process is running with a given process name, you can also use that name instead of the process ID as in the following examples:
procdump -accepteula -ma w3wp.exe
procdump -accepteula -ma dotnet.exe
(The -ma
option specifies that ProcDump should write a dump file with all process memory. The default dump format only includes thread and handle information.)
c:\devtools\sysinternals\procdump
System Information
It may also be useful to generate a System Information report using the .NET Agent Diagnostics utility as detailed here.
Other useful information may be available in the Windows Event Viewer.
Writing to STDOUT
It is possible to configure the .NET Core agent to output all logging to the console. Note that this is only potentially of use when running in a Linux environment or when running on Windows outside of IIS from a command line.
To enable output to STDOUT, add the following to the contrast_security.yaml
file:
agent:
logger:
stdout: true
or set the following equivalent environment variable:
CONTRAST__AGENT__LOGGER__STDOUT=true
To contact the Contrast Support Team, please submit a ticket to our online support portal.