How to get logs from the .NET Core agent

Objective

The .NET Core agent logs only INFO level messages by default, but when troubleshooting specific issues it is often necessary to change that level to obtain additional detail on what the agent is seeing and doing. 

In certain scenarios, bad instrumentation can cause a web server process to crash or a specific page to error out. Deeper logging and perhaps a dump of the worker process(es) will often be needed for successful troubleshooting of such issues.

The article details the steps needed to gather additional logs and process dumps as needed.

Process

Changing the logging level

You can change which information is logged by setting the logging level in your contrast_security.yaml file to TRACE, together with optional additional .NET specific debug options as follows (or use the corresponding environment variable options by converting the yaml below using our Agent Configuration Editor):

agent:
  logger:
    level: TRACE
  dotnet:
    debug:
      log_method_sigs: true
      log_modified_il: true
      log_discovered_routes: true
      exception_logging:
        mode: ALL
        capture_stack_traces: true

To troubleshoot issues specific to route discovery, add the following to the above options:

agent:
  dotnet:
    debug:
      log_discovered_routes: true

To troubleshoot issue specific to application performance, set the logger level to INFO and enable performance timing logging:

agent:
  logger:
    level: INFO
  dotnet:
    debug:
      performance_timings:
        enable: true

 It is also very helpful to provide detailed information on the .NET Core environment.  The following command produces most of what support will need.

dotnet --info

Process Dumps

There are two primary types of issue for which Contrast needs to gather process dumps:

• Process Crash
• Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage

The method for collecting a process dump varies by platform:

For Linux Based Operating Systems

Process Crash

If the process is crashing while the Contrast agent is attached or trying to attach, these environment variables can be set to create a dump on crash:

export COMPlus_DbgEnableMiniDump=1
export COMPlus_DbgMiniDumpType=2

Exercise the application to reproduce the crash, and any resulting dumps will be located at /tmp/coredump.[pid]

Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage

If the application runs but a page throws an error or fails to load, a dump can be created manually using either the dotnet createdump utility, or the Contrast agent diagnostics utility.  IN either case, you will first need to determine the pid of the running application like so:

$ ps -ax | grep dotnet
 9186 pts/0    SLl+   0:00 dotnet run
 9241 pts/0    SLl+   0:01 /usr/share/dotnet/dotnet /home/bob/.nuget/packages/microsoft.aspnetcore.razor.design/2.1.1/tools/rzc.dll server -p razor-issues-2406
 9258 pts/0    SLl+   0:06 dotnet exec /home/bob/Desktop/SampleApp/bin/Debug/netcoreapp2.2/SampleApp.dll

In this case we want the `dotnet exec` process (with pid 9258) because it is the process running the application.

Using createdump

The location of the `createdump` command may differ based on which version of .NET Core is installed but is generally located in /usr/share/dotnet/shared/Microsoft.NETCore.App/<SDK version>/

createdump -h [pid]

The dump will be written to /tmp/coredump.[pid]

For Example:

$ /usr/share/dotnet/shared/Microsoft.NETCore.App/5.0.6/createdump -h 9258
Writing full dump to file /tmp/coredump.9258
Written 12285857792 bytes (2999477 pages) to core file 

Using the agent diagnostics

The .NET Core agent comes bundled with a diagnostics utility that includes an option to generate a dump file of any dotnet process.  To make use of that utility, use thecreate-dump option to generate a dump of type "Full" as detailed here.

For Example:

$ ./contrast-dotnet-diagnostics create-dump --type Full --pid 9258
2023-05-01 18:04:23.2944|INFO|NLogManager|Applying new log level 'warn' (Warn).
Diagnostics running as '.NET Core' on Linux (x64), Non-Azure.

Preparing to create Full dump file for process 15517.
The directory was not specified. Falling back to the current directory of '/home/bob/Contrast.NET.Core/Contrast.NET.Core_2.1.5.0/diagnostics/linux-x64'.
Detected .NET Core app running on Linux.
Dump file successfully written to '/home/bob/Contrast.NET.Core/Contrast.NET.Core_2.1.5.0/diagnostics/linux-x64/15517-20230501-180423-358.dmp'.

Agent is failing to load

On Linux, if the agent fails to load then there will be no logs created.  The preferred method for troubleshooting this issue is to use the diagnostics utility that is packaged with the agent - details are documented here - but if that yields insufficient detail, then the strace utility (system call tracer) on Linux can help us diagnose why the agent is not loading into their process.

 As an example, this command runs `dotnet SampleApp.dll` and outputs all kernel calls to the strace_output.txt file.

strace -o strace_output.txt dotnet SampleApp.dll

For Windows Based Operating Systems:

First identify how the dotnet core app is running to determine which process to inspect.  The possible options are:

• IIS In-Process.
• IIS Out-of-process
• Standalone process.

If the application is hosted in IIS or Azure App Services, check the web.config file.  The hostingModel element of the aspNetCore tag will indicate whether it’s in-process or out-of-process.

If the application is running in InProcess mode, you will need the w3wp.exe process.  If running OutOfProcess or Standalone then you will need the dotnet.exe process. 

If you are unsure, identifying this process is most easily achieved with SysInternals Process Explorer on Windows or using the "Advanced Tools" blade (Kudu) in Azure App Services then the "Process Explorer" tab. Look for the dotnet.exe or w3wp.exe processes (in Azure App Services - note that you can ignore the w3wp.exe process labeled SCM as that is the process running the Kudu console itself), then check the loaded DLLs. Look for Contrast.SensorsNetCore.dll as a loaded module,

Process Crash

If the application process crashes with the Contrast .NET Core agent, complete the following steps to gather information to send to Contrast Support.  Note that these steps only apply to Windows and not Azure App Services.

Check whether process dumps have already been generated

If Windows Error Reporting (WER) is enabled, by default IIS will put its crash dumps in

“C:\ProgramData\Microsoft\Windows\WER\ReportQueue” or “C:\ProgramData\Microsoft\Windows\WER\ReportArchive” directory.

A message to that effect can be seen in the normal Event Viewer (Windows Logs->Application) log. There will be an Error entry for the crash itself, followed by an Information message that says the crash dump was saved. If this information message is missing, WER is likely disabled, so you should use the ProcDump method instead (see below).

Configure ProcDump to generate dump files

• Set up the ProcDump utility to capture crash dumps automatically.
  • Download the current version of ProcDump from the Microsoft documentation site to the Windows server with the agent.
  • Create a folder to store the dumps and then set ProcDump as the default debugger on occurrence of a crash by running the following commands from an administrator command prompt - for example:

    md c:\dumps 
    procdump.exe -accepteula -ma -i c:\dumps

  • Install the latest .NET Core agent.
  • Stop the application running with Contrast
  • Enable additional logging as detailed above.
  • Start the application running with Contrast
  • Exercise the application to reproduce the crash.

Once you've reproduced the crash, gather the following items and include them in your bug report:

• Agent Logs: All files in the agent log directory, C:\ProgramData\Contrast\dotnet-core\logs; right click on the LOGS folder > Send To > Compressed (zip) folder.
• Windows Event Log: Event Viewer > Windows Logs > Application > Save All Events As > "MyEvents.evtx"
• Crash Dumps: Create a zip file of each w3wp process dump file in C:\dumps(e.g., w3wp.exe_171002_151601.dmp). Dump files can be quite large.

You can then "uninstall" ProcDump (revert the default debugger to the Windows default) by running procdump.exe -u.

Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage

Determine whether there was an unhandled exception

Use the following indicators to determine if the .NET Framework agent is causing an application error:

• You've observed the application working normally without the agent.
• You've observed a page of the application "crashing" (returning a 500 error) with the agent.
• There are no errors for ."NET Runtime" and "Application Error" in the Windows Event Log.
• There may be warnings for "ASP.NET" in the Windows Event Log, for example:

   Source: ASP.NET 4.0.30319.0
   Date: 10/9/2017 9:22:46 AM
   Event ID: 1309
   Task Category: Web Event
   Level: Warning
   Keywords: Classic
   User: N/A
   Computer: FOO.COMPUTER.COM
   Description:
   Event code: 3005
   Event message: An unhandled exception has occurred.
   Event time: 09/10/2017 9:22:46 AM
   Event time (UTC): 09/10/2017 2:22:46 PM
   Event ID: f706787c1f1247e6a87b777a90413c3d
   Event sequence: 9
   Event occurrence: 1
   Event detail code: 0
   Application information:
   Application domain: /LM/W3SVC/1/ROOT/FOO-1-131520325424796488
   Trust level: Full
   Application Virtual Path: /Foo
   Application Path: E:\MCMSFiles\inetpub\wwwroot\Foo\
   Machine name: FOO
   Process information:
   Process ID: 176840
   Process name: w3wp.exe
   Account name: System
   Exception information:
   Exception type: ArgumentOutOfRangeException
   Exception message: Index was out of range. Must be non-negative and less than the size of the collection.
   Parameter name: index
   at System.Collections.ArrayList.get_Item(Int32 index)
   at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index)
   at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686
   at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468
   at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
  
   Request information:
   Request URL: https://www.foo-staging.com/Foo/Fabrikam.aspx
   Request path: /Foo/Fabrikam.aspx
   User host address: 1.2.3.4
   User: msteeber
   Is authenticated: True
   Authentication Type: 
   Thread account name: System
  
   Thread information:
   Thread ID: 19
   Thread account name: System
   Is impersonating: False
   Stack trace: at System.Collections.ArrayList.get_Item(Int32 index)
   at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index)
   at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686
   at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468
   at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
  

As the process hasn't crashed, ProcDump won't automatically capture process dumps. Instead, you must gather the process dump manually as follows:

Using the agent diagnostics

The .NET Core agent comes bundled with a diagnostics utility that includes an option to generate a dump file of any .NET Core process.  To make use of that utility:

• Find the Process ID of the process running the application (see here)
• Use the .NET Agent Diagnostics Utility create-dump option to generate a dump of type "Full" as detailed here.

Using SysInternals ProcDump

• • Find the Process ID of the process running the application (see here)

  • Download the current version of ProcDump from the Microsoft documentation site to the Windows server with the agent.

  • If the exception type is know (e.g. in the above example, a ArgumentOutOfRangeException was thrown) you can configure ProcDump to automatically capture a dump on when it encounters that exception.  Full details of the available options are here, but for example, to capture a dump when an exception occurs in the process with process ID pid, where that exception contains the term "Argument":

    procdump -ma -e -w -f Argument [pid]

(The -w option will wait for the specified process to start if it is not running) 

• • If the exception type is not known, or you wish to capture a dump to troubleshoot excessive resource usage or some other issue that does not result in an exception, for example, then from an administrator command prompt, run the following command:

    procdump -accepteula -ma [pid]

If only a single process is running with a given process name, you can also use that name instead of the process ID as in the following examples:

procdump -accepteula -ma w3wp.exe
procdump -accepteula -ma dotnet.exe

(The -ma option specifies that ProcDump should write a dump file with all process memory. The default dump format only includes thread and handle information.)

System Information

It may also be useful to generate a System Information report using the .NET Agent Diagnostics utility as detailed here.

Other useful information may be available in the Windows Event Viewer.

Writing to STDOUT

It is possible to configure the .NET Core agent to output all logging to the console.  Note that this is only potentially of use when running in a Linux environment or when running on Windows outside of IIS from a command line.

To enable output to STDOUT, add the following to the contrast_security.yaml file:

agent:
  logger:
    stdout: true

or set the following equivalent environment variable:

CONTRAST__AGENT__LOGGER__STDOUT=true

To contact the Contrast Support Team, please submit a ticket to our online support portal.