How to get logs from the .NET Framework agent

Objective

The .NET Framework agent logs only INFO level messages by default, but when troubleshooting specific issues it is often necessary to change that level to obtain additional detail on what the agent is seeing and doing. 

In certain scenarios, bad instrumentation can cause a web server process to crash or a specific page to error out. Deeper logging and perhaps a dump of the worker process(es) will often be needed for successful troubleshooting of such issues.

The article details the steps needed to gather additional logs and process dumps as needed.

Process

Changing the logging level

You can change which information is logged by setting the logging level in your contrast_security.yaml file to TRACE, together with optional additional .NET specific debug options as follows (or use the corresponding environment variable options by converting the yaml below using our Agent Configuration Editor):

agent:
  logger:
    level: TRACE
  dotnet:
    debug:
      log_method_sigs: true
      log_modified_il: true
      log_discovered_routes: true
      exception_logging:
        mode: ALL
        capture_stack_traces: true

To troubleshoot issues specific to route discovery, add the following to the above options:

agent:
  dotnet:
    debug:
      log_discovered_routes: true

To troubleshoot issue specific to application performance, leave the logger level at the default value (INFO, ERROR etc.) and just set the following option:

agent:
  dotnet:
    debug:
      performance_timings:
        enable: true

Process Dumps

There are two primary types of issue for which Contrast needs to gather process dumps:

• Process Crash
• Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage

Process Crash

Determine whether the worker process crashed

Check your scenario against the following indicators to determine whether the worker process crashed.

• The web application is unresponsive after installing the .NET Framework agent.

• The Windows Event Log (Event Viewer > Windows Logs > Application) has Error entries for the .NET Runtime and Application Error.

  • The ".NET Runtime" error has details such as:
    

    Application: w3wp.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an internal error in the .NET Runtime at IP XXXXXXXXX with exit code YYYYYYY

  • The "Application Error" entry has details similar to:

    Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
    Faulting module name: clr.dll, version: 4.7.2114.0, time stamp: 0x59a63e48
    Exception code: 0xc0000005
    Fault offset: 0x00000000002ff61c
    Faulting process id: 0x3724
    Faulting application start time: 0x01d337d711f21e68
    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
    Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
    Report Id: 4fc99650-a3ca-11e7-80e8-005056bd4248
    

Once you confirm that the observed bug is a process crash, you're ready to gather information to file a bug.

Gather information on the process crash

Complete the following steps to gather information to send to Contrast Support:

Check whether process dumps have already been generated

If Windows Error Reporting (WER) is enabled, by default IIS will put its crash dumps in

“C:\ProgramData\Microsoft\Windows\WER\ReportQueue” or “C:\ProgramData\Microsoft\Windows\WER\ReportArchive” directory.

A message to that effect can be seen in the normal Event Viewer (Windows Logs->Application) log. There will be an Error entry for the crash itself, followed by an Information message that says the crash dump was saved. If this information message is missing, WER is likely disabled, so you should use the ProcDump method instead (see below).

Configure ProcDump to generate dump files

• Set up the ProcDump utility to capture crash dumps automatically.
  • Download the current version of ProcDump from the Microsoft documentation site to the Windows server with the agent.
  • Create a folder to store the dumps and then set ProcDump as the default debugger on occurrence of a crash by running the following commands from an administrator command prompt - for example:

    md c:\dumps 
    procdump.exe -ma -i c:\dumps

  • Install the latest .NET Framework agent.
  • Stop the .NET Framework agent service.
  • Enable additional logging as detailed above.
  • Start the .NET Framework agent service.
  • Exercise the application to reproduce the crash.

Once you've reproduced the crash, gather the following items and include them in your bug report:

• Agent Logs: All files in the agent log directory, C:\ProgramData\Contrast\dotnet\logs; right click on the LOGS folder > Send To > Compressed (zip) folder.
• Windows Event Log: Event Viewer > Windows Logs > Application > Save All Events As > "MyEvents.evtx"
• Crash Dumps: Create a zip file of each w3wp process dump file in C:\dumps(e.g., w3wp.exe_171002_151601.dmp). Dump files can be quite large.

You can then "uninstall" ProcDump (revert the default debugger to the Windows default) by running procdump.exe -u.

Unhandled Managed Exception/Page Error/500 or Excessive Resource Usage

Determine whether there was an unhandled exception

The above process also helps the .NET Framework engineering team resolve issues such as application errors caused by the .NET agent. Use the following indicators to determine if the .NET Framework agent is causing an application error:

• You've observed the application working normally without the agent.
• You've observed a page of the application "crashing" (returning a 500 error) with the agent.
• There are no errors for ."NET Runtime" and "Application Error" in the Windows Event Log.
• There may be warnings for "ASP.NET" in the Windows Event Log, for example:

   Source: ASP.NET 4.0.30319.0
   Date: 10/9/2017 9:22:46 AM
   Event ID: 1309
   Task Category: Web Event
   Level: Warning
   Keywords: Classic
   User: N/A
   Computer: FOO.COMPUTER.COM
   Description:
   Event code: 3005
   Event message: An unhandled exception has occurred.
   Event time: 09/10/2017 9:22:46 AM
   Event time (UTC): 09/10/2017 2:22:46 PM
   Event ID: f706787c1f1247e6a87b777a90413c3d
   Event sequence: 9
   Event occurrence: 1
   Event detail code: 0
   Application information:
   Application domain: /LM/W3SVC/1/ROOT/FOO-1-131520325424796488
   Trust level: Full
   Application Virtual Path: /Foo
   Application Path: E:\MCMSFiles\inetpub\wwwroot\Foo\
   Machine name: FOO
   Process information:
   Process ID: 176840
   Process name: w3wp.exe
   Account name: System
   Exception information:
   Exception type: ArgumentOutOfRangeException
   Exception message: Index was out of range. Must be non-negative and less than the size of the collection.
   Parameter name: index
   at System.Collections.ArrayList.get_Item(Int32 index)
   at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index)
   at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686
   at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468
   at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
  
   Request information:
   Request URL: https://www.foo-staging.com/Foo/Fabrikam.aspx
   Request path: /Foo/Fabrikam.aspx
   User host address: 1.2.3.4
   User: msteeber
   Is authenticated: True
   Authentication Type: 
   Thread account name: System
  
   Thread information:
   Thread ID: 19
   Thread account name: System
   Is impersonating: False
   Stack trace: at System.Collections.ArrayList.get_Item(Int32 index)
   at System.Web.UI.WebControls.DataListItemCollection.get_Item(Int32 index)
   at Fabrikam.SetTabCount(Int32 index, NullableInt32 summaryCount) in C:\Foo\Fabrikam.aspx.cs:line 1686
   at Fabrikam.GetSummaryCounts() in C:\Foo\Fabrikam.aspx.cs:line 1468
   at Fabrikam.OnPreRender(EventArgs e) in C:\Foo\Fabrikam.aspx.cs:line 549
   at System.Web.UI.Control.PreRenderRecursiveInternal()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
  

As the process hasn't crashed, ProcDump won't automatically capture process dumps. Instead, you must gather the process dump manually as follows:

Using the agent diagnostics

The .NET Framework agent comes bundled with a diagnostics utility that includes an option to generate a dump file of any .NET Framework process.  To make use of that utility:

• Find the Process ID of the worker process that you need.

  • IIS Manager > Worker Processes: Find the "Application Pool Name" you need, and take note of the value in the "Process Id".

• Use the .NET Agent Diagnostics Utility create-dump option to generate a dump of type "Full" as detailed here.

Using SysInternals ProcDump

• Find the Process ID of the worker process that you need.

  • IIS Manager > Worker Processes: Find the "Application Pool Name" you need, and take note of the value in the "Process Id".

• Download the current version of ProcDump from the Microsoft documentation site to the Windows server with the agent.

• If the exception type is known (e.g. in the above example, a ArgumentOutOfRangeException was thrown) you can configure ProcDump to automatically capture a dump on when it encounters that exception.  Full details of the available options are here, but for example, to capture a dump when an exception occurs in the process with process ID NNNNN, where that exception contains the term "Argument":

  procdump -ma -e -f Argument NNNNN

• If the exception type is not known, or you wish to capture a dump to troubleshoot excessive resource usage, for example: from an administrator command prompt, replace NNNNN with the process ID from the previous step and run the following command:

  procdump -ma NNNNN

(The -ma option specifies that ProcDump should write a dump file with all process memory. The default dump format only includes thread and handle information.)

System Information

It may also be useful to generate a System Information report using the .NET Agent Diagnostics utility as detailed here.

Other useful information may be available in the Windows Event Viewer.

Other Bugs

If you encountered a bug other than a process crash or unhandled exception - maybe the .NET Tray has an inaccurate state, or the agent found a false positive - please contact us via the Submit a Request link at the bottom of this page. Contrast doesn't usually need process dumps, but trace-level logs and a detailed description of the problem are very helpful when it's time to fix these bugs.