procmon-parser 0.2.0

Parser to Procmon configuration and log files formats

Parsing Procmon files with Python

Procmon (https://docs.microsoft.com/en-us/sysinternals/downloads/procmon) is a very powerful monitoring tool for Windows, capable of capturing file system, registry, process/thread and network activity.

Procmon uses internal file formats for configuration (PMC) and logs (PML). Prior to procmon-parser, PMC files could only be parsed and generated by the Procmon GUI, and PML files could be read only using the Procmon GUI, or converting them to CSV or XML using Procmon command line.

The goals of procmon-parser are:

• Parsing & Building PMC files - making it possible to dynamically add/remove filter rules, which can significantly reduce the size of the log file over time as Procmon captures millions of events.
• Parsing PML files (limited support) - making it possible to directly load the raw PML file into convenient python objects instead of having to convert the file to CSV/XML formats prior to loading.

PMC (Process Monitor Configuration) Parser

Usage

Loading configuration of a pre-exported Procmon configuration:

>>> from procmon_parser import load_configuration, dump_configuration, Rule
>>> with open("ProcmonConfiguration.pmc", "rb") as f:
...     config = load_configuration(f.read())
>>> config["DestructiveFilter"]
0
>>> config["FilterRules"]
[Rule(Column.PROCESS_NAME, RuleRelation.IS, "System", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Procmon64.exe", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Procmon.exe", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Procexp64.exe", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Procexp.exe", RuleAction.EXCLUDE), Rule(Column.PROCESS_NAME, RuleRelation.IS, "Autoruns.exe", RuleAction.EXCLUDE), Rule(Column.OPERATION, RuleRelation.BEGINS_WITH, "IRP_MJ_", RuleAction.EXCLUDE), Rule(Column.OPERATION, RuleRelation.BEGINS_WITH, "FASTIO_", RuleAction.EXCLUDE), Rule(Column.RESULT, RuleRelation.BEGINS_WITH, "FAST IO", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "pagefile.sys", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Volume", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$UpCase", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Secure", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Root", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$MftMirr", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Mft", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$LogFile", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.CONTAINS, "$Extend", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Boot", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$Bitmap", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$BadClus", RuleAction.EXCLUDE), Rule(Column.PATH, RuleRelation.ENDS_WITH, "$AttrDef", RuleAction.EXCLUDE), Rule(Column.EVENT_CLASS, RuleRelation.IS, "Profiling", RuleAction.EXCLUDE)]

Adding some new rules

>>> new_rules = [Rule('Path', 'contains', '1337', 'include'), Rule('Process_Name', 'is', 'python.exe')]
>>> config["FilterRules"] = new_rules + config["FilterRules"]

Dropping filtered events

>>> config["DestructiveFilter"] = 1

Dumping the new configuration to a file

>>> with open("ProcmonConfiguration1337.pmc", "wb") as f:
...     dump_configuration(config, f)

File Format

A PMC file is the configuration file for Procmon, which can be exported using the graphical interface, and later imported in another Procmon instance. This file contains a sequence of records, where every record has a name and a and a value. The known record options are:

• Columns - a list of the width of the GUI columns in pixels.
• ColumnCount - the number of columns to show.
• ColumnMap - an ordered list of the column types to show.
• DbgHelpPath
• Logfile - an optional path to a PML file to store the captured events.
• HighlightFG
• HighlightBG
• LogFont
• BoookmarkFont (they have a typo...)
• AdvancedMode
• Autoscroll
• HistoryDepth
• Profiling
• DestructiveFilter - whether to drop events that the current filters exclude.
• AlwaysOnTop
• ResolveAddresses
• SourcePath
• SymbolPath
• FilterRules - a list of filter rules that can be used to show only interesting events.
• HighlightRules

For the raw binary format of PMC files you can refer to configuration_format.py.

PML (Process Monitor Log) Parser

Usage

procmon-parser exports a ProcmonLogsReader class for reading logs directly from a PML file:

>>> from procmon_parser import ProcmonLogsReader
>>> f = open("LogFile.PML", "rb")
>>> pml_reader = ProcmonLogsReader(f)
>>> len(pml_reader)  # number of logs
53214

>>> first_event = next(pml_reader)  # reading the next event in the log
>>> first_event
Event("dwm.exe", 932, 1568, "Registry", "RegQueryValue", 7400, 2020-07-12T01:18:10.775242900, 0, "", "HKCU\Software\Microsoft\Windows\DWM\ColorPrevalence", {})

>>> print(first_event.process)  #  Accessing the process of the event
"C:\Windows\system32\dwm.exe", 932
>>> for module in first_event.process.modules[:3]:
...     print(module)  # printing information about some modules
"C:\Windows\system32\dwm.exe", address=0x7ff6fa980000, size=0x18000
"C:\Windows\system32\d3d10warp.dll", address=0x7fff96700000, size=0x76c000
"C:\Windows\system32\wuceffects.dll", address=0x7fff9a920000, size=0x3f000

>>> first_event.stacktrace  # get a list of the stack frames addresses from the event
[18446735291098361031, 18446735291098336505, 18446735291095097155, 140736399934388, 140736346856333, 140736346854333, 140698742953668, 140736303659045, 140736303655429, 140736303639145, 140736303628747, 140736303625739, 140736303693867, 140736303347333, 140736303383760, 140736303385017, 140736398440420, 140736399723393]
>>>

File Format

For the raw binary format of PML files you can refer to logs_format.py and logs_details_format.py.

Currently the parser can handle PML files saved by Procmon.exe of versions 3.4.0 or higher.

Unsupported features

The PML format is very complex so there are some features that are not supported yet:

• Reading logs from multiple related PML files.
• Detail column is not fully supported for all operation types.

Pull requests to support these missing features are very welcome :)

Tests

To test that the parser works as expected, There are two fairly large PML files, on Windows 7 32 bit and Windows 10 64 bit,

Contributing

procmon-parser is developed on GitHub at eronnen/procmon-parser. Feel free to report an issue or send a pull request, use the issue tracker.