mft2es

A command-line tool and Python library for parsing Windows Master File Table ($MFT) and importing the results into Elasticsearch.

mft2es leverages the Rust-based parser pymft-rs, making it faster than pure-Python parsers in many cases.

Usage

mft2es can be used as a standalone command-line tool or integrated directly into your Python scripts.

$ mft2es '/path/to/your/$MFT'

from mft2es import mft2es

mft2es("/path/to/your/$MFT")

Arguments

mft2es can process multiple files at once:

$ mft2es 'file1/$MFT' 'file2/$MFT' 'file3/$MFT'

mft2es can recursively process all MFT and $MFT files under a specified directory:

$ tree .
mftfiles/
  ├── $MFT
  └── subdirectory/
    ├── $MFT
    └── subsubdirectory/
      └── $MFT

$ mft2es /mftfiles/ # The path is recursively expanded to all MFT and $MFT files.

Options

--version, -v

--help, -h

--quiet, -q
  Suppress standard output
  (default: False)

--multiprocess, -m:
  Enable multiprocessing for faster processing.
  (default: False)

--size:
  Number of records to process per chunk (default: 500)

--host:
  Elasticsearch host address (default: localhost)

--port:
  Elasticsearch port number (default: 9200)

--index:
  Destination index name (default: mft2es)

--scheme:
  Protocol scheme to use (http or https) (default: http)

--pipeline:
  Elasticsearch ingest pipeline to use (default: )

--timeline:
  Enable MACB timeline analysis mode
  (default: False)

--tags:
  Comma-separated tags to add to each record for identification
  (e.g., hostname, domain name) (default: )

--login:
  Username for Elasticsearch authentication

--pwd:
  Password for Elasticsearch authentication

--no-verify-certs:
  Disable TLS certificate verification (default: False)

--ca-certs:
  Path to a CA certificate bundle for TLS verification (default: None)

Examples

When using from the command line:

$ mft2es '/path/to/your/$MFT' --host=localhost --port=9200 --index=foobar --size=500

When using from a Python script:

mft2es("/path/to/your/$MFT", host="localhost", port=9200, index="foobar", size=500)

With Elasticsearch authentication:

$ mft2es '/path/to/your/$MFT' --host=localhost --port=9200 --index=foobar --login=elastic --pwd=******

With timeline analysis mode:

$ mft2es '/path/to/your/$MFT' --timeline --index=mft-timeline

With tags for host identification:

$ mft2es '/path/to/your/$MFT' --tags "WORKSTATION-1,DOMAIN-ABC" --index=host-analysis

Appendix

mft2json

mft2es also includes mft2json, a command-line tool for converting Windows Master File Table records into JSON files. 🍣 🍣 🍣

$ mft2json '/path/to/your/$MFT' -o /path/to/output/target.json

mft2json also supports line-delimited output. --format jsonl (or ndjson) writes one record per line without holding the entire dataset in memory. When no output path is specified, the default extension is .jsonl:

$ mft2json '/path/to/your/$MFT' --format jsonl

With tags for host identification:

$ mft2json '/path/to/your/$MFT' --tags "WORKSTATION-1,DOMAIN-ABC" -o /path/to/output/target.json

You can also convert $MFT records directly into a Python list[dict]:

from mft2es import mft2json

result: list[dict] = mft2json("/path/to/your/$MFT")

Timeline Analysis

mft2es supports timeline analysis mode that creates MACB (Modified, Accessed, Changed, Birth) timeline records for forensic investigation.

$ mft2es '/path/to/your/$MFT' --timeline --index=mft-timeline

Output Format Examples

Standard Mode

[
  {
    "header": {
      "signature": [
        70,
        73,
        76,
        69
      ],
      "usa_offset": 48,
      "usa_size": 3,
      "metadata_transaction_journal": 172848302,
      "sequence": 1,
      "hard_link_count": 1,
      "first_attribute_record_offset": 56,
      "flags": "ALLOCATED",
      "used_entry_size": 416,
      "total_entry_size": 1024,
      "base_reference": {
        "entry": 0,
        "sequence": 0
      },
      "first_attribute_id": 6,
      "record_number": 0
    },
    "attributes": {
      "StandardInformation": {
        "header": {
          "type_code": "StandardInformation",
          "record_length": 96,
          "form_code": 0,
          "residential_header": {
            "index_flag": 0
          },
          "name_size": 0,
          "name_offset": null,
          "data_flags": "(empty)",
          "instance": 0,
          "name": ""
        },
        "data": {
          "created": "2019-03-11T16:42:33.593750Z",
          "modified": "2019-03-11T16:42:33.593750Z",
          "mft_modified": "2019-03-11T16:42:33.593750Z",
          "accessed": "2019-03-11T16:42:33.593750Z",
          "file_flags": "FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM",
          "max_version": 0,
          "version": 0,
          "class_id": 0,
          "owner_id": 0,
          "security_id": 256,
          "quota": 0,
          "usn": 0
        }
      },
      "FileName": {
        "header": {
          "type_code": "FileName",
          "record_length": 104,
          "form_code": 0,
          "residential_header": {
            "index_flag": 1
          },
          "name_size": 0,
          "name_offset": null,
          "data_flags": "(empty)",
          "instance": 3,
          "name": ""
        },
        "data": {
          "parent": {
            "entry": 5,
            "sequence": 5
          },
          "created": "2019-03-11T16:42:33.593750Z",
          "modified": "2019-03-11T16:42:33.593750Z",
          "mft_modified": "2019-03-11T16:42:33.593750Z",
          "accessed": "2019-03-11T16:42:33.593750Z",
          "logical_size": 16384,
          "physical_size": 16384,
          "flags": "FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM",
          "reparse_value": 0,
          "name_length": 4,
          "namespace": "Win32AndDos",
          "name": "$MFT",
          "path": "$MFT"
        }
      },
      "DATA": {
        "header": {
          "type_code": "DATA",
          "record_length": 72,
          "form_code": 1,
          "residential_header": {
            "vnc_first": 0,
            "vnc_last": "0x198f",
            "unit_compression_size": 0,
            "allocated_length": 62390272,
            "file_size": 62390272,
            "valid_data_length": 62390272,
            "total_allocated": null
          },
          "name_size": 0,
          "name_offset": null,
          "data_flags": "(empty)",
          "instance": 1,
          "name": ""
        },
        "data": null
      },
      "BITMAP": {
        "header": {
          "type_code": "BITMAP",
          "record_length": 80,
          "form_code": 1,
          "residential_header": {
            "vnc_first": 0,
            "vnc_last": 0,
            "unit_compression_size": 0,
            "allocated_length": 12288,
            "file_size": 8200,
            "valid_data_length": 8200,
            "total_allocated": null
          },
          "name_size": 0,
          "name_offset": null,
          "data_flags": "(empty)",
          "instance": 5,
          "name": ""
        },
        "data": null
      }
    },
    "tags": ["mft", "WORKSTATION-1", "DOMAIN-ABC"]
  },
  ...
]

Timeline Mode

[
  {
    "@timestamp": "2007-06-30T12:50:52.252395Z",
    "event": {
      "action": "mft-standardinformation-m",
      "category": [
        "file"
      ],
      "type": [
        "change"
      ],
      "kind": "event",
      "provider": "mft",
      "module": "windows",
      "dataset": "windows.mft"
    },
    "windows": {
      "mft": {
        "record": {
          "number": 0,
          "name": "$MFT",
          "path": "$MFT"
        },
        "header": {
          "signature": [
            70,
            73,
            76,
            69
          ],
          "usa_offset": 48,
          "usa_size": 3,
          "metadata_transaction_journal": 77648146,
          "sequence": 1,
          "hard_link_count": 1,
          "first_attribute_record_offset": 56,
          "flags": "ALLOCATED",
          "used_entry_size": 424,
          "total_entry_size": 1024,
          "base_reference": {
            "entry": 0,
            "sequence": 0
          },
          "first_attribute_id": 6
        },
        "attribute": {
          "type": "StandardInformation",
          "macb_type": "M",
          "header": {
            "record_length": 96,
            "form_code": 0,
            "residential_header": {
              "index_flag": 0
            },
            "name_size": 0,
            "name_offset": null,
            "data_flags": "(empty)",
            "instance": 0,
            "name": ""
          },
          "data": {
            "file_flags": "FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM",
            "max_version": 0,
            "version": 0,
            "class_id": 0,
            "owner_id": 0,
            "security_id": 256,
            "quota": 0,
            "usn": 0
          }
        }
      }
    },
    "log": {
      "file": {
        "path": "/path/to/your/MFT"
      }
    },
    "tags": [
      "mft"
    ]
  },
  ...
]

Installation

From PyPI

$ pip install mft2es

With uv

$ uv add mft2es

From GitHub Releases

Standalone binaries built with Nuitka are available from GitHub Releases for systems without a Python environment.

$ chmod +x ./mft2es
$ ./mft2es {{options...}}

> mft2es.exe {{options...}}

Contributing

The source code for mft2es is hosted on GitHub: https://github.com/sumeshi/mft2es. Please report issues and feature requests. 🍣 🍣 🍣

Included in

• Tsurugi Linux [Lab] — included in selected releases.

Thank you for your interest in mft2es!

License

Released under the MIT License.

Third-party licenses

The standalone binaries distributed via GitHub Releases may bundle the following third-party libraries. These libraries remain under their original licenses.

Apache-2.0

• elasticsearch-py / elasticsearch — licensed under the Apache License 2.0.
  • Bundled version: elasticsearch==9.4.1
  • License text: https://github.com/elastic/elasticsearch-py/blob/main/LICENSE

MIT

• mft / pymft-rs — licensed under the MIT License.

  • Bundled version: mft==0.6.1
  • License text: https://github.com/omerbenamram/pymft-rs/blob/master/pyproject.toml

• urllib3 — licensed under the MIT License.

  • Bundled version: urllib3==2.6.3
  • License text: https://github.com/urllib3/urllib3/blob/main/LICENSE.txt

Apache-2.0 OR MIT, with MPL-2.0 components

• orjson — licensed under Apache-2.0 OR MIT, and contains source code licensed under MPL-2.0.
  • Bundled version: orjson==3.11.9
  • License text:
    • https://github.com/ijl/orjson/blob/master/LICENSE-APACHE
    • https://github.com/ijl/orjson/blob/master/LICENSE-MIT
    • https://github.com/ijl/orjson/blob/master/LICENSE-MPL-2.0

MIT and MPL-2.0

• tqdm — licensed under MIT, with MPL-2.0-covered files/components.
  • Bundled version: tqdm==4.67.3
  • License text: https://github.com/tqdm/tqdm/blob/master/LICENCE