mft2es 1.4.0

A library for fast import of Windows Master File Table($MFT) into Elasticsearch.

mft2es

Fast import of Windows Master File Table($MFT) into Elasticsearch.

mft2es uses Rust library pymft-rs, so it runs much faster than traditional software.

Usage

mft2es can be executed from the command line or incorporated into a Python script.

$ mft2es /path/to/your/$MFT

or

from mft2es import mft2es

if __name__ == '__main__':
  filepath = '/path/to/your/$MFT'
  mft2es(filepath)

Args

mft2es supports simultaneous import of multiple files.

$ mft2es foo/MFT bar/MFT

Additionally, it also allows for recursive import under the specified directory.

$ tree .
mftfiles/
  ├── MFT
  └── subdirectory/
    ├── MFT
    └── subsubdirectory/
      ├── MFT
      └── $MFT

$ mft2es /mftfiles/ # The Path is recursively expanded to all MFT, and $MFT.

Options

--version, -v

--help, -h

--quiet, -q
  Flag to suppress standard output
  (default: False)

--multiprocess, -m:
  Enable multiprocessing for faster execution
  (default: False)

--size:
  Chunk size for processing (default: 500)

--host:
  ElasticSearch host address (default: localhost)

--port:
  ElasticSearch port number (default: 9200)

--index:
  Destination index name for importing (default: mft2es)

--scheme:
  Protocol scheme to use (http or https) (default: http)

--pipeline
  Elasticsearch Ingest Pipeline to use (default: )

--login:
  The login to use if Elastic Security is enabled (default: )

--pwd:
  The password associated with the provided login (default: )

Examples

When using from the commandline interface:

$ mft2es /path/to/your/$MFT --host=localhost --port=9200 --index=foobar --size=500

When using from the python-script:

if __name__ == '__main__':
    mft2es('/path/to/your/$MFT', host=localhost, port=9200, index='foobar', size=500)

With credentials for Elastic Security:

$ mft2es /path/to/your/$MFT --host=localhost --port=9200 --index=foobar --login=elastic --pwd=******

Note: The current version does not verify the certificate.

Appendix

Mft2json

An additional feature: :sushi: :sushi: :sushi:

Convert Windows MFT to a JSON file.

$ mft2json /path/to/your/$MFT -o /path/to/output/target.json

Convert Windows Event Logs to a Python List[dict] object.

from mft2es import mft2json

if __name__ == '__main__':
  filepath = '/path/to/your/$MFT'
  result: List[dict] = mft2json(filepath)

Output Format

The structures is not well optimized for searchable with Elasticsearch. I'm waiting for your PR!!

[
  {
    "header": {
      "signature": [
        70,
        73,
        76,
        69
      ],
      "usa_offset": 48,
      "usa_size": 3,
      "metadata_transaction_journal": 172848302,
      "sequence": 1,
      "hard_link_count": 1,
      "first_attribute_record_offset": 56,
      "flags": "ALLOCATED",
      "used_entry_size": 416,
      "total_entry_size": 1024,
      "base_reference": {
        "entry": 0,
        "sequence": 0
      },
      "first_attribute_id": 6,
      "record_number": 0
    },
    "attributes": {
      "StandardInformation": {
        "header": {
          "type_code": "StandardInformation",
          "record_length": 96,
          "form_code": 0,
          "residential_header": {
            "index_flag": 0
          },
          "name_size": 0,
          "name_offset": null,
          "data_flags": "(empty)",
          "instance": 0,
          "name": ""
        },
        "data": {
          "created": "2019-03-11T16:42:33.593750Z",
          "modified": "2019-03-11T16:42:33.593750Z",
          "mft_modified": "2019-03-11T16:42:33.593750Z",
          "accessed": "2019-03-11T16:42:33.593750Z",
          "file_flags": "FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM",
          "max_version": 0,
          "version": 0,
          "class_id": 0,
          "owner_id": 0,
          "security_id": 256,
          "quota": 0,
          "usn": 0
        }
      },
      "FileName": {
        "header": {
          "type_code": "FileName",
          "record_length": 104,
          "form_code": 0,
          "residential_header": {
            "index_flag": 1
          },
          "name_size": 0,
          "name_offset": null,
          "data_flags": "(empty)",
          "instance": 3,
          "name": ""
        },
        "data": {
          "parent": {
            "entry": 5,
            "sequence": 5
          },
          "created": "2019-03-11T16:42:33.593750Z",
          "modified": "2019-03-11T16:42:33.593750Z",
          "mft_modified": "2019-03-11T16:42:33.593750Z",
          "accessed": "2019-03-11T16:42:33.593750Z",
          "logical_size": 16384,
          "physical_size": 16384,
          "flags": "FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM",
          "reparse_value": 0,
          "name_length": 4,
          "namespace": "Win32AndDos",
          "name": "$MFT",
          "path": "$MFT"
        }
      },
      "DATA": {
        "header": {
          "type_code": "DATA",
          "record_length": 72,
          "form_code": 1,
          "residential_header": {
            "vnc_first": 0,
            "vnc_last": "0x198f",
            "unit_compression_size": 0,
            "allocated_length": 62390272,
            "file_size": 62390272,
            "valid_data_length": 62390272,
            "total_allocated": null
          },
          "name_size": 0,
          "name_offset": null,
          "data_flags": "(empty)",
          "instance": 1,
          "name": ""
        },
        "data": null
      },
      "BITMAP": {
        "header": {
          "type_code": "BITMAP",
          "record_length": 80,
          "form_code": 1,
          "residential_header": {
            "vnc_first": 0,
            "vnc_last": 0,
            "unit_compression_size": 0,
            "allocated_length": 12288,
            "file_size": 8200,
            "valid_data_length": 8200,
            "total_allocated": null
          },
          "name_size": 0,
          "name_offset": null,
          "data_flags": "(empty)",
          "instance": 5,
          "name": ""
        },
        "data": null
      }
    }
  }
  ...
]

Installation

from PyPI

$ pip install mft2es

from GitHub Releases

The version compiled into a binary using Nuitka is also available for use.

$ chmod +x ./mft2es
$ ./mft2es {{options...}}

> mft2es.exe {{options...}}

Contributing

The source code for mft2es is hosted at GitHub, and you may download, fork, and review it from this repository(https://github.com/sumeshi/mft2es).
Please report issues and feature requests. :sushi: :sushi: :sushi:

License

mft2es is released under the MIT License.

Powered by following libraries:

• pymft-rs
• Nuitka