Parse a list of indicators into a dictionary or JSON structure for programmatic use.

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for bonifield_tools from gravatar.com bonifield_tools

Unverified details

These details have not been verified by PyPI
Meta
  • Requires: Python >=3.11
Report project as malware

Project description

ParseIOC

Parse a list of indicators into a dictionary, JSON structure, or Sqlite database for programmatic use.

Handles:

  • emails and their domains
  • URL domains, and username+password and ports if present
  • ipv4 and ipv6 addresses and networks
  • md5, sha256, and sha512 hashes

...with the goal of mapping this dictionary to fields in a SIEM or similar database.

This enables programatic use of an IOC file to quickly query specific values in a SIEM, such as source.ip or file.hash.sha256.

This tool will support more formats (imphash, ssdeep, ja3+) "eventually".

Otherwise-unidentifiable strings are categorized as "file".

Setup

uv add parse_ioc
# or
pip3 install parse_ioc

Usage

from parse_ioc import ParseIOC, map_fields, parse_multi, to_sqlite

Categorize a single indicator

i = ParseIOC("https://192.168.20.20/bad.txt")
i.to_dict
i.to_json

Categorize a list or file of indicators

setting mode="single" will produce the same results as calling ParseIOC(ioc) by itself

p = parse_multi(ioc_list, mode="combined")
# or
p = parse_multi("ioc_examples.txt", mode="combined")

print(json.dumps(p, indent=4))

Map a list or file of indicators to a TOML of SIEM fields

m = map_fields("ioc_examples.txt", "map_ecs.toml")
print(json.dumps(m, indent=4))

Create Sqlite database of parsed indicators

The schema is simply "ioc" and "type"; the dict/JSON keys, are the type values

to_sqlite("ioc_examples.txt", "iocs.db")

Gotchas

  • Domain names must have a period (.) otherwise the string will be considered a file
  • This regex in the domain checker, determines if the parsed domain name drops further down to the "file" bucket or not:
file_extensions = re.compile(r"\.(?:exe|dll|msi|bat|cmd|elf|scr|cpl|ps1|vbs|pdf|docx|xlsx|pptx|doc|xls|ppt|rtf|csv|txt|log|xml|zip|rar|7z|tar|gz|iso|img|dmg|cab|png|jpg|jpeg|gif|ico|bmp|svg|mp3|mp4|wav|avi|mov|js|php|css|html|htm|sql|conf|ini|yaml|yml|json)$", re.IGNORECASE)

Known Issues

  • need to streamline and optimize code
  • remove both regex statements created by AI

Expected sample output from running parse_ioc.py directly

======================== categorize a single indicator =========================
.to_dict: <class 'dict'> {'ioc': '192.168.20.20', 'ioc_type': 'ipv4'}
.to_json: <class 'str'> {"ioc": "192.168.20.20", "ioc_type": "ipv4"}
================ parse a list of IOCs into a combined structure ================
{
    "email": [
        "bob@email.local"
    ],
    "domain": [
        "anotherwebsite.local",
        "securewebsite.local",
        "n--nhk-u63b1cko2lyc6jrwxgom6k.com",
        "website.local",
        "email.local"
    ],
    "port": [
        9443,
        8443
    ],
    "credentials": [
        "username:password"
    ],
    "ipv4": [
        "192.168.20.20",
        "192.168.1.1"
    ],
    "ipv4_network": [
        "192.168.1.0/24"
    ]
}
================ parse a file of IOCs into a combined structure ================
{
    "email": [
        "bad.username@subdomain.bad.local",
        "bob@email.local"
    ],
    "domain": [
        "anotherwebsite.local",
        "securewebsite.local",
        "bad.local",
        "n--nhk-u63b1cko2lyc6jrwxgom6k.com",
        "website.local",
        "subdomain.bad.local",
        "email.local"
    ],
    "ipv4": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1"
    ],
    "port": [
        9443,
        8443
    ],
    "credentials": [
        "username:password"
    ],
    "ipv4_network": [
        "192.168.1.0/24",
        "8.8.8.0/24"
    ],
    "ipv6": [
        "fe80::"
    ],
    "file_path_linux": [
        "/home/bob/file.txt"
    ],
    "file": [
        "aaaaaaaaaaaaaaaa",
        "file.txt",
        "bob documents.pdf",
        "rc:\\users\\bob smith\\desktop\\file.txt:alt.exe",
        "not-an-ioc"
    ],
    "file_path_windows": [
        "c:/users/bob smith/desktop/file.txt:alt.exe",
        "c:\\users\\bob\\desktop\\test.txt",
        "file.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file2.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file.txt"
    ],
    "md5": [
        "6cd3556deb0da54bca060b4c39479839"
    ],
    "sha256": [
        "315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3"
    ],
    "sha512": [
        "c1527cd893c124773d811911970c8fe6e857d6df5dc9226bd8a160614c0cd963a4ddea2b94bb7d36021ef9d865d5cea294a82dd49a0bb269f51f6e7a57f79421"
    ]
}
=== yield dictionaries from a list or file, instead of a combined structure ====
{'ioc': 'bob@email.local', 'ioc_type': 'email', 'extra': [{'ioc': 'email.local', 'ioc_type': 'domain'}]}
{'ioc': 'bad.username@subdomain.bad.local', 'ioc_type': 'email', 'extra': [{'ioc': 'subdomain.bad.local', 'ioc_type': 'domain'}]}
{'ioc': '8.8.8.8', 'ioc_type': 'ipv4'}
{'ioc': '1.2.3.4', 'ioc_type': 'ipv4'}
{'ioc': 'website.local', 'ioc_type': 'domain'}
{'ioc': 'anotherwebsite.local', 'ioc_type': 'domain', 'extra': [{'ioc': 9443, 'ioc_type': 'port'}]}
{'ioc': 'securewebsite.local', 'ioc_type': 'domain', 'extra': [{'ioc': 'username:password', 'ioc_type': 'credentials'}, {'ioc': 8443, 'ioc_type': 'port'}]}
{'ioc': '192.168.1.1', 'ioc_type': 'ipv4'}
{'ioc': '192.168.1.0/24', 'ioc_type': 'ipv4_network'}
{'ioc': '8.8.8.0/24', 'ioc_type': 'ipv4_network'}
{'ioc': '192.168.20.20', 'ioc_type': 'ipv4'}
{'ioc': 'fe80::', 'ioc_type': 'ipv6'}
{'ioc': 'n--nhk-u63b1cko2lyc6jrwxgom6k.com', 'ioc_type': 'domain'}
{'ioc': 'bad.local', 'ioc_type': 'domain'}
{'ioc': '/home/bob/file.txt', 'ioc_type': 'file_path_linux'}
{'ioc': 'rc:\\users\\bob smith\\desktop\\file.txt:alt.exe', 'ioc_type': 'file'}
{'ioc': 'file.txt', 'ioc_type': 'file'}
{'ioc': 'file.txt:alt.exe', 'ioc_type': 'file_path_windows'}
{'ioc': 'bob documents.pdf', 'ioc_type': 'file'}
{'ioc': '/home/bob/file.txt', 'ioc_type': 'file_path_linux'}
{'ioc': 'c:\\users\\bob\\desktop\\test.txt', 'ioc_type': 'file_path_windows'}
{'ioc': 'c:\\users\\bob smith\\desktop\\file.txt', 'ioc_type': 'file_path_windows'}
{'ioc': 'c:\\users\\bob smith\\desktop\\file.txt', 'ioc_type': 'file_path_windows'}
{'ioc': 'c:\\users\\bob smith\\desktop\\file.txt:alt.exe', 'ioc_type': 'file_path_windows'}
{'ioc': 'c:\\users\\bob smith\\desktop\\file.txt:alt.exe', 'ioc_type': 'file_path_windows'}
{'ioc': 'c:\\users\\bob smith\\desktop\\file.txt:alt.exe', 'ioc_type': 'file_path_windows'}
{'ioc': 'c:\\users\\bob smith\\desktop\\file2.txt:alt.exe', 'ioc_type': 'file_path_windows'}
{'ioc': 'c:/users/bob smith/desktop/file.txt:alt.exe', 'ioc_type': 'file_path_windows'}
{'ioc': 'not-an-ioc', 'ioc_type': 'file'}
{'ioc': 'aaaaaaaaaaaaaaaa', 'ioc_type': 'file'}
{'ioc': '6cd3556deb0da54bca060b4c39479839', 'ioc_type': 'md5'}
{'ioc': '315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3', 'ioc_type': 'sha256'}
{'ioc': 'c1527cd893c124773d811911970c8fe6e857d6df5dc9226bd8a160614c0cd963a4ddea2b94bb7d36021ef9d865d5cea294a82dd49a0bb269f51f6e7a57f79421', 'ioc_type': 'sha512'}
======== provide IOC (file or list) and TOML config (path) map_fields() ========
{
    "email.from.address": [
        "bad.username@subdomain.bad.local",
        "bob@email.local"
    ],
    "email.sender.address": [
        "bad.username@subdomain.bad.local",
        "bob@email.local"
    ],
    "email.to.address": [
        "bad.username@subdomain.bad.local",
        "bob@email.local"
    ],
    "email.reply_to.address": [
        "bad.username@subdomain.bad.local",
        "bob@email.local"
    ],
    "email.cc.address": [
        "bad.username@subdomain.bad.local",
        "bob@email.local"
    ],
    "email.bcc.address": [
        "bad.username@subdomain.bad.local",
        "bob@email.local"
    ],
    "destination.domain": [
        "anotherwebsite.local",
        "securewebsite.local",
        "bad.local",
        "n--nhk-u63b1cko2lyc6jrwxgom6k.com",
        "website.local",
        "subdomain.bad.local",
        "email.local"
    ],
    "url.domain": [
        "anotherwebsite.local",
        "securewebsite.local",
        "bad.local",
        "n--nhk-u63b1cko2lyc6jrwxgom6k.com",
        "website.local",
        "subdomain.bad.local",
        "email.local"
    ],
    "tls.client.server_name": [
        "anotherwebsite.local",
        "securewebsite.local",
        "bad.local",
        "n--nhk-u63b1cko2lyc6jrwxgom6k.com",
        "website.local",
        "subdomain.bad.local",
        "email.local"
    ],
    "dns.question.registered_domain": [
        "anotherwebsite.local",
        "securewebsite.local",
        "bad.local",
        "n--nhk-u63b1cko2lyc6jrwxgom6k.com",
        "website.local",
        "subdomain.bad.local",
        "email.local"
    ],
    "file.origin_referrer_url": [
        "anotherwebsite.local",
        "securewebsite.local",
        "bad.local",
        "n--nhk-u63b1cko2lyc6jrwxgom6k.com",
        "website.local",
        "subdomain.bad.local",
        "email.local"
    ],
    "file.origin_url": [
        "anotherwebsite.local",
        "securewebsite.local",
        "bad.local",
        "n--nhk-u63b1cko2lyc6jrwxgom6k.com",
        "website.local",
        "subdomain.bad.local",
        "email.local"
    ],
    "source.ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "destination.ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "dns.resolved_ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "host.ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "network.forwarded_ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "related.ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "client.ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "server.ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "server.nat.ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "threat.enrichments.indicator.ip": [
        "192.168.20.20",
        "1.2.3.4",
        "8.8.8.8",
        "192.168.1.1",
        "192.168.1.0/24",
        "8.8.8.0/24",
        "fe80::"
    ],
    "file.path": [
        "/home/bob/file.txt",
        "c:/users/bob smith/desktop/file.txt:alt.exe",
        "c:\\users\\bob\\desktop\\test.txt",
        "file.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file2.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file.txt"
    ],
    "file.name": [
        "/home/bob/file.txt",
        "c:/users/bob smith/desktop/file.txt:alt.exe",
        "c:\\users\\bob\\desktop\\test.txt",
        "file.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file2.txt:alt.exe",
        "c:\\users\\bob smith\\desktop\\file.txt"
    ],
    "dll.hash.md5": [
        "6cd3556deb0da54bca060b4c39479839"
    ],
    "email.attachments.file.hash.md5": [
        "6cd3556deb0da54bca060b4c39479839"
    ],
    "file.hash.md5": [
        "6cd3556deb0da54bca060b4c39479839"
    ],
    "process.hash.md5": [
        "6cd3556deb0da54bca060b4c39479839"
    ],
    "dll.hash.sha256": [
        "315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3"
    ],
    "email.attachments.file.hash.sha256": [
        "315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3"
    ],
    "file.hash.sha256": [
        "315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3"
    ],
    "process.hash.sha256": [
        "315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3"
    ],
    "dll.hash.sha512": [
        "c1527cd893c124773d811911970c8fe6e857d6df5dc9226bd8a160614c0cd963a4ddea2b94bb7d36021ef9d865d5cea294a82dd49a0bb269f51f6e7a57f79421"
    ],
    "email.attachments.file.hash.sha512": [
        "c1527cd893c124773d811911970c8fe6e857d6df5dc9226bd8a160614c0cd963a4ddea2b94bb7d36021ef9d865d5cea294a82dd49a0bb269f51f6e7a57f79421"
    ],
    "file.hash.sha512": [
        "c1527cd893c124773d811911970c8fe6e857d6df5dc9226bd8a160614c0cd963a4ddea2b94bb7d36021ef9d865d5cea294a82dd49a0bb269f51f6e7a57f79421"
    ],
    "process.hash.sha512": [
        "c1527cd893c124773d811911970c8fe6e857d6df5dc9226bd8a160614c0cd963a4ddea2b94bb7d36021ef9d865d5cea294a82dd49a0bb269f51f6e7a57f79421"
    ],
    "source.port": [
        9443,
        8443
    ],
    "destination.port": [
        9443,
        8443
    ]
}
========================== output to Sqlite database ===========================
exported 34 indicators to out.db
('bad.username@subdomain.bad.local', 'email')
('bob@email.local', 'email')
('n--nhk-u63b1cko2lyc6jrwxgom6k.com', 'domain')
('website.local', 'domain')
('email.local', 'domain')
('securewebsite.local', 'domain')
('bad.local', 'domain')
('subdomain.bad.local', 'domain')
('anotherwebsite.local', 'domain')
('8.8.8.8', 'ipv4')
('192.168.20.20', 'ipv4')
('192.168.1.1', 'ipv4')
('1.2.3.4', 'ipv4')
('9443', 'port')
('8443', 'port')
('username:password', 'credentials')
('192.168.1.0/24', 'ipv4_network')
('8.8.8.0/24', 'ipv4_network')
('fe80::', 'ipv6')
('/home/bob/file.txt', 'file_path_linux')
('file.txt', 'file')
('rc:\\users\\bob smith\\desktop\\file.txt:alt.exe', 'file')
('aaaaaaaaaaaaaaaa', 'file')
('bob documents.pdf', 'file')
('not-an-ioc', 'file')
('c:\\users\\bob smith\\desktop\\file.txt', 'file_path_windows')
('c:/users/bob smith/desktop/file.txt:alt.exe', 'file_path_windows')
('file.txt:alt.exe', 'file_path_windows')
('c:\\users\\bob smith\\desktop\\file.txt:alt.exe', 'file_path_windows')
('c:\\users\\bob smith\\desktop\\file2.txt:alt.exe', 'file_path_windows')
('c:\\users\\bob\\desktop\\test.txt', 'file_path_windows')
('6cd3556deb0da54bca060b4c39479839', 'md5')
('315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3', 'sha256')
('c1527cd893c124773d811911970c8fe6e857d6df5dc9226bd8a160614c0cd963a4ddea2b94bb7d36021ef9d865d5cea294a82dd49a0bb269f51f6e7a57f79421', 'sha512')

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for bonifield_tools from gravatar.com bonifield_tools

Unverified details

These details have not been verified by PyPI
Meta
  • Requires: Python >=3.11

Release history Release notifications | RSS feed

This version

0.3.0

0.2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parse_ioc-0.3.0.tar.gz (21.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

parse_ioc-0.3.0-py3-none-any.whl (21.5 kB view details)

Uploaded Python 3

File details

Details for the file parse_ioc-0.3.0.tar.gz.

File metadata

  • Download URL: parse_ioc-0.3.0.tar.gz
  • Upload date:
  • Size: 21.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for parse_ioc-0.3.0.tar.gz
Algorithm Hash digest
SHA256 43082f121a877d24e0c95808c1ff1264623751fcc1052bac5f97d61ab3d46852
MD5 a22f2bfd8a76a1d150f88f6c07093642
BLAKE2b-256 0c37d813df57697adf72b72c2d9dcb955929f2fcde6262213cc624563f73198e

See more details on using hashes here.

File details

Details for the file parse_ioc-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: parse_ioc-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 21.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for parse_ioc-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 66345c09d76ef647048f788d78b6b698ea470400873b1579c3b82404577b95cb
MD5 3b73eccc626164743fce599ceabccfdd
BLAKE2b-256 2e3ab9a27c954daa262d956ed8de6f95c35322fc4cad5f8a20adef60b79772c2

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page