A tool to search for data patterns in (encoded) binary data
Project description
MetExt : Metadata Extractor
MetExt (Metadata Extractor) is a Python library and CLI utility that allows to process data in different formats and extract data patterns of interest.
Installation
MetExt requires Python v3.5+.
You can install MetExt from PyPi.
$ python -m pip install metext
Or you can clone the repository install the project locally.
$ git clone https://github.com/espoem/MetExt.git
$ cd MetExt
$ python -m pip install -e .
Dependencies can also be installed from the requirements.txt
file.
$ python -m pip install -r requirements.txt
CLI
Help
metext --help
Usage
metext [-h] [-i INPUT] [-f FILE] [-r] [-o OUTPUT]
[-d [{quopri,yenc,base58ripple,base64url,pem,base58btc,base85,hexdump,base91,binhex,percent,ascii85,base64,mime,base32,hex,z85,raw,base32crockford,gzip,uu,base32hex} [{quopri,yenc,base58ripple,base64url,pem,base58btc,base85,hexdump,base91,binhex,percent,
ascii85,base64,mime,base32,hex,z85,raw,base32crockford,gzip,uu,base32hex} ...]]]
[-e [{bch,xrp,sha256,bip32-xkey,url,ipv4,pem,sha512,ltc,email,chainlink,md5,uuid,btc-wif,sha1,urn,issn,eth,isbn,json,base64,ada,usdt,base32,doi,magnet,sha224,ipv6,uri,data_uri,btc,mac,dot,sha384} [{bch,xrp,sha256,bip32-xkey,url,ipv4,pem,sha512,ltc,email,
chainlink,md5,uuid,btc-wif,sha1,urn,issn,eth,isbn,json,base64,ada,usdt,base32,doi,magnet,sha224,ipv6,uri,data_uri,btc,mac,dot,sha384} ...]]]
[-F {text,csv,json}] [--version]
Input paths support wildcards. If no input is provided, standard input is processed, i.e. input can be piped into metext. MetExt process input data in best effort, i.e. it tries to decode the data if encoded. The range of decoding to consider can be set via the argument -d
. The list of patterns to search for is set with the argument -e
. If not set, all patterns are searched for.
If no decoding is wanted to apply, set -d raw
.
API
List supported modules
from metext import list_decoders_names, list_extractors_names, list_printers_names
if __name__ == "__main__":
print(list_decoders_names())
print(list_extractors_names())
print(list_printers_names())
Analyse data for patterns
from metext import analyse, print_analysis_output
if __name__ == "__main__":
# file-like object
with open("filepath", "rb") as fp:
result = analyse(fp, "raw") # without decoding, find all supported patterns
print(result)
print_analysis_output(result) # by default prints to STDOUT in JSON format
print_analysis_output(
result, "output_filepath", "csv"
) # prints to "output_filepath" in CSV format
# input is list of filepaths
# consider either no, or base64 encoding
# extract ipv4 patterns
result = analyse(["filepath"], ["raw", "base64"], "ipv4")
# input is string
# limit decoding to provided decoders with custom kwargs
# limit patterns to extractors with custom kwargs
input_data = "some string"
result = analyse(
input_data,
[
(
"base64",
{
"charset": "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/"
},
)
],
[("base64", {})],
)
# input is bytes
# decode data in best effort
# extract all patterns
input_data = "some byte string"
result = analyse(input_data)
Modules
The tool's functionality is divided into individual modules (plugins) of a particular type. A new module can be created by inheriting from one of plugins' bases in plugin_base.py file. A plugin must have defined a unique pair PLUGIN_TYPE, PLUGIN_NAME.
Supported decoders
Decoder | Description |
---|---|
hex | Hexadecimal values, optionally with delimiters or prefixes |
hexdump | Hexdump format produced by tools hexdump and xdd |
base32 | base32 encoding as defined in RFC 4648 |
base32hex | base32hex encoding as defined in RFC 4648 |
base32crockford | Crockford's Base 32 encoding |
base58btc | Base 58 encoding using Bitcoin alphabet |
base58ripple | Base 58 encoding using Ripple alphabet |
base64 | base64 encoding as defined in RFC 4648. |
base64url | base64url encoding as defined in RFC 4648 |
base85 | base85 encoding as defined in RFC 1924 |
ascii85 | Ascii85 encoding |
z85 | Z85 encoding |
base91 | Base 91 encoding |
binhex | BinHex 4.0 encoding |
pem | Decodes Base 64 data in PEM format |
gzip | Decompress GZIP data |
mime | Decodes MIME message body |
percent | URL-encoding as defined in RFC 3986 |
quopri | Quoted-printable encoding as defined in RFC 2045 |
uu | UUEncoding |
yenc | yEnc encoding |
Supported pattern extractors
Extractor | Description |
---|---|
base32 | base32-encoded data |
base64 | base64-encoded data |
btc | Bitcoin P2PKH, P2SH and Bech32 mainnet addresses |
btc-wif | Bitcoin private key in wallet import format (WIF) |
bip32-key | Extended private and public keys as defined in BIP32 |
eth | Ethereum address |
ltc | Litecoin legacy and Bech32 address |
xrp | XRP (Ripple) classic address |
usdt | Tether (Omni, Ethereum) address |
bch | Bitcoin Cash legacy and CashAddr address |
chainlink | Chainlink (Ethereum) address |
ada | Cardano address |
dot | Polkadot address |
E-mail address | |
uuid | Universally unique identifier (UUID, GUID) |
md5 | MD5 hex digest |
sha1 | SHA-1 hex digest |
sha224 | SHA-224 hex digest |
sha256 | SHA-256 hex digest |
sha384 | SHA-384 hex digest |
sha512 | SHA-512 hex digest |
ipv4 | IPv4 address |
ipv6 | IPv6 address |
isbn | ISBN identifier |
issn | ISSN identifier |
doi | DOI identifier |
json | JSON non-empty object or array |
mac | MAC address |
pem | Data in PEM format |
uri | Generic URI with a registered scheme |
url | URL with http(s) or ftp scheme |
urn | URN |
data_uri | Data URI |
magnet | BitTorrent magnet link |
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for metext-0.0.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1be72e5c7625a6a760381526c2a865759c1b70c37a7086d8f0346a9cef870721 |
|
MD5 | 44a7f3dff8db465c003f62ac8c795ad1 |
|
BLAKE2b-256 | 3c38a6f878b210e3b7a7b4be0e7521c916a14a635d7e391469bdfb915be1a49a |