Skip to main content

command line tool to collect useful information from urls, directories, and files

Project description

The Hamburglar


There are 2 versions of hamburglar, full and lite. The main branch is the full version, and hamburglar lite is on a separate branch.


Full fledged scraping tool for artifact retrieval from multiple sources. There are some dependencies, so install them first:

pip3 install -r requirements.txt

Hamburglar also has the option of checking against file signatures during a hexdump. It will get skipped if not set up. To get it working, you will need to first create the database and a user:

CREATE USER 'hamman'@'localhost' IDENTIFIED BY 'deadbeef';
GRANT ALL PRIVILEGES ON fileSign.signatures TO 'hamman'@'localhost';

Then, run magic_sig_scraper. This can be run on a cronjob to regularly update it, or just run it once:


Hamburglar Lite

Multithreaded and recursive directory scraping script. Stores useful information with the filepath and finding. Hamburglar lite will never require external packages, and will always remain as a single script. Setup is as simple as requesting the file and using it:


This is designed to be quickly downloaded and executed on a machine.


usage: [-h] [-g] [-x] [-v] [-w] [-i] [-o FILE] [-y YARA] path

positional arguments:
  path                  path to directory, url, or file, depending on flag

optional arguments:
  -h, --help            show this help message and exit
  -g, --git             sets hamburglar into git mode
  -x, --hexdump         give hexdump of file
  -v, --verbose         increase output verbosity
  -w, --web             sets Hamburgler to web request mode, enter url as path
  -i, --ioc             uses iocextract to parse contents
  -o FILE, --out FILE   write results to FILE
  -y YARA, --yara YARA  use yara ruleset for checking

Directory Traversal

  • python3 ~/Directory/
    • This will recursively scan for files in the given directory, then analyzes each file for a variety of findings using regex filters

Single File Analysis

  • python3 ~/Directory/file.txt
    • This will recursively scan for files in the given directory, then analyzes each file for a variety of findings using regex filters

YARA Rule Based Analysis

  • python3 -y rules/ ~/Directory
    • This will compile the yara rule files in the rules directory and then check them against every item in Directory.

Git Scraping Mode

  • python3 -g
    • Adding -y <rulepath> will allow the repo to be scraped using yara rules

Web Request Mode

  • python3 -w
    • Adding a -w to tells the script to handle the path as a url.
    • Currently this does not spider the page, it just analyzes the requested html content

IOC Extraction

  • python3 -w -i
    • Adding a -i will use iocextract to extract any ioc's from the requested url

Hex Dump Mode

  • python3 -x ~/file-to-dump
    • This just does a hex dump and nothing more right now -- could be piped into a file
    • This will eventually be used for binary analysis


  • Adding -v will set the script into verbose mode, and -h will show details of available arguments
  • Adding -o FILENAME will set the results filename, this is especially useful in scripting situations where you might want multiple results tables (ie github repo spidering)


  • whitelistOn: turns on or off whitelist checking
  • maxWorkers: number of worker threads to run concurrently when reading file stack
  • whitelist: list of files or directories to exclusively scan for (if whitelistOn=True)
  • blacklist: list of files, extensions, or directories to block in scan
  • regexList: dictionary of regex filters with filter type as the key

The Hamburglar can find

  • ipv4 addresses (public and local)
  • emails
  • private keys
  • urls
  • ioc's (using iocextract)
  • cryptocurrency addresses
  • anything you can imagine using regex filters and yara rules

Example output:

    "/home/adam/Dev/test/email.txt": {
        "emails": "{''}"
    "/home/adam/Dev/test/email2.txt": {
        "emails": "{''}"
    "/home/adam/Dev/test/ips.txt": {
        "ipv4": "{'', ''}"
    "/home/adam/Dev/test/test2/email.txt": {
        "emails": "{'', ''}"
    "/home/adam/Dev/test/test2/ips.txt": {
        "ipv4": "{'', ''}"
    "/home/adam/Dev/test/test2/links.txt": {
        "site": "{''}"


Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hamburglar-1.0.1.tar.gz (28.7 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page