filecrawler 0.1.1

File Crawler index files and search credentials.

Knows More

FileCrawler officially supports Python 3.7+.

Main features

• List all file contents
• Index file contents at Elasticsearch
• Do OCR at several file types (with tika lib)
• Look for hard-coded credentials
• Much more...

Parsers:

• PDF files
• Microsoft Office files (Word, Excel etc)
• X509 Certificate files
• Image files (Jpg, Png, Gif etc)
• Java packages (Jar and war)
• Disassembly APK Files with APKTool
• Compressed files (zip, tar, gzip etc)

Extractors:

• AWS credentials
• Github and gitlab credentials

Installing

pip install -U filecrawler

Running

Config file

Create a sample config file with default parameters

filecrawler --create-config -v

Edit the configuration file config.yml with your desired parameters

Note: You must adjust the Elasticsearch URL parameter before continue

Run

filecrawler --index-name filecrawler --path /mnt/client_files --crawler --elastic -T 30 -v

Help

$ filecrawler -h

File Crawler v0.1.1 by Helvio Junior
File Crawler index files and search credentials.
https://github.com/helviojunior/filecrawler
    
usage: 
    filecrawler module [flags]

Available Modules:
  --crawler                  Crawler folder and files

Global Flags:
  --index-name [index name]  Crawler name
  --path [folder path]       Folder path to be indexed
  --config [config file]     Configuration file. (default: ./fileindex.yml)
  --db [sqlite file]         Filename to save status of indexed files. (default: ~/.filecrawler/{index_name}/indexer.db)
  -T [tasks]                 number of connects in parallel (per host, default: 16)
  --create-config            Create config sample
  --clear-session            Clear old file status and reindex all files
  -h, --help                 show help message and exit
  -v                         Specify verbosity level (default: 0). Example: -v, -vv, -vvv

Use "filecrawler [module] --help" for more information about a command.

Credits

This project was inspired of:

1. FSCrawler
2. Gitleaks

Note: Some part of codes was ported from this 2 projects

To do

Check the TODO file