File Crawler index files and search hard-coded credentials.
Project description
Knows More
FileCrawler officially supports Python 3.7+.
Main features
- List all file contents
- Index file contents at Elasticsearch
- Do OCR at several file types (with tika lib)
- Look for hard-coded credentials
- Much more...
Parsers:
- PDF files
- Microsoft Office files (Word, Excel etc)
- X509 Certificate files
- Image files (Jpg, Png, Gif etc)
- Java packages (Jar and war)
- Disassembly APK Files with APKTool
- Compressed files (zip, tar, gzip etc)
- SQLite3 database
Extractors:
- AWS credentials
- Github and gitlab credentials
Installing
Dependencies
apt install default-jre default-jdk libmagic-dev git
Installing FileCrawler
Installing from last release
pip install -U filecrawler
Installing development package
pip install -i https://test.pypi.org/simple/ FileCrawler
Running
Config file
Create a sample config file with default parameters
filecrawler --create-config -v
Edit the configuration file config.yml with your desired parameters
Note: You must adjust the Elasticsearch URL parameter before continue
Run
filecrawler --index-name filecrawler --path /mnt/client_files --crawler --elastic -T 30 -v
Help
$ filecrawler -h
File Crawler v0.1.1 by Helvio Junior
File Crawler index files and search credentials.
https://github.com/helviojunior/filecrawler
usage:
filecrawler module [flags]
Available Modules:
--crawler Crawler folder and files
Global Flags:
--index-name [index name] Crawler name
--path [folder path] Folder path to be indexed
--config [config file] Configuration file. (default: ./fileindex.yml)
--db [sqlite file] Filename to save status of indexed files. (default: ~/.filecrawler/{index_name}/indexer.db)
-T [tasks] number of connects in parallel (per host, default: 16)
--create-config Create config sample
--clear-session Clear old file status and reindex all files
-h, --help show help message and exit
-v Specify verbosity level (default: 0). Example: -v, -vv, -vvv
Use "filecrawler [module] --help" for more information about a command.
How-to install ELK from scratch
Credits
This project was inspired of:
Note: Some part of codes was ported from this 2 projects
To do
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
FileCrawler-0.1.2.tar.gz
(23.4 MB
view hashes)