Skip to main content

Unleash Metadata Intelligence with MetaDetective. Your Assistant Beyond Metagoofil.

Project description

Contributors Forks Stargazers Issues MIT License GitHub unittest Workflow Status

MetaDetective Logo

🕵️‍♂️ MetaDetective

Unleash Metadata Intelligence with MetaDetective.
Bridging the chasm in metadata extraction and analysis.

📘 Explore the full documentation · 🎥 View Demo · 🐞 Report Bug · 🛠️ Request Feature

MetaDetective Demo Screenshot

📜 Table of Contents

Click to collapse/expand
  1. 📖 About
  2. 🛠️ Installation
  3. 🎮 Usage
  4. ❗ Troubleshooting
  5. 🤝 Contributing
  6. 📜 License
  7. 📞 Contact

🕵️‍♂️ About MetaDetective

MetaDetective: Advanced metadata extraction and web scraping.

Metadata, in the realm of cybersecurity, is more than just embedded information; it's a gateway to insightful perspectives, often unveiling crucial leads in OSINT and pentesting. MetaDetective was born out of the need to offer a potent solution, especially as prominent tools like Metagoofil on Kali Linux shifted their focus away from direct metadata analysis.

As key tools like Metagoofil on Kali Linux shifted their trajectory away from pure metadata analysis, the exigency for a robust alternative took center stage. Enter MetaDetective.

🧠 Tailored Metadata Analysis

Drawing inspiration from the foundational tools like Metagoofil, MetaDetective emerges as a revitalized and improved iteration, dedicated to providing efficient metadata extraction and presentation. It stands out as a comprehensive Python 3 tool, purposely designed to bridge the existing gaps in metadata analysis.

📊 Streamlined Data Presentation

Beyond mere extraction, MetaDetective prides itself on its capability to meticulously categorize and showcase metadata. Whether dealing with an individual file or an array of them, the tool ensures users grasp the entire spectrum of data, both in its breadth and depth.

🌐 Web Scraping

While Metagoofil once leaned on Google searches—a method riddled with IP restrictions and the labyrinth of proxy workarounds—MetaDetective pioneers a path with direct web scraping. By targeting sites directly, it sidesteps disruptions, delivering a dataset that's not just richer, but also more precise, spotlighting potential data leaks.

MetaDetective Scraping Demo Screenshot

🔍 Complementary Utility for OSINT and Pentesting

MetaDetective isn't just a standalone behemoth. It's crafted for seamless integration and synergy with tools like Metagoofil. A quintessential addition to every pentester's and OSINT researcher's toolkit, MetaDetective magnifies data acquisition prowess and broadens the horizons of analysis.

(🔼 Back to top)

🚀 Installation

Before diving into the installation process, ensure you meet the following prerequisites.

Prerequisites

  1. Python 3: Ensure Python 3 is installed on your system before initiating the installation process.

  2. Exiftool: Given its simplicity, MetaDetective doesn't rely on any external dependencies or libraries. However, it does necessitate exiftool. Ensure you have exiftool set up on your system.

⚠️ Note: MetaDetective has been rigorously tested with Python 3.11.4 on Linux alongside exiftool version 12.56. While it may function with other versions, compatibility is guaranteed only with these specific configurations.

Installation methods

  1. Git clone the repository:
git clone https://github.com/franckferman/MetaDetective.git
  1. Direct download: To skip cloning and directly download the script (designed for simplicity and flexibility, it doesn't depend on any external packages, so if you only need the script, you can also directly download it):
curl -O https://raw.githubusercontent.com/franckferman/MetaDetective/stable/src/MetaDetective/MetaDetective.py
  1. Pip Installation:
  • Create & Activate a Virtual Environment:
python3 -m venv MetaDetectiveEnv
source MetaDetectiveEnv/bin/activate
  • Install MetaDetective:
pip install MetaDetective
  1. Docker integration:

For a Docker-based setup, refer to our Docker-specific guide: MetaDetective Docker Setup.

(🔼 Back to top)

🎮 Usage

Ensure you adapt your command according to how you've set up MetaDetective.

Getting started

Kick off with the built-in help to explore MetaDetective's functionalities:

python3 src/MetaDetective/MetaDetective.py -h

Command examples

🕵️ File analysis:

Task Command
Analyze all files in directory python3 src/MetaDetective/MetaDetective.py -d examples/
Specific types & ignore patterns python3 src/MetaDetective/MetaDetective.py -d examples/ -i ^admin anonymous -t doc pdf
Display all results for each file python3 src/MetaDetective/MetaDetective.py -d examples/ -t all --display all

🔎 Export function:

Task Command
Default export (HTML) python3 src/MetaDetective/MetaDetective.py -d examples/ --export
Formatted display, txt export python3 src/MetaDetective/MetaDetective.py -d examples ---format formatted -e txt -o ~/

🌐 Web Scraping:

Task Command
Scan without downloading python3 src/MetaDetective/MetaDetective.py --scraping --scan --url https://example.com/
Download to specified directory python3 src/MetaDetective/MetaDetective.py --scraping --download-dir ~ --url https://example.com/
Download with set depth python3 src/MetaDetective/MetaDetective.py --scraping --depth 1 --download-dir ~ --url https://example.com/

Additional parameters

🌐 Web Scraping:

To initiate the web scraping mode, use the --scraping flag. Remember, this option doesn't function independently. It requires either a scanning or downloading parameter.

  • Activating web scraping mode:
python3 src/MetaDetective/MetaDetective.py --scraping
  • Scanning and displaying statistics: Ensure both the URL and --scan flags are used.
python3 src/MetaDetective/MetaDetective.py --scraping --scan --url https://example.com
  • Downloading web content: Indicate the desired directory using --download-dir and provide the target URL.
python3 src/MetaDetective/MetaDetective.py --scraping --download-dir ~ --url https://example.com
  • Adjusting scraping depth: Use the --depth flag to specify how deeply the scraper should navigate through links.
python3 src/MetaDetective/MetaDetective.py --scraping --scan --url https://aulnay-sous-bois.fr --depth 1
Additional Flags:
  • External link tracking: Use --follow-extern to allow tracking of external links (those outside the base URL). Typically not advised, but might be useful in certain contexts.

  • Thread management: Use --threads to specify the number of threads for concurrent operations.

  • Rate limiting: Use --rate to control the maximum number of requests per second.

🕵️ File analysis & Metadata Analyzer:

Basic Commands:

To begin analyzing files, you'll use either the -d or -f flag.

  • -d or --directory: Select a directory containing one or multiple files.
  • -f or --files: Choose a single or multiple specific files.

Analyze the contents of a directory.

python3 src/MetaDetective/MetaDetective.py -d examples

Analyze the contents of a file.

python3 src/MetaDetective/MetaDetective.py -f examples/MetaDetective.docx

Analyze the contents of multiple files.

python3 src/MetaDetective/MetaDetective.py -f examples/MetaDetective-APTX_4869_report.pdf examples/MetaDetective-Kogoro_s_Choice.pdf
Specifying data type

You can filter to analyze specific file types:

Task Command
Specify a data type python3 src/MetaDetective/MetaDetective.py -d directory -t pdf
Add multiple data types python3 src/MetaDetective/MetaDetective.py -d directory -t pdf doc
Include all types python3 src/MetaDetective/MetaDetective.py -d directory -t all
Ignoring specific results:

If you want to omit specific keywords from the displayed metadata, use the -i or --ignore flag. For instance, you might want to exclude common usernames like "admin" during the reconnaissance phase of your pentest. Regex patterns are supported, e.g., ^BeginBy.

Task Command
Exclude specific results python3 src/MetaDetective/MetaDetective.py -d directory -i anonymous
Exclude multiple terms python3 src/MetaDetective/MetaDetective.py -d directory -i anonymous admin administrateur
Regex exclusions python3 src/MetaDetective/MetaDetective.py -d directory -i anonymous ^admin
Display options

Adapt the display of your results to suit your preferences:

Task Command
Show each file's metadata python3 src/MetaDetective/MetaDetective.py --display all
Singular results without duplicates python3 src/MetaDetective/MetaDetective.py --display singular
Format options

Modify your display further with these:

Task Command
Stylish display python3 src/MetaDetective/MetaDetective.py --display all --format formatted
Simpler look python3 src/MetaDetective/MetaDetective.py --display all --format concise

🔎 Export options

MetaDetective provides flexibility in exporting analysis results.

By default, using the --export or -e option will save your results in an HTML format. This design ensures a visually appealing report for your analysis.

If you prefer a .txt format, that's possible too. Switch between formats using the -e or --export flag followed by the desired format: -e txt or -e pdf.

The export will, by default, use a predefined name appended with a timestamp. To customize this name, you can append a suffix using the -c or --custom flag.

Further, the --out or -o argument lets you specify the directory path for your exported data.

Be aware: The display and format options, as previously discussed, will influence the presentation of your exported document, whether in HTML or TXT format. Data representation might differ between the two formats.

Task Description Command
HTML Export (Default) Produces an HTML file named: MetaDetective_Export-<TIMESTAMP>.html. python3 src/MetaDetective/MetaDetective.py -d directory -e
TXT Format Export Save results in TXT format. python3 src/MetaDetective/MetaDetective.py -d directory --export txt
Custom Filename Suffix Add a custom suffix to the filename. python3 src/MetaDetective/MetaDetective.py -d directory -e --custom Pentest-MD_2
Specify Output Directory Define the directory for data export. python3 src/MetaDetective/MetaDetective.py -d directory -e -o directory

MetaDetective HTML Export Demo Screenshot

Note: The export format can greatly affect data presentation and accessibility. Opt for the format that aligns with your requirements.

(🔼 Back to top)

🔧 Troubleshooting

Encountering issues? Don't worry. If you come across any problems or have questions, please don't hesitate to submit a ticket for assistance: Submit an issue on GitHub

(🔼 Back to top)

🤝 Contributing

We truly appreciate and welcome community involvement. Your contributions, feedback, and suggestions play a crucial role in improving the project for everyone. If you're interested in contributing or have ideas for enhancements, please feel free to open an issue or submit a pull request on our GitHub repository. Every contribution, no matter how big or small, is highly valued and greatly appreciated!

(🔼 Back to top)

📚 License

This project is licensed under the GNU Affero General Public License, Version 3.0. For more details, please refer to the LICENSE file in the repository: Read the license on GitHub

(🔼 Back to top)

📞 Contact

ProtonMail LinkedIn

(🔼 Back to top)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

MetaDetective-1.0.7.tar.gz (62.4 kB view details)

Uploaded Source

Built Distribution

MetaDetective-1.0.7-py3-none-any.whl (42.5 kB view details)

Uploaded Python 3

File details

Details for the file MetaDetective-1.0.7.tar.gz.

File metadata

  • Download URL: MetaDetective-1.0.7.tar.gz
  • Upload date:
  • Size: 62.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for MetaDetective-1.0.7.tar.gz
Algorithm Hash digest
SHA256 17f0e76041a16c9728e524a04e5c3478916b22d76eb17aca841ec6b4d01dad4a
MD5 478c0b2a85c643217b07089ea867a062
BLAKE2b-256 a00dc5168001e3f1e0f5c841753e57d593050e9782525b8c0520d63f9ca84612

See more details on using hashes here.

File details

Details for the file MetaDetective-1.0.7-py3-none-any.whl.

File metadata

File hashes

Hashes for MetaDetective-1.0.7-py3-none-any.whl
Algorithm Hash digest
SHA256 3eccd583b6acb5063cbbe2438276376a7de2c1325e0422aeeb2ab52d9e33b825
MD5 7bb0c21778c3988e5b841939f432a88c
BLAKE2b-256 9cfbf1eb2226a7bfee66e0cf34f8d85b5242de81c18b23362da6eaf03ee56761

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page