websential 0.1.0

Advanced Python library for website URL discovery, security protection detection, database identification, and URL classification.

Websential

Websential is a Python library designed for advanced website reconnaissance and security analysis. It allows you to:

• Collect all visible and hidden URLs from a website.
• Detect website protections such as Cloudflare, reCAPTCHA, WAFs, and more.
• Identify the underlying database type based on response content.
• Classify URLs into functional groups like login, token, payment, OAuth, and more.

---

Installation

pip install websential

---

Usage

from Websential import urldetect, urlfilter

url = "https://www.example.com"

Collect URLs from the site

urls = urldetect.urls(url) print("Collected URLs:", urls)

Filter URLs by type (e.g., token)

token_urls = urlfilter.type(urls, 'token') print("Token-related URLs:", token_urls)

Detect website protections

protections = urldetect.protection(url) print("Detected protections:", protections)

Detect database type

databases = urldetect.database(url) print("Detected databases:", databases)

---

Features

1. URL Detection

   • Fetches all HTTP/HTTPS links on a website.
   • Parses robots.txt to detect allowed/disallowed paths and sitemaps.

2. Protection Detection

   • Detects multiple protection mechanisms including: Cloudflare, reCAPTCHA v2/v3, hCaptcha, Datadome, Akamai, Imperva, F5 BIG-IP, Fortinet, Wordfence, and more.

3. Database Detection

   • Detects common databases like MySQL, PostgreSQL, Oracle, SQL Server, MongoDB, SQLite by analyzing HTTP responses and headers.

4. URL Classification

   • Groups URLs based on functional categories: login/auth, token, payment, OAuth, verification, accounts, registration, admin panels, etc.

---

License

MIT License