EndpointScanner: Website Endpoint Scanner and Rate Limit Tester (Version 7.5)

Source code at: EndpointScanner Repository

The dev branch hosts new features that may not be the full update, main branch will only contain stable updates to install.

Note: endpoints and paths in this documentation mean the same thing.

A fast automated website reconnaissance tool for red teaming in cybersecurity that extracts paths, endpoints, files, subdomains, and even external links from modern websites. Automated IDOR and broken access control vulnerability testing through replacing variables with 1 in endpoints/paths. Has a built in rate limit tester that can test on any endpoint with 4 HTTP methods, and can bypass simple WAFs/captchas and SPAs.

Navigation

• Benchmark against popular tools
• How it works
• All flags
• Installation
• Example Commands
• Release Notes
• Legal Disclaimer

Feel free to contact me at sphericalflower@gmail.com to ask for help on how to use the tool or to leave feedback.

How it works

• Uses curl_cffi to bypass simple captchas via TLS/JA3 fingerperint impersonation
• Uses playwright-stealth browser instead of usual Playwright to remove giveaways in the browser. (more details about playwright near the bottom of the README)
• Scans extra map files like robots.txt and sitemap.xml for more paths
• Uses a fake path to test which are real paths and which are shells. (websites like SPAs give a lot of trouble to current tools)
• Scrapes all JavaScript, HTML and XML files to find paths in the website.
• Has a hardcoded set of paths that should never exist in a website to test. (e.g. .env.local, .git/config)
• Differentiates paths by website endpoints, assets, redirects etc.
• Replaces {example} variables in paths with a custom string to be able to test the paths, and can also test for broken access control.
• Checks server uptime and prints out JS Stack of the website (these are extra details, a flag is needed to show the extra details.)
• Has a rate limit tester by sending n number requests to a certain endpoint with 4 HTTP methods, custom headers, body, variable in request body.

Tool flags

All passable flags for the tool:

Installation

You can install EndpointScanner via PyPI.

Installation via PyPI (or pip)

Main Package

You MUST have python 3.9 or above to use this tool! To install the official endpointscanner Python package:

Command for MacOS/Linux:

pip install endpointscanner

Command for Windows Command Prompt:

pip install endpointscanner

Playwright Installation

After that, install chromium on playwright (playwright will be installed when you install endpointscanner):

Command for MacOS/Linux:

playwright install chromium

Command for Windows Command Prompt:

py -m playwright install chromium

Example Commands

Example command to run to scan a site (show inaccessible endpoints, show assets, show progress as it finds endpoints, and show files that it got endpoints from):

endpointscanner https://example.com -s -m -sp -ss

Example command to run to test a site (Assuming you are testing 5000 requests and creating 5k accounts)

endpointscanner example.com -r 5000 -t /signup -rt POST -rb '{"username":"ExampleUser{X}", "pass":"ExamplePassword"}' -rv '{X}' --force -rh 'POST /signup HTTP/2
Host: example.com
Cookie: clearedcaptcha=true
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:151.0) Gecko/20100101 Firefox/151.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.9
Prefer: safe
Accept-Encoding: gzip, deflate, br
Content-Type: application/json
Content-Length: 60
Origin: https://example.com
Referer: https://example.com/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Priority: u=0
Te: trailers'

Example command to only show the original endpoint without adjusting the variable and output to a file:

endpointscanner example.com -oo -o examplescan.txt

Release notes

New features in update 7.5

• -rat/--ratelimit-await-time flag, as th default is 50s and for a PoC with 50-100 requests 50 seconds is way too long.
• -p/--pipeable flag to make the output of the tool pipeable into other automated payload testing tools. (E.g. sqlmap)
• -ps/--path-sub flag to change the automated path normalization from replacing variables to 1 to any custom string. Defaults to 1.
• Merged asset-manifest.json, web-manifest.json, manifest.json into one loop.
• added -de/--depth flag
• Made the sitemap loop (which previous only scanned .xml files) be able to scan .xml, .txt, .rss, .atom files.
  • Able to scan .gz and .zip files if the flag -pz/--parse-zip is passed. (Detects based off the file extension)
• Added endpoints to the SENSITIVE_ENDPOINT set like actuator endpoints and swagger ui
• Added automatic input where if you don't answer input questions after a set time (default 90 seconds) (e.g. an input where you must say if you want to sort endpoints or not). Can be disabled via the -nai/--no-auto-input flag.
• Fixed a bug in the rate-limiting tester where the rate limit testing function would crash if the number of requests was too high by using a queue.
• Replace /* paths in robots.txt with '1' as * means everything, showing you the original path. This will also be affected by the -ps and -oo flag.
• Added a section to the sorted endpoints which are 'protected endpoints', for endpoints that return 401/403 or other status codes showing it exists but is protected.
• Let the scanner scan .mjs and .cjs files besides just .js
• Allow text files for the -esl flag
• Changed the -or flag to the -ed flag, now you have to pass -ed for the extra details, instead of extra details being shown by default (for example, JS Stack detected). Things like found path count will still be shown.
• Added -nhb/--no-headless-browser flag to completely disable the headless browser and use a simple GET request instead.
• Added -fm flag, --filter-more. Will filter stuff like www.w3.org if passed as results like http://www.w3.org/2000/svg are present in almost all websites.
• Added -orlt flag to only perform a rate-limiting test.
• Changed scraper to scrape all href and src attributes instead of only looking for code/image files.
• Allow user to add an entire domain, or only one file as an external script loader (e.g. examplegiganticcdn.com/example.js)
• Moved structure file scraping into mapfiles.py
• Moved code file scraping into scrapefiles.py
• Moved path sorting into filtersort.py
• Optimised sorting endpoints
• Added -oe/--only-endpoints, -ox/--only-external and -os/--only-sub flag to only print endpoints, external links, and subdomains respectively
• Rate limit tester now uses curl_cffi instead of httpx

Bug Fixes/Code improvements

• Fixed regex problems in the map file check for:
  • openid-configuration
  • asset-manifest.json
  • web-manifest.json
  • manifest.json
  • sw.js
  • service-worker.js
• Changed previous -nhb flag from showing browser UI to -nh/-no-headless to be more clear
• Fixed an issue where the scraping would find data MIME types (e.g. application/x-javascript) and mistake them for real endpoints
• Fixed a scope bug in the tool
• Improve regex.
• Remove a function removing trailing slashes in paths, as some are directories.
• Added -o cli for people who don't realise it prints to CLI by default.
• Changed -d to -ds flag

Plans for next version and the future

Version 7.6:

• Fixing a URL parameter problem in the tool
• Further improving regex
• Scanning openapi.json for paths
• properly checking sensitive endpoints

Future plans (May be added in the future version):

none right now

Weaknesses

• On Server-Side Rendered websites like PHP websites, there are literally no frontend code files for EndpointScanner to scan hence it will fail on those types of websites. (Keep in mind EndpointScanner is still top-tier when it comes to the modern web)
• If there is a login page, the script will either show that all of the pages require login, or label all of them as 403 (if sorting is enabled).
• If there are shells (e.g. React SPA shells) in the page, it may give false positives for sensitive endpoints. If you see sensitive endpoints in the scan, they may not actually be exposed on the website if the website has a shell. (E.g. .gitignore, .env.local). This will be fixed in update 7.5.1

ai assisted code btw

Legal Disclaimer

Note that this tool is strictly meant for authorised testing and security research. Running this script on websites where you are not permitted to do so can result in legal action. The author of this script assumes no responsibility for any misuse or legal consequences from running this script. Ensure you have received permission from the owner of the target website before performing tests or scans on their website.