MkDocs Anti AI Scraper Plugin
This plugin tries to prevent AI scrapers from easily ingesting your website's contents. It is probably implemented pretty badly and by design it can be bypassed by anyone that invests a bit of time, but it is probably better than nothing.
Installation
Install the plugin with pip:
pip install mkdocs-anti-ai-scraper-plugin
Then add the plugin to your mkdocs.yml:
plugins:
- search
- anti_ai_scraper
Or with all config options:
plugins:
- search
- anti_ai_scraper:
robots_txt: True
sitemap_xml: True
encode_html: True
debug: False
Implemented Techniques
Technique | Scraper Protection | Impact on human visitors | Enabled by default --- | --- | --- Add robots.txt | weak | none | yes Remove sitemap.xml | very weak | none | yes Encode HTML | only against simple HTML parser based scrapers | slows down page loading, may break page events | true
Add robots.txt
This technique is enabled by default, and can be disabled by setting the option robots_txt: False in mkdocs.yml.
If enabled, it adds a robots.txt with the following contents to the output directory:
User-agent: *
Disallow: /
This hints to crawlers that they should not crawl your site.
This technique does not hinder normal users from using the site at all.
However, the robots.txt is not enforcing anything.
It just tells well-behaved bots how you would like them to behave.
Many AI bots may just ignore it (Source).
Remove sitemap.xml
This technique is enabled by default, and can be disabled by setting the option robots_txt: False in mkdocs.yml.
If enabled, it removes the sitemap.xml and sitemap.xml.gz files.
This prevents leaking the paths to pages not referenced by your navigation.
Encode HTML
This technique is enabled by default, and can be disabled by setting the option robots_txt: False in mkdocs.yml.
If enabled, it encodes (zip + ASCII85) each page's contents and will decode it in the user's browser with JavaScript.
This obscures the page contents to simple scrapers that just download and parse your HTML.
It will not work against any bots that use remote controlled browsers (using selenium or other tech).
The decoding takes some time and will result in browser events (like onload) being fired before the page is decoded.
This may break some functionality, that listens to these events and expects them to happen.
Planned Techniques
- remove sitemap.xml(.gz): just obscures a bit, the nav will still point to most pages.
- Encoding the page contents and decode with JS: Will prevent basic HTML parsers from getting the contents, but anything using a browser (selenium, pupeteer, etc) will still work.
- Encrypt page contents and adding client side "CAPTCHA" to generate the key: Should help against primitive browser based bots.
It would probably make sense to just let the user solve the CAPTCHA once and cache the key as a cookie or in
localStorage. - Bot detection JS: Will be a cat and mouse game, but should help against badly written crawlers
Suggestions welcome: If you know bot detection mechanisms, that can be used with static websites, feel free to open an issue :D
Problems and Considerations
- Similar to the encryption plugin, the encryption of the search index is hard. So best disable search to prevent anyone from accessing its index.
- Obviously, to protect your contents from scraping, you should not have their source code hosted in public repos ;D
- By blocking bots, you also prevent search engines like Google from properly endexing your site.
Notable changes
Version 0.1.0
- Added
encode_htmloption - Added
sitemap_xmloption
Version 0.0.1
- Added
robots_txtoption
Development Commands
This repo is managed using poetry.
You can install poetry with pip install poetry or pipx install poetry.
Clone repo:
git clone git@github.com:six-two/mkdocs-anti-ai-scraper-plugin.git
Install/update extension locally:
poetry install
Build test site:
poetry run mkdocs build
Serve test site:
poetry run mkdocs serve
Release
Set PyPI API token (only needed once):
poetry config pypi-token.pypi YOUR_PYPI_TOKEN_HERE
Build extension:
poetry build
Upload extension:
poetry publish
Metadata
Release files for mkdocs-anti-ai-scraper-plugin 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mkdocs_anti_ai_scraper_plugin-0.1.0.tar.gz | 4.0 kB | Without provenance | Details for mkdocs_anti_ai_scraper_plugin-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mkdocs_anti_ai_scraper_plugin-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for mkdocs_anti_ai_scraper_plugin-0.1.0-py3-none-any.whl |
Total release size: 8.8 kB
Release files / mkdocs_anti_ai_scraper_plugin-0.1.0.tar.gz
| Download URL | mkdocs_anti_ai_scraper_plugin-0.1.0.tar.gz |
|---|---|
| Size | 4.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e31f41d4593d2557d2a727547d8a3726d214793f729dc7c81e9b711bf52aca79
|
|
BLAKE2b-256 checksum How to use checksums |
d76c762190c005e081223db2ae03320844e4fb42ace280bd1f9fdeea608e8cd8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.1.4 CPython/3.9.6 Darwin/24.6.0
|
Release files / mkdocs_anti_ai_scraper_plugin-0.1.0-py3-none-any.whl
| Download URL | mkdocs_anti_ai_scraper_plugin-0.1.0-py3-none-any.whl |
|---|---|
| Size | 4.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2ddcefc73d47db96038cd7f88b59dde7016ebb8356c4c180bf540cf380361ee5
|
|
BLAKE2b-256 checksum How to use checksums |
6fa8c5075deff5fe2d3f0a3231f6d2381dbc83f6f8cba9d727aa182cf03cf170
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.1.4 CPython/3.9.6 Darwin/24.6.0
|
Security for mkdocs-anti-ai-scraper-plugin 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.