Datasette plugin that blocks all robots using robots.txt
Project description
datasette-block-robots
Datasette plugin that blocks all robots using robots.txt
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-block-robots
Usage
Having installed the plugin, /robots.txt
on your Datasette instance will return the following:
User-agent: *
Disallow: /
Here's a demo of the plugin in action: https://sqlite-generate-demo.datasette.io/robots.txt
Configuration
By default the plugin will block all access to the site, using Disallow: /
.
You can instead block access to specific areas of the site by adding the following to your metadata.json
configuration file:
{
"plugins": {
"datasette-block-robots": {
"disallow": ["/mydatabase"]
}
}
}
This will result in a /robots.txt
that looks like this:
User-agent: *
Disallow: /mydatabase
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-block-robots
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv
:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for datasette-block-robots-0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 206fe3727ffb2712d0a8ccd99e094f1bc281a05680f31d96014e4e07ec732475 |
|
MD5 | 8965fe2c9f2606e50bd9c638085c4dd5 |
|
BLAKE2b-256 | fc91bf373a025db914e987f3b74b83f3f110b520e9ea5442ceb6a55d6788d89f |
Hashes for datasette_block_robots-0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4d85cb668a39f6357fba1e2cb53b2c5892c1a16cd239095c4e8bae1c23a47b8 |
|
MD5 | 8cb6fe58f67ce87b3ecc67262eb8bf2a |
|
BLAKE2b-256 | 0ea9763047c5b2dea6b9c1ad0958fa2972e083d29fc3561b64d5e434a7d14859 |