Datasette plugin that blocks all robots using robots.txt
Project description
datasette-block-robots
Datasette plugin that blocks robots and crawlers using robots.txt
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-block-robots
Usage
Having installed the plugin, /robots.txt
on your Datasette instance will return the following:
User-agent: *
Disallow: /
This will request all robots and crawlers not to visit any of the pages on your site.
Here's a demo of the plugin in action: https://sqlite-generate-demo.datasette.io/robots.txt
Configuration
By default the plugin will block all access to the site, using Disallow: /
.
If you want the index page to be indexed by search engines without crawling the database, table or row pages themselves, you can use the following:
{
"plugins": {
"datasette-block-robots": {
"allow_only_index": true
}
}
}
This will return a /robots.txt
like so:
User-agent: *
Disallow: /db1
Disallow: /db2
With a Disallow
line for every attached database.
To block access to specific areas of the site using custom paths, add this to your metadata.json
configuration file:
{
"plugins": {
"datasette-block-robots": {
"disallow": ["/mydatabase/mytable"]
}
}
}
This will result in a /robots.txt
that looks like this:
User-agent: *
Disallow: /mydatabase/mytable
Alternatively you can set the full contents of the robots.txt
file using the literal
configuration option. Here's how to do that if you are using YAML rather than JSON and have a metadata.yml
file:
plugins:
datasette-block-robots:
literal: |-
User-agent: *
Disallow: /
User-agent: Bingbot
User-agent: Googlebot
Disallow:
This example would block all crawlers with the exception of Googlebot and Bingbot, which are allowed to crawl the entire site.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-block-robots
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv
:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for datasette-block-robots-1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a47299e4c8142439ba1532039cee6f8fdb76fad786cc445f9f4b0229dce359df |
|
MD5 | 7534660d5866459777e526073859054b |
|
BLAKE2b-256 | 058c44ab0d679a2de225d384686e584e2bf92be88bead6e1963628ed938975aa |
Hashes for datasette_block_robots-1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3556381c5d61230927cc3e21c1a831b7afb0194d7a90f22c6777d0f8dc13d8f5 |
|
MD5 | 3e527d88838b24188eb54eca2c1bc2f9 |
|
BLAKE2b-256 | c8c3705f5da8d7d6fac43936a9bc2a026a5ebe94cfb18a50a08a091df50d7f92 |