Django X-Robots-Tag Middleware

Simple Django middleware to send the X-Robots-Tag header on all responses. Useful for ensuring a site in development is not accidentally picked up and indexed until it is ready.

Works under both WSGI and ASGI, and is fully type annotated.

Requirements

• Python 3.10+
• Django 5.2+

Installation

From PyPI:

$ python -m pip install django-x-robots-tag-middleware

Usage

1. Add the middleware to MIDDLEWARE in your settings.py:

   MIDDLEWARE = [
       # ...
       "x_robots_tag_middleware.middleware.XRobotsTagMiddleware",
   ]
   

2. Add X_ROBOTS_TAG to your settings.py with the directives you want:

   X_ROBOTS_TAG = ["noindex", "nofollow"]
   

   The setting may be a list of directives (joined with commas) or a plain string:

   X_ROBOTS_TAG = "noindex, nofollow"
   

   If X_ROBOTS_TAG is unset, None, or empty, no header is added — so you can enable the header per environment without changing your middleware list:

   # settings/staging.py
   X_ROBOTS_TAG = ["noindex", "nofollow"]
   
   # settings/production.py
   X_ROBOTS_TAG = None
   

Details about the values for the tag and what they do can be found in the Google documentation.

Development

This project uses uv for packaging and dependency management (with the uv_build backend) and Ruff for linting and formatting.

$ uv sync --group dev                    # create .venv from uv.lock
$ uv run python manage.py test           # run the test suite
$ uv run coverage run manage.py test     # ...with coverage
$ uv run coverage report
$ uv run ruff check . && uv run ruff format .
$ uv build                               # build the sdist and wheel

To test against a specific Django version:

$ uv pip install "Django~=6.0.0"
$ uv run --no-sync python manage.py test

Optionally, install the pre-commit hooks:

$ uvx pre-commit install

The tests run against the x_robots_tag_middleware_demo project included in this repo.

License

MIT — see LICENSE.