Skip to main content

Django robots.txt generator

Project description

Fork of dimka665 https://github.com/dimka665/django-url-robots

Django robots.txt generator. Based on using decorated django.conf.urls.url. It gets urlpatterns and replaces ambiguous parts by *.

Installation & Usage

The recommended way to install django2-url-robots is with pip

  1. Install from PyPI with easy_install or pip:

    pip install django2-url-robots
  2. Add 'django2_url_robots' to your INSTALLED_APPS:

    INSTALLED_APPS = (
        ...
        'django2_url_robots',
        ...
        )
  3. Add django2_url_robots view to your root URLconf:

    urlpatterns += [
        url(r'^robots\.txt$', django2_url_robots.views.robots_txt),
        ]
  4. Describe rules by boolean keyword argument robots_allow using for it django2_url_robots.utils.url instead django.conf.urls.url:

    from django2_url_robots.utils import url
    
    urlpatterns += [
       url('^profile/private$', views.some_view, robots_allow=False),
       ]

django2-url-robots tested with Django-2.0.5+. Encodes unicode characters by percent-encoding.

Settings

In this moment there are only one option to define template of robots.txt file:

urlpatterns += [
    url(r'^robots\.txt$', django2_url_robots.views.robots_txt, {'template': 'my_awesome_robots_template.txt'}),
    ]

Example

robots_template.txt:

User-agent: *
Disallow: /*  # disallow all
{{ rules|safe }}

urls.py:

from django.conf.urls import include

urlpatterns = [
    url(r'^profile', include('django2_url_robots.tests.urls_profile')),
]

urls_profile.py:

from django2_url_robots.utils import url

urlpatterns = [
    url(r'^s$', views.some_view, name='profiles', robots_allow=True),
    url(r'^/(?P<nick>\w+)$', views.some_view),
    url(r'^/(?P<nick>\w+)/private', views.some_view, name='profile_private', robots_allow=False),
    url(r'^/(?P<nick>\w+)/public', views.some_view, name='profile_public', robots_allow=True),
    ]

Resulting robots.txt:

User-agent: *
Disallow: /*  # disallow all
Allow: /profiles$
Disallow: /profile/*/private*
Allow: /profile/*/public*

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django2-url-robots-3.2.2.tar.gz (4.9 kB view details)

Uploaded Source

Built Distribution

django2_url_robots-3.2.2-py3-none-any.whl (6.8 kB view details)

Uploaded Python 3

File details

Details for the file django2-url-robots-3.2.2.tar.gz.

File metadata

  • Download URL: django2-url-robots-3.2.2.tar.gz
  • Upload date:
  • Size: 4.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/51.0.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

File hashes

Hashes for django2-url-robots-3.2.2.tar.gz
Algorithm Hash digest
SHA256 fe8028f242325e3de6ec59c9982b0e04b294f81f145dbe17038be9e7346f0639
MD5 dd4a045c5ab31b83b0abd9290f7f5985
BLAKE2b-256 8dbd9f307f3ac66a6955041fedb2f8af399dad0be35e521cba8e14c03548c2f8

See more details on using hashes here.

File details

Details for the file django2_url_robots-3.2.2-py3-none-any.whl.

File metadata

  • Download URL: django2_url_robots-3.2.2-py3-none-any.whl
  • Upload date:
  • Size: 6.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/51.0.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9

File hashes

Hashes for django2_url_robots-3.2.2-py3-none-any.whl
Algorithm Hash digest
SHA256 f93f3ca88ac9963dfab1d694ea89564f2e9abbf25e8453c61cb2942b722cb3b3
MD5 655da0d2bb2df980532929100b0f1e7a
BLAKE2b-256 ed2cef130863c305510646e560933aa9b3e48deabda6228437ddf4ed0f1de1a3

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page