Skip to main content

django-icv-sitemaps

CI PyPI Python Django License: MIT

Django's built-in django.contrib.sitemaps loads every URL into memory at request time. On a site with tens of thousands of pages that means slow responses, high memory pressure, and no incremental updates when content changes. At a million URLs it simply does not work.

django-icv-sitemaps replaces that approach entirely. Sitemaps are built in the background by Celery tasks, written atomically to any Django storage backend (local, S3, GCS), and served as static files. Only sections whose content has changed are ever rebuilt. The full protocol is covered: standard, image, video, and news sitemaps, automatic file splitting, gzip compression, and search engine pinging, plus a complete set of web discovery files (robots.txt, llms.txt, ads.txt, security.txt, humans.txt) managed from the database.

Part of the ICV-Django ecosystem, but fully standalone: no other ICV packages required.


Features

  • Background generation: sitemaps are generated by Celery tasks (optional), written to Django storage backends (local, S3, GCS), and served statically
  • Incremental updates: post_save/post_delete signals mark affected sections as stale; only changed sections are regenerated
  • All four sitemap types: standard, image, video, and news sitemaps with correct XML namespaces per the sitemap protocol
  • Automatic splitting: files are split at 50,000 URLs or 50 MB per the protocol limits
  • SitemapMixin: declare any Django model as sitemap-includable with a small set of class attributes
  • Auto-sections: ICV_SITEMAPS_AUTO_SECTIONS wires signal handlers automatically, like Django's ICV_SEARCH_AUTO_INDEX
  • robots.txt: dynamic, database-driven rules merged with settings; includes Sitemap: directive automatically
  • llms.txt: AI crawler guidance served at /llms.txt
  • ads.txt / app-ads.txt: IAB-format authorised seller declarations
  • security.txt: RFC 9116 compliant, served at /.well-known/security.txt
  • humans.txt: team credits
  • URL redirects: database-driven redirect rules (301/302/307/308/410) with exact, prefix, and regex matching, priority ordering, expiry, hit tracking, and CSV import/export
  • 404 tracking: automatic detection of recurring 404s with hit counts and referrer tracking; create redirect rules directly from admin
  • RedirectMiddleware: opt-in middleware; a redirect rule (301/302/307/308) is evaluated before Django's URL resolver, a 410 rule only once the resolver has returned a 404, so it can never shadow a live view; fail-open design never breaks the request cycle
  • Search engine ping: Google, Bing, Yandex notified on content changes (conditional on checksum comparison)
  • Multi-tenancy: all discovery files are tenant-scoped; sitemap paths include tenant prefix to prevent collisions; tenant IDs are sanitised to prevent path-traversal attacks
  • Gzip support: compressed .xml.gz output with correct headers
  • Atomic writes: temp file then rename; no partially-written files served
  • 6 management commands: setup, generate, ping, validate, stats, redirects
  • Django admin: all 8 models registered with actions, list filters, and read-only views
  • Celery graceful degradation: tasks work synchronously when Celery is not installed
  • Testing utilities: 8 factory-boy factories, pytest fixtures, and helpers in icv_sitemaps.testing

Installation

pip install django-icv-sitemaps

Add to INSTALLED_APPS:

INSTALLED_APPS = [
    # ...
    "icv_sitemaps",
]

Run migrations:

python manage.py migrate icv_sitemaps

Include the URL configuration:

# urls.py
from django.urls import include, path

urlpatterns = [
    path("", include("icv_sitemaps.urls")),
    # ...
]

This registers all discovery file endpoints at the root (/sitemap.xml, /robots.txt, /llms.txt, /ads.txt, /app-ads.txt, /.well-known/security.txt, /humans.txt).


Quick Start

1. Make your model sitemap-includable

# myapp/models.py
from django.db import models
from icv_sitemaps.mixins import SitemapMixin


class Article(SitemapMixin, models.Model):
    sitemap_section_name = "articles"
    sitemap_changefreq = "weekly"
    sitemap_priority = 0.7

    title = models.CharField(max_length=200)
    slug = models.SlugField(unique=True)
    is_published = models.BooleanField(default=True)
    updated_at = models.DateTimeField(auto_now=True)

    def get_absolute_url(self):
        return f"/articles/{self.slug}/"

    @classmethod
    def get_sitemap_queryset(cls):
        return cls.objects.filter(is_published=True)

2. Configure auto-sections

# settings.py
ICV_SITEMAPS_BASE_URL = "https://example.com"

ICV_SITEMAPS_AUTO_SECTIONS = {
    "articles": {
        "model": "blog.Article",
        "sitemap_type": "standard",
        "changefreq": "weekly",
        "priority": 0.7,
    },
    "product_images": {
        "model": "catalogue.ProductImage",
        "sitemap_type": "image",
    },
    "videos": {
        "model": "media.Video",
        "sitemap_type": "video",
    },
    "breaking_news": {
        "model": "news.BreakingStory",
        "sitemap_type": "news",
    },
}

3. Set up and generate

# Create SitemapSection records from config
python manage.py icv_sitemaps_setup

# Generate all sitemaps
python manage.py icv_sitemaps_generate --all

# Validate output
python manage.py icv_sitemaps_validate

# Check stats
python manage.py icv_sitemaps_stats

4. Automatic regeneration

When an Article is saved or deleted, its section is marked stale. The regenerate_stale_sitemaps task picks it up on the next run.

# Celery beat schedule (optional)
from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    "icv-sitemaps-regenerate-stale": {
        "task": "icv_sitemaps.tasks.regenerate_stale_sitemaps",
        "schedule": crontab(minute="*/15"),
    },
    "icv-sitemaps-regenerate-all": {
        "task": "icv_sitemaps.tasks.regenerate_all_sitemaps",
        "schedule": crontab(hour=3, minute=0),
    },
    "icv-sitemaps-cleanup-logs": {
        "task": "icv_sitemaps.tasks.cleanup_generation_logs",
        "schedule": crontab(hour=4, minute=0),
    },
    "icv-sitemaps-cleanup-orphans": {
        "task": "icv_sitemaps.tasks.cleanup_orphan_files",
        "schedule": crontab(day_of_week=0, hour=5, minute=0),
    },
}

Sitemap Types

Standard

Standard XML sitemaps with <loc>, <lastmod>, <changefreq>, and <priority> per the sitemaps.org protocol.

Image

Uses the http://www.google.com/schemas/sitemap-image/1.1 namespace. Configure image fields on your mixin:

class ProductImage(SitemapMixin, models.Model):
    sitemap_section_name = "product_images"
    sitemap_type = "image"
    sitemap_image_field = "image_url"
    sitemap_image_caption_field = "caption"
    sitemap_image_title_field = "title"

Video

Uses the http://www.google.com/schemas/sitemap-video/1.1 namespace:

class Video(SitemapMixin, models.Model):
    sitemap_section_name = "videos"
    sitemap_type = "video"
    sitemap_video_url_field = "video_url"
    sitemap_video_thumbnail_field = "thumbnail_url"
    sitemap_video_title_field = "title"
    sitemap_video_description_field = "description"
    sitemap_video_duration_field = "duration_seconds"

News

Uses the http://www.google.com/schemas/sitemap-news/0.9 namespace. Entries older than ICV_SITEMAPS_NEWS_MAX_AGE_DAYS (default 2) are automatically excluded:

class BreakingStory(SitemapMixin, models.Model):
    sitemap_section_name = "breaking_news"
    sitemap_type = "news"
    sitemap_news_publication_name = "Example News"
    sitemap_news_language = "en"
    sitemap_news_title_field = "headline"
    sitemap_news_date_field = "published_at"

Static Sections: URLs without a model

Every section above resolves a Django model. Sites also have pages with no model behind them: the homepage, a pricing page, marketing landing pages. A section_type="static" section covers exactly this, sourcing its URLs from a declared list or a callable instead of a queryset.

ICV_SITEMAPS_AUTO_SECTIONS = {
    "marketing-pages": {
        "section_type": "static",
        "sitemap_type": "standard",
        "changefreq": "weekly",
        "priority": "0.7",
        "settings": {"url_provider": "myapp.sitemaps:marketing_urls"},
    },
}
# myapp/sitemaps.py
def marketing_urls():
    """Return an iterable of sitemap entry dicts for pages with no model."""
    return [
        {"loc": "/", "changefreq": "daily", "priority": 1.0},
        {"loc": "/pricing/", "changefreq": "weekly", "priority": 0.8},
        {"loc": "/about/"},
    ]

Two ways to declare a static section's URLs, read from the section's settings JSONField:

Key Type Description
url_provider str Dotted path ("module.path:function" or "module.path.function") to a no-argument callable returning an iterable of entry dicts. Takes precedence over urls when both are present.
urls list[dict] An inline list of entry dicts, for a URL set that doesn't need a callable.

Each entry dict is the same shape a model section produces internally: {"loc": str, "lastmod": ..., "changefreq": ..., "priority": ..., "images"/"video"/"news": ...}. Only loc is required.

from icv_sitemaps.services import create_section

create_section(
    "marketing-pages",
    url_provider="myapp.sitemaps.marketing_urls",
)

# or an inline list, no callable needed
create_section(
    "legal-pages",
    urls=[{"loc": "/terms/"}, {"loc": "/privacy/"}],
)

The callable owns host and locale concerns. Generation often runs as a background Celery task, outside any request context, so url_provider must build fully-qualified or base-relative URLs itself: it cannot rely on the current request's host or the active locale. For a multi-host or multi-locale site, reverse routes explicitly (e.g. pass urlconf=... to django.urls.reverse and wrap in translation.override(...)).

Static sections do not go stale automatically. They have no model, so post_save/post_delete never fires for them; connect_auto_section_signals() skips them silently. Their content changes on deploy, not on data writes, so call mark_section_stale("marketing-pages") from your deploy pipeline, or rely on the periodic regenerate_all_sitemaps task to pick them up.


Discovery contract: the mixin is a convenience, not a requirement

SitemapMixin is the fastest way to make a model sitemap-includable, but it is optional. Sitemap generation has two requirements, both read via duck-typed getattr(), never by checking for the mixin:

  1. The model is registered in ICV_SITEMAPS_AUTO_SECTIONS (see above).
  2. The instance provides the get_sitemap_*() methods that _extract_entry() reads, plus a get_sitemap_queryset() classmethod that generate_section() calls to enumerate rows.
Method Required? Read as Default when absent
get_sitemap_url() Yes called directly none: raises, entry is skipped
get_sitemap_lastmod() No getattr(instance, ..., lambda: None)() None (omits <lastmod>)
get_sitemap_changefreq() No getattr(instance, ..., lambda: "daily")() "daily"
get_sitemap_priority() No getattr(instance, ..., lambda: 0.5)() 0.5
get_sitemap_images() Only for sitemap_type="image" getattr(instance, ..., list)() []
get_sitemap_video() Only for sitemap_type="video" getattr(instance, ..., lambda: None)() None
get_sitemap_news() Only for sitemap_type="news" getattr(instance, ..., lambda: None)() None
get_sitemap_queryset() (classmethod) No try/except AttributeError falls back to model_class.objects.all()

None of these checks test for SitemapMixin. Any object providing the methods it needs for its sitemap_type works, with no inheritance from an icv-sitemaps base class required. SitemapMixin.get_sitemap_url() itself just delegates to your model's own get_absolute_url(), which is the usual Django convention, so most models need no override at all.

This follows ADR-025 principle 2 (discovery is pull-based, by protocol): a package that enumerates objects owned elsewhere must accept any object satisfying the documented protocol, never require a mixin. Content packages that want sitemap coverage (a CMS page model, for example) are never obliged to know icv_sitemaps exists beyond implementing these methods.

A model implementing the protocol directly, without SitemapMixin:

# blog/models.py
from django.db import models

class Article(models.Model):
    title = models.CharField(max_length=200)
    slug = models.SlugField(unique=True)
    is_published = models.BooleanField(default=True)
    updated_at = models.DateTimeField(auto_now=True)

    def get_absolute_url(self) -> str:
        return f"/articles/{self.slug}/"

    def get_sitemap_url(self) -> str:
        return self.get_absolute_url()

    def get_sitemap_lastmod(self):
        return self.updated_at

    def get_sitemap_changefreq(self) -> str:
        return "weekly"

    def get_sitemap_priority(self) -> float:
        return 0.7

    @classmethod
    def get_sitemap_queryset(cls):
        return cls.objects.filter(is_published=True)
# settings.py
ICV_SITEMAPS_AUTO_SECTIONS = {
    "articles": {"model": "blog.Article", "sitemap_type": "standard"},
}

That is the whole contract for a standard sitemap. SitemapMixin exists because most models want the same class-attribute-to-method mapping (and the image/video/news field wiring); reach for it when that saves you writing boilerplate, skip it when you would rather implement the methods directly.


Discovery Files

robots.txt

Database-driven rules managed via Django admin or the service layer:

from icv_sitemaps.services import add_robots_rule

# Block AI crawlers from /private/
add_robots_rule("GPTBot", "disallow", "/private/")
add_robots_rule("CCBot", "disallow", "/")

# Block all bots from /admin/
add_robots_rule("*", "disallow", "/admin/")

Extra directives from settings are appended after database rules:

ICV_SITEMAPS_ROBOTS_EXTRA_DIRECTIVES = [
    "Crawl-delay: 10",
]

Raw bulk_create bypasses cache invalidation

RobotsRule's cache is invalidated by a post_save/post_delete signal handler, and Django's bulk_create emits neither signal. Writing RobotsRule rows directly with RobotsRule.objects.bulk_create(...) leaves the cached robots.txt content stale until ICV_SITEMAPS_CACHE_TIMEOUT expires (default 3600 seconds), even though the rows already exist in the database.

Call RobotsRule.objects.bulk_create(...) yourself followed by invalidate_robots_cache(tenant_id=...):

from icv_sitemaps.services import invalidate_robots_cache
from icv_sitemaps.models import RobotsRule

RobotsRule.objects.bulk_create([...])
invalidate_robots_cache(tenant_id="acme")

ads.txt / app-ads.txt

IAB-format authorised seller declarations:

from icv_sitemaps.services import add_ads_entry

add_ads_entry("google.com", "pub-1234567890", "DIRECT", certification_id="f08c47fec0942fa0")
add_ads_entry("adnetwork.com", "pub-9876543210", "RESELLER")

# For app-ads.txt
add_ads_entry("google.com", "pub-1234567890", "DIRECT", is_app_ads=True)

Raw bulk_create bypasses cache invalidation

AdsEntry's cache is invalidated by a post_save/post_delete signal handler, and Django's bulk_create emits neither signal. Writing AdsEntry rows directly with AdsEntry.objects.bulk_create(...) leaves the cached ads.txt or app-ads.txt content stale until ICV_SITEMAPS_CACHE_TIMEOUT expires (default 3600 seconds), even though the rows already exist in the database. AdsEntry maps to two distinct cache keys depending on is_app_ads, so pass the matching value.

Call AdsEntry.objects.bulk_create(...) yourself followed by invalidate_ads_cache(is_app_ads=..., tenant_id=...):

from icv_sitemaps.services import invalidate_ads_cache
from icv_sitemaps.models import AdsEntry

AdsEntry.objects.bulk_create([...])  # is_app_ads=False rows
invalidate_ads_cache(is_app_ads=False, tenant_id="acme")

llms.txt, security.txt, humans.txt

Free-form text content managed via DiscoveryFileConfig:

from icv_sitemaps.services import set_discovery_file_content

set_discovery_file_content("llms_txt", """# llms.txt
# AI training and crawl guidance for example.com

Allow: /blog/
Disallow: /private/
""")

set_discovery_file_content("security_txt", """Contact: mailto:security@example.com
Expires: 2027-01-01T00:00:00.000Z
Preferred-Languages: en
""")

set_discovery_file_content("humans_txt", """/* TEAM */
Lead: Nigel Copley
Site: example.com
""")

Raw bulk_create bypasses cache invalidation

DiscoveryFileConfig's cache is invalidated by a post_save/post_delete signal handler, and Django's bulk_create emits neither signal. Writing DiscoveryFileConfig rows directly with DiscoveryFileConfig.objects.bulk_create(...) leaves the cached content stale until ICV_SITEMAPS_CACHE_TIMEOUT expires (default 3600 seconds), even though the rows already exist in the database. Each file_type is cached under its own key, so pass the matching value.

Call DiscoveryFileConfig.objects.bulk_create(...) yourself followed by invalidate_discovery_cache(file_type, tenant_id=...):

from icv_sitemaps.services import invalidate_discovery_cache
from icv_sitemaps.models import DiscoveryFileConfig

DiscoveryFileConfig.objects.bulk_create([...])  # all file_type="llms_txt"
invalidate_discovery_cache("llms_txt", tenant_id="acme")

URL Redirects & 404 Tracking

Redirect Rules

Database-driven redirect rules evaluated by RedirectMiddleware. A 301/302/307/308 rule is evaluated before Django's URL resolver, so it wins even for a path a view would otherwise serve: useful for a promo route or masking a page mid-migration. A 410 rule is different: it asserts the resource is permanently removed, which is incoherent for a path the resolver can still serve, so it is only evaluated once the resolver has already returned a 404. A 410 rule can never shadow a live view.

from icv_sitemaps.services import add_redirect

# Permanent redirect
add_redirect("/old-page/", "/new-page/", 301)

# Temporary redirect
add_redirect("/promo/", "/summer-sale/", 302)

# 410 Gone: page permanently removed
add_redirect("/deleted-product/", "", 410)

# Prefix match: all paths under /blog/2023/ redirect
add_redirect("/blog/2023/", "/archive/2023/", 301, match_type="prefix")

# Regex match
add_redirect(r"/product/\d+/", "/products/", 301, match_type="regex")

# Bulk import from CSV
from icv_sitemaps.services import bulk_import_redirects

with open("redirects.csv") as f:
    import csv
    rows = list(csv.DictReader(f))
    result = bulk_import_redirects(rows)
    # {"created": 150, "updated": 3, "errors": []}

# Bulk create (insert-only, faster for a large one-off import)
from icv_sitemaps.services import bulk_create_redirects

result = bulk_create_redirects(rows)
# {"created": 150, "errors": []}

Raw bulk_create bypasses cache invalidation

RedirectRule's cache is invalidated by a post_save/post_delete signal handler, and Django's bulk_create emits neither signal. Writing RedirectRule rows directly with RedirectRule.objects.bulk_create(...) leaves the cached prefix/regex rule list stale until ICV_SITEMAPS_REDIRECT_CACHE_TIMEOUT expires (default 300 seconds), even though the rows already exist in the database. Exact-match rules are unaffected: check_redirect always resolves an exact match with a direct database query, never from the cache.

Either use bulk_create_redirects above, which does one bulk_create(ignore_conflicts=True) and invalidates the cache once at the end, or call RedirectRule.objects.bulk_create(...) yourself followed by invalidate_redirect_cache(tenant_id=...):

from icv_sitemaps.services import invalidate_redirect_cache
from icv_sitemaps.models import RedirectRule

RedirectRule.objects.bulk_create([...])
invalidate_redirect_cache(tenant_id="acme")

Enable the Middleware

# settings.py
MIDDLEWARE = [
    # ... security/WAF middleware first ...
    "icv_sitemaps.middleware.RedirectMiddleware",
    "django.middleware.common.CommonMiddleware",
    # ...
]

ICV_SITEMAPS_REDIRECT_ENABLED = True

404 Tracking

Enable automatic 404 tracking to identify broken URLs:

# settings.py
ICV_SITEMAPS_404_TRACKING_ENABLED = True
ICV_SITEMAPS_404_TRACKING_SAMPLE_RATE = 1.0   # Track all 404s (reduce for high traffic)
ICV_SITEMAPS_404_IGNORE_PATTERNS = [
    r"\.(?:css|js|ico|png|jpg|jpeg|gif|svg|woff2?|ttf|eot|map)$",
]

Review top 404s and create redirects:

from icv_sitemaps.services import get_top_404s

# Top 50 unresolved 404s with at least 5 hits
for entry in get_top_404s(min_hits=5):
    print(f"{entry.path}: {entry.hit_count} hits, referrers: {entry.referrers}")

Or from the command line:

python manage.py icv_sitemaps_redirects --top-404s
python manage.py icv_sitemaps_redirects --list
python manage.py icv_sitemaps_redirects --import redirects.csv
python manage.py icv_sitemaps_redirects --export redirects.csv
python manage.py icv_sitemaps_redirects --prune   # Remove expired rules

Gone Resolution Hook

ICV_SITEMAPS_GONE_RESOLVER lets a consumer answer "is this already-404 path deliberately gone?" from their own data, without materialising a RedirectRule row per deleted object. It is called only on the 404 path, and only after a gone RedirectRule lookup has already found nothing, so a hand-authored rule always takes precedence over the resolver.

# myapp/services.py
def is_gone(request):
    if Product.deleted_objects.filter(old_url=request.path).exists():
        return 410
    return None

# settings.py
ICV_SITEMAPS_GONE_RESOLVER = "myapp.services.is_gone"

The callable takes the request and returns 410 or None; 410 is the only supported status code. Any other return value, including other status codes, is logged as a warning and treated as None (the 404 passes through unchanged): this hook exists to express "this URL is gone", not to rewrite arbitrary responses. An exception raised by the callable is caught, logged, and treated the same as None (fail-open, never breaks the response). A path resolved this way is served the same 410 response as a matching 410 RedirectRule and is not additionally recorded by the 404 tracker.


Configuration

Settings Reference

All settings are namespaced under ICV_SITEMAPS_*. Every setting has a sensible default so the package works out of the box for local development.

Setting Type Default Description
ICV_SITEMAPS_BASE_URL str "" Base URL for absolute sitemap URLs (e.g. "https://example.com"). Required: raises ImproperlyConfigured at generation time if empty
ICV_STORAGES_ALIAS str "default" Which alias in your STORAGES setting icv-sitemaps writes generated files to and reads them back from. Fleet-wide convention (ADR-037); falls back to Django's own storages["default"]
ICV_CACHES_ALIAS str "default" Which alias in your CACHES setting icv-sitemaps caches through. Fleet-wide convention (ADR-037); falls back to Django's own caches["default"]
ICV_AUTH_USER_MODEL str settings.AUTH_USER_MODEL Which user model icv-sitemaps' last_modified_by FK targets. Fleet-wide convention (ADR-037); a project that configures nothing gets AUTH_USER_MODEL
ICV_TENANT_MODEL str "auth.Group" Which tenant model every tenant_ref FK targets (issue #50). The single ecosystem knob (ADR-025 T2, ADR-037); "auth.Group" is a functional floor so migration import never crashes when unset, and icv_sitemaps.W003 warns while it is active. Set it once, before first migrate; see "Multi-Tenancy" below
ICV_SITEMAPS_STORAGE_PATH str "sitemaps/" Base path within the storage backend
ICV_SITEMAPS_MAX_URLS_PER_FILE int 50000 Maximum URLs per file (protocol limit: 50,000)
ICV_SITEMAPS_MAX_FILE_SIZE_BYTES int 52428800 Maximum file size in bytes (protocol limit: 50 MB)
ICV_SITEMAPS_BATCH_SIZE int 5000 Queryset iteration batch size
ICV_SITEMAPS_GZIP bool True Compress files with gzip
ICV_SITEMAPS_PING_ENGINES list ["google", "bing"] Engines to ping after regeneration
ICV_SITEMAPS_PING_ENABLED bool True Enable/disable pinging
ICV_SITEMAPS_AUTO_SECTIONS dict {} Auto-register model sections (see Quick Start)
ICV_SITEMAPS_ROBOTS_EXTRA_DIRECTIVES list [] Extra lines appended to robots.txt
ICV_SITEMAPS_ROBOTS_SITEMAP_URL str "" Override sitemap URL in robots.txt (auto-detected if empty)
ICV_SITEMAPS_CACHE_TIMEOUT int 3600 Cache TTL for discovery files (seconds)
ICV_SITEMAPS_TENANT_PREFIX_FUNC str "" Dotted path to tenant prefix callable
ICV_SITEMAPS_ASYNC_GENERATION bool True Use Celery for background generation
ICV_SITEMAPS_STREAMING_THRESHOLD int 100000 URL count above which streaming generation is used
ICV_SITEMAPS_NEWS_MAX_AGE_DAYS int 2 Maximum age for news entries (Google requires < 2 days)
ICV_SITEMAPS_REDIRECT_ENABLED bool False Enable redirect middleware evaluation (opt-in)
ICV_SITEMAPS_REDIRECT_CACHE_TIMEOUT int 300 Cache TTL for redirect rule lookups (seconds)
ICV_SITEMAPS_404_TRACKING_ENABLED bool False Enable 404 tracking in the redirect middleware
ICV_SITEMAPS_404_TRACKING_SAMPLE_RATE float 1.0 Fraction of 404s to track (0.0--1.0)
ICV_SITEMAPS_404_IGNORE_PATTERNS list [r"\.(?:css|js|...)$"] Regex patterns for paths to ignore when tracking 404s
ICV_SITEMAPS_GONE_RESOLVER str "" Dotted path to a callable answering "is this already-404 path deliberately gone?"

Auto-Sections Configuration

Each key in ICV_SITEMAPS_AUTO_SECTIONS is the section name. The value is a configuration dict:

Key Type Default Description
section_type str "model" "model" or "static". A "static" section reads settings.url_provider/settings.urls instead of model
model str required for "model" "app_label.ModelName". Not used for "static" sections
sitemap_type str "standard" standard, image, video, or news
changefreq str "daily" Default change frequency
priority float 0.5 Default priority (0.0--1.0)
on_save bool True Mark section stale on model save. No-op for "static" sections (no model to hang the signal on)
on_delete bool True Mark section stale on model delete. No-op for "static" sections
settings.url_provider str none "static" sections only: dotted path to a no-argument callable returning entry dicts. Takes precedence over settings.urls
settings.urls list[dict] none "static" sections only: inline list of entry dicts

Service Functions

All functions are importable from icv_sitemaps.services:

from icv_sitemaps.services import (
    # Sitemap generation
    generate_section,
    generate_all_sections,
    generate_index,
    mark_section_stale,
    get_generation_stats,
    # Section management
    create_section,
    delete_section,
    # Search engine ping
    ping_search_engines,
    # robots.txt
    render_robots_txt,
    add_robots_rule,
    get_robots_rules,
    invalidate_robots_cache,
    # ads.txt
    render_ads_txt,
    add_ads_entry,
    invalidate_ads_cache,
    # Discovery files
    get_discovery_file_content,
    set_discovery_file_content,
    invalidate_discovery_cache,
    # Redirects
    check_redirect,
    add_redirect,
    bulk_import_redirects,
    bulk_create_redirects,
    invalidate_redirect_cache,
    record_404,
    get_top_404s,
)

Management Commands

Command Purpose
icv_sitemaps_setup [--dry-run] Create SitemapSection records from ICV_SITEMAPS_AUTO_SECTIONS and verify storage
icv_sitemaps_generate [--section NAME] [--all] [--index-only] [--force] [--tenant ID] Generate sitemaps; defaults to stale sections only
icv_sitemaps_ping [--url URL] [--tenant ID] Ping search engines
icv_sitemaps_validate [--section NAME] Validate generated sitemaps against protocol
icv_sitemaps_stats [--tenant ID] Show generation statistics
icv_sitemaps_redirects [--list] [--import FILE] [--export FILE] [--prune] [--top-404s] Manage redirect rules

Signals

All signals are defined in icv_sitemaps.signals:

Signal When
sitemap_section_generated After a section is successfully generated
sitemap_generation_complete After all sections are generated
sitemap_section_deleted After a section and its files are deleted
sitemap_pinged After search engines are pinged
sitemap_section_stale After a section is marked stale
redirect_rule_saved After a redirect rule is saved
redirect_rule_deleted After a redirect rule is deleted
redirect_matched When a redirect rule matches a request

Celery Tasks

Task Purpose Schedule
regenerate_stale_sitemaps Regenerate stale sections Every 15 minutes
regenerate_all_sitemaps Full regeneration Daily at 03:00
ping_engines_task Ping search engines After generation
cleanup_generation_logs Delete old logs (30-day default) Daily at 04:00
cleanup_orphan_files Remove unreferenced storage files Weekly
cleanup_expired_redirects Delete expired redirect rules Daily
cleanup_redirect_logs Delete old resolved 404 logs (90-day default) Weekly

Multi-Tenancy

Enable tenant-scoped discovery files by setting ICV_SITEMAPS_TENANT_PREFIX_FUNC to a dotted path to a callable that returns the tenant identifier:

# myapp/tenancy.py
def get_tenant_id(request):
    return getattr(request, "tenant_id", "")

# settings.py
ICV_SITEMAPS_TENANT_PREFIX_FUNC = "myapp.tenancy.get_tenant_id"

The callable must either return a safe tenant identifier (matching [\w\-]+) or a falsy value for single-tenant use; if it raises, or returns anything else, tenant resolution fails closed rather than falling back to the single-tenant bucket: views raise TenantResolutionError (a 500), and RedirectMiddleware passes the request through unmodified.

Each tenant gets isolated robots.txt, ads.txt, sitemaps, and all other discovery files. Sitemap files are stored with tenant-prefixed paths (e.g. sitemaps/acme/products-0.xml).

Tenant foreign key (issue #50)

The six tenant-keyed models (SitemapSection, RobotsRule, AdsEntry, DiscoveryFileConfig, RedirectRule, RedirectLog) also carry a nullable tenant_ref foreign key, resolved from ICV_TENANT_MODEL (default "auth.Group", a functional floor so migration import never crashes when unset). This is a fleet-standard shape (ADR-019 section 2), not this package implementing tenancy: tenant_id remains the scoping key every filter, storage path, and cache key uses; tenant_ref adds referential integrity and a column a django-boundary RLS policy can key on.

The two are kept consistent by the package itself. On clean() and save() of any of the six models:

  • If tenant_ref is unset, nothing changes.
  • If tenant_ref is set and tenant_id is blank, tenant_id is derived as str(tenant_ref_id).
  • If tenant_ref is set and tenant_id already holds a different value, save()/clean() raises ValidationError rather than silently overwriting it.

A consumer's tenant resolver should return str(request.tenant.pk), so the derived tenant_id always agrees with the FK. A consumer wiring django-boundary row-level security on one of these tables passes the FK's column explicitly, since it is not named tenant_id:

from boundary.migrations_ops import CreateTenantPolicy

CreateTenantPolicy("sitemapsection", tenant_column="tenant_ref_id")

Set ICV_TENANT_MODEL once, before the first migrate, and leave it unchanged thereafter (it is baked into swappable migration metadata, the get_user_model() pattern). icv_sitemaps.W003 warns at manage.py check time while the auth.Group floor is active; icv_sitemaps.E001 errors if the configured value cannot resolve to a model.

# settings.py
ICV_TENANT_MODEL = "myapp.Tenant"

Production Configuration

# settings.py
ICV_SITEMAPS_BASE_URL = "https://example.com"
STORAGES = {
    "default": {"BACKEND": "django.core.files.storage.FileSystemStorage"},
    "sitemaps": {"BACKEND": "storages.backends.s3boto3.S3Boto3Storage"},
    "staticfiles": {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"},
}
ICV_STORAGES_ALIAS = "sitemaps"
ICV_SITEMAPS_STORAGE_PATH = "sitemaps/"
ICV_SITEMAPS_GZIP = True
ICV_SITEMAPS_PING_ENGINES = ["google", "bing"]
ICV_SITEMAPS_BATCH_SIZE = 10000
ICV_SITEMAPS_ASYNC_GENERATION = True

Testing

The package provides testing utilities for consuming projects:

from icv_sitemaps.testing import (
    SitemapSectionFactory,
    StaticSitemapSectionFactory,
    SitemapFileFactory,
    SitemapGenerationLogFactory,
    RobotsRuleFactory,
    AdsEntryFactory,
    DiscoveryFileConfigFactory,
    RedirectRuleFactory,
    RedirectLogFactory,
)

To run the package's own tests:

git clone https://github.com/icvoss/django-icv-sitemaps.git
cd django-icv-sitemaps
pip install -e ".[dev]"
pytest tests/ -v

Models

Model Purpose
SitemapSection Logical sitemap section (e.g. "products", "articles") with staleness tracking
SitemapFile Individual generated XML file with URL count and checksum
SitemapGenerationLog Audit trail for generation runs
RobotsRule Database-driven robots.txt directives
AdsEntry ads.txt / app-ads.txt authorised seller entries
DiscoveryFileConfig Content store for llms.txt, security.txt, humans.txt
RedirectRule HTTP redirect and 410 Gone rules with pattern matching
RedirectLog Aggregated 404 tracking with hit counts and referrers

URL Endpoints

URL Content-Type Description
/sitemap.xml application/xml Sitemap index
/sitemaps/<filename> application/xml Individual sitemap files
/robots.txt text/plain Robots exclusion protocol
/llms.txt text/plain AI crawler guidance
/ads.txt text/plain Authorised digital sellers
/app-ads.txt text/plain Authorised app sellers
/.well-known/security.txt text/plain Security contact (RFC 9116)
/security.txt 301 redirect Redirects to /.well-known/security.txt
/humans.txt text/plain Team credits

Requirements

  • Python 3.11+
  • Django 5.1+
  • httpx 0.27+ (for search engine pings)
  • Celery 5.3+ (optional, for background generation)

Licence

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django_icv_sitemaps-3.2.0.tar.gz (170.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

django_icv_sitemaps-3.2.0-py3-none-any.whl (116.5 kB view details)

Uploaded Python 3

File details

Details for the file django_icv_sitemaps-3.2.0.tar.gz.

File metadata

  • Download URL: django_icv_sitemaps-3.2.0.tar.gz
  • Upload date:
  • Size: 170.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for django_icv_sitemaps-3.2.0.tar.gz
Algorithm Hash digest
SHA256 08a22cafe72e84af8fd2c0f7505b503ddb28141059755cbacf2fef4955c66d73
MD5 f832d4be6756659844a02a1d21b9d4e2
BLAKE2b-256 405470f65bb53a48f1f0b8c5b74cdafa15786f9aa9c856f20624601c9e1ac48c

See more details on using hashes here.

Provenance

The following attestation bundles were made for django_icv_sitemaps-3.2.0.tar.gz:

Publisher: publish.yml on icvoss/django-icv-sitemaps

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file django_icv_sitemaps-3.2.0-py3-none-any.whl.

File metadata

File hashes

Hashes for django_icv_sitemaps-3.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 f3a7c6c839fd3453bcd309ba66e06cf04d21827416f112ba60250310027e4960
MD5 d284f539dd1b6ec4b850e058f3f918b5
BLAKE2b-256 8e6da1698011aea1a1384daf1fe77672b3ebad72490f828e4f460604cfe18d53

See more details on using hashes here.

Provenance

The following attestation bundles were made for django_icv_sitemaps-3.2.0-py3-none-any.whl:

Publisher: publish.yml on icvoss/django-icv-sitemaps

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

3.4.0

2 files

3.3.1

2 files

3.3.0

2 files

This release

3.2.0 This release

2 files

3.1.1

2 files

3.1.0

2 files

3.0.0

2 files

2.0.0

2 files

1.0.0

2 files

0.8.0

2 files

0.7.0

2 files

0.6.1

2 files

0.6.0

2 files

0.5.1

2 files

0.5.0

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.0

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page