django-controlled-vocabularies

A Django app for managing, publishing, and consuming SKOS controlled vocabularies — built for research organisations. Curators create and edit vocabularies through a web interface; developers attach concepts to their own models and serve standards-compliant RDF at stable URIs. No hand-edited RDF, no code releases to add a term, no triplestore to operate.

Status: early development (pre-0.1). The data model and design are sketched out (see docs/brainstorm.md); the first release targets consumption (import, models, a concept relationship field, and RDF export) ahead of the editing interface. See GOALS.md.

Scope & philosophy

Research data infrastructure runs on controlled vocabularies, but Django has no native way to manage or consume them, and the existing editors are foreign stacks, unmaintained, or deployment-heavy. This app closes that gap by treating a vocabulary as relational data, not a document: concepts are Django models, the database is the source of truth, and RDF is a projection produced only at the import/export boundary. That turns everything Django already does well (forms, permissions, indexed search, referential integrity) onto vocabulary management, and lets vocabularies scale to tens of thousands of concepts and evolve as data rather than code.

It deliberately is:

• SKOS-focused.
• A Django app — installable into any Django project (notably FairDM) and runnable standalone.
• Both a manager (author, edit, version, publish) and a consumer (attach concepts to your models via a field; serve RDF at stable URIs).
• Multilingual — a concept holds labels and definitions in any number of languages, with one preferred label per language. Each vocabulary picks the language that anchors its concept URIs, and a curator can override any concept's slug.
• A graph, not a flat list — concepts link to one another through a broader/narrower hierarchy (navigable both ways, and a concept may sit under several broader concepts) and a symmetric related association, all within one vocabulary.
• Organisable into collections — a curator can gather a vocabulary's concepts into named collections, optionally in a deliberate order, separate from the hierarchy and from any other collection a concept belongs to.

It deliberately is not:

• A general RDF/OWL toolkit — the SKOS-only scope is intentional.
• A triplestore or SPARQL endpoint — the relational database is the store.
• A replacement for rdflib — rdflib is used only to parse and serialize at the boundary.
• A reasoner — no OWL inference.
• An editor for external vocabularies — imported external vocabularies are read-only references.
• A way to extend a published vocabulary with your own terms — that is out of scope.
• A faithful mirror of imported external vocabularies — imports are normalised to what the app supports (e.g. its configured languages); unsupported languages and constructs are not stored.

Tie-breaks, when principles collide: the database is the source of truth over RDF fidelity in memory · lossless round-tripping over schema neatness · stable concept URIs over convenient identifiers · vocabulary-as-data over vocabulary-as-code · SKOS fidelity over generality.

Configuration

Every vocabulary, concept, and collection has a URI — its identity, always present. It is static, held exactly as given and never recomputed by the app, once it is fixed: a record imported from an external vocabulary keeps the identifier its publisher assigned it. A record authored here instead has a dynamic URI, composed from a base address, until it is published, at which point that becomes static too. Set the base address in your Django settings:

CONTROLLED_VOCABULARIES_BASE_URI = "https://vocab.example.org/vocabularies"

If you leave it unset, the app falls back to http://localhost:8000/vocabularies so it still runs out of the box — set a real address before you rely on it anywhere it is seen. This site's own address for viewing a record is always composed from this base and the record's slugs — a concept's is {base}/{scheme-slug}/{concept-slug} — even when that record's static URI points elsewhere, and the slugs follow the labels while a vocabulary is unpublished.

A record's place in its vocabulary is what keeps its URI distinct, and the database enforces that: a vocabulary's slug is unique across the site, and a concept's or collection's slug is unique within its vocabulary. A composed URI therefore cannot collide, because the parts it is built from cannot. A stored static_uri carries its own unique index per model on top of that.

Nothing stops you changing a static_uri once it is set. That is deliberate. A published identifier is not supposed to move, but keeping it still is a matter of not editing it, rather than of the model refusing every write path. Make the field non-editable wherever you expose it once a record is published, and treat a value that differs between two imports of the same vocabulary as a problem with the source file.

An externally assigned static URI must use one of a small set of accepted schemes — http, https, urn, doi, info, ark, tag, hdl, and oai by default, since those are what real SKOS vocabularies actually use. If your vocabularies carry identifiers in another scheme, add it explicitly:

CONTROLLED_VOCABULARIES_ALLOWED_URI_SCHEMES = [
    "http", "https", "urn", "doi", "info", "ark", "tag", "hdl", "oai", "my-custom-scheme",
]

A stored identifier is later rendered as a link, so schemes that can carry executable content (javascript, data, vbscript) are refused even if you add them to this setting.

Attaching a concept to your model

ConceptField is a ForeignKey to Concept, optionally restricted to the vocabularies you name. vocabulary takes three shapes:

from django.db import models

from controlled_vocabularies.fields import ConceptField


class Specimen(models.Model):
    name = models.CharField(max_length=200)
    rock_type = ConceptField(vocabulary="rock-type")                     # one vocabulary
    dominant_material = ConceptField(vocabulary=["igneous", "mineral"])  # several
    keyword = ConceptField(null=True, blank=True)                        # no vocabulary named

vocabulary names the owning ConceptScheme by its slug rather than by a database relation. A field declaration is read when Python imports the module, and a vocabulary is rows in a table that may not exist yet, so the slug is all the field can carry.

vocabulary="rock-type" restricts to one vocabulary. A list restricts to the union of the named vocabularies, and the refusal names every vocabulary the field accepts. Leaving vocabulary out entirely restricts nothing: any concept in the database is a valid choice. That last shape is for a field drawing on whatever a project has imported rather than one fixed list.

When the field names at least one vocabulary, only a concept whose scheme.slug matches is offered as a form choice or accepted by full_clean(). A concept from any other vocabulary is refused. Deleting a concept a record still references is refused whichever shape you declare (on_delete=PROTECT), whether the delete reaches it directly or cascades down from its scheme.

Reading a concept back:

specimen.rock_type              # the attached Concept, or None
specimen.get_rock_type_label()  # its preferred label in the active language, falling
                                # back to the vocabulary's default language
specimen.get_rock_type_uri()    # its URI

If "rock-type" has not been imported yet, which is the state of a fresh install before the vocabulary's own import step has run, nothing about the field declaration fails. manage.py check reports a warning (controlled_vocabularies.W001) naming the model, the field, and the missing slug. If the vocabulary genuinely arrives in a later deployment step, silence the warning with SILENCED_SYSTEM_CHECKS.

Reading get_<field>_label() on every row of a list costs a query for the concept and one for its scheme, plus one for its labels when the active language is not the vocabulary's default, per row. select_related("rock_type__scheme") and prefetch_related("rock_type__labels") collapse that back to a fixed number of queries for the whole list.

Attaching several concepts to your model

ConceptsField is the many-to-many counterpart: several concepts on one record instead of one. vocabulary takes the same three shapes it takes on ConceptField, and means the same thing by each:

from django.db import models

from controlled_vocabularies.fields import ConceptsField


class Sample(models.Model):
    name = models.CharField(max_length=200)
    rock_types = ConceptsField(vocabulary=["igneous", "sedimentary"])  # several vocabularies
    keywords = ConceptsField(blank=True)  # no vocabulary named

One slug restricts to that vocabulary, a list restricts to the union of the named vocabularies, and leaving vocabulary out — as keywords does above — restricts nothing. A concept outside the named vocabularies is refused, and the refusal names every vocabulary the field accepts.

Reading concepts back:

sample.rock_types                    # the many-to-many manager
sample.get_rock_types_labels()       # a list of preferred labels, active language with fallback
sample.get_rock_types_uris()         # a list of URIs, same order

Both accessors return an empty list for a record holding nothing, including one that has not been saved yet, rather than raising.

Required and optional. blank=True, as on keywords above, makes the field optional: a record can hold zero concepts. Without it, as on rock_types, the field is required, and required means at least one concept attached. Django gives a many-to-many field no hook into ordinary field validation, so this is enforced when full_clean() runs rather than by the database — a ModelForm or an explicit full_clean() call catches an empty required field, a plain .save() does not. A record cannot hold any concepts before it has a primary key, so the check is skipped on an unsaved instance. If your own model overrides full_clean() after declaring the field, your override shadows this check, and you'll need to call it yourself or reproduce it.

Query cost. get_<field>_labels() and get_<field>_uris() issue a query for the field's concepts, and a further query per concept for its scheme and its labels — the same per-concept cost ConceptField's singular accessors carry, multiplied by however many concepts a record holds. Over a list of records that's at least one query per record. prefetch_related("rock_types__scheme", "rock_types__labels") collapses that back to a fixed number of queries for the whole list.

What every shape guarantees, and what the unrestricted one gives up. Naming one vocabulary, several, or none all give a record the same three things: a concept it already holds cannot be deleted out from under it, its attached concepts read back by label and by URI, and required still means at least one. Naming one or several vocabularies adds a fourth guarantee on top — a concept outside those vocabularies is refused, both by a form built from the model and by the relation itself, in either direction, at the moment the membership is written.

A field naming no vocabulary gives up only that fourth guarantee, and nothing else. Said plainly, so it needn't be inferred:

• It places no restriction on which concepts a record can hold — any concept in the database is accepted.
• A form built from the model offers every concept in the database as a choice, not a filtered subset. On an installation with several vocabularies imported, that can be a genuinely long list of choices in an ordinary select widget — sooner than a restricted field would ever reach.
• manage.py check has no named vocabulary that could be missing, so it reports nothing for this field.

It still deletes-protects, reads back, and enforces "required" exactly as a restricted field does.

The delete guarantee's real boundary. "Cannot be deleted out from under it" holds for anything that goes through the Django ORM — a single instance, and a bulk queryset .delete() alike, since Django applies on_delete=PROTECT to both the same way. It does not hold for a DELETE issued directly against the database outside Django — raw SQL, a database console, a migration that drops rows without going through the ORM — because Django never writes an ON DELETE clause into the schema for any relation. The protection lives in application code, not in a database constraint.

Choosing a concept by typing

ConceptField and ConceptsField render as a search-as-you-type control by default. A project declaring either gets it without naming a widget, writing an endpoint, or configuring anything per field — an ordinary ModelForm built from the model already renders it.

Getting it working outside the admin takes three steps, done once per project rather than once per field. In the order a developer does them:

1. Include the route. The package carries its own search endpoint in controlled_vocabularies.urls; include it once, at an address of your choosing:

   from django.urls import include, path
   
   urlpatterns = [
       ...
       path("vocab/", include("controlled_vocabularies.urls")),
   ]
   

2. Add "django_tomselect" to INSTALLED_APPS. The control's templates and static assets live inside that package, and Django only finds another package's templates and static files through an installed app — no amount of work in this package moves that.

3. Add "django_tomselect.middleware.TomSelectMiddleware" to MIDDLEWARE. Without it, the control still renders, but as a plain <select> carrying no search box and no JavaScript at all — nothing errors, nothing warns at render time, and the control simply doesn't work.

Skip any of the three and manage.py check reports it: a missing route (controlled_vocabularies.W002), a missing INSTALLED_APPS entry (controlled_vocabularies.W003), or a missing middleware entry (controlled_vocabularies.W004), each naming exactly what to add. A render that reaches the missing-route case despite the warning raises ImproperlyConfigured naming both the route and the INSTALLED_APPS entry, rather than the underlying library's own NoReverseMatch against a URL pattern you never wrote.

A search returns, per concept, its identifier, its preferred label in the active language, and the name of the vocabulary it belongs to — nothing else, no editorial notes or alternative labels. The results are restricted to whatever vocabularies the field declares, the same restriction that already governs which concepts a form accepts; a field naming no vocabulary searches every concept in the database. The endpoint carries no permission rule by default — it serves the same concept data the rest of this package publishes at stable URIs. A project holding vocabularies it does not want searched restricts the include instead, at whatever access level fits.

The control needs a browser running JavaScript. Without it, both fields still work as an ordinary required-relation form field; only the search-as-you-type behaviour is unavailable.

Choosing a concept in the admin

A model that declares ConceptField or ConceptsField gets the same search-as-you-type control on its Django admin pages once it's registered — nothing further to configure. ModelAdmin builds the field the same way it builds any other, and the wiring is the three steps above: the route, django_tomselect in INSTALLED_APPS, and its middleware. The admin adds nothing to that list.

A concept is chosen on these pages, never created or edited there. Django ordinarily puts an add, change, delete or view link beside a related field so a person filling in a form can manage the related record without leaving it. None of the four appears beside a concept field, whether or not this package's own models are registered in the admin, and whatever permissions the signed-in person holds. Authoring stays wherever concepts themselves are curated — a controlled vocabulary stops being controlled the moment a data-entry form can invent or edit one.

Where the admin presents the field read-only — listed in readonly_fields, or because the signed-in person may view the page but not change it — the concept is shown by its preferred label and no control renders. This is Django's own read-only presentation, not this package's: with Concept registered, a single concept links to its own change page. Several concepts, on a ConceptsField, always render as plain text. The package neither adds to this nor suppresses it.

A project can still ask for a different control the way it would for any other related field:

• autocomplete_fields renders Django's own autocomplete widget instead — Django requires the related model to be registered and searchable, as it does for any field named this way.
• raw_id_fields renders Django's plain identifier input, with the magnifying-glass lookup link Django gives every field named this way. That link opens a list to pick from — it selects a concept, it does not create one.
• A ModelForm declaring its own widget for the field via Meta.widgets, passed to ModelAdmin.form, renders that widget.
• readonly_fields renders the read-only presentation described above.

The first three still refuse the add, change, delete and view affordances, the same as the default control, and saving and validation carry on unchanged: a concept outside the field's named vocabularies is refused, and a legitimate one still saves.

One piece of text on these pages is Django's rather than this package's. Under a multi-value field the admin adds Hold down "Control", or "Command" on a Mac, to select more than one. It is the same sentence the admin puts under any multiple-select field, and it does not describe this control: concepts are added by typing and picking, and removed one at a time.

Importing a published vocabulary

import_skos() reads a SKOS file — Turtle, RDF/XML, or JSON-LD — and creates or updates the vocabulary it declares, along with every concept it contains: identity, labels, documentary notes, broader/narrower and related relationships, and collection membership. Every record is matched by its static URI, never by name, so importing the same file twice updates the same rows rather than duplicating them.

from controlled_vocabularies.exchange import import_skos

report = import_skos("rocks.ttl")

The caller may name a target ConceptScheme for a file that declares no vocabulary of its own, or have it checked against one the file does declare:

report = import_skos("rocks.ttl", scheme=my_scheme)

Re-running an import upserts rather than deleting and recreating. For a record the file still contains, the file is authoritative for that record's own content — labels, notes, relationships, and collection membership end up matching the file exactly, including the removal of a value the file no longer carries. A record the file does not mention at all is left completely untouched and named in the report instead, never deleted.

Labels and notes are matched by language, not only by predicate. A file's en value fills a site configured for en-gb, and a file's en-gb value fills a site configured for en. Matching runs on the base language, the first subtag of the tag, case-insensitively and in both directions, and an exact tag match always wins over a variant.

Where a file offers several variants of one configured language, what happens next depends on how many values that kind of label or note can hold:

• A preferred label holds one per language, so the variant the vocabulary predominantly publishes in is the one kept, with ties broken by language code. Each variant that lost is named in the report under its own published language.
• Alternative labels, hidden labels, and notes have no such limit, so every variant the file offers is stored.

A value stored under a variant match is named in the report as a substitution rather than applied silently. That matters most where two variants differ by script. A site configured for zh-Hans importing a vocabulary published only as zh-Hant receives that content, in a script its readers may not be able to read. The report is what makes that visible, so a curator can decide what to do about it.

The package stores content for every language code in settings.LANGUAGES. A project that declares none inherits Django's own default list of 99 languages, so a vocabulary published in sixty languages imports into all sixty. Narrowing LANGUAGES is how a site limits what an import stores.

A vocabulary's slug and a concept's slug are both derived from their own published identifier — its fragment where it has one, otherwise the last segment of its path — assigned once, on first import, and never recomputed by a later one. Renaming a record, or a vocabulary's name arriving in a different language on a later import, never moves that record's local address; only the publisher reassigning the identifier itself does. A record authored on this site rather than imported has no published identifier to derive from, and keeps deriving its slug from its label, as it always has.

That permanence has a cost worth naming: a vocabulary published under opaque codes gets an opaque local address — /v-113/00123 rather than /soil-types/clay. It is accepted because the address is correct and stays correct, which is what every consumer of a URL needs from it, and because a readable address that can move under data already pointing at it is worse than a stable one that cannot.

The call returns an ImportReport, a plain dataclass rather than rendered text, so a caller can inspect what happened without parsing anything:

• created / updated — the URIs of every vocabulary, concept, and collection the run wrote.

• set_aside — a SetAsideEntry per value the run could not store, each carrying a closed, translatable reason (an unconfigured language, a notation, a mapping to another vocabulary, a predicate the models have no place for, a missing relationship or collection member, and so on) and the data needed to render a message about it. Nothing a file contains is ever dropped in silence — a value the app cannot store is always named here.

  Published files are often not well-behaved, and the reasons cover that too. Where a pair of concepts is stated as both broader and related, the hierarchical statement wins, because SKOS declares the two disjoint, and the related one is set aside. A second preferred label in one language is set aside rather than refused at the database. An identifier whose own segment yields no usable slug is set aside naming the slug as the problem, not the identifier. None of these stops the rest of the vocabulary from importing.

• absent_from_source — the URIs of records that exist here but that the file no longer mentions.

• normalized — a NormalizedEntry per value the run stored, but under a different predicate than the one the file asserted (a foreign dcterms:description read as a concept's definition, for example).

• fatal — populated only on a failed run: a FatalFinding per reason the whole import was refused (a missing or blank-node identity, or a vocabulary that could not be resolved). A failed run raises SkosImportFailed, carrying this same report, and leaves the database exactly as it was before the run started.

A run either succeeds in full or changes nothing: every problem in a file is collected before any of it is written, and a fatal one rolls the whole run back. The import_skos management command wraps this for use from a terminal (see below). There is no web-facing entry point yet.

Reading a file never reassigns identity. A concept or collection whose URI is already held by a different vocabulary stays where it is. So does one whose URI is held by a record of another kind, such as a collection in one file and a concept in another. Both are set aside and reported. Moving a record between vocabularies is a curatorial decision, not a side effect of reading a file.

An imported file is treated as untrusted input. RDF/XML is scanned for entity expansion and external references before a parser sees it. A JSON-LD document is refused rather than fetched if its @context names a remote location, whether that is a plain string reference or an @import reference tucked inside an inline object context. import_skos() itself never makes a network request when reading a file — the command's own URL fetch (see below) happens first, and only the fetched or local bytes ever reach this scan. An ordinary inline JSON-LD context, carrying no such reference, imports normally.

import_skos() raises one of two exceptions, both ValidationError subclasses carrying a translatable message. SkosImportError covers every reason a file cannot be turned into usable SKOS at all: not found, not in a supported serialization, unparseable, or refused by the safety scan above. SkosImportFailed covers the case where the file parses but the run collects one or more fatal problems (a missing or blank-node identity, or a vocabulary that cannot be resolved), and carries the same ImportReport its fatal bucket names them in. UnsafeRdfXmlError and UnsafeJsonLdError, the two exceptions the safety scan itself raises, are exported SkosImportError subclasses, so code that only catches (SkosImportError, SkosImportFailed) already catches a file the safety scan refuses too.

Importing from the command line

The import_skos management command wraps import_skos() for use from a terminal or a deployment script:

python manage.py import_skos rocks.ttl

The source can be a local filesystem path or an http:///https:// URL, told apart by the value itself rather than by a flag. A URL is fetched under a fixed 30-second read timeout, a fixed 50 MiB response ceiling and a fixed ten-minute deadline for the whole transfer, over a connection that only ever speaks http and https. The fetched document's identifiers resolve against the address it was served from, following any redirect — so a vocabulary published behind a PURL or a /latest alias is stored under the URIs its publisher assigned, and a re-import updates the same concepts the first import created rather than making a second copy.

• --format names the source's serialization (turtle, xml, or json-ld), for a source whose extension or Content-Type does not.
• --dry-run performs the entire import and reports the outcome exactly as a live run would, then leaves the database exactly as it was beforehand — useful for seeing what a file would set aside before deciding whether to configure a language for it.
• --verbosity, Django's own option, prints bucket counts by default — how many records were created, updated, set aside, normalized, or absent from the source, plus the set-aside account grouped by reason and by language. At 2 or above, every individual set-aside entry prints too. At 0 nothing prints at all, as with any Django command.

A refusal exits non-zero and prints every reason the run was refused. A run that sets values aside still exits zero: importing a vocabulary published in more languages than a site is configured for always sets some aside, so that outcome is treated as normal rather than as a failure a deployment script should stop on.

Relationship to other packages

Supersedes and retires skos-builder and django-research-vocabs, consolidating vocabulary authoring, management, and Django consumption into one app.

License

MIT — see LICENSE.