Skip to main content

bikiDATA

Developer-friendly Queries over RDF triples

Using this tool you can hit the ground running, and make some useful queries over RDF data using simple JSON calls.

Getting started TL;DR

You have a n-triple file called myfile.nt.

Install bikidata by saying pip install bikidata

Import it into bikidata with: python -m bikidata myfile.nt

And now, in a python prompt, you can query things, for example:

import bikidata

r = bikidata.query({
    "filters": [
        {"p":"fts",
         "o":"something"
        }
    ]
})

or

r = bikidata.query({
    "filters": [
        { "p":"id",
          "o":"<http://example.com/id/123>"
        }
    ]
})

or

r = bikidata.query({
    "filters": [
        {"p":"fts",
         "o":"something"
        },
        { "op":"not",
          "p":"<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>",
          "o":"<https://swapi.co/vocabulary/Species>"
        }
    ]
})

For more examples, see the file: examples.ipynb

Full-text search

The fts filters are answered by a Tantivy index, that is built together with the database, in a directory next to it (bikidata.duckdb.fts, or set BIKIDATA_FTS_PATH). The index has one document per entity, with:

  • its labels (rdfs:label, skos:prefLabel, schema:name, ...), weighted highest
  • all its other literals
  • the labels of the IRIs it links to, weighted lowest
  • the literals of the blank nodes it references, unrolled up to 3 levels deep. So a painting can be found by the name of its artist, even when the production event in between is a blank node.

All the terms of a query have to match, and the Tantivy query syntax can be used, for example "night watch" for a phrase or rembrandt OR vermeer. The filter "p": "fts 1" returns the entities that link to the matches, "p": "fts <iri>" only searches the literals of property <iri>. The results are sorted by score.

To (re)build the index of an existing database, for example one that was built with an earlier version of bikidata that used the DuckDB FTS extension:

import bikidata
bikidata.build_fts(stemmer="porter")  # or "german", "french", ..., or "none"

bikidata.fts_search("rembrandt", size=10)  # [(entity, score), ...]

Note that the index is not updated by insert_async and delete_async, rebuild it with build_fts after changes.

SPARQL queries

Next to the JSON queries, the data can also be queried with SPARQL 1.1. The SPARQL query is translated into a single SQL query that runs on the same DuckDB tables, no extra index is needed.

import bikidata

r = bikidata.sparql("""
PREFIX schema: <http://schema.org/>
SELECT ?person ?name WHERE {
    ?person a schema:Person ;
            schema:name ?name .
    FILTER(LANG(?name) = "en")
}
ORDER BY ?name
LIMIT 10
""")

for row in r["results"]["bindings"]:
    print(row["person"]["value"], row["name"]["value"])
  • SELECT queries return a dict in the SPARQL 1.1 Query Results JSON Format.
  • ASK queries return {"head": {}, "boolean": True}.
  • CONSTRUCT and DESCRIBE queries return a list of (s, p, o) tuples, each term encoded as in N-Triples.

Optional parameters are db_path (defaults to the BIKIDATA_DB database), base for resolving relative IRIs and union_default_graph. Triples loaded from a .trig file are in named graphs, that you can query with GRAPH. By default the default graph only contains the triples that are not in a named graph, pass union_default_graph=True to use the union of all the graphs as the default graph.

To see the SQL a SPARQL query is translated into, use bikidata.sparql_engine.to_sql(query).

The implementation is tested with the W3C SPARQL 1.0 and 1.1 query test suites, see tests/test_w3c.py, where the few tests that do not pass are listed in tests/w3c_known_failures.txt. Not supported are SERVICE (federated queries), SPARQL Update, and entailment regimes.

Redis support

When querying non-trivial datasets of a few million triples, or handling many concurrent users, we do not want to open a new database connection for each query, and cache the results in memory. By using Redis we can scale the number of bikidata workers horizontally, and share the cache between them. The query_async function can now be awaited in an async context. This function places the query in a Redis queue, and awaits the result.

To use this, you need to have a Redis server running, and install the redis python package:

pip install redis

And then you need to run the bikidata worker in a separate terminal:

python -m bikidata worker

It is possible to run multiple workers, and they will share the load.

Then, in your code, you can await the query_async function in stead of the regular query function:

Metadata

Release files for bikidata 0.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bikidata 0.5
File Size Uploaded Provenance
bikidata-0.5.tar.gz 52.6 kB Without provenance Details

Built distribution (wheel)

Table of built distributions (wheels) for bikidata 0.5
File Interpreter ABI Platform
bikidata-0.5-py3-none-any.whl Python 3 none any Details

Total release size: 98.7 kB

Release files / bikidata-0.5.tar.gz

Download URL bikidata-0.5.tar.gz
Size 52.6 kB
Tags Source
SHA-256 checksum
How to use checksums
df82f630e073872b415c73d2ceef174439f1960379c9c8a29763b8fa79805a81
BLAKE2b-256 checksum
How to use checksums
7b31c5801d2acb652ea8e52e7694516490190ebf55359de943c177601feb7838
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / bikidata-0.5-py3-none-any.whl

Download URL bikidata-0.5-py3-none-any.whl
Size 46.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7f2994507b5404b0905ca3e1d47d8b09ac55368591b4f3eada7b3400c4329a3e
BLAKE2b-256 checksum
How to use checksums
b17133d88ac7f128ba9edc252881d531fa061d7026b842dab05a69cfc503cbb1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for bikidata 0.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.5 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page