Skip to main content

A Store back-end for rdflib to allow for reading and querying HDT documents

Project description

rdflib-htd logo

Build Status PyPI version

A Store back-end for rdflib to allow for reading and querying HDT documents.

Online Documentation

Requirements

  • Python version 3.6.4 or higher

  • pip

  • gcc/clang with c++11 support

  • Python Development headers ..

    You should have the Python.h header available on your system.For example, for Python 3.6, install the python3.6-dev package on Debian/Ubuntu systems.

Installation

Installation using pipenv or a virtualenv is strongly advised!

Manual installation

Requirement: pipenv

git clone https://github.com/Callidon/pyHDT
cd pyHDT/
./install.sh

Getting started

You can use the rdflib-hdt library in two modes: as an rdflib Graph or as a raw HDT document.

HDT Document usage

from rdflib_hdt import HDTDocument

# Load an HDT file. Missing indexes are generated automatically.
# You can provide the index file by putting them in the same directory than the HDT file.
document = HDTDocument("test.hdt")

# Display some metadata about the HDT document itself
print(f"Number of RDF triples: {document.total_triples}")
print(f"Number of subjects: {document.nb_subjects}")
print(f"Number of predicates: {document.nb_predicates}")
print(f"Number of objects: {document.nb_objects}")
print(f"Number of shared subject-object: {document.nb_shared}")

# Fetch all triples that matches { ?s foaf:name ?o }
# Use None to indicates variables
triples, cardinality = document.search_triples((None, FOAF("name"), None))

print(f"Cardinality of (?s foaf:name ?o): {cardinality}")
for s, p, o in triples:
  print(triple)

# The search also support limit and offset
triples, cardinality = document.search_triples((None, FOAF("name"), None), limit=10, offset=100)
# etc ...

An HDT document also provides support for evaluating joins over a set of triples patterns.

from rdflib_hdt import HDTDocument
from rdflib import Variable
from rdflib.namespace import FOAF, RDF

document = HDTDocument("test.hdt")

# find the names of two entities that know each other
tp_a = (Variable("a"), FOAF("knows"), Variable("b"))
tp_b = (Variable("a"), FOAF("name"), Variable("name"))
tp_c = (Variable("b"), FOAF("name"), Variable("friend"))
query = set([tp_a, tp_b, tp_c])

iterator = document.search_join(query)
print(f"Estimated join cardinality: {len(iterator)}")

# Join results are produced as ResultRow, like in the RDFlib SPARQL API
for row in iterator:
   print(f"{row.name} knows {row.friend}")

Handling non UTF-8 strings in python

If the HDT document has been encoded with a non UTF-8 encoding the previous code won’t work correctly and will result in a UnicodeDecodeError. More details on how to convert string to str from C++ to Python here

To handle this, we doubled the API of the HDT document by adding:

  • search_triples_bytes(...) return an iterator of triples as (py::bytes, py::bytes, py::bytes)

  • search_join_bytes(...) return an iterator of sets of solutions mapping as py::set(py::bytes, py::bytes)

  • convert_tripleid_bytes(...) return a triple as: (py::bytes, py::bytes, py::bytes)

  • convert_id_bytes(...) return a py::bytes

Parameters and documentation are the same as the standard version

from rdflib_hdt import HDTDocument

document = HDTDocument("test.hdt")
it = document.search_triple_bytes("", "", "")

for s, p, o in it:
print(s, p, o) # print b'...', b'...', b'...'
# now decode it, or handle any error
try:
   s, p, o = s.decode('UTF-8'), p.decode('UTF-8'), o.decode('UTF-8')
except UnicodeDecodeError as err:
   # try another other codecs, ignore error, etc
   pass

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rdflib_hdt-3.1.tar.gz (235.9 kB view details)

Uploaded Source

Built Distribution

rdflib_hdt-3.1-py3.7-linux-x86_64.egg (7.4 MB view details)

Uploaded Source

File details

Details for the file rdflib_hdt-3.1.tar.gz.

File metadata

  • Download URL: rdflib_hdt-3.1.tar.gz
  • Upload date:
  • Size: 235.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for rdflib_hdt-3.1.tar.gz
Algorithm Hash digest
SHA256 0db95fe58e276fe58668cae6ef94dd7b7b30bf1e9f89dd06d3941208a17fcc44
MD5 f7e619415939ff0ee56d6405160fef0f
BLAKE2b-256 ea11f83aedf9517a20fe10fbd2e1ccd147760322b8cce36eac9efedb8b3783be

See more details on using hashes here.

File details

Details for the file rdflib_hdt-3.1-py3.7-linux-x86_64.egg.

File metadata

File hashes

Hashes for rdflib_hdt-3.1-py3.7-linux-x86_64.egg
Algorithm Hash digest
SHA256 5efa586ae8934b4c968c4f7b9ec14864b08efd6b9a3342f2463982172f6519ed
MD5 cb6993ccd6f278349d2e2d188c701fc4
BLAKE2b-256 3b04d584ae3e684d8522ce77baf1c155bb35ad2ca0462d5e1b6b675f8c3b3c87

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page