pyheimdall-heurist 1.0.0

Heurist connector for pyHeimdall

What is this?

Heimdall is a tool for converting more easily one or more databases from one format to another. It leverages modules called “connectors”, responsible for conversion of data between specific databases schemas and the HERA format.

This repository contains a connector to Heurist’s own XML export format (HML).

Why should I use it?

You can use this connector, along with the pyheimdall software, to retrieve any data exported from Heurist. You can then aggregate this data into your research corpus easily, for example using other Heimdall connectors.

Take note, however that some legal restrictions might apply to data retrieved from any scientific database.

Plus, if at the end of your project, you share your data, please cite the original data properly.

How can I use it?

Setup

This pyHeimdall connector is available as a PyPI package named pyheimdall-heurist. You can install it using the pip package manager:

pip install pyheimdall-heurist

You can use pip to either upgrade or uninstall this connector, too:

pip install --upgrade pyheimdall-heurist
pip uninstall pyheimdall-heurist

Usage

import heimdall

tree = heimdall.getDatabase(format='heurist:xml', url='Export_db_AAAAMMDDHHMMSS.xml')
heimdall.createDatabase(tree, format='csv', url='.')

Please note that you shouldn’t use pyheimdall-heurist functions directly. As long as the package is installed on your system, pyHeimdall will automatically discover its features and allow you to use them as long as any other default or external connector.

Is it tested?

Well, yes … and no.

As you can see in our badge list, or in this coverage report, the code coverage by our unit tests is not too shabby. However, to validate their behaviour, these tests take the file /tests/resources/heurist.xml as an input, which simulates an exported XML (“HML”) file from Heurist.

And this is where the problem lies.

At the time of writing (early 2025), Heurist still has an outdated comunity management and is full of undocumented behaviours. Among these, the “HML” file remains unspecified, the link to its XSD is broken, and Heurist doesn’t even behave as expected, when one feeds it its own output. So, this connector really tries its best to deduce how stuff seems to work in most cases.

This as two limits, though.

• First, those assumptions can be wrong, or incomplete. Still at the time of writing, the Heurist core development team seems unable to document its own output, so … well.

• Second, even if everybody seems fine today, the Heurist core development team is known to force unto its userbase new (sometimes breaking) updates without any prior notice, documentation or regression testing. Of course, this means that the /test/resources/heurist.xml file may become outdated with time, and this connector won’t automagically warn you of any problem.

Unit testing with a static test file is, of course, not ideal. But (you know the drill: “at time of writing”) Heurist is devoid of any machine-usable API, let alone RESTful. So there isn’t really any other choice.

As you may understand, by design of its own creator, third party softwares trying to be interoperable with Heurist are just … well, f•••ed.

To conclude: depending on when you read this, you can install this connector, try to load a recently exported XML (HML) file, and if it doesn’t work, you’re probably f•••ed, too.

That is, unless you contribute, of course.

How can I contribute?

PyHeimdall welcomes any feedback or proposal. Details can be accessed here

License

GNU Affero General Public License version 3.0 or later