datagrepper
Datagrepper is a web application and JSON API to retrieve historical messages sent via Fedora Messaging. Datanommer is a seperate project and service that consumes messages from the Fedora Messaging queue and puts them in a database. These messages is what datagrepper queries.
Datagrepper is curently running in production at https://apps.fedoraproject.org/datagrepper/
Development Environment
Vagrant allows contributors to get quickly up and running with a datagrepper development environment by automatically configuring a virtual machine.
The datagrepper Vagrant environment configures configures and enables a datanommer service and database. The datanommer instance is configured to be empty when first provisioned, but to consume messages from the stage Fedora Messaging queue.
Install vagrant
To get started, run the following commands to install the Vagrant and Virtualization packages needed, and start the libvirt service:
$ sudo dnf install ansible libvirt vagrant-libvirt vagrant-sshfs vagrant-hostmanager
$ sudo systemctl enable libvirtd
$ sudo systemctl start libvirtd
Checkout and Provision
Next, check out the datagrepper code and run vagrant up:
$ git clone https://github.com/fedora-infra/datagrepper
$ cd datanommer
$ vagrant up
Interacting with your development datagrepper
After successful provisioning of the Datagrepper vagrant setup, the datagrepper web application will be accessible from your host machine's web browser at
Using the development environment
SSH into your newly provisioned development environment:
$ vagrant ssh
The vagrant setup also defines 4 handy commands to interact with the service that runs the datagrepper flask application:
$ datagrepper-start
$ datagrepper-stop
$ datagrepper-restart
$ dataprepper-logs
Additionally, the following commands are also available for interacting with the datanommer service:
$ datanommer-consumer-start
$ datanommer-consumer-stop
$ datanommer-consumer-restart
$ datanommer-consumer-logs
Running the tests
Datanommer is comprised of 3 seperate modules in this single repository. There is a handy script in the top directory of this repo to run the tests on all 3 modules:
$ ./runtests.sh
However, tests can also be run on a single module by invotking tox in that modules' directory. For example:
$ cd datanommer.models/
$ tox
Note, that the tests use virtual environments that are not created from scratch with every subsequent run of the tests. Therefore, when changes happen to dependencies, the tests may fail to run correctly. To recreate the virtual envrionments, run the tests commands with the -r flag, for example:
$ ./runtests.sh -r
or
$ cd datanommer.models/
$ tox -r
Release files for datagrepper 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| datagrepper-1.1.0.tar.gz | 3.1 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| datagrepper-1.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 6.3 MB
Release files / datagrepper-1.1.0.tar.gz
| Download URL | datagrepper-1.1.0.tar.gz |
|---|---|
| Size | 3.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3b461cbb5f54ef1b133bb462a8f2ed872c8832ae4bddce110c086d822b98ec35
|
|
BLAKE2b-256 checksum How to use checksums |
d92b6d36f46610750d75b755cee90e156eece31e13e474d4427943d8563de035
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.5 Linux/6.4.15-200.fc38.x86_64
|
Release files / datagrepper-1.1.0-py3-none-any.whl
| Download URL | datagrepper-1.1.0-py3-none-any.whl |
|---|---|
| Size | 3.1 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8c774363ca9094294ad02d6b49373e6feb5e8491b8536ddd607b48bc0ba7b832
|
|
BLAKE2b-256 checksum How to use checksums |
f6e600e2e1c58501b32ee93e3542866742ecc7299c2c754590e451709943c561
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.5 Linux/6.4.15-200.fc38.x86_64
|
Security for datagrepper 1.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
