Karp backend
Project description
Karp backend
This in the version 6 of Karp backend, for the legacy version (v5).
Setup
This project uses poetry and MariaDB.
-
Run
make install
ormake install-dev
for a develop-install -
Install MariaDB and create a database
-
Setup environment variables (can be placed in a
.env
file in the root and then ?poetry run
sets those):export DB_DATABASE=<name of database> export DB_USER=<database user> export DB_PASSWORD=<user's password> export DB_HOST=localhost export AUTH_JWT_PUBKEY_PATH=/path/to/pubkey
-
Activate the virtual environment by running:
poetry shell
-
Run
karp-cli db up
to initialize database -
Run
make serve
ormake serve-w-reload
to start development serveror
poetry shell
and thenuvicorn asgi:app
-
To setup Elasticsearch, download Elasticsearch 6.x and start it
-
Install elasticsearch python libs for the right version
- If you use Elasticsearch 6.x, run
source <VENV_NAME>/bin/activate
andpip install -e .[elasticsearch6]
- If you use Elasticsearch 6.x, run
-
Add environment variables
export ES_ENABLED=true
export ELASTICSEARCH_HOST=localhost:9200
export SEARCH_CONTEXT=es6_search_service
Create test resources
poetry shell
and then:karp-cli entry-repo create karp/tests/data/config/places.json
karp-cli resource create karp/tests/data/config/places.json
karp-cli entries add places tests/data/places.jsonl
- Do the same for
municipalities
karp-cli resource publish places
karp-cli resource publish municipalities
Pre-processing data before publishing
** TODO: review this **
Can be used to have less downtime, because sometimes the preprocessing may
be faster on another machine than the machine that talks to Elasticsearch.
Do create
and import
on both machines, with the same data. Use
machine 1 to preprocess and use result on machine 2.
-
Create resource and import data as usual.
-
Run
karp-cli preprocess --resource_id places --version 2 --filename places_preprocessed
places_preprocessed
will contain a pickled dataset containing everything that is needed -
Run
karp-cli publish_preprocessed --resource_id places --version 2 --data places_preprocessed
-
Alternatively run
karp-cli reindex_preprocessed --resource_id places --data places_preprocessed
, if the resource was already published.
Technologies
Python
- Poetry >= 3.10
- FastAPI
- SQLAlchemy
- Typer
- Elasticsearch
- Elasticsearch DSL
Databases
- MariaDB
- Elasticsearch
Development
Version handling
Version can be bumped with bump2version
.
Usage:
- Increase patch number
a.b.X => a.b.(X+1)
:make bumpversion
orbumpversion patch
- Increase minor number
a.X.c => a.(X+1).0
:make bumpversion-minor
orbumpversion minor
- Increase major number
X.b.c => (X+1).0.0
:make bumpversion-major
orbumpversion major
- To custom version
a.b.c => X.Y.Z
:bumpversion --new-version X.Y.Z
bumpversion
is configured in .bumpversion.cfg
.
The version is changed in the following files:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file karp_backend-6.1.12.tar.gz
.
File metadata
- Download URL: karp_backend-6.1.12.tar.gz
- Upload date:
- Size: 201.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ed523c1ad1f9b7b85b6eb5cf0a31136af03bf672df7da9fcb16481d0ede89a9 |
|
MD5 | e2ec006fe3c95f6f1815209125cbf24b |
|
BLAKE2b-256 | 744777d45f3f80019f1ed72327044bb023819dd2d54de325decc43683c21c7bd |
File details
Details for the file karp_backend-6.1.12-py3-none-any.whl
.
File metadata
- Download URL: karp_backend-6.1.12-py3-none-any.whl
- Upload date:
- Size: 280.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | adbecc93721b8a8dee29ed349e9b612220e49b407384685b751d3890bf6f8b9f |
|
MD5 | 0acedbebae23a1e8522ef119b8b7618f |
|
BLAKE2b-256 | 9d50128435b10e5619e94d5aa2fc05d16c0614ccf70dc8717b71b81a7c6da5ac |