udata-search-service

A search service for udata. The idea is to have search service separated from the udata MongoDB. The indexation update is made using real-time messages with Kafka.

See the following architecture schema: Udata Search Service architecture schema

Getting started

You can follow this recommended architecture for your code:

$WORKSPACE
├── fs
├── udata
│   ├── ...
│   └── setup.py
│		└── udata.cfg
├── udata-front
│   ├── ...
│   └── setup.py
└── udata-search-service
    ├── ...
    └── pyproject.toml

Clone the repository:

cd $WORKSPACE
git clone git@github.com:opendatateam/udata-search-service.git

Start the different services using docker-compose:

cd udata-search-service
docker-compose up

This will start:

an elasticsearch
a kafka broker
a zookeper
a kafka consumer
a search app

Initialize the elasticsearch indices on setup.

# Locally
udata-search-service init-es

# In the docker context
docker-compose run --entrypoint /bin/bash web -c 'udata-search-service init-es'

You can feed the elasticsearch by publishing messages to Kafka. Using udata, when you modify objects, indexation messages will be sent and will be consumed by the kafka consumer. If you want to reindex your local mongo base in udata, you can run:

cd $WORKSPACE/udata/
source ./venv/bin/activate
udata search index

After a reindexation, you'll need to change the alias by using the following command:

# Locally
udata-search-service set-alias <index-suffix>

# In the docker context
docker-compose run --entrypoint /bin/bash web -c 'udata-search-service set-alias <index-suffix>'

You can query the search service with the search service api, ex: http://localhost:5000/api/1/datasets/?q=toilettes%20à%20rennes

Development

You can create a virtualenv, activate it and install the requirements with the following commands.

python3 -m venv venv
source venv/bin/activate
make deps
make install

You can start the Elasticsearch and Kafak broker and zookeper using the docker compose. You can start the consumer locally with the following command:

udata-search-service consume-kafka

You can start the web search service with the following command:

udata-search-service run

Troubleshooting

If the elasticsearch service exits with an error 137, it is killed due to out of memory error. You should read the following points.
If you are short on RAM, you can limit heap memory by setting ES_JAVA_OPTS=-Xms750m -Xmx750m as environment variable when starting the elasticsearch service.
If you are on MAC and still encounter RAM memory issues, you should increase Docker limit memory to 4GB instead of default 2GB.
If you are on Linux, you may need to double the vm.max_map_count. You can set it with the following command: sysctl -w vm.max_map_count=262144.
If you are on Linux, you may encounter permissions issues. You can either create the volume or change the user to the current user using chown.

Metadata

Release files for udata-search-service 1.0.0.dev127

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for udata-search-service 1.0.0.dev127
File	Size	Uploaded	Provenance
udata-search-service-1.0.0.dev127.tar.gz	39.0 kB	Mar 30, 2022	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for udata-search-service 1.0.0.dev127
File	Interpreter	ABI	Platform	Reset
udata_search_service-1.0.0.dev127-py2.py3-none-any.whl 17.9 kB Mar 30, 2022 Without provenance	Python 3, Python 2	none	any	Details

Total release size: 57.0 kB

Release files / udata-search-service-1.0.0.dev127.tar.gz

Download URL	udata-search-service-1.0.0.dev127.tar.gz
Size	39.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`de8af38838f0ef7b73b9be4352694a196a8998548e0928d3000fb5c72c0525e3`
BLAKE2b-256 checksum How to use checksums	`d6bbe010bb13b75f892eae5219cf9cfaaa2295eb7d86c6966af75df03b534341`
Upload date	Mar 30, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`python-requests/2.27.1`

Release files / udata_search_service-1.0.0.dev127-py2.py3-none-any.whl

Download URL	udata_search_service-1.0.0.dev127-py2.py3-none-any.whl
Size	17.9 kB
Tags	Python 2 Python 3
SHA-256 checksum How to use checksums	`6da25c126c61bdeb5da9fedc922ec4eea5ba56b4f1f3bc038d71654fa92d3733`
BLAKE2b-256 checksum How to use checksums	`126ac806ca1d09cda7669a05a5c19eb902c8606971429d75957659a8f92dca6d`
Upload date	Mar 30, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`python-requests/2.27.1`

Security for udata-search-service 1.0.0.dev127

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.