The Oshepherd guiding the Ollama(s) inference orchestration.
Project description
oshepherd
The Oshepherd guiding the Ollama(s) inference orchestration.
A centralized Flask API service, using Celery (RabbitMQ + Redis) to orchestrate multiple Ollama workers.
Install
pip install oshepherd
Usage
-
Setup RabbitMQ and Redis:
Create instances for free for both: * cloudamqp.com * redislabs.com
-
Setup Flask API Server:
# define configuration env file # use credentials for redis and rabbitmq cp .api.env.template .api.env # start api oshepherd start-api --env-file .api.env
-
Setup Celery/Ollama Worker(s):
# install ollama https://ollama.com/download ollama run mistral # define configuration env file # use credentials for redis and rabbitmq cp .worker.env.template .worker.env # start worker oshepherd start-worker --env-file .worker.env
Words of advice 🚨
This package is in alpha, its architecture and api might change in the near future. Currently this is getting tested in a closed environment by real users, but haven't been audited, nor tested thorugly. Use it at your own risk.
Disclaimer on Support
As this is an alpha version, support and responses might be limited. We'll do our best to address questions and issues as quickly as possible.
Contribution Guidelines
We welcome contributions! If you find a bug or have suggestions for improvements, please open an issue or submit a pull request.
Conda Support
To run and build locally you can use conda:
conda create -n oshepherd python=3.8
conda activate oshepherd
pip install -r requirements.txt
# install oshepherd
pip install -e .
Tests
Follow usage instructions to start api server and celery worker using a local ollama, and then run the tests:
pytest -s tests/
Reporting Issues
Please report any issues you encounter on the GitHub issues page. Before creating a new issue, take a moment to search through the existing issues to avoid duplicates.
Author
Currently, mnemonica.ai is sponsoring the development of this tool.
License
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file oshepherd-0.0.4.tar.gz
.
File metadata
- Download URL: oshepherd-0.0.4.tar.gz
- Upload date:
- Size: 8.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.8.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd585c916a705d2c0f9c52b85b455bc546e9b71aeb058395332558da2ecb88af |
|
MD5 | fc0dd94297cfa9103ebf1a0e524bbe81 |
|
BLAKE2b-256 | 0c51ff278da9806885c259c8594622900b836bd6b09fb1ccbe0dc4aef01d9ba0 |
File details
Details for the file oshepherd-0.0.4-py3-none-any.whl
.
File metadata
- Download URL: oshepherd-0.0.4-py3-none-any.whl
- Upload date:
- Size: 9.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.8.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4fdbf51c787d68927a0f2bb03042945ae7736afc19d5cdd263a23862364166f8 |
|
MD5 | 0adbd8b757d8d6c636596196229bf1b4 |
|
BLAKE2b-256 | c1d61816ab606a57189920394c77af58183aecc8860f39465021eb73cf0ebf75 |