okik 0.0.3

A Python package to serve python functions, classes, or .py files on a local server or cloud-based environment.

About

Okik is a command-line interface (CLI) that allows users to run various inference services such as LLM, RAG(WIP), or anything in between using various frameworks on any *cloud. With Okik, you can easily run these services directly on any cloud without the hassle of managing your own infra.

Installation

To install Okik, follow these steps:

1. Clone the repository: git clone https://github.com/okikorg/okik.git
2. Navigate to the project directory: cd okik
3. Install Okik using pip: pip install .

Quick Start

To run Okik, simply execute the following command in your terminal: okik

██████  ██   ██ ██ ██   ██
██    ██ ██  ██  ██ ██  ██
██    ██ █████   ██ █████
██    ██ ██  ██  ██ ██  ██
██████  ██   ██ ██ ██   ██



Simplify. Deploy. Scale.
Type 'okik --help' for more commands.

Initialise the project

okik init

Quick Example

Write this in your main.py file:

from okik.endpoints import service, endpoint, app
import asyncio
from typing import Any
from sentence_transformers import SentenceTransformer
import sentence_transformers
from torch.nn.functional import cosine_similarity as cosine
import torch
import random

# your service configuration
@service(
    replicas=1,
    resources={"accelerator": {"type": "A40", "device": "cuda", "count": 1, "memory": 4}},
    backend="okik" # <- provisioning backend is okik
)
class Embedder:
    def __init__(self):
        self.model = SentenceTransformer("paraphrase-MiniLM-L6-v2", cache_folder=".okik/cache")

    @endpoint()
    def embed(self, sentence: str):
        logits = self.model.encode(sentence)
        return logits

    @endpoint()
    def similarity(self, sentence1: str, sentence2: str):
        logits1 = self.model.encode(sentence1, convert_to_tensor=True)
        logits2 = self.model.encode(sentence2, convert_to_tensor=True)
        return cosine(logits1.unsqueeze(0), logits2.unsqueeze(0))

    @endpoint()
    def version(self):
        return sentence_transformers.__version__

    @endpoint(stream=True)
    async def stream_data(self) -> Any:
        async def data_generator():
            for i in range(10):
                yield f"data: {i}\n"
                await asyncio.sleep(1)
        return data_generator()

# Mock LLM Service Example
@service(replicas=1)
class MockLLM:
    def __init__(self):
        pass

    @endpoint(stream=True) # <- streaming response enabled for use cases like chatbot
    async def stream_random_words(self, prompt: str = "Hello"):
        async def word_generator():
            words = ["hello", "world", "fastapi", "stream", "test", "random", "words", "python", "async", "response"]
            for _ in range(10):
                word = random.choice(words)
                yield f"{word}\n"
                await asyncio.sleep(0.4)
        return word_generator()

Verify the routes

# run the okik routes to check all available routes
okik routes

Serving the app

# run the okik run to start the server in production mode
okik server
# or run in dev mode
okik server --dev --reload
#or
okik server -d -r

Test the app

curl -X POST http://0.0.0.0:3000/embedder/version
# or if you like to use httpie then
http POST 0.0.0.0:3000/embedder/version

# or test the stream endpoint
curl -X POST http://0.0.0.0:3000/mockllm/stream_random_words -d '{"prompt": "Hello"}'
# or if you like to use httpie then
http POST 0.0.0.0:3000/mockllm/stream_random_words prompt="hello" --stream

Build the app

okik build -a "your_awesome_app" -t latest

Status

Okik is currently in development so expect sharp edges and bugs. Feel free to contribute to the project by submitting a pull request.