kodosumi 0.9.3

kodosumi framework to execute and orchestrate agentic services safe and at scale

kodosumi

kodosumi is the runtime environment to manage and execute agentic services at scale. The system is based on Ray - a distributed computing framework - and a combination of litestar and fastapi to deliver agentic services to users or other agents. Similar to Ray, kodosumi follows a Python first agenda.

kodosumi is one component of a larger eco system with masumi and sokosumi.

introduction

kodosumi consists of three main building blocks. First, a Ray cluster to execute agentic services at scale. kodosumi builds on top of Ray and actively manages the lifecycle and events of service executions from starting to finished or error. No matter you name your code an application, flow, service or script: The third building block is your application which runs on top of kodosumi.

The following architecture shows the relation between the three building blocks: 1) your service on top of 2) kodosumi which operates 3) a distributed compute cluster with Ray secure and at scale.

You build and deploy your Flow by providing an endpoint (HTTP route) and an entrypoint (Python callable) to kodosumi (left bottom box in the diagram). kodosumi delivers features for access control, flow control and manages flow execution with Ray head node and worker nodes. kodosumi spooler gathers flow execution results and outputs into the event stream.

Deep-dive into endpoints and how these translate into entrypoints of flows which operationalize the business logic of agentic services or agents in the broader sense.

If you need further background information read why kodosumi

installation

The following quick guide

1. installs kodosumi and all prerequisites along the kodosumi example flows
2. starts Ray and kodosumi on your localhost
3. deploys example flows

This installation has been tested with versions ray==2.46.0 and python==3.12.6.

If you want to skip the examples then continue with the kodosumi development workflow and start implementing your custom agentic service with the kodosumi framework.

install and run examples

STEP 1 - clone and install kodosumi-examples

Clone and install the kodosumi-examples into your Python Virtual Environment. The kodosumi and Ray packages are automatically installed as a dependency.

git clone https://github.com/masumi-network/kodosumi-examples.git
cd ./kodosumi-examples
git checkout dev
pip install .

Since some of the examples utilize additional frameworks like CrewAI and langchain the installation of the kodosumi examples takes a while. All dependencies of the examples are installed. Please note that these dependencies are managed by Ray in production. See deployment.

STEP 2 - prepare runtime environment

You need an OpenAI API key to run some of the examples. Specify the API key in .env.

# ./env
OPENAI_API_KEY=<-- enter-your-key -->

STEP 3 - start ray

Start Ray head node on your localhost. Load environment variables with dotenv before starting ray:

dotenv run -- ray start --head

Check ray status and visit ray dashboard at http://localhost:8265. For more information about ray visit ray's documentation.

STEP 4 - deploy

You have various options to deploy and run the example services. kodosumi-examples repository ships with the following examples in agentic.examples:

• hymn - creates a hymn based on a given topic. The example demonstrates the use of CrewAI and OpenAI
• prime - calculates prime number gaps. Distributes the tasks across the Ray cluster and demonstrates performance benefits.
• throughput - real-time experience of different event stream pressures with parameterized BPMs (beats per minute).
• form - demonstrates form elements supported by kodosumi.

You can run any of these examples. The next steps focus on agentic.examples.hymn.

alternative 1: run with uvicorn

You can launch each example service as a python module.

uvicorn agentic.examples.hymn.app:app --port 8011

This starts a uvicorn (Asynchronous Server Gateway Interface) server at http://localhost:8011. All HTTP endpoints of app are available at URL http://localhost:8011/openapi.json. Launch another terminal session, source the Python Virtual Environment and register this URL with kodosumi panel:

koco start --register http://localhost:8011/openapi.json

Visit kodosumi admin panel at http://localhost:3370. The default user is defined in config.py and reads name=admin and password=admin. Launch the Hymn Creator from the service screen and revisit results at the timeline screen.

You can start another service prime in a new terminal with

uvicorn agentic.examples.prime.app:app --port 8012

Register this service with kodosumi panel config with both service endpoints

http://localhost:8011/openapi.json
http://localhost:8012/openapi.json

You can specify multiple registers at koco start

koco start --register http://localhost:8011/openapi.json --register http://localhost:8012/openapi.json

Running your services as standalone uvicorn applicaitons is best practice to facilitate debugging.

alternative 2: deploy and run with Ray serve

Run your services as Ray serve deployments. This is the preferred approach to deploy services in production. The downside of this approach is that you have to use remote debugging tools and attach to session breakpoints for debugging (see Using the Ray Debugger).

Ray Serve is built on top of Ray, so it easily scales to many machines and offers flexible scheduling support such as fractional GPUs so you can share resources and serve many applications at low cost.

With Ray serve you either run or deploy your services. Instead of the mechanics with uvicorn which refers the app application object, Ray serve demands the bound fast_app object. To test and improve your service run it with

serve run agentic.examples.hymn.app:fast_app

In contrast to the previous command a serve deploy command is used to deploy your Serve application to the Ray cluster. It sends a deploy request to the cluster and the application is deployed asynchronously. This command is typically used for deploying applications in a production environment.

serve deploy agentic.examples.hymn.app:fast_app

Using Ray serve run or deploy the --register must connect to Ray's proxy URL /-/routes. With serve run or deploy the port defaults to 8000 and you start koco start with the Ray serve endpoint http://localhost:8000/-/routes.

koco start --register http://localhost:8000/-/routes

multi-service setup with Serve config files

serve run and serve deploy feature single services. Running multiple uvicorn services is possible but soon gets dity and quirky. For multi-service deployments use Ray serve config files.

In directory ./data/config create a file config.yaml with serve's overarching configuration, for example

proxy_location: EveryNode

http_options:
  host: 127.0.0.1
  port: 8001

grpc_options:
  port: 9001
  grpc_servicer_functions: []

logging_config:
  encoding: TEXT
  log_level: DEBUG
  logs_dir: null
  enable_access_log: true

Alongside this file config.yaml create service configuration files. For each service deployment create a dedicated configuration file:

hymn.yaml

name: hymn
route_prefix: /hymn
import_path: agentic.examples.hymn.app:fast_app
runtime_env: 
  pip:
  - crewai
  - crewai_tools
  env_vars:
    OTEL_SDK_DISABLED: "true"
    OPENAI_API_KEY: <-- enter-you-API-key -->

prime.yaml

name: prime
route_prefix: /prime
import_path: agentic.examples.prime.app:fast_app

throughput.yaml

name: throughput
route_prefix: /throughput
import_path: agentic.examples.throughput.app:fast_app
runtime_env: 
  pip:
  - lorem-text

Test this deployment set with

koco deploy ./data/config/config.yaml --test

With success, stop ray serve and perform a Ray serve deployment

serve shutdown --yes
koco deploy ./data/config/config.yaml

Restart koco start with the Ray serve endpoint http://localhost:8001/-/routes as configured in config.yaml.

koco start --register http://localhost:8001/-/routes

If one or more Ray serve applications are not yet available when kodosumi starts, you need to refresh the list of registered flows. Visit control screen in the admin panel and click RECONNECT.

Adding and removing deployments is operationalized with config files in ./data/config. All files alongside config.yaml are deployed. You can test your deployment setup with koco deploy ./data/config/config.yaml --test.

Where to get from here?

• continue with kodosumi development workflow
• see the admin panel screenshots
• read about basic concepts and terminology