Cuzco 0.1.dev1

No project description provided

Cuzco : An Emperor With and Without Clothes.

Accelerate your AI development. No inference credits? No problem.

What the heck is this thing?..

Cuzco is a Python package that let's you...

• Treat Huggingface's and Ollama's inference endpoint as a single endpoint.
• Declaratively summon language models.
• Automatically fall back to local inference once you've exhausted your Huggingface inference credits.

Maximal System Requirements

• Docker Engine ( see the 'example-docker' folder for a potential image build. )
• A valid Huggingface token with read permissions.
• As much freekin compute as possible if you plan on running backup inference.

Bare Naked Requirements

• Python >= 3.12
• Ollama ( See official installation https://ollama.com/download )
• As much freekin compute as possible.

Usage in Dockerfile

ENTRYPOINT[ "python", "-m", "cuzco"]

or

ENTRYPOINT[ "python", "-m", "cuzco", "--entrypoint", "your_optional_plugin_with_cool_custom_tools_and_or_interface" ]

Extending Cuzco for Your Project.

With nearly any language model a pip install away, it's time to focus on tool building. In your pyproject.toml, you can define new entrypoints under [project.entry-points.user_defined]. See Cuzco's pyproject.toml and src/example_plugin for guidance.

LanguageModelAgentGenerator the secret sauce.

The LanguageModelAgentGenerator class is a Generator that continually updates your inference endpoint. When building plugins, customize it's tools or subclass it to your liking.

interfaces.py

interfaces.py contains various subclasses of the GradioUI class. By default CuzcoApp is the interface implemented when launch is called. When building plugins, customize these classes to your liking.

config.toml Variables

defaults

• GRADIO_PORT = 7860
• LANGUAGE_MODEL_FLAVOR = "" # Must begin with '-' if defined.
• HF_ACCESS_TOKEN = ""
• OLLAMA_PORT = 11434

protected defaults

• GRADIO_SERVER = "0.0.0.0"
• OLLAMA_START = ["ollama", "serve"]
• OLLAMA_FETCH = ["ollama", "pull"]
• OLLAMA_HOST = "http://localhost"

Enviromental Variables and Overrides

• GRADIO_PORT : Optional[int]
• HF_ACCESS_TOKEN : str
• LANGUAGE_MODEL_PROVIDER : str,
• LANGUAGE_MODEL_VERSION : float
• LANGUAGE_MODEL_PARAMETERS : str , suffix must be a 'b'. When using Qwen, the number of parameters ideally should be 32 billion and no less than 7 billion. Use can use 1.5 billion parameters to test Cuzco is working as intended.
• LANGUAGE_MODEL_FLAVOR : Optional[str] , prefix must be a '-' such as '-coder'.
• OLLAMA_PORT : Optional[int]

Gotchas

• Not all models are compatible with both Huggingface and Ollama. If you require Huggingface inference and back-up inference, double check https://ollama.com/library and https://huggingface.co/models for model cross-availability.

P.S.

I wish to convey my deep gratitude to the folks at Docker, Huggingface and Meta. You made this project possible.