TinyAGI 0.0.1

A modular package for interacting with llama.cpp models via a server or CLI, supporting LoRA and JSON configurations.

TinyAGI

A modular Python package that provides both a Flask API server and a command-line interface (CLI) for interacting with llama.cpp models, including support for LoRA addons and configuration via JSON files.

Features

• Flask API Server: Serve models via HTTP endpoints.
• Command-Line Interface (CLI): Interact with models directly from the terminal.
• LoRA Support: Utilize LoRA adapters with your models.
• JSON Configuration: Manage settings and parameters via JSON files.
• Modular Design: Use components independently in your projects.

Installation

Install from PyPI:

pip install TinyAGI

Usage

Configuration

Create a config.json file in your working directory:

{
  "model_path": "path/to/your/model.bin",
  "lora_path": "path/to/your/lora_adapter.bin",
  "model_params": {
    "n_ctx": 2048,
    "n_gpu_layers": 20
  },
  "inference_params": {
    "temperature": 0.7,
    "top_p": 0.95,
    "max_tokens": 512
  }
}

Running the Server

Start the Flask API server:

tinyagi-server

The server will be accessible at http://localhost:5000.

Using the CLI

Generate Text from a Prompt

tinyagi-cli generate --prompt "Why is the sky blue?"

Stream Output

tinyagi-cli generate --prompt "Tell me a story." --stream

Generate Embeddings

tinyagi-cli embed --input "Text to embed"

Reload the Model Configuration

tinyagi-cli reload --config new_config.json

API Endpoints

• POST /chat: Chat with the model using messages.
• POST /generate: Generate text from a prompt.
• POST /embed: Generate embeddings.
• POST /reload: Reload the model configuration.
• GET /config: Get the current configuration.

Example: Chat Endpoint

Request:

curl -X POST http://localhost:5000/chat \
  -H 'Content-Type: application/json' \
  -d '{
        "messages": [
            {"role": "user", "content": "Hello! Who are you?"}
        ]
    }'

Response:

{
  "response": "Assistant: I am an AI language model developed to assist with your questions."
}

Contributing

Contributions are welcome! Please open an issue or submit a pull request on GitHub.

License

This project is licensed under the MIT License.

Contact

For any inquiries or support, please contact SullyGreene.