createllm

A Python package that enables users to create and train their own Language Learning Models (LLMs) from scratch using custom datasets. This package provides a simplified approach to building, training, and deploying custom language models tailored to specific domains or use cases.

🎯 Core Purpose

createllm allows you to:

• Train custom language models on your specific text data
• Create domain-specific LLMs for specialized applications
• Build and experiment with different model architectures
• Deploy trained models for text generation tasks

✨ Key Features

• 🔨 Build LLMs from scratch using your own text data
• 🚀 Multi-threaded training for faster model development
• 📊 Real-time training progress tracking
• 🎛️ Configurable model architecture
• 💾 Easy model saving and loading
• 🎯 Custom text generation capabilities
• 📈 Built-in performance monitoring

📋 Requirements

pip install torch torchvision tqdm dill

🚀 Quick Start Guide

1. Prepare Your Training Data

Place your training text in a file. The model learns from this text to generate similar content.

my_training_data.txt
├── Your custom text
├── Can be articles
├── Documentation
└── Any text content you want the model to learn from

2. Train Your Custom LLM

from createllm import ModelConfig, GPTTrainer, TextFileProcessor

# Initialize model configuration
config = ModelConfig(
    vocab_size=None,  # Will be automatically set based on your data
    n_embd=384,      # Embedding dimension
    block_size=256,  # Context window size
    n_layer=4,       # Number of transformer layers
    n_head=4        # Number of attention heads
)

# Create trainer instance
trainer = GPTTrainer(
    text_file="path/to/my_training_data.txt",
    learning_rate=3e-4,
    batch_size=64,
    max_iters=5000,
    eval_interval=500,
    saved_path="path/to/save/model"
)

# Start training
trainer.trainer()  # This will automatically process text and train the model

3. Use Your Trained Model

from createllm import LLMModel

# Load your trained model
model = LLMModel("path/to/saved/model")

# Generate text
generated_text = model.generate("Your prompt text")
print(generated_text)

📝 Example Use Cases

1. Domain-Specific Documentation Generator

# Train on technical documentation
trainer = GPTTrainer(
    text_file="technical_docs.txt",
    saved_path="tech_docs_model"
)
trainer.trainer()

2. Custom Writing Style Model

# Train on specific author's works
trainer = GPTTrainer(
    text_file="author_works.txt",
    saved_path="author_style_model"
)
trainer.trainer()

3. Specialized Content Generator

# Train on specific content type
trainer = GPTTrainer(
    text_file="specialized_content.txt",
    saved_path="content_model"
)
trainer.trainer()

⚙️ Model Configuration Options

Customize your model architecture based on your needs:

config = ModelConfig(
    n_embd=384,     # Larger for more complex patterns
    block_size=256, # Larger for longer context
    n_layer=8,      # More layers for deeper understanding
    n_head=8,       # More heads for better pattern recognition
    dropout=0.2     # Adjust for overfitting prevention
)

💡 Training Tips

1. Data Quality

   • Clean your training data
   • Remove irrelevant content
   • Ensure consistent formatting

2. Resource Management

   trainer = GPTTrainer(
       batch_size=32,     # Reduce if running out of memory
       max_iters=5000,    # Increase for better learning
       eval_interval=500  # Monitor training progress
   )
   

3. Model Size vs Performance

   • Smaller models (n_layer=4, n_head=4): Faster training, less complex patterns
   • Larger models (n_layer=8+, n_head=8+): Better understanding, more resource intensive

🔍 Monitoring Training

The training process provides real-time feedback:

step 0: train loss 4.1675, val loss 4.1681
step 500: train loss 2.4721, val loss 2.4759
step 1000: train loss 1.9842, val loss 1.9873
step 1500: train loss 1.1422, val loss 1.1422
...

📁 Saved Model Structure

saved_model/
├── model.pt           # Model weights
├── encoder.pickle    # Text encoder
├── decoder.pickle    # Text decoder
└── config.json      # Model configuration

⚠️ Limitations

• Training requires significant computational resources
• Model quality depends on training data quality
• Larger models require more training time and resources

🤝 Contributing

Contributions are welcome! Please feel free to submit pull requests.

📫 Support

For issues and questions, please open an issue in the GitHub repository.

📄 License

This project is licensed under the MIT License.

🙏 Acknowledgments

Based on the GPT architecture with modifications for custom training and ease of use.