🗂️ LlamaIndex 🦙
LlamaIndex (GPT Index) is a data framework for your LLM application.
PyPI:
- LlamaIndex: https://pypi.org/project/llama-index/.
- GPT Index (duplicate): https://pypi.org/project/gpt-index/.
LlamaIndex.TS (Typescript/Javascript): https://github.com/run-llama/LlamaIndexTS.
Documentation: https://gpt-index.readthedocs.io/.
Twitter: https://twitter.com/llama_index.
Discord: https://discord.gg/dGcwcsnxhU.
Ecosystem
- LlamaHub (community library of data loaders): https://llamahub.ai
- LlamaLab (cutting-edge AGI projects using LlamaIndex): https://github.com/run-llama/llama-lab
🚀 Overview
NOTE: This README is not updated as frequently as the documentation. Please check out the documentation above for the latest updates!
Context
- LLMs are a phenomenal piece of technology for knowledge generation and reasoning. They are pre-trained on large amounts of publicly available data.
- How do we best augment LLMs with our own private data?
We need a comprehensive toolkit to help perform this data augmentation for LLMs.
Proposed Solution
That's where LlamaIndex comes in. LlamaIndex is a "data framework" to help you build LLM apps. It provides the following tools:
- Offers data connectors to ingest your existing data sources and data formats (APIs, PDFs, docs, SQL, etc.)
- Provides ways to structure your data (indices, graphs) so that this data can be easily used with LLMs.
- Provides an advanced retrieval/query interface over your data: Feed in any LLM input prompt, get back retrieved context and knowledge-augmented output.
- Allows easy integrations with your outer application framework (e.g. with LangChain, Flask, Docker, ChatGPT, anything else).
LlamaIndex provides tools for both beginner users and advanced users. Our high-level API allows beginner users to use LlamaIndex to ingest and query their data in 5 lines of code. Our lower-level APIs allow advanced users to customize and extend any module (data connectors, indices, retrievers, query engines, reranking modules), to fit their needs.
💡 Contributing
Interested in contributing? See our Contribution Guide for more details.
📄 Documentation
Full documentation can be found here: https://gpt-index.readthedocs.io/en/latest/.
Please check it out for the most up-to-date tutorials, how-to guides, references, and other resources!
💻 Example Usage
pip install llama-index
Examples are in the examples folder. Indices are in the indices folder (see list of indices below).
To build a simple vector store index:
import os
os.environ["OPENAI_API_KEY"] = 'YOUR_OPENAI_API_KEY'
from llama_index import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(documents)
To query:
query_engine = index.as_query_engine()
query_engine.query("<question_text>?")
By default, data is stored in-memory.
To persist to disk (under ./storage):
index.storage_context.persist()
To reload from disk:
from llama_index import StorageContext, load_index_from_storage
# rebuild storage context
storage_context = StorageContext.from_defaults(persist_dir='./storage')
# load index
index = load_index_from_storage(storage_context)
🔧 Dependencies
The main third-party package requirements are tiktoken, openai, and langchain.
All requirements should be contained within the setup.py file. To run the package locally without building the wheel, simply run pip install -r requirements.txt.
📖 Citation
Reference to cite if you use LlamaIndex in a paper:
@software{Liu_LlamaIndex_2022,
author = {Liu, Jerry},
doi = {10.5281/zenodo.1234},
month = {11},
title = {{LlamaIndex}},
url = {https://github.com/jerryjliu/llama_index},
year = {2022}
}
2 maintainers
Release files for gpt-index 0.8.42
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| gpt_index-0.8.42.tar.gz | 566.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| gpt_index-0.8.42-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.4 MB
Release files / gpt_index-0.8.42.tar.gz
| Download URL | gpt_index-0.8.42.tar.gz |
|---|---|
| Size | 566.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
93be6b0be13ce590b7a48ddf9f250989e0175351e42c8a0bf86026831542fc4f
|
|
BLAKE2b-256 checksum How to use checksums |
766b9f8f065684d7d87d5dee438474ededb3860b87de07f9fffc64f4df5bc532
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / gpt_index-0.8.42-py3-none-any.whl
| Download URL | gpt_index-0.8.42-py3-none-any.whl |
|---|---|
| Size | 877.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a55f783c1ef16337cb8fbaea72ee32c24cd49b085b32cacbfb96262d54a0faa6
|
|
BLAKE2b-256 checksum How to use checksums |
800ec4343de79214aec067a7174fa0e495f9ff54f5beeb8b98680831b541fd3c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
