Skip to main content

Compress your prompt and context to let LLMs deal with 2x more content.

Project description

Logo of Selective Context

Selective Context for Large Language Models

Selective Context is a novel approach for optimizing the usage of context in Large Language Models (LLMs) by filtering out less informative content. This approach allows LLMs to process long documents and maintain extended conversations more efficiently without compromising their performance on various tasks.

This repository contains the code and data for the paper: Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering.

Updates!!

Try our demo on Huggingface Space.

Key Features

  • Efficient Context Management: Selective Context maximizes the utility of fixed context length in LLMs, allowing them to process long documents and extended conversations more efficiently.
  • Informativeness Evaluation: Our method employs a base language model to compute self-information for lexical units (sentences, phrases, or tokens) in a context and use it to evaluate their informativeness.
  • Extensive Evaluation: We provide extensive evaluations of Selective Context on three data sources (arxiv papers, BBC news articles, and conversation transcripts) and four different NLP tasks (summarization, question answering, original context reconstruction, and conversation).

Getting Started

To get started, follow these steps:

  1. Install selective-context via Pypi:

    pip install selective-context
    
  2. Import SelectiveContext:

    from selective_context import SelectiveContext
    
  3. Compress your prompt and context:

    sc = SelectiveContext(model_type='gpt2', lang='en')
    context, reduced_content = sc(text)
    
  4. You can also adjust the reduce ratio:

    context, reduced_content = sc(text, reduce_ratio = 0.5)
    
  5. If you prefer to try with web interface, try our streamlit app:

    streamlit run app/app.py
    

    Or directly visit our Space on Hugging Face Hub.

Code Structure

  • selective_context.py: A demo for performing context reduction using Selective Context.
  • context_manager.py: The main module for managing context and implementing the Selective Context algorithm.
  • main.py: The main script for running experiments and evaluating the effectiveness of Selective Context.
  • qa_manager.py: A helper module for managing question answering tasks during the experiments.

Experiments

To reproduce the experiments from the paper, run the following command:

python main.py

This will run the experiments on arxiv papers, BBC news articles, and conversation transcripts with four different NLP tasks: summarization, question answering, original context reconstruction, and conversation.

Citation

If you find this repository helpful or use our method in your research, please consider citing our paper:

@misc{li2023unlocking,
      title={Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering}, 
      author={Yucheng Li},
      year={2023},
      eprint={2304.12102},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

This project is licensed under the MIT License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

selective-context-0.1.3.tar.gz (6.5 kB view details)

Uploaded Source

Built Distribution

selective_context-0.1.3-py3-none-any.whl (6.5 kB view details)

Uploaded Python 3

File details

Details for the file selective-context-0.1.3.tar.gz.

File metadata

  • Download URL: selective-context-0.1.3.tar.gz
  • Upload date:
  • Size: 6.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.8.16

File hashes

Hashes for selective-context-0.1.3.tar.gz
Algorithm Hash digest
SHA256 5571da117dd6dc50969da8f4cc4ec39c3cfea3a0fdaa4df39fa039a85cbf1789
MD5 7e8d5b9c8f9cf4d7bc1c89f37f994cfa
BLAKE2b-256 86eeb61552ff31b774027ef3e0bb8c1ac1423cee7d6afb2432cf20a27c3b9862

See more details on using hashes here.

File details

Details for the file selective_context-0.1.3-py3-none-any.whl.

File metadata

File hashes

Hashes for selective_context-0.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 25c80655d5e06b37beecfdb4d7ba777f7eaaa0b5922d3b9cda83f42dc1aa6589
MD5 8a1400a0ecfccf232a524d80aa281b70
BLAKE2b-256 9c8e16757100c091cd0575a1bbfbbbf2621c0f116c8acddf5a42e04c07210072

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page