medplexity 0.1.6

medplexity helps with evaluation of LLMs for medical use-cases.

Medplexity

Medplexity explorer • Frontend GitHub repository • Substack

Medplexity is a python library to help with evaluation of LLMs for medical applications.

It is designed to help with the following tasks:

• Evaluating performance of LLMs on existing medical datasets and benchmarks. E.g. MedQA, PubMedQA, etc.
• Comparing performance of different prompts, models, and architectures.
• Exporting results of evaluation for visualisation and further analysis.

The goal is to help answer questions like "How much better would GPT-4 perform given a vector database to load certain resources?".

🔧 Quick install

pip install medplexity

📖 Documentation

Documentation can be found here.

Example

See our "Getting Started" notebook for a full example with MedMCQA dataset.

Contributions

Contributions are welcome! Check out the todos below, and feel free to open a pull request. Remember to install pre-commit to be compliant with our standards:

pre-commit install

Feel free to raise any questions on Discord

Todos

Some initial todos include:

• Multiple-Choice datasets
  • Add MedMCQA dataset
  • Add PubMedQA dataset
  • Add MedQA dataset
  • Add MMLU dataset
• Long-form question answering datasets
  • Add HealthSearchQA dataset
  • Add MedicationQA dataset
  • Add LiveQA dataset
• Explore datasets for multi-modality, specifically vision tasks for GPT-4V.
• LLMs
  • Wrapper for OpenAI
  • Wrapper for deepinfra
  • Wrapper for Google PALM
  • Wrapper for HuggingFace text-gen
• Jupyter notebook quickstart
• Example with langchain integration
• Visualisation of results
  • Add export of evaluations
  • Frontend for exploring exported results

Explorer

In addition to the library, we are also building a web app to explore the results of evaluations. The explorer is available at medplexityai.com. It's also open-sourced, see the frontend repository.

📜 License

Medplexity is licensed under the MIT License. See the LICENSE file for more details.