ComProScanner

A comprehensive Python package for extracting composition-property data from scientific articles for building databases

Overview

ComProScanner is a multi-agent framework designed to extract composition-property relationships from scientific articles in materials science. It automates the entire workflow from metadata collection to data extraction, evaluation, and visualization.

Key Features:

• 🏗️ Data extraction from texts, tables and figures.
• 📚 Multi-publisher support (Elsevier, Springer, Wiley, IOP, local PDFs)
• 🤖 Agentic extraction using CrewAI framework
• 🔍 RAG-powered context retrieval for cost-effective automation with accuracy
• 📊 Comprehensive evaluation and visualization tools
• 🎯 Customizable extraction workflows
• 🌐 Knowledge graph generation

Installation

Install from PyPI:

pip install comproscanner

Or install from source:

git clone https://github.com/slimeslab/ComProScanner.git
cd comproscanner
pip install -e .

Quick Start

Here's a complete example extracting piezoelectric coefficient (d₃₃) data:

from comproscanner import ComProScanner

# Initialize scanner
scanner = ComProScanner(main_property_keyword="piezoelectric")

# Collect metadata
scanner.collect_metadata(
    base_queries=["piezoelectric", "piezoelectricity"],
    extra_queries=["ceramics", "applications"]
)

# Process articles
property_keywords = {
    "exact_keywords": ["d33"],
    "substring_keywords": [" d 33 "]
}

scanner.process_articles(
    property_keywords=property_keywords,
    source_list=["elsevier", "springer"]
)

# Extract composition-property data
scanner.extract_composition_property_data(
    main_extraction_keyword="d33"
)

Workflow

The ComProScanner workflow consists of four main stages:

1. Metadata Retrieval - Find relevant scientific articles
2. Article Collection - Extract full-text from various publishers
3. Information Extraction - Use LLM agents to extract structured data
4. Post Processing & Dataset Creation - Evaluate, clean, and visualize results

Documentation

📖 Full documentation is available at slimeslab.github.io/ComProScanner

• Installation Guide
• Quick Start Tutorial
• User Guide
• RAG Configuration

Core Capabilities

Supported Publishers

• Elsevier (via TDM API)
• Springer Nature (via TDM API)
• Wiley (via TDM API)
• IOP Publishing (via SFTP bulk access)
• Local PDFs (any publication)

Data Extraction

• Composition-property relationships
• Material families
• Synthesis methods and precursors
• Characterization techniques
• Synthesis steps

Evaluation Methods

• Semantic Evaluation - Using semantic similarity measures
• Agentic Evaluation - LLM-powered contextual analysis

Visualization

• Data Visualization
• Evaluation Visualization

Requirements

• Python 3.12 or 3.13
• TDM API keys for desired publishers (Elsevier, Springer, Wiley)
• LLM API keys (OpenAI, Anthropic, Google, etc.)
• Optional: Neo4j for knowledge graph visualization

Citation

If you use ComProScanner in your research, please cite the following papers:

@article{roy2026comproscanner,
      title={ComProScanner: a multi-agent based framework for composition-property structured data extraction from scientific literature},
      author={Roy, Aritra and Grisan, Enrico and Buckeridge, John and Gattinoni, Chiara},
      journal={Digital Discovery},
      volume={5},
      number={4},
      pages={1794--1808},
      year={2026},
      publisher={Royal Society of Chemistry},
      doi  ="10.1039/D5DD00521C",
      url  ="https://doi.org/10.1039/D5DD00521C"
}
@misc{roy2026comproscanner_vlm,
      title={Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracy}, 
      author={Aritra Roy and Enrico Grisan and Chiara Gattinoni and John Buckeridge},
      year={2026},
      eprint={2606.00065},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      doi={10.48550/arXiv.2606.00065},
      url={https://arxiv.org/abs/2606.00065}, 
}

Changelog

See the CHANGELOG for details on what has changed in each version.

Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright © 2025-2026 SLIMES Lab

Contact

Author: Aritra Roy

• 🌐 Website: aritraroy.live
• 📧 Email: contact@aritraroy.live
• 🐙 GitHub: @aritraroy24
• 𝕏 Twitter: @aritraroy24

Project Links:

• 📦 PyPI: pypi.org/project/comproscanner
• 📖 Documentation: slimeslab.github.io/ComProScanner
• 🐛 Issues: github.com/slimeslab/ComProScanner/issues

Made with ❤️ by SLIMES Lab