nano-crucible 2.0.0

Python Client for Crucible - National Archive for NSRC Observations

nano-crucible : National Archive for NSRC Observations

A Python client library and CLI tool for Crucible - the Molecular Foundry's data lakehouse for scientific research. Crucible stores experimental and synthetic data from DOE Nanoscale Science Research Centers (NSRCs), along with comprehensive metadata about samples, projects, instruments, and users.

🔬 What is Crucible?

Crucible is the centralized data infrastructure for the Molecular Foundry and other DOE Nanoscale Science Research Centers, providing:

• Unified data storage for experimental and synthetic data
• Rich metadata capture for to associate to datasets
• Sample provenance tracking with parent-child relationships

✨ Features

🐍 Python API

• Dataset Management: Create, query, update, and download datasets
• Sample Tracking: Manage samples with hierarchical relationships and provenance
• Metadata: Store and retrieve scientific metadata and experimental parameters
• Linking: Connect datasets, samples, and create relationships programmatically

🖥️ Command-Line Interface

• crucible config: One-time setup and configuration management
• crucible upload: Upload datasets with automatic parsing and metadata extraction
• crucible open: Open resources in the Crucible Web Explorer with one command
• crucible link: Create relationships between datasets and samples

📦 Installation

From PyPI (Recommended)

pip install nano-crucible

From GitHub (Latest Development)

pip install git+https://github.com/MolecularFoundryCrucible/nano-crucible

For Development

git clone https://github.com/MolecularFoundryCrucible/nano-crucible.git
cd nano-crucible
pip install -e .

🚀 Quick Start

Python API

Creating and Uploading Datasets

from crucible import CrucibleClient, BaseDataset
from crucible.config import config

# Get client
client = config.client

# Method 1: Create dataset (no files)
dataset = client.create_new_dataset(
    unique_id = "my-unique-dataset-id",  # Optional, auto-generated if None
    dataset_name="High-Temperature Synthesis",
    measurement="XRD",
    project_id="nanomaterials-2024",
    public=False,
    scientific_metadata={
        "temperature_C": 800,
        "pressure_bar": 1.0,
        "duration_hours": 12,
        "atmosphere": "nitrogen"
    },
    keywords=["synthesis", "high-temperature", "oxides"]
)

# Method 2: Upload dataset with files using BaseDataset
dataset = BaseDataset(
    unique_id="my-unique-dataset-id",  # Optional, auto-generated if None
    dataset_name="Electron Microscopy Images",
    measurement="TEM",
    project_id="nanomaterials-2024",
    public=False,
    instrument_name="TEM-2100",
    data_format="TIFF",
    file_to_upload="/path/to/image.tiff"
)

# Upload with metadata and files
result = client.create_new_dataset_from_files(
    dataset=dataset,
    scientific_metadata={
        "magnification": 50000,
        "voltage_kV": 200,
        "spot_size": 3
    },
    keywords=["TEM", "imaging", "nanoparticles"],
    files_to_upload=["/path/to/image.tiff", "/path/to/calibration.txt"],
    thumbnail="/path/to/thumbnail.png",  # Optional
    ingestor='ApiUploadIngestor',
    wait_for_ingestion_response=True
)

print(f"Dataset created: {result['created_record']['unique_id']}")

Linking Resources

# Link two datasets
client.link_datasets("parent-dataset", "child-dataset")
# Link two samples
client.link_samples("parent-sample", "child-sample")
# Link sample to dataset
client.add_sample_to_dataset("dataset-id", "sample-id")

Command-Line Interface

1. Initial Configuration

# One-time setup
crucible config init

# View your configuration
crucible config show

# Update settings
crucible config set api_key YOUR_NEW_KEY

Get your API key at: https://crucible.lbl.gov/api/v1/user_apikey

2. Upload Data with Parsers

# Upload with generic dataset
crucible upload -i data.txt -pid my-project \
    --metadata '{"temperature=300,pressure=1.0"}' \
    --keywords "experiment,test"

# Upload specific dataset (e.g. LAMMPS simulation)
# Works only if the parser exists
crucible upload -i simulation.lmp -t lammps -pid my-project

3. Link Resources

# Link two datasets
crucible link -p parent_dataset_id -c child_dataset_id
# Link two samples
crucible link -p parent_sample_id -c child_sample_id
# Link sample to dataset
crucible link -d dataset_id -s sample_id

4. Open in Browser

# Open the Crucible Web Explorer
crucible open
# Open to a specific resource
crucible open RESOURCE_MFID

📖 Documentation

• CLI Documentation: See cli/README.md
• Parser Documentation: See parsers/README.md
• API Reference: Coming soon

🤝 Contributing

We welcome contributions! Areas where you can help:

• New parsers for additional data formats
• Bug reports and feature requests
• Documentation improvements
• Example notebooks and tutorials

📄 License

This project is licensed under the BSD-3-Clause License - see the LICENSE file for details.

🔗 Links

• Crucible API: https://crucible.lbl.gov/api/v1
• Crucible Web Interface: https://crucible-graph-explorer.run.app

💬 Support

For issues, questions, or feature requests:

• GitHub Issues: https://github.com/MolecularFoundryCrucible/nano-crucible/issues
• Email: [e-mail], [e-mail], [e-mail]

---

nano-crucible is developed and maintained by the Data Group at the Molecular Foundry at Lawrence Berkeley National Laboratory.