Table remodeler

The table remodeler provides a flexible, operation-based framework for transforming tabular data files through JSON-configurable pipelines. Originally extracted from the hed-python remodeling tools, this package operates as a standalone tool while maintaining compatibility with HED (Hierarchical Event Descriptors) annotations via the hedtools dependency.

Key features:

• Operation-based architecture for reproducible data transformations
• JSON-configurable pipelines for batch processing
• Support for HED-annotated event files (via hedtools package)
• Built-in backup and restore functionality
• Both programmatic API and command-line interface
• Extensible: create custom operations by extending BaseOp

Installation

pip install table-remodeler

For development with all extras:

git clone https://github.com/hed-standard/table-remodeler.git
cd table-remodeler
pip install -e ".[dev,test,docs,examples]"

Or install specific optional dependencies:

• Development tools: pip install -e ".[dev]"
• Testing: pip install -e ".[test]"
• Documentation: pip install -e ".[docs]"
• Jupyter examples: pip install -e ".[examples]"

Quick start

Python API

from remodeler import Dispatcher

# Define operations
operations = [
    {
        "operation": "remove_columns",
        "parameters": {"column_names": ["unnecessary_col"]}
    },
    {
        "operation": "rename_columns",
        "parameters": {"column_mapping": {"old_name": "new_name"}}
    }
]

# Execute operations
dispatcher = Dispatcher(operations, data_root="/path/to/dataset")
dispatcher.run_operations()

Command line

# Run remodeling operations
run_remodel /path/to/data /path/to/operations.json

# Create backup before modifications
run_remodel_backup /path/to/data --backup-name my_backup

# Restore from backup
run_remodel_restore /path/to/data --backup-name my_backup

Available operations

Data Transformation

• factor_column - Create factor columns from value mappings
• merge_consecutive - Merge consecutive rows with same values
• remap_columns - Remap column values using lookup tables
• remove_columns - Remove specified columns
• remove_rows - Remove rows based on criteria
• rename_columns - Rename columns
• reorder_columns - Reorder columns
• split_rows - Split rows based on criteria

HED-specific operations

• factor_hed_tags - Factor HED tags into separate columns
• factor_hed_type - Factor by HED tag types
• summarize_hed_tags - Summarize HED tag usage
• summarize_hed_type - Summarize HED types
• summarize_hed_validation - Validate HED annotations
• summarize_definitions - Extract HED definitions
• summarize_sidecar_from_events - Generate sidecar from events

Analysis operations

• summarize_column_names - List column names across files
• summarize_column_values - Summarize unique values per column

Documentation

Full API and developer documentation is available at https://www.hedtags.org/table-remodeler.

Users of the table-remodeler should look at the HED remodeling quickstart and HED remodeling tools.

Building Documentation

To build the documentation locally:

1. Install documentation dependencies:

   pip install -e .[docs]
   

2. Build the HTML documentation:

   sphinx-build -b html docs docs/_build/html
   

3. View the documentation by opening docs/_build/html/index.html in your web browser.

Requirements

• Python 3.10+
• pandas
• numpy
• hedtools (for HED-specific operations)

License

MIT License - see LICENSE file for details

Contributing

Contributions are welcome! Please see the development setup in the documentation.

Related Projects

• hed-python - Core HED tools
• hed-schemas - HED schema vocabularies
• hed-specification - HED specification