MicroMole

    MicroMole is a Python package that provides tools to collect and store data from various sources such as local files, databases, website APIs, and more. It offers a unified interface for data ingestion, transformation, and persistence, enabling seamless integration of diverse data streams into your applications or pipelines. With support for multiple collectors and storage backends, MicroMole simplifies data management tasks for developers and data scientists.
    
    ## Features
    
    - **Data Collectors**: Modules for fetching data from local files (CSV, JSON, XML, etc.), databases (SQL, NoSQL), web APIs (REST, GraphQL), and other sources like streams or sensors.
    - **Storage Backends**: Support for storing collected data in files, databases, caches, or cloud storage with configurable options.
    - **Unified API**: Consistent methods for collection, processing, and storage across different sources.
    - **Data Transformation**: Built-in tools for cleaning, filtering, and transforming data during collection.
    - **Configuration-Driven**: Use YAML or JSON configs to define sources, transformations, and destinations.
    - **Error Handling and Logging**: Robust mechanisms for retries, validation, and detailed logging.
    - **Extensible**: Easily add custom collectors or storage adapters.
    - **Cross-Platform Compatibility**: Works on Windows, macOS, and Linux.
    
    ## Installation
    
    You can install MicroMole via pip:
    
    ```bash
    pip install micromole
    ```
    
    Alternatively, clone the repository and install from source:
    
    ```bash
    git clone https://github.com/<USER_OR_ORG>/micromole.git
    cd micromole
    pip install -e .
    ```
    
    ### Requirements
    
    - Python 3.<MIN_VERSION> or higher
    - Dependencies: requests, pandas, sqlalchemy, pyyaml (automatically installed via pip where applicable)
    
    ## Quick Start
    
    Import the module, configure a collector, and fetch data:
    
    ```python
    import micromole
    
    # Load configuration for a source
    config = micromole.load_config('path/to/api.yaml')
    
    # Initialize collector
    collector = micromole.Collector(config)
    
    # Collect data
    data = collector.fetch()
    
    # Store data
    micromole.store(data, 'path/to/storage.db')
    ```
    
    ## Usage
    
    ### Loading Configurations
    
    MicroMole uses YAML files to define data sources and storage. A sample YAML for a web API might look like:
    
    ```yaml
    source_type: api
    url: https://api.example.com/data
    method: GET
    headers:
      Authorization: Bearer <TOKEN>
    params:
      query: value
    transform:
      filter: key == 'active'
    storage:
      type: database
      engine: sqlite
      file: data.db
    ```
    
    Use `micromole.load_config(yaml_path)` to parse and validate the config.
    
    ### Collecting Data
    
    ```python
    # From local file
    file_config = micromole.load_config('file.yaml')
    file_collector = micromole.Collector(file_config)
    file_data = file_collector.fetch()
    
    # From database
    db_config = micromole.load_config('db.yaml')
    db_collector = micromole.Collector(db_config)
    db_data = db_collector.query('SELECT * FROM table')
    ```
    
    ### Storing Data
    
    ```python
    # Store to file
    micromole.store(data, 'output.csv', format='csv')
    
    # Store to database
    micromole.store(data, 'sqlite:///data.db', table='results')
    ```
    
    ### Supported Sources
    
    - Local Files: CSV, JSON, Excel, Parquet, etc.
    - Databases: SQLite, PostgreSQL, MySQL, MongoDB, etc.
    - Web APIs: RESTful services, with authentication support.
    - Others: Streams, FTP, cloud APIs (configurable via extensions).
    
    ## Examples
    
    ### Example 1: Collecting from API and Storing to Database
    
    ```python
    import micromole
    
    config = micromole.load_config('weather_api.yaml')
    collector = micromole.Collector(config)
    weather_data = collector.fetch()
    
    # Transform data
    transformed = micromole.transform(weather_data, filter=lambda x: x['temp'] > 20)
    
    # Store
    micromole.store(transformed, 'weather.db', table='daily')
    ```
    
    ### Example 2: Batch Collection from Multiple Sources
    
    ```python
    import micromole
    
    configs = ['file1.yaml', 'api2.yaml', 'db3.yaml']
    data_list = []
    
    for cfg in configs:
        config = micromole.load_config(cfg)
        collector = micromole.Collector(config)
        data = collector.fetch()
        data_list.append(data)
    
    # Combine and store
    combined = micromole.merge(data_list)
    micromole.store(combined, 'combined.json', format='json')
    ```
    
    ## Configuration Guide
    
    Each YAML config must include:
    
    - `source_type`: String identifier (e.g., 'file', 'api', 'database')
    - `connection`: Dictionary of parameters (e.g., path, url, credentials)
    - `transform`: Optional list of transformations (filter, map, etc.)
    - `storage`: Optional details for immediate storage after collection
    
    For advanced customization, refer to the [docs/config-reference.md](docs/config-reference.md).
    
    ## Contributing
    
    Contributions are welcome! Please follow these steps:
    
    1. Fork the repository.
    2. Create a feature branch (`git checkout -b feature/<FEATURE_NAME>`).
    3. Commit your changes (`git commit -am 'Add some feature'`).
    4. Push to the branch (`git push origin feature/<FEATURE_NAME>`).
    5. Open a Pull Request.
    
    See [CONTRIBUTING.md](CONTRIBUTING.md) for more details.
    
    ## License
    
    This project is licensed under the <LICENSE_TYPE> License - see the [LICENSE](LICENSE) file for details.
    
    ## Acknowledgments
    
    - Built with inspiration from open-source data ingestion communities.
    - Thanks to contributors of underlying libraries like requests, pandas, sqlalchemy.