sampling-mining-workflows-dsl 0.0.4

About The Repository

This repository provides a Python internal DSL (Domain Specific Language) to model sampling workflows for MSR (Mining Software Repositories) studies. The DSL offers a structured and intuitive approach to define, execute, and analyze complex data sampling and processing workflows commonly used in software engineering research.

Project Structure

PythonWorkflowDSL/
├── pyproject.toml                                    # Python package configuration
├── README.md                                         # Project documentation
└── src/
    └── sampling_mining_workflows_dsl/               # Main DSL package
        ├── __init__.py                              # Package initialization
        ├── CompleteWorkflow.py                      # Complete workflow implementation
        ├── toolbox.py                               # Utility functions and tools
        ├── Workflow.py                              # Core workflow class
        ├── WorkflowBuilder.py                       # Builder pattern for workflows
        ├── analysis/                                # Statistical analysis modules
        │   └── ...                                  # Chi-square, coverage, distribution analysis
        ├── constraint/                              # Constraint system
        │   └── ...                                  # Boolean constraints and comparators
        ├── element/                                 # Data element management
        │   ├── loader/                             # Data loader implementations
        │   └── writer/                             # Data writer implementations
        ├── exec_visualizer/                        # Execution visualization
        │   └── ...                                  # Local server and visualization tools
        ├── metadata/                               # Metadata system
        │   └── ...                                  # Boolean, date, string metadata types
        ├── operator/                               # Workflow operators
        │   ├── clustering/                         # Clustering operators
        │   ├── selection/                          # Selection operators (filter/sampling)
        │   └── set_algebra/                        # Set algebra operations
        └── test/                                   # Test files and examples
            └── ...                                  # JSON test data and workflow examples

Getting Started

Prerequisites

Before using this DSL, ensure you have Python 3.8+ installed on your system.

Recommended: install and use uv

Starting from scratch

1. uv init
2. uv pip install https://github.com/RomainLefeuvre/samplingMiningWorkflowDSL.git

you can also use the published version on pypi

1. pip install sampling-mining-workflows-dsl

Template Repository

A complete template with examples is available at:

git@github.com:RomainLefeuvre/msr_papers_sampling_workflows.git

Clone the template to get started quickly:

git clone https://github.com/RomainLefeuvre/SamplinMiningWorkflowDSLTemplate.git

Dev environment installation

If you want to contribute to this repository, you can clone and setup locally this project :

1. Clone this repository:

   git clone https://github.com/RomainLefeuvre/SamplingMiningWorflowDSL.git
   cd SamplingMiningWorflowDSL
   

2. (Recommended) Install dependencies with uv:

uv sync

3. Activate the virtual environment created by uv:

source .venv/bin/activate

(back to top)

Usage

Basic Workflow Example

from sampling_mining_workflows_dsl.WorkflowBuilder import WorkflowBuilder
from sampling_mining_workflows_dsl.element.loader.LoaderFactory import LoaderFactory
from sampling_mining_workflows_dsl.element.writer.WriterFactory import WritterFactory
from sampling_mining_workflows_dsl.metadata.Metadata import Metadata

# Define metadata for your data
id_metadata = Metadata.of_string("id")
language_metadata = Metadata.of_string("language")
commit_count_metadata = Metadata.of_integer("commitNb")

# Create loaders and writers
json_loader = LoaderFactory.json_loader
json_writer = WritterFactory.json_writer

# Create a basic sampling workflow
workflow = (
    WorkflowBuilder()
    .input(json_loader("input_data.json", id_metadata, commit_count_metadata, language_metadata))
    .filter_operator("commitNb > 1000")
    .random_selection_operator(50)
    .output(json_writer("output_data.json"))
)

# Execute the workflow
results = workflow.execute_workflow()

License

Distributed under the GNU General Public License v3.0. See LICENSE for more information.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

(back to top)