DataOpsHub 6.6.12

A Python toolkit for streamlined data operations and database management

DataOpsHub

DataOpsHub is a comprehensive Python toolkit designed specifically for healthcare data operations and management. It provides a robust set of tools for managing medical records, user data, form validation, database operations, and web automation in healthcare environments. The package emphasises data integrity, security, and compliance with healthcare standards including HL7 v2 compatibility.

Features

• Healthcare Data Models:

  • Comprehensive user management with medical data relationships
  • Vital signs tracking and monitoring
  • Laboratory results management
  • Medication history and prescription tracking
  • HL7 v2 compatibility for healthcare data interchange

• Database Operations:

  • Advanced SQLAlchemy-based database handling
  • Multi-database support (MySQL, PostgreSQL, SQLite)
  • Automated file-to-database loading (CSV, Excel, ODS)
  • Comprehensive CRUD operations with error handling
  • Data filtering and querying with temporal range support

• Form Validation and Data Entry:

  • WTForms-based form validation for healthcare applications
  • Comprehensive field validation (email, medical IDs, dates, demographics)
  • Spanish DNI validation support
  • JSON and form data validation with error reporting
  • Medical data entry forms with specialised validations

• Security and Configuration:

  • Automated secure password generation
  • Comprehensive security configuration management
  • Custom exception hierarchy for precise error handling
  • Environment-based configuration with security best practices

• Web Automation:

  • Selenium WebDriver setup and testing utilities
  • Headless browser configuration for automated testing
  • Support for Firefox and Chrome browsers in CI/CD environments

Installation

Prerequisites

Before installing, please ensure the following dependencies are available on your system:

• External Tools (required for full functionality):

  • Database server (MySQL, PostgreSQL, or SQLite)
  • Web browser (Firefox/Chrome for web automation features)

• Required Third-Party Libraries:

  pip install sqlalchemy pandas numpy scipy wtforms email-validator selenium
  

  Or via Anaconda (recommended channel: conda-forge):

  conda install -c conda-forge sqlalchemy pandas numpy scipy
  pip install wtforms email-validator selenium
  

• Database Drivers (choose based on your database):

  # For MySQL
  pip install pymysql
  
  # For PostgreSQL
  pip install psycopg2-binary
  
  # SQLite is included with Python
  

• Internal Package Dependencies:

  pip install filewise paramlib
  pip install pygenutils                    # Core functionality
  pip install pygenutils[arrow]             # With arrow support (optional)
  

For regular users (from PyPI)

pip install dataopshub

For contributors/developers (with latest Git versions)

# Install with development dependencies (includes latest Git versions)
pip install -e .[dev]

# Alternative: Use requirements-dev.txt for explicit Git dependencies
pip install -r requirements-dev.txt
pip install -e .

Benefits of the new approach:

• Regular users: Simple pip install dataopshub with all dependencies included
• Developers: Access to latest Git versions for development and testing
• PyPI compatibility: All packages can be published without Git dependency issues

Troubleshooting

If you encounter import errors:

1. For PyPI users: The package should install all dependencies automatically. If you get import errors, try:

   pip install --upgrade dataopshub
   

2. For developers: Make sure you've installed the development dependencies:

   pip install -e .[dev]
   

3. Common issues:

   • Missing dependencies: For regular users, all dependencies are included. For developers, use pip install -e .[dev]
   • Python version: Ensure you're using Python 3.10 or higher
   • Database drivers: Install the appropriate database driver for your database system

Verify Installation

To verify that your installation is working correctly:

try:
    import DataOpsHub
    from filewise.general.introspection_utils import get_type_str
    from pygenutils.arrays_and_lists.data_manipulation import flatten_list
    from paramlib.config_params import DB_ERROR_CODE_DICT
    
    print("✅ All imports successful!")
    print(f"✅ DataOpsHub version: {DataOpsHub.__version__}")
    print("✅ Installation is working correctly.")
    
except ImportError as e:
    print(f"❌ Import error: {e}")
    print("💡 For regular users: pip install dataopshub")
    print("💡 For developers: pip install -e .[dev]")

Usage

Basic Example - User Management

from DataOpsHub.models.user import User
from DataOpsHub.services.user_service import UserService
from DataOpsHub.databases.database_handler import init_db
from DataOpsHub.security.config import DATABASE_CREDENTIALS

# Initialize database
engine, Session = init_db(DATABASE_CREDENTIALS)
session = Session()

# Create user service
user_service = UserService(session)

# Create a new user
user_data = {
    'user_id': 'U001',
    'first_name': 'John',
    'last_name': 'Doe',
    'email': '[e-mail]',
    'date_of_birth': '1990-01-01',
    'gender': 'Male'
}

result = user_service.create_user(user_data)
print(result['message'])  # "User created successfully"

Advanced Example - Medical Data Management

from DataOpsHub.models.user import VitalSigns, LabResults
from datetime import datetime

# Add vital signs for a user
vital_signs = VitalSigns(
    user_id='U001',
    heart_rate=72,
    blood_pressure='120/80',
    temperature=98.6,
    oxygen_saturation=98
)

# Add lab results
lab_result = LabResults(
    user_id='U001',
    test_name='Complete Blood Count',
    result='Normal',
    unit='cells/mcL',
    reference_range='4,500-11,000'
)

session.add(vital_signs)
session.add(lab_result)
session.commit()

# Get user with all medical data
user_data = user_service.get_user_with_related_data('U001')
print(f"User: {user_data['user'].first_name}")
print(f"Vital signs: {len(user_data['vital_signs'])}")
print(f"Lab results: {len(user_data['lab_results'])}")

Form Validation Example

from DataOpsHub.data_entry_forms.forms.forms import RegistrationForm
from DataOpsHub.data_entry_forms.validators.form_field_validations import validate_field

# Validate a registration form
form_data = {
    'email': '[e-mail]',
    'first_name': 'John',
    'last_name': 'Doe'
}

form = RegistrationForm(data=form_data)
if form.validate():
    print("Form validation successful!")
else:
    print("Validation errors:", form.errors)

# Validate individual fields
email_result = validate_field('email', '[e-mail]', {}, [])
print(f"Email valid: {not email_result['has_error']}")

Database File Upload Example

from DataOpsHub.databases.database_handler import upload_file_data

# Upload CSV data to database
upload_file_data(
    input_file_list='medical_data.csv',
    config=DATABASE_CREDENTIALS,
    database_type='mysql',
    if_exists='append',
    dtype_dict={'patient_id': 'str', 'age': 'int64', 'temperature': 'float64'}
)

Security Example

from DataOpsHub.security.automated_passwords import generate_automated_password
from DataOpsHub.security.exceptions import ValidationError

# Generate secure password
password = generate_automated_password(33, 127, 16)
print(password)

# Handle custom exceptions
try:
    # Some operation that might fail
    pass
except ValidationError as e:
    print(f"Validation failed: {e.message}")

Project Structure

The package is organised into specialised sub-packages for healthcare operations:

DataOpsHub/
├── constants/
│   ├── error_messages.py         # Standardised error messages
│   └── table_mappings.py         # Database table field mappings
├── data_entry_forms/
│   ├── forms/
│   │   └── forms.py              # WTForms-based form classes
│   └── validators/
│       └── form_field_validations.py  # Comprehensive validation functions
├── databases/
│   ├── database_handler.py       # Core database operations and utilities
│   └── upload_data.py           # File-to-database upload utilities
├── models/
│   ├── base.py                  # Base model with HL7 compatibility
│   └── user.py                  # User and medical data models
├── security/
│   ├── automated_passwords.py   # Secure password generation
│   ├── config.py               # Security configuration management
│   └── exceptions.py           # Custom exception hierarchy
├── services/
│   └── user_service.py         # User management service layer
└── web_automation/
    └── webdriver_Firefox-Chrome_setup_test.py  # WebDriver testing utilities

Key Functions

Database Operations

• init_db() - Initialize database with healthcare models
• upload_file_data() - Load CSV/Excel/ODS files into database tables
• filter_data() - Advanced data filtering with temporal ranges
• create_user(), update_user(), delete_user() - User CRUD operations

Medical Data Models

• User - Comprehensive user model with medical relationships
• VitalSigns - Vital signs tracking and monitoring
• LabResults - Laboratory test results management
• MedicationsHistory - Medication and prescription tracking

Form Validation

• RegistrationForm - User registration with validation
• validate_field() - Individual field validation
• validate_date_range() - Medical date range validation
• verify_spanish_dni() - Spanish ID validation

Security Functions

• generate_automated_password() - Secure password generation
• Custom exception classes for precise error handling
• Environment-based security configuration

Web Automation

• test_firefox_webdriver() - WebDriver setup testing
• configure_firefox_options() - Headless browser configuration

Healthcare Standards Compliance

DataOpsHub is designed with healthcare standards in mind:

• HL7 v2 Compatibility: All models include HL7 v2 fields for healthcare data interchange
• Medical Data Integrity: Comprehensive validation for medical identifiers and data
• Audit Trails: Automatic timestamp tracking for all medical records
• Security: Healthcare-grade security configuration and password policies
• Data Relationships: Proper medical data relationships between users and their health records

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

Development Guidelines

• Follow healthcare data security best practices
• Ensure all medical data models maintain HL7 compatibility
• Add comprehensive validation for any new form fields
• Include proper error handling and custom exceptions
• Write tests for database operations and data validation

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• SQLAlchemy team for robust database operations
• WTForms community for form validation framework
• Healthcare standards organisations for HL7 specifications
• Open-source contributors to the healthcare data ecosystem

Contact

For any questions or suggestions, please open an issue on GitHub or contact the maintainers.

Version

Current version: 6.6.12

For detailed changelog, see CHANGELOG.md.

Security Notice

This package handles sensitive healthcare data. Please ensure:

• Use secure database connections in production
• Implement proper access controls and authentication
• Follow HIPAA compliance guidelines where applicable
• Regularly update security configurations and dependencies
• Use environment variables for sensitive configuration data