dmsc 0.1.8

Dunimd Middleware Service - A high-performance Rust middleware framework with modular architecture

Dunimd Middleware Service

English | 简体中文

Help Documentation | Changelog | Security | Contributing | Code of Conduct

DMSC (Dunimd Middleware Service) — A high-performance Rust middleware framework that unifies backend infrastructure. Built for enterprise-scale with modular architecture, built-in observability, and distributed systems support.

🏗️ Core Architecture

📐 Modular Design

DMSC adopts a highly modular architecture with 18 core modules, enabling on-demand composition and seamless extension:

Module	Description
auth	Authentication & authorization (JWT, OAuth, permissions)
cache	Multi-backend cache abstraction (Memory, Redis, Hybrid)
config	Multi-source configuration management with hot reload
core	Runtime, error handling, and service context
database	Database abstraction with PostgreSQL, MySQL, SQLite support
device	Device control, discovery, and intelligent scheduling
fs	Secure file system operations and management
gateway	API gateway with load balancing, rate limiting, and circuit breaking
grpc	gRPC server and client support with Python bindings (requires grpc feature)
hooks	Lifecycle event hooks (Startup, Shutdown, etc.)
log	Structured logging with tracing context integration
module_rpc	Inter-module RPC communication for distributed method calls
observability	Metrics, tracing, and Grafana integration
database.orm	Type-safe ORM with repository pattern, query builder, and Python bindings
protocol	Protocol abstraction layer for multi-protocol support (requires pyo3 feature)
queue	Distributed queue abstraction (Kafka, RabbitMQ, Redis, Memory)
service_mesh	Service discovery, health checking, and traffic management
validation	Input validation and data sanitization utilities
ws	WebSocket server support with Python bindings (requires websocket feature)
c	C/C++ FFI bindings for cross-language integration (requires c feature)
java	Java JNI bindings for Java application integration (requires java feature)

Note: Some modules require specific feature flags:

• grpc: gRPC support (--features grpc)
• websocket: WebSocket support (--features websocket)
• protocol: Protocol abstraction layer (--features protocol or full)
• c: C/C++ FFI bindings (--features c)
• java: Java JNI bindings (--features java)

🚀 Key Features

🔍 Distributed Tracing

• W3C Trace Context standard implementation
• Full-chain TraceID/SpanID propagation
• Baggage data transmission for business context
• Multi-language compatibility (Java, Go, Python)
• Automatic span creation via #[tracing::instrument] attribute

📊 Enterprise Observability

• Native Prometheus metrics export
• Counter, Gauge, Histogram, Summary metric types
• Out-of-the-box Grafana dashboard integration
• Real-time performance statistics with quantile calculation
• Full-stack metrics (CPU, memory, I/O, network)

🤖 Intelligent Device Management

• Auto-discovery and registration
• Efficient resource pool management
• Policy-based scheduling with priority support
• Dynamic load balancing
• Complete device lifecycle management

📝 Structured Logging

• JSON and text format support
• Configurable sampling rates
• Intelligent log rotation
• Automatic tracing context inclusion
• DEBUG/INFO/WARN/ERROR log levels

⚙️ Flexible Configuration

• Multi-source loading (files, environment variables, runtime)
• Hot configuration updates
• Modular architecture for on-demand composition
• Plugin-based extension mechanism

📁 Secure File System

• Unified project root directory management
• Atomic file operations
• Categorized directory structure
• JSON data persistence
• Secure path handling

🛠️ Installation & Environment

Prerequisites

• Rust: 1.65+ (2021 Edition)
• Cargo: 1.65+
• Platforms: Linux (x64, arm64), macOS (x64, arm64), Windows (x64, arm64), Android (arm64-v8a, armeabi-v7a, x86_64)

Build Dependencies

Some features require additional system dependencies:

Dependency	Required For	Installation
protoc	etcd feature (Protocol Buffers)	Protocol Buffers
CMake + C++ compiler	kafka feature (rdkafka)	See instructions below

Installing protoc

Windows:

# Using chocolatey
choco install protoc

# Or download from GitHub releases
# https://github.com/protocolbuffers/protobuf/releases

macOS:

brew install protobuf

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y protobuf-compiler

Linux (CentOS/RHEL):

sudo yum install -y protobuf-compiler

Installing CMake and C++ compiler (for Kafka support)

Windows:

# CMake is usually installed with Visual Studio
# Or download from: https://cmake.org/download/

# Using chocolatey
choco install cmake

macOS:

# CMake and C++ compiler (Xcode Command Line Tools)
xcode-select --install

# Or using Homebrew
brew install cmake

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y cmake build-essential

Linux (CentOS/RHEL):

sudo yum install -y cmake gcc-c++ make

Quick Setup

Add DMSC to your project's Cargo.toml:

[dependencies]
dmsc = "0.1.8"

Or use cargo add:

cargo add dmsc

⚡ Quick Start

Core API Usage

use dmsc::prelude::*;

#[tokio::main]
async fn main() -> DMSCResult<()> {
    // Build service runtime
    let app = DMSCAppBuilder::new()
        .with_config("config.yaml")?
        .with_logging(DMSCLogConfig::default())
        .with_observability(DMSCObservabilityConfig::default())
        .build()?;
    
    // Run business logic
    app.run(|ctx: &DMSCServiceContext| async move {
        ctx.logger().info("service", "DMSC service started")?;
        // Your business code here
        Ok(())
    }).await
}

Observability Example

use dmsc::prelude::*;
use dmsc::observability::{DMSCTracer, DMSCSpanKind, DMSCSpanStatus};

#[tracing::instrument(name = "user_service", skip(ctx))]
async fn get_user(ctx: &DMSCServiceContext, user_id: u64) -> DMSCResult<User> {
    let user = fetch_user_from_db(user_id).await?;
    Ok(user)
}

Or using DMSCTracer directly:

use dmsc::prelude::*;
use dmsc::observability::DMSCTracer;

async fn get_user(ctx: &DMSCServiceContext, user_id: u64) -> DMSCResult<User> {
    let tracer = DMSCTracer::new(1.0);
    let _span = tracer.span("get_user")
        .with_attribute("user_id", user_id.to_string())
        .start();
    let user = fetch_user_from_db(user_id).await?;
    Ok(user)
}

🔧 Configuration

Configuration Example

# config.yaml
service:
  name: "my-service"
  version: "1.0.0"

logging:
  level: "info"
  file_format: "json"
  file_enabled: true
  console_enabled: true

observability:
  metrics_enabled: true
  tracing_enabled: true
  prometheus_port: 9090

resource:
  providers: ["cpu", "gpu", "memory"]
  scheduling_policy: "priority_based"

Configuration Sources

DMSC supports multiple configuration sources in order of priority (lowest to highest):

1. Configuration files (YAML, TOML, JSON)
2. Custom configuration via code
3. Environment variables (prefixed with DMSC_)

🧪 Development & Testing

Running Tests

Multi-Language Testing

DMSC provides comprehensive testing across all supported languages:

• Rust: Core library tests with cargo test
• Python: Python binding tests with pytest
• Java: JNI binding tests with standard Java test runner
• C/C++: C API tests with native compilers

Test Coverage

The tests verify:

• ✅ Builder pattern behavior in all languages
• ✅ Method chaining returns appropriate instances (language-specific)
• ✅ Runtime creation and lifecycle management
• ✅ Error handling and edge cases
• ✅ Cross-language API consistency

Running Rust Tests

# Run all Rust tests
cargo test

# Run specific test modules
cargo test --lib app_builder
cargo test --lib app_runtime

# Run with verbose output
cargo test -- --nocapture

# Run with all features
cargo test --all-features

Running Python Tests

# Install Python package in development mode
cd python
pip install -e .

# Run all Python tests
python -m pytest tests/Python/ -v

# Run specific test classes
python -m pytest tests/Python/test_core.py::TestDMSCAppBuilderWrapper -v
python -m pytest tests/Python/test_core.py::TestDMSCAppRuntimeWrapper -v

Running Java Tests

# Build JNI library
cargo build --release --no-default-features --features java

# Compile and run Java tests
cd java
javac -d build/classes/java/test -cp build/classes/java/main \
  src/test/java/TestAll.java src/test/java/TestAppBuilder.java

java -cp build/classes/java/test:build/classes/java/main \
  -Djava.library.path=../target/release TestAll

API Behavior Across Languages

Language	Builder Pattern	Method Chaining	Reason
Rust	Returns Self	Consumes original	Native builder pattern
Python	Returns self	Same instance	Pythonic wrapper for PyO3
Java	Returns new instance	Immutable builder	Java best practice
C	Returns new pointer	Memory management	C idioms

❓ Frequently Asked Questions

Q: How to add a new module? A: Implement the DMSCModule trait and register it via DMSCAppBuilder::with_module.

Q: How to configure logging level? A: Set logging.level in the configuration file, supporting DEBUG/INFO/WARN/ERROR levels.

Q: How to enable metrics export? A: Set observability.metrics_enabled: true and configure prometheus_port in the configuration file.

Q: How to extend configuration sources? A: Implement a custom configuration loader and register it with DMSCConfigManager.

Q: How to handle asynchronous tasks? A: Use DMSCAppBuilder::with_async_module to add async modules, the framework handles async lifecycle automatically.

🌏 Community & Citation

• Welcome to submit Issues and PRs!
• Github: https://github.com/mf2023/DMSC.git
• Gitee: https://gitee.com/dunimd/dmsc.git
• GitCode: https://gitcode.com/dunimd/dmsc.git

📄 License & Open Source Agreements

🏛️ Project License

This project uses Apache License 2.0 open source agreement, see LICENSE file.

📋 Dependency Package Open Source Agreements

Open source packages and their agreement information used by this project:

Dependencies License

📦 Package	📜 License	📦 Package	📜 License
serde	MIT/Apache-2.0	serde_json	MIT/Apache-2.0
serde_yaml	MIT/Apache-2.0	tokio	MIT
futures	MIT/Apache-2.0	futures-util	MIT/Apache-2.0
http	MIT/Apache-2.0	hyper	MIT
prometheus	MIT/Apache-2.0	redis	MIT
lapin	MIT/Apache-2.0	rdkafka	MIT
yaml-rust	MIT/Apache-2.0	toml	MIT/Apache-2.0
etcd-client	MIT	sysinfo	MIT
async-trait	MIT/Apache-2.0	dashmap	MIT
chrono	MIT	uuid	MIT/Apache-2.0
rand	MIT/Apache-2.0	notify	CC0-1.0
jsonwebtoken	MIT	reqwest	MIT/Apache-2.0
urlencoding	MIT	parking_lot	MIT/Apache-2.0
log	MIT/Apache-2.0	tracing	MIT
pyo3	MIT/Apache-2.0	jni	MIT/Apache-2.0
safer-ffi	MIT	tempfile	MIT/Apache-2.0
thiserror	MIT/Apache-2.0	hex	MIT/Apache-2.0
base64	MIT/Apache-2.0	regex	MIT/Apache-2.0
url	MIT/Apache-2.0	aes-gcm	MIT/Apache-2.0
ring	ISC	lazy_static	MIT/Apache-2.0
libloading	ISC	zeroize	MIT/Apache-2.0
zeroize_derive	MIT/Apache-2.0	secrecy	MIT
erased-serde	MIT	data-encoding	MIT
crc32fast	MIT/Apache-2.0	generic-array	MIT
bincode	MIT	typenum	MIT/Apache-2.0
html-escape	MIT	rustls	MIT/Apache-2.0
rustls-pemfile	MIT/Apache-2.0	webpki	ISC
rustls-native-certs	MIT/Apache-2.0	tokio-rustls	MIT/Apache-2.0
bytes	MIT	tonic	MIT
prost	MIT/Apache-2.0	prost-types	MIT/Apache-2.0
tokio-stream	MIT	tower	MIT
async-stream	MIT	tokio-tungstenite	MIT
tungstenite	MIT	num-bigint	MIT/Apache-2.0
oqs	MIT/Apache-2.0	sm-crypto	MIT
openssl-sys	Apache-2.0	tokio-postgres	MIT/Apache-2.0
rusqlite	MIT	sqlx	MIT/Apache-2.0
criterion	MIT/Apache-2.0