dmsc 0.1.6

Dunimd Middleware Service - A high-performance Rust middleware framework with modular architecture

Dunimd Middleware Service

English | 简体中文

Help Documentation | Changelog | Security | Contributing | Code of Conduct

DMSC (Dunimd Middleware Service) — A high-performance Rust middleware framework that unifies backend infrastructure. Built for enterprise-scale with modular architecture, built-in observability, and distributed systems support.

🏗️ Core Architecture

📐 Modular Design

DMSC adopts a highly modular architecture with 18 core modules, enabling on-demand composition and seamless extension:

Module	Description
auth	Authentication & authorization (JWT, OAuth, permissions)
cache	Multi-backend cache abstraction (Memory, Redis, Hybrid)
config	Multi-source configuration management with hot reload
core	Runtime, error handling, and service context
database	Database abstraction with PostgreSQL, MySQL, SQLite support
device	Device control, discovery, and intelligent scheduling
fs	Secure file system operations and management
gateway	API gateway with load balancing, rate limiting, and circuit breaking
grpc	gRPC server and client support with Python bindings (requires grpc feature)
hooks	Lifecycle event hooks (Startup, Shutdown, etc.)
log	Structured logging with tracing context integration
module_rpc	Inter-module RPC communication for distributed method calls
observability	Metrics, tracing, and Grafana integration
database.orm	Type-safe ORM with repository pattern, query builder, and Python bindings
protocol	Protocol abstraction layer for multi-protocol support (requires pyo3 feature)
queue	Distributed queue abstraction (Kafka, RabbitMQ, Redis, Memory)
service_mesh	Service discovery, health checking, and traffic management
validation	Input validation and data sanitization utilities
ws	WebSocket server support with Python bindings (requires websocket feature)

Note: Some modules require specific feature flags:

• grpc: gRPC support (--features grpc)
• websocket: WebSocket support (--features websocket)
• protocol: Protocol abstraction layer (--features protocol or full)

🚀 Key Features

🔍 Distributed Tracing

• W3C Trace Context standard implementation
• Full-chain TraceID/SpanID propagation
• Baggage data transmission for business context
• Multi-language compatibility (Java, Go, Python)
• Automatic span creation via #[tracing::instrument] attribute

📊 Enterprise Observability

• Native Prometheus metrics export
• Counter, Gauge, Histogram, Summary metric types
• Out-of-the-box Grafana dashboard integration
• Real-time performance statistics with quantile calculation
• Full-stack metrics (CPU, memory, I/O, network)

🤖 Intelligent Device Management

• Auto-discovery and registration
• Efficient resource pool management
• Policy-based scheduling with priority support
• Dynamic load balancing
• Complete device lifecycle management

📝 Structured Logging

• JSON and text format support
• Configurable sampling rates
• Intelligent log rotation
• Automatic tracing context inclusion
• DEBUG/INFO/WARN/ERROR log levels

⚙️ Flexible Configuration

• Multi-source loading (files, environment variables, runtime)
• Hot configuration updates
• Modular architecture for on-demand composition
• Plugin-based extension mechanism

📁 Secure File System

• Unified project root directory management
• Atomic file operations
• Categorized directory structure
• JSON data persistence
• Secure path handling

🛠️ Installation & Environment

Prerequisites

• Rust: 1.65+ (2021 Edition)
• Cargo: 1.65+
• Platforms: Linux, macOS, Windows

Build Dependencies

Some features require additional system dependencies:

Dependency	Required For	Installation
protoc	etcd feature (Protocol Buffers)	Protocol Buffers
CMake + C++ compiler	kafka feature (rdkafka)	See instructions below

Installing protoc

Windows:

# Using chocolatey
choco install protoc

# Or download from GitHub releases
# https://github.com/protocolbuffers/protobuf/releases

macOS:

brew install protobuf

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y protobuf-compiler

Linux (CentOS/RHEL):

sudo yum install -y protobuf-compiler

Installing CMake and C++ compiler (for Kafka support)

Windows:

# CMake is usually installed with Visual Studio
# Or download from: https://cmake.org/download/

# Using chocolatey
choco install cmake

macOS:

# CMake and C++ compiler (Xcode Command Line Tools)
xcode-select --install

# Or using Homebrew
brew install cmake

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y cmake build-essential

Linux (CentOS/RHEL):

sudo yum install -y cmake gcc-c++ make

Quick Setup

Add DMSC to your project's Cargo.toml:

[dependencies]
dmsc = { git = "https://github.com/mf2023/DMSC" }

Or use cargo add:

cargo add dmsc --git https://github.com/mf2023/DMSC

⚡ Quick Start

Core API Usage

use dmsc::prelude::*;

#[tokio::main]
async fn main() -> DMSCResult<()> {
    // Build service runtime
    let app = DMSCAppBuilder::new()
        .with_config("config.yaml")?
        .with_logging(DMSCLogConfig::default())
        .with_observability(DMSCObservabilityConfig::default())
        .build()?;
    
    // Run business logic
    app.run(|ctx: &DMSCServiceContext| async move {
        ctx.logger().info("service", "DMSC service started")?;
        // Your business code here
        Ok(())
    }).await
}

Observability Example

use dmsc::prelude::*;
use dmsc::observability::{DMSCTracer, DMSCSpanKind, DMSCSpanStatus};

#[tracing::instrument(name = "user_service", skip(ctx))]
async fn get_user(ctx: &DMSCServiceContext, user_id: u64) -> DMSCResult<User> {
    let user = fetch_user_from_db(user_id).await?;
    Ok(user)
}

Or using DMSCTracer directly:

use dmsc::prelude::*;
use dmsc::observability::DMSCTracer;

async fn get_user(ctx: &DMSCServiceContext, user_id: u64) -> DMSCResult<User> {
    let tracer = DMSCTracer::new(1.0);
    let _span = tracer.span("get_user")
        .with_attribute("user_id", user_id.to_string())
        .start();
    let user = fetch_user_from_db(user_id).await?;
    Ok(user)
}

🔧 Configuration

Configuration Example

# config.yaml
service:
  name: "my-service"
  version: "1.0.0"

logging:
  level: "info"
  file_format: "json"
  file_enabled: true
  console_enabled: true

observability:
  metrics_enabled: true
  tracing_enabled: true
  prometheus_port: 9090

resource:
  providers: ["cpu", "gpu", "memory"]
  scheduling_policy: "priority_based"

Configuration Sources

DMSC supports multiple configuration sources in order of priority (lowest to highest):

1. Configuration files (YAML, TOML, JSON)
2. Custom configuration via code
3. Environment variables (prefixed with DMSC_)

🧪 Development & Testing

Running Tests

# Run all tests
cargo test

# Run specific test module
cargo test cache

# Run with verbose output
cargo test -- --nocapture

❓ Frequently Asked Questions

Q: How to add a new module? A: Implement the DMSCModule trait and register it via DMSCAppBuilder::with_module.

Q: How to configure logging level? A: Set logging.level in the configuration file, supporting DEBUG/INFO/WARN/ERROR levels.

Q: How to enable metrics export? A: Set observability.metrics_enabled: true and configure prometheus_port in the configuration file.

Q: How to extend configuration sources? A: Implement a custom configuration loader and register it with DMSCConfigManager.

Q: How to handle asynchronous tasks? A: Use DMSCAppBuilder::with_async_module to add async modules, the framework handles async lifecycle automatically.

🌏 Community & Citation

• Welcome to submit Issues and PRs!
• Gitee: https://gitee.com/dunimd/dmsc.git
• Github: https://github.com/mf2023/DMSC.git

📄 License & Open Source Agreements

🏛️ Project License

This project uses Apache License 2.0 open source agreement, see LICENSE file.

📋 Dependency Package Open Source Agreements

Open source packages and their agreement information used by this project:

Dependencies License

📦 Package	📜 License
serde	Apache 2.0
serde_json	MIT
serde_yaml	MIT
tokio	MIT
prometheus	Apache 2.0
redis	MIT
hyper	MIT
lapin	Apache 2.0
futures	MIT
yaml-rust	MIT
toml	MIT
etcd-client	MIT
sysinfo	MIT
async-trait	MIT
dashmap	MIT
chrono	MIT
uuid	Apache 2.0
rand	MIT
notify	MIT
jsonwebtoken	MIT
reqwest	MIT
urlencoding	MIT
parking_lot	MIT
log	MIT
pyo3	Apache 2.0
tempfile	MIT
tracing	MIT
thiserror	MIT
hex	MIT
base64	MIT
regex	MIT
url	Apache 2.0
aes-gcm	Apache 2.0
ring	Apache 2.0
lazy_static	MIT
libloading	MIT
zeroize	MIT/Apache-2.0
secrecy	MIT
data-encoding	MIT
crc32fast	MIT
generic-array	MIT
bincode	MIT
typenum	MIT
html-escape	MIT
rustls	Apache 2.0/MIT
rustls-pemfile	Apache 2.0/MIT
webpki	ISC
rustls-native-certs	Apache 2.0/MIT
bytes	Apache 2.0
tonic	MIT
prost	Apache 2.0
tokio-stream	MIT
tower	MIT
async-stream	MIT
tokio-tungstenite	MIT
tungstenite	MPL-2.0
num-bigint	MIT/Apache-2.0
oqs	MIT/Apache-2.0