log-similarity 0.1.0

Similarity-driven log clustering for error log analysis

Log Similarity SDK

A powerful Python SDK for similarity-driven log clustering and error log analysis. Automatically groups similar log messages into clusters and identifies anomalous logs that don't match existing patterns.

Features

• Intelligent Log Clustering: Automatically groups similar logs based on semantic similarity
• Multi-layer Preprocessing: Advanced pattern normalization for universal log formats
• Unmatched Log Detection: Identifies error logs that don't fit any existing cluster
• High Performance: Efficient length-based indexing and similarity calculation
• Flexible Configuration: Customizable similarity thresholds and preprocessing rules

Installation

pip install log-similarity

Or install from source:

git clone https://github.com/suiwenfeng/log-similarity-sdk.git
cd log-similarity-sdk
pip install -e .

Quick Start

1. Build Clusters from Base Error Logs

from log_similarity import SimilarityClusterer

# Sample base error logs
base_logs = [
    "User 12345 logged in from 192.168.1.100",
    "User 67890 logged in from 10.0.0.1",
    "Database connection timeout after 30 seconds",
    "Database connection timeout after 45 seconds",
    "GET /api/users returned 404",
    "GET /api/posts returned 500",
]

# Create and train clusterer
clusterer = SimilarityClusterer(similarity_threshold=0.85)
clusterer.fit(base_logs, verbose=True)

# View cluster summary
clusterer.print_summary()

2. Find Unmatched Error Logs

# New error logs to check
new_logs = [
    "User 11111 logged in from 172.16.0.1",  # Similar to cluster 1
    "GET /api/products returned 404",         # Similar to cluster 3
    "Critical: Out of memory error",          # NEW - doesn't match any cluster
    "Fatal: Disk space exhausted",            # NEW - doesn't match any cluster
]

# Find logs that don't match any existing cluster
unmatched = clusterer.find_unmatched_logs(new_logs, verbose=True)

# Process unmatched logs
for log in unmatched:
    print(f"Unmatched: {log['original']}")
    print(f"Template:  {log['template']}")
    print(f"Similarity: {log['similarity']:.2f}\n")

API Reference

SimilarityClusterer

Main class for log clustering and analysis.

Constructor

SimilarityClusterer(
    similarity_threshold: float = 0.90,
    merge_threshold: float = 0.95,
    length_tolerance: float = 0.3,
    preprocessor_config: Optional[PreprocessorConfig] = None
)

Parameters:

• similarity_threshold: Minimum similarity score to match a cluster (default: 0.90)
• merge_threshold: Threshold for merging logs (default: 0.95)
• length_tolerance: Token length tolerance for comparison (default: 0.3)
• preprocessor_config: Custom preprocessing configuration

Methods

fit(log_messages, verbose=True)

Build clusters from training logs.

Parameters:

• log_messages (List[str]): List of log messages to cluster
• verbose (bool): Print progress information

Returns:

• self: The clusterer instance (for method chaining)

Example:

clusterer.fit(base_error_logs, verbose=True)

predict(log_message)

Predict which cluster a single log belongs to.

Parameters:

• log_message (str): Log message to predict

Returns:

• dict: Prediction result with keys:
  • cluster_id: Matched cluster ID or None
  • template: Log template
  • similarity: Similarity score
  • match_type: 'matched' or 'unmatched'
  • original: Original log message
  • preprocessed: Preprocessed log

Example:

result = clusterer.predict("User 999 logged in from 10.0.0.5")
if result['match_type'] == 'matched':
    print(f"Matched cluster: {result['cluster_id']}")
else:
    print("No matching cluster found")

predict_batch(log_messages, verbose=False)

Predict clusters for multiple logs.

Parameters:

• log_messages (List[str]): List of log messages
• verbose (bool): Show progress

Returns:

• List[dict]: List of prediction results

find_unmatched_logs(log_messages, verbose=False)

Find logs that don't match any existing cluster.

Parameters:

• log_messages (List[str]): Logs to check
• verbose (bool): Show progress and statistics

Returns:

• List[dict]: List of unmatched logs with keys:
  • original: Original log message
  • preprocessed: Preprocessed log
  • template: Extracted template
  • similarity: Highest similarity score

Example:

unmatched = clusterer.find_unmatched_logs(new_error_logs, verbose=True)
print(f"Found {len(unmatched)} new error patterns")

get_all_clusters()

Get all clusters with their details.

Returns:

• List[dict]: Sorted list of clusters (by frequency)

save(filepath)

Save clusterer state to JSON file.

Parameters:

• filepath (str): Path to save file

Advanced Usage

Custom Preprocessing

from log_similarity import PreprocessorConfig, LogPreprocessor

# Create custom config
config = PreprocessorConfig()
# Modify patterns as needed
# config.dynamic_patterns.append((r'custom_pattern', '<CUSTOM>'))

clusterer = SimilarityClusterer(preprocessor_config=config)

Adjusting Similarity Threshold

# Stricter matching (higher threshold)
strict_clusterer = SimilarityClusterer(similarity_threshold=0.95)

# More lenient matching (lower threshold)
lenient_clusterer = SimilarityClusterer(similarity_threshold=0.80)

Batch Processing

# Process large log files
with open('error_logs.txt', 'r') as f:
    logs = f.readlines()

# Build clusters from first 10k logs
clusterer.fit(logs[:10000], verbose=True)

# Find unmatched in remaining logs
unmatched = clusterer.find_unmatched_logs(logs[10000:], verbose=True)

# Save results
with open('unmatched_errors.txt', 'w') as f:
    for log in unmatched:
        f.write(log['original'] + '\n')

Use Cases

1. Error Log Monitoring: Detect new error patterns in production logs
2. Log Deduplication: Group similar logs to reduce noise
3. Anomaly Detection: Identify unusual log patterns
4. Log Template Extraction: Automatically extract log templates
5. Log Analysis: Understand log distribution and patterns

Algorithm

The SDK uses a similarity-driven clustering algorithm with:

1. Multi-layer Preprocessing: Normalizes dynamic parts (IPs, timestamps, UUIDs, etc.)
2. Template Extraction: Identifies static and dynamic parts using Logram parser
3. Similarity Calculation: Combines Jaccard similarity, prefix matching, and length ratio
4. Efficient Indexing: Length-based indexing for fast cluster lookup

Requirements

• Python >= 3.7
• No external dependencies (pure Python)

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Support

For issues and questions, please open an issue on GitHub.