spam-analyzer 1.0.0b0

A simple email analyzer

A fast spam filter written in Python inspired by SpamAssassin integrated with machine learning.

Table of Contents

• Table of Contents
• What is Spam Analyzer?
• Installation
• Usage
  • CLI
  • Python
• Contributing
• License

What is spam-analyzer?

spam-analyzer is a CLI (Command Line Interface) application that aims be a viable alternative to spam filter services.

This program can classify the email given in inputs in spam or non-spam using a machine learning algorithm (Random Forest), the model is trained using a dataset of 19900 emails. Anyway it could be wrong sometimes, if you want to improve the accuracy of the model you can train it with your persolized dataset.

The main features of spam-analyzer are:

1. spam recognition with the option to display a detailed analysis of the email
2. JSON output
3. it can be used as a library in your Python project to extract features from an email
4. it is written in Python with its most modern features to ensure software correctness

What is spam and how does spam-analyzer know it?

The analysis takes in consideration the following main aspects:

• the headers of the email
• the body of the email
• the attachments of the email

The most significant parts are the headers and the body of the email. The headers are analyzed to extract the following features:

• SPF (Sender Policy Framework)
• DKIM (DomainKeys Identified Mail)
• DMARC (Domain-based Message Authentication, Reporting & Conformance)
• If the sender domain is the same as the first in received headers
• The subject of the email
• The send date
• If the send date is compliant to the RFC 2822 and if it was sent from a valid time zone
• The date of the first received header

While the body is analyzed to extract the following features:

• If there are links
• If there are images
• If links are only http or https
• The percentage of the body that is written in uppercase
• The percentage of the body that contains blacklisted words
• The polarity of the body calculated with TextBlob
• The subjectivity of the body calculated with TextBlob
• If it contains mailto links
• If it contains javascript code
• If it contains html code
• If it contains html forms

About attachments we only know if they are present or not and if they are executable files.

The task could be solved in a programmatic way, chaining a long set of if statements based on the features extracted from the email. However, this approach is not scalable and it is not easy to maintain. Moreover, it is not possible to improve the accuracy of the model without changing the code and, the most important, the analysis would be based on the conaissance of the programmer and not on the data. Since we live in the data era, we should use the data to solve the problem, not the programmer's knowledge. So I decided to use a machine learning algorithm to solve the problem using all the features extracted from the email.

Installation

spam-analyzer is available on PyPI, so you can install it with pip:

pip install spam-analyzer

For the latest version, you can install it from the source code:

git clone https://github.com/matteospanio/spam-analyzer.git
cd spam-analyzer
pip install .

Usage

CLI

spam-analyzer can be used as a CLI application:

usage: spam-analyzer [-h] -f FILE [-l WORDLIST] [-v] [-V] [-fmt FORMAT] [-o FILE] [--destination-dir DIRECTORY]

A simple program to analyzer emails

options:
  -h, --help            show this help message and exit
  -f FILE, --file FILE  The file or directory to analyze
  -l WORDLIST, --wordlist WORDLIST
                        A file containing the spam wordlist
  -v, --verbose         More program output
  -V, --version         Show program version
  -fmt FORMAT, --output-format FORMAT
                        Format output in a different way
  -o FILE, --output-file FILE
                        Write output to a file (works only for json format)
  --destination-dir DIRECTORY
                        The directory where copy your classified emails

• spam-analyzer -f <file>: classify the email given in input
• spam-analyzer -f <file> -v: classify the email given in input and display a detailed analysis¹
• spam-analyzer -f <file> -fmt json: classify the email given in input and display the result in JSON format (useful for integration with other programs)
• spam-analyzer -f <file> -fmt json -o <file>: classify the email given in input and write the result in JSON format in the file given in input²
• spam-analyzer -f <file> -l <file>: classify the email given in input using the wordlist given in input
• spam-analyzer -f <directory> --destination-dir <directory>: classify all the emails in the directory given in input and copy them in the directory given in input splitted in spam and non-spam folders

Configuration

spam-analyzer is thought to be highly configurable: on its first execution it will create a configuration file in ~/.config/spamanalyzer/ with some other default files. You can change the configuration file to customize the behavior of the program. At the moment of writing there are only paths to the wordlist and the model, but in the future there will be more options (e.g. senders blacklist and whitelist, a default path where to copy classified emails,...).

Python

from spamanalyzer import MailAnalyzer

analyzer = MailAnalyzer(wordlist_path="path/to/wordlist.txt")
analysis = analyzer.analyze("path/to/email.txt")

The spamanalyzer library provides a really simple interface to extract features from an email. The MailAnalyzer class provides the analyze method that takes in input the path to the email and returns a MailAnalysis object containing the analysis of the email.

Furthermore, the MailAnalysis class provides the is_spam method that returns True if the email is spam, False otherwise. Further examples are available in the folder examples of the source code.

Contributing

Contributions are welcome! Please read the contribution guidelines first.

License

spam-analyzer is licensed under the GPLv3 license.

1. The --verbose option is available only for the first use case, it will not work in combination with the --output-format option. ↩

2. You should use the --output-file instead of the > operator to write the output in a file, because the spam-analyzer program prints some other information on the standard output while processing the email(s). ↩