isbg 2.3.1

a script that makes it easy to scan an IMAP inbox for spam usingSpamAssassin and get your spam moved to another folder.

isbg is a script and a python 3 module that makes it easy to scan an IMAP inbox for spam using SpamAssassin and get your spam moved to another folder.

Unlike the normal mode of deployments for SpamAssassin, isbg does not need to be involved in mail delivery, and can run on completely different machines to where your mailbox actually is. So this is the perfect tool to take good care of your ISP mailbox without having to leave it.

The full documentation can be found here.

Table of Contents

• Support

• Features

• Installation

  • Dependencies

  • Install via pip (easiest method)

  • Install from source

  • Uninstall from source

  • Building Windows Installer

• Usage

  • SpamAssassin

    • Configure your spamassassin

  • CLI Options

  • How does it work?

  • Multiple accounts

  • Saving your password

  • SSL

  • Read and Seen flags

  • Gmail Integration

  • Ignored emails

  • Partial runs

• Contact and about

• License

Support

Please note, the current release of isbg only supports python 3. Python 2 is compatible with older releases.

Features

• Works with all common IMAP servers

• Works on Linux, MacOS X and Windows (even smartphones!)

• Can do IMAP over SSL

• Can remember your password

• Will work painlessly against multiple IMAP accounts and servers

• Is not involved in the mail delivery process, and so can run on any machine that can contact your IMAP server

• Highly configurable

• Sensible defaults so you don’t have to do any configuring :-)

• Possibility to skip spam detection to stick only to the teach feature

• Don’t fail when meeting horrible and bad formed mail

• Lock file to prevent multiple instance to run at the same time (for cron jobs)

Installation

isbg install a python package module and also a script to use it isbg, it also install another script to unwrap messages: isbg_sa_unwrap.

Dependencies

isbg is written in the Python language. Python is installed by default on most Linux systems. You can can find out more about Python at Python home page.

Make sure you have SpamAssassin installed. All the necessary information can be found on the SpamAssassin wiki. SpamAssassin should be on your $PATH (it installs in /usr/bin/ by default)

To run, isbg also depends on some python modules.

• docopt for command line options.

• cchardet or chardet for encoding detection.

• xdg to found the .cache directory. xdg is not required, if it’s not installed, isbg will try to found .cache.

Install via pip (easiest method)

Install as user:

$ pip3 install isbg --user

Install as root:

$ sudo pip3 install isbg

To list the files installed:

$ pip3 show isbg --files

And to uninstall it:

$ pip3 uninstall isbg

Install from source

Download a copy of the isbg repository onto your machine:

$ cd ~
$ git clone https://gitlab.com/isbg/isbg.git

Install program dependencies:

$ cd /isbg
$ python3 setup.py install -r installed_files.txt

Files will be installed under /usr/local/. installed_files.txt contains a list of all the dependencies installed.

Uninstall from source

Uninstall program dependencies:

$ tr '\n' '\0' < installed_files.txt | xargs -0 rm -vf --

Building Windows Installer

Run the following to build a windows installer:

$ python3 setup.py bdist_wininst

Usage

SpamAssassin

If you have never used SpamAssassin before, you’ll probably be quite nervous about it being too good and taking out legitimate email, or not taking out enough spam. It has an easily adjustable threshold to change how aggressive it is. Run the following command to create your preferences file:

$ spamassassin  </dev/null >/dev/null
Created user preferences file: /home/rogerb/.spamassassin/user_prefs

You can then edit $HOME/.spamassassin/user_prefs and change the thresholds.

You can also edit the system-wide settings in /etc/spamassassin/locals.cf.

If you want to use the --learnspambox or --learnhambox, you’ll have to configure your spamassassin.

Configure your spamassassin

If you want to use --learnspambox or --learnhambox features, you have to add this configuration:

Allow Tell

You have to start spamd with the --allow-tell option.

On Debian systems (Debian and Ubuntu), you have to edit /etc/default/spamassassin and replace:

OPTIONS="-D --create-prefs --max-children 5 --helper-home-dir"

by:

OPTIONS="-D --allow-tell --create-prefs --max-children 5 \
--helper-home-dir"

Don’t forget to restart your spamd server after that (sudo service spamassassin restart on Debian).

Setup a DNS cacher

By default, SpamAssassin will perform many DNS lookups for NetworkTests to significantly improve scoring of messages primarily by DNSBlocklists like Spamhaus, SORBS, etc. If you don’t use a DNS cacher and you run isbg often, chances are you will hit the maximum number of free accepted queries on certain hosts and will get blocked.

To improve performance, scoring and make sure DNSBlocklists don’t start blocking you, it is a good idea to set up a DNS cacher.

Using spamc to get better performances

By default, SpamAssassin is spooled up and down each and every time isbg has an email to scan.

You can improve isbg’s performance by running SpamAssassin as a daemon and by telling isbg to use that instead of running a regular process.

On most Linux distributions, you can run SpamAssassin as a daemon this way:

$ systemctl enable spamassassin
$ systemctl start spamassassin

You can then run isbg with the --spamc option to make use of the daemon.

CLI Options

The default behavior of isbg is to not make any changes your Inbox unless you specify specific command line options. Consequently you can experiment without worry at the beginning.

Your first step is to create a new folder to receive suspected spam. I use one named ‘spam’.

Run isbg with the --help option to see what options are available or check its manual page with $ man isbg [1].

You can also unwrap SpamAssassin bundled emails with isbg_sa_unwrap, you can use the --help option to check the available options or $ man isbg_sa_unwrap to check its manual page [2].

[1]

The rendered man page can be read in the “Manual page for isbg” section of this documentation.

[2]

The rendered man page can be read in the “Manual page for isbg_sa_unwrap” section of this documentation.

How does it work?

IMAP assigns each message in a folder a unique id. isbg scans the folder for messages it hasn’t seen before, and for each one, downloads the message and feeds it to SpamAssassin. If SpamAssassin says the message is spam, then the SpamAssassin report is uploaded into your spam folder. Unless you specify the --noreport option, in which case the message is copied from your Inbox to the Spam folder (the copy happens on the IMAP server itself so this option is good if you are on a low bandwidth connection).

Multiple accounts

By default isbg saves the list of seen IMAP message unique IDs in a file in your home directory. It is named .isbg-trackXXXX where XXXX is a 16 byte identifier based on the IMAP host, username and port number. Consequently you can just run isbg against different servers/accounts and it will automatically keep the tracked UIDs separate. You can override the filename with --trackfile.

To run isbg for multiple accounts one after another, it is possible to use bash scripts like the ones in the examples directory. Since these scripts contain passwords and are thus sensitive data, make sure the file permissions are very restrictive.

Saving your password

If you don’t want isbg to prompt you for your password each time, you can specify the --savepw option. This will save the password in a file in your home directory. The file is named $HOME/.cache/isbg/.isbg-XXXX where XXXX is a 16 byte identifier based on the IMAP host, username and port number (the same as for the multiple accounts description above). You can override the filename with --passwdfilename.

The password is obfuscated, so anyone just looking at the contents won’t be able to see what it is. However, if they study the code to isbg then they will be able to figure out how to de-obfuscate it, and recover the original password. (isbg needs the original password each time it is run as well).

Consequently you should regard this as providing minimal protection if someone can read the file.

SSL

isbg can do IMAP over SSL if your version of Python has been compiled with SSL support. Since Python 2.6, SSL comes built in with Python.

However you should be aware that the SSL support does NOT check the certificate name nor validate the issuer. If an attacker can intercept the connection and modify all the packets flowing by, then they will be able to pose as the IMAP server. Other than that, the connection will have the usual security features of SSL.

Read and Seen flags

There are two flags IMAP uses to mark messages, Recent and Seen. Recent is sent to the first IMAP client that connects after a new message is received. Other clients or subsequent connections won’t see that flag. The Seen flag is used to mark a message as read. IMAP clients explicitly set Seen when a message is being read.

Pine and some other mailers use the Recent flag to mark new mail. Unfortunately this means that if isbg or any other IMAP client has even looked at the Inbox, the messages won’t be shown as new. It really should be using Seen.

The IMAP specification does not permit clients to change the Recent flag.

Gmail Integration

Gmail has a few unique ways that they interact with a mail client. isbg must be considered to be a client due to interacting with the Gmail servers over IMAP, and thus, should conform to these special requirements for proper integration.

There are two types of deletion on a Gmail server.

• Type 1: Move a message to ‘[Gmail]/Trash’ folder.

  This “removes all labels” from the message. It will no longer appear in any folders and there will be a single copy located in the trash folder. Gmail will “empty the trash” after the received email message is 30 days old.

  You can also do a “Normal IMAP delete” on the message in the trash folder to cause it to be removed permanently without waiting 30 days.

• Type 2: Normal IMAP delete flag applied to a message.

  This will “remove a single label” from a message. It will no longer appear in the folder it was removed from but will remain in other folders and also in the “All Mail” folder.

  Enable Gmail integration mode by passing --gmail in conjunction with --delete on the command line when invoking isbg. These are the features which are tweaked:

  • The --delete command line switch will be modified so that it will result in a Type 1 delete.

  • The --deletehigherthan command line switch will be modified so that it will results in a Type 1 delete.

  • If --learnspambox is used along with the --learnthendestroy option, then a Type 1 delete occurs leaving only a copy of the spam in the Trash.

  • If --learnhambox is used along with the --learnthendestroy option, then a Type 2 delete occurs, only removing the single label.

Reference information was taken from gmail IMAP usage.

Ignored emails

By default, isbg ignores emails that are bigger than 120,000 bytes since spam are not often that big. If you ever get emails with score of 0 on 5 (0.0/5.0), it is likely that SpamAssassin is skipping it due to size.

Default maximum size can be changed with the use of the --maxsize option.

Partial runs

By default, isbg scans 50 emails for operation: spam learn, ham learn and spam detection. If you want to change the default, you can use the --partialrun option specifying the number. isbg tries to read first the new messages and tracks the before seen to not reprocess them.

This is useful when your inbox has a lot of emails, since deletion and mail tracking are only performed at the end of the run and full scans can take too long.

If you want that isbg does track all the emails you can disable the partialrun with --partialrun=0.

Contact and about

Please join our isbg mailing list if you use isbg or contribute to it! The mailing list will be used to announce project news and to discuss the further development of isbg.

You can also hang out with us on IRC, at #isbg on Freenode.

See the CONTRIBUTORS file in the git repository for more information on who wrote and maintains this software.

License

This program is licensed under the GNU General Public License version 3.

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.