pymap 0.13.1

Lightweight, asynchronous IMAP serving in Python.

pymap

Lightweight, asynchronous IMAP serving in Python.

This project attempts to simplify the complexity of the IMAP protocol into a set of clean Python APIs that can be implemented by pluggable backends. Everything runs in an asyncio event loop.

API Documentation

Table of Contents

• Install and Usage
  • dict Plugin
  • maildir Plugin
  • redis Plugin
• Admin Tool
• Supported Extensions
• Development and Testing
  • Type Hinting

Install and Usage

$ pip install pymap
$ pymap --help
$ pymap dict --help
$ pymap maildir --help

dict Plugin

The dict plugin uses in-memory dictionary objects to store mail and metadata. While the server is running, all concurrent and future connections will see the same data, including added and removed messages, but no changes will persist if the server is restarted.

You can try out the dict plugin with demo data:

$ pymap --port 1143 --debug dict --demo-data

In another terminal, connect to port 1143 and run some commands:

* OK [CAPABILITY IMAP4rev1 STARTTLS AUTH=PLAIN AUTH=LOGIN AUTH=CRAM-MD5 BINARY UIDPLUS MULTIAPPEND IDLE APPENDLIMIT=1000000000] Server ready 163.1.168.192.in-addr.arpa
. login demouser demopass
. OK Authentication successful.
. select INBOX
* OK [PERMANENTFLAGS (\Answered \Deleted \Draft \Flagged \Seen)] Flags permitted.
* FLAGS (\Answered \Deleted \Draft \Flagged \Recent \Seen)
* 4 EXISTS
* 1 RECENT
* OK [UIDNEXT 105] Predicted next UID.
* OK [UIDVALIDITY 4097774359] UIDs valid.
* OK [UNSEEN 4] First unseen message.
. OK [READ-WRITE] Selected mailbox.
. logout
* BYE Logging out.
. OK Logout successful.

Here are some other commands to try:

. uid fetch 1:* all
. list "" *
. create "A New Folder"
. store * +FLAGS (\Deleted)
. expunge

Add new messages using the append command:

. append INBOX (\Flagged) {38+}
From: [e-mail]

test message!

maildir Plugin

The maildir plugin uses on-disk storage for mail and metadata. For mail data, it uses the eponymous Maildir format. However, since Maildir alone is not enough for modern IMAP usage, it is extended with additional data as described in Dovecot's MailboxFormat/Maildir, with the intention of being fully compatible.

For login, the plugin uses a simple formatted text file, e.g.:

john::s3cretp4ssword
sally:mail/data:sallypass
susan:/var/mail/susan:!@#$%^:*

The colon-delimited fields are the user ID, the mailbox path, and the password. The mailbox path may be empty, relative, or absolute. An empty mailbox path will use the user ID as a relative path.

Try out the maildir plugin:

$ pymap --port 1143 --debug maildir /path/to/users.txt

Once started, check out the dict plugin example above to connect and see it in action. The biggest difference is, when stop and restart the pymap server, your mail messages remain intact.

redis Plugin

The redis plugin uses the Redis data structure store for mail and metadata. It requires aioredis and will not appear in the plugins list without it.

$ pip install aioredis
$ pymap redis --help

Keys are composed of a heirarchy of prefixes separated by /. For example, the key containing the flags of a message might be:

/eacb1cf1558741d0b5419b3f838882f5/mbx/Fdaddd3075d7b42e78a7edb1d87ee5800/msg/9173/flags

In this example, the eacb1cf1558741d0b5419b3f838882f5 and Fdaddd3075d7b42e78a7edb1d87ee5800 prefixes are randomly generated IDs acting as the namespaces for the login user and mailbox, respectively, and the message has UID 9173.

The default way to create logins is with a redis hash with a password field. For example:

127.0.0.1:6379> HSET /john password "s3cretp4ssword"
(integer) 1
127.0.0.1:6379> HSET /sally password "sallypass"
(integer) 1

For compatibility with dovecot's auth dict, a JSON object can be used instead of a redis hash with the --users-json command-line argument.

127.0.0.1:6379> SET /susan "{\"password\": \"!@#$%^:*\"}"
(integer) 1

Try out the redis plugin:

$ pymap --port 1143 --debug redis redis://localhost

Once started, check out the dict plugin example above to connect and see it in action.

Admin Tool

The pymap-admin tool can be used to perform various admin functions against a running pymap server. This is a separate grpc service using grpclib listening on a UNIX socket, typically /tmp/pymap/admin-<pid>.sock.

The admin tool and service have extra dependencies you must install first:

pip install grpclib protobuf

append Command

To append a message directly to a mailbox, without using IMAP, use the append admin command. First, check out the help:

$ pymap-admin append --help

As a basic example, you can append a message to a dict plugin backend like this:

$ cat <<EOF | pymap-admin append demouser
> From: [e-mail]
>
> test message!
> EOF
validity: 1784302999
uid: 101

The output is the UID validity value of the mailbox the message was appended to, and the UID of the appended message. In this example, demouser is the login user and the default mailbox is INBOX.

Supported Extensions

In addition to RFC 3501, pymap supports a number of IMAP extensions to give clients easier and more powerful use.

RFC 2177

Adds the IDLE capability and command, which lets clients wait (without issuing commands) and receive mailbox updates as they happen without polling.

RFC 2180

Defines some conventions for handling multi-access in scenarios such as EXPUNGE and mailbox deletion.

RFC 3502

Adds the MULTIAPPEND capability, allowing multiple messages to be atomically appended to a mailbox with a single APPEND command.

RFC 3516

Adds the BINARY extension, providing better support for the encoding and transfer of binary data.

RFC 4315

Adds the UIDPLUS capability, which adds the UID EXPUNGE command and defines the APPENDUID and COPYUID giving clients more insight into the messages added to a mailbox.

RFC 4466

No additional functionality by itself, but allows pymap to be extended easily and more robustly handle bad client implementations.

RFC 5530

Adds additional IMAP response codes that can help tell an IMAP client why a command failed.

RFC 7889 (partial)

Adds the APPENDLIMIT= capability, declaring the maximum message size a server will accept from an APPEND command. Mailbox-specific limitations defined by the RFC are not supported.

RFC 8474

Adds the OBJECTID capability, assigning unique IDs to mailboxes, messages, and threads to improve client caching and display.

Development and Testing

You will need to do some additional setup to develop and test plugins. First off, I suggest activating a venv. Then, install the test requirements and a local link to the pymap package:

$ pip install -r test/requirements.txt
$ pip install -e .

Run the tests with py.test:

$ py.test

If you intend to create a pull request, you should make sure the full suite of tests run by CI/CD is passing:

$ py.test
$ mypy pymap test
$ flake8 pymap test

A py.test run executes both unit and integration tests. The integration tests use mocked sockets to simulate the sending and receiving of commands and responses, and are kept in the test/server/ subdirectory.

Type Hinting

This project makes heavy use of Python's type hinting system, with the intention of a clean run of mypy:

mypy pymap

No code contribution will be accepted unless it makes every effort to use type hinting to the extent possible and common in the rest of the codebase. There is no need to attempt --strict mode.

The MIT License (MIT)

Copyright (c) 2019 Ian Good

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.