emrap 1.0.1

eMail Requestor and Processor

eMail Requestor and Processor

OVERVIEW

The general intention for this package is to automate searching and extracting text from emails and take actions using found values.

Currently there is support for searching in Gmail and sending HTTP requests prior or following email search. HTTP responses can also be searched and text found in either emails or responses can be dynamically substituted in subsequent HTTP requests.

The general flow looks like this:

1. Send the specified pre-search requests and extract values from the responses matching the given regular expressions
2. Search for emails matching the given query
3. Send the specified post-search requests and extract values from the responses matching the given regular expressions. Named placeholders in those requests are replaced with whatever has matched in emails or responses to pre-search requests.
4. Repeat: go back to 1. but now named placeholders in the pre-search requests are replaced with whatever has matched in emails or responses to post-search requests from the previous run.

An example usage is a password reset where you send a request to get a password reset, fetch the email and extract the reset code, send another request with the code in it.

SETUP

Installation

pip3 install emrap

Creating an OAuth client

This package uses the Gmail OAuth2 API and requires credentials for an OAuth client, which you can create in the Google dev console (no GSuite needed). Then you need to authorize the app to access the emails of the target account (which can be any Google account, not just the one that owns the client credentials). Follow these steps:

1. Go to https://console.developers.google.com/apis/ and create a project.

2. Go to https://console.developers.google.com/apis/credentials/consent and configure your app. Select the project if not selected, then choose between Internal or External app. Enter your app name and click on Save and continue several times until you finish that wizard and then Back to Dashboard.

3. Go to https://console.developers.google.com/apis/credentials and create credentials of type OAuth client ID. Select Desktop App as the Application Type and name it.

4. Click on the download button to download your credentials JSON file (does not work in Firefox). Rename it to credentials.json and put it in the same directory as the python client.

5. Go to https://console.developers.google.com/apis/library/gmail.googleapis.com?q=gmail api and enable to Gmail API.

6. Start the client which will open the OAuth consent screen in the browser. Follow the prompts to allow it, ignoring the warning it's unverified (you'll need to get it verified if you want to use it for non-testing purposes). You'll only need to authorise this once.

USAGE

Import actions.HttpAutomator and run it. See the demos for examples.

usage: demo.py --creds-file CREDS_FILE [--token-file TOKEN_FILE]
               [--query QUERY]
               [--latest | --earliest | --range MIN MAX MIN MAX]
               [--extracts directive:value [directive:value ...]
               [directive:value [directive:value ...] ...]]
               [--mime-type MIME_TYPE]
               [--pre-search PRE_SEARCH [PRE_SEARCH ...] | --pre-search-async
               PRE_SEARCH_ASYNC [PRE_SEARCH_ASYNC ...]]
               [--post-search POST_SEARCH [POST_SEARCH ...] |
               --post-search-async POST_SEARCH_ASYNC [POST_SEARCH_ASYNC ...]]
               [--origin ORIGIN | --is-ssl IS_SSL] [--max-threads MAX_THREADS]
               [--timeout TIMEOUT] [--min-wait MIN_WAIT] [--max-wait MAX_WAIT]
               [--success-codes 2XX 3XX [2XX 3XX ...] | --fail-codes 4XX 5XX
               [4XX 5XX ...]] [--retry [N]] [--no-verify] [--no-redirect]
               [--proxy PROXY | --raw-socket] [--output OUTPUT]
               [--repeat REPEAT] [--cache-file CACHE_FILE] [--verbose]
               [--help]

HTTP request automator with email search capability. An automation app to send
HTTP requests which may generate an email, search in gmail and extract value,
then optionally send more requests. Good for automarting password resets for
example. It can be used just to search in emails. Supports dynamic text
substitution with values extracted from previous responses or emails.

Options related to authentication:
  --creds-file CREDS_FILE, -c CREDS_FILE
                        JSON file containing OAuth client credentials
                        (default: None)
  --token-file TOKEN_FILE, -t TOKEN_FILE
                        Previously saved token pickle file containing user
                        access token. If it doesn't exist it is created.
                        (default: None)

Options related to filtering emails:
  --query QUERY, -q QUERY
                        Search query to filter emails. See
                        https://support.google.com/mail/answer/7190?hl=en.
                        (default: None)
  --latest, -l          Take only the latest email that matches. (default:
                        None)
  --earliest, -e        Take only the earliest email that matches. (default:
                        None)
  --range MIN MAX MIN MAX, -r MIN MAX MIN MAX
                        Take only emails number MIN to MAX (including) when
                        sorted in descending order by time. (default: None)

Options related to extracting text from responses or
            email:
  --extracts directive:value [directive:value ...] [directive:value [directive:value ...] ...], -E directive:value [directive:value ...] [directive:value [directive:value ...] ...]
                        This specifies how and where to search for and extract
                        text and gives a name each extracted set of values
                        that can be used to substitute in HTTP requests. The
                        flag is given once per substitution definition and can
                        be given as many times as needed. It takes one or more
                        of the following directives (each as a separate
                        argument). There should not be a space around either
                        side of the colon unless intended, e.g. if the
                        'default' directive value should contain leading
                        spaces.
                            name:{unique name} HTTP requests can then use
                                '{%name%}' which will be substituted by
                                whatever matched name. By default the first
                                match is taken, specify another one by
                        appending
                                _N, e.g. '{%name_2%}' for the second one.
                                The results table will list all matches along
                                with the name regardless if any HTTP requests
                                were supplied.
                            stages:pre|fetch|post fetch matches in emails and
                                pre|post in pre- or post- email search. If not
                                given, then all three stages are searched.
                            where:to_address,from_address,subject,raw_headers,
                        body,content,time
                                Any of these can be given. to/from_address,
                                subject and body apply to emails, raw_headers
                                and content to HTTP responses, time applies to
                                both. Defaults to search headers/subject and
                                body/content.
                            regex:{arbitrary regex} Emails (stage fetch) are
                                searched in descending order of time and
                                requests (stages pre and post) are searched in
                                order they were received.
                            regex_group:{number} Default is to take the whole
                                match.
                            regex_flags:{flags} One letter per flag, e.g. IM
                        for
                                case Insensitive and Multiline (see doc on
                                python.re).
                            default:{arbitrary text} Text to use if regex
                                doesn't match. Default is empty. (default:
                        None)
  --mime-type MIME_TYPE, -M MIME_TYPE
                        Preferred MIME type to search in. Only applies to
                        searching in multipart emails. (default: text/plain)

Options related to sending requests. HTTP requests are
            read from text files as raw HTTP requests. Dynamic
            replacements are supported.:
  --pre-search PRE_SEARCH [PRE_SEARCH ...], -b PRE_SEARCH [PRE_SEARCH ...]
                        List of files to send as HTTP requests before email
                        search. Requests will be sent synchronously, one after
                        the other. (default: None)
  --pre-search-async PRE_SEARCH_ASYNC [PRE_SEARCH_ASYNC ...], -B PRE_SEARCH_ASYNC [PRE_SEARCH_ASYNC ...]
                        List of files to send as HTTP requests before email
                        search. Requests will be sent by concurrent threads.
                        (default: None)
  --post-search POST_SEARCH [POST_SEARCH ...], -a POST_SEARCH [POST_SEARCH ...]
                        List of files to send as HTTP requests after email
                        search. Executed for each email that matched the
                        search. Requests will be sent synchronously, one after
                        the other. (default: None)
  --post-search-async POST_SEARCH_ASYNC [POST_SEARCH_ASYNC ...], -A POST_SEARCH_ASYNC [POST_SEARCH_ASYNC ...]
                        List of files to send as HTTP requests after email
                        search. Executed for each email that matched the
                        search. Requests will be sent by concurrent threads.
                        (default: None)
  --origin ORIGIN, -O ORIGIN
                        Specify origin as {protocol}://{host}[:{port}]. This
                        overrides the Host header for all requests. (default:
                        None)
  --is-ssl IS_SSL       Use SSL. This overrides guessing from the port number
                        in the Host header for all requests. If no port is
                        present in the Host header, it defaults to HTTP on
                        port 80. (default: None)
  --max-threads MAX_THREADS
                        Maximum number of threads to start in async mode.
                        (default: 5)
  --timeout TIMEOUT, -T TIMEOUT
                        Timeout in seconds for response (default: None)
  --min-wait MIN_WAIT, -w MIN_WAIT
                        Min seconds to wait before sending next request.
                        (default: 0)
  --max-wait MAX_WAIT, -W MAX_WAIT
                        Max seconds to wait before sending next request.
                        (default: 0)
  --success-codes 2XX 3XX [2XX 3XX ...]
                        List of codes considered a success. X matches any
                        digit. (default: ['2XX'])
  --fail-codes 4XX 5XX [4XX 5XX ...]
                        List of codes considered a failure. X matches any
                        digit. (default: None)
  --retry [N], -R [N]   Retry N times on failed response. N defaults to 1 if
                        omitted. (default: 0)
  --no-verify           Do not verify SSL certificate and ignore warnings.
                        (default: True)
  --no-redirect         Do not follow redirections. (default: True)
  --proxy PROXY, -x PROXY
                        HTTP proxy to use in the format of {hostname}:{port}.
                        (default: None)
  --raw-socket, -S      Send the request content over a raw IPv4 socket.
                        Default is to use python's requests package which may
                        add some headers as well as change Content-Length.
                        Using a proxy, following redirections and SSL
                        verification is not supported when using raw sockets.
                        (default: False)
  --repeat REPEAT, -n REPEAT
                        How many times to repeat each
                        [request->]search[->request cycle]. Extracted text
                        from previous cycles is available for substitution.
                        (default: 1)

Options related to displaying output:
  --output OUTPUT, -o OUTPUT
                        Text file to print summary table to. '-' means
                        standard output. (default: None)

Global options :
  --cache-file CACHE_FILE, -C CACHE_FILE
                        Previously saved cache to restore from. If it doesn't
                        exist it is created. (default: None)
  --verbose, -v         Be verbose. Can be given multiple times to increase
                        verbosity. Twice is for debugging, three times is to
                        flood your screen with virtually useless information.
                        (default: None)
  --help, -h            Show this help message and exit.