twempest 0.2.9

Twitter to text via template. Somehow that abbreviates to "twempest".

Twempest is a command-line program written in Python.

Motivation

Twempest was born of the need to periodically echo my Twitter tweets as posts on first a Jekyll-based and then Pelican-built blog. That’s it, that’s all.

Licence

Copyright 2018 Dave Rogers. Licensed under the GNU General Public License, version 3. Refer to the attached LICENSE file or see http://www.gnu.org/licenses/ for details.

Change Log

The current version is 0.2.9. Twempest is semver-ish in its versioning scheme.

Twempest is currently a beta release candidate in preparation for version 1.0.0.

Installation

Homebrew (macOS)

On macOS, Homebrew will take care of installing any dependencies, including Python 3.

brew tap yukondude/tap
brew install twempest

PyPI (POSIX)

On *NIX, you will first need to install Python 3.6 (or higher) using your preferred method.

pip3 install twempest

Unicode Locale Configuration

If the Unicode locale is not configured in a particular environment (e.g., cron), you will see the following error:

RuntimeError: Click will abort further execution because Python 3
was configured to use ASCII as encoding for the environment.
Consult http://click.pocoo.org/python3/for mitigation steps.

To solve this, you must explicitly export the Unicode locale in the environment. For example, insert something like the following near the top of your crontab file:

LC_ALL=en_CA.UTF-8
LANG=en_CA.UTF-8

Use the locale appropriate for your language and region. The error message should show the list of available locales.

Development Setup

1. Make sure that Python 3.6 or better is installed.

2. Install pipenv: pip install pipenv

3. Clone the Twempest repo: git clone https://github.com/yukondude/Twempest.git ; cd Twempest

4. Install dependencies: pipenv install --dev --three

5. Install the project in development mode: pipenv run ./setup.py develop

6. Run the unit tests to make sure everything is copacetic: pipenv run ./setup.py test

7. Pour a snifter of Château de Montifaud and light up a Laranja Reserva Toro.

While developing, you will always have to run the Twempest CLI from within the pipenv virtual environment: pipenv run twempest

Usage

Run twempest --help to view the usage instructions:

Usage: twempest [OPTIONS] TEMPLATE

  Download a sequence of recent Twitter tweets and convert these, via the
  given template file, to text format. Twempest uses the Jinja template
  syntax throughout: http://jinja.pocoo.org/docs/2.9/templates/

Options:
  -c, --config-path TEXT  Twempest configuration directory path, which must be
                          writable, and must also contain the twempest.conf
                          file.  [default: ~/.twempest]
  -A, --abbreviated       Render the abbreviated form of the tweet text rather
                          than the full, extended, version.
  -a, --append            Append rendered tweet(s) to existing file(s) rather
                          than skipping past with a warning.
  -n, --count TEXT        Maximum number of tweets to retrieve. The actual
                          number may be lower.  [default: 200]
  -D, --dry-run           Display all configuration options and template
                          contents without retrieving tweets.
  -i, --image-path TEXT   The directory path (template tags allowed) to write
                          downloaded image (media type == 'photo') files. The
                          directory path will be created if it doesn't exist.
                          Media file names use the --render-file name followed
                          by a number and the appropriate file extension. If
                          omitted, media files will not be downloaded.
  -u, --image-url TEXT    The URL path (template tags allowed) to use for all
                          image files downloaded via the --image-path option.
  --pickle                Serialize a list of the rendered tweet statuses as a
                          standard Python pickle byte stream. The stream will
                          be written to 'twempest.p' in the current working
                          directory.
  -q, --quiet             Suppress warning messages.
  -f, --render-file TEXT  The file name (template tags allowed) for the
                          rendered tweets. If omitted, tweets will be rendered
                          to STDOUT.
  -p, --render-path TEXT  The directory path (template tags allowed) to write
                          the rendered tweet files. The directory path will be
                          created if it doesn't exist.  [default: .]
  -@, --replies           Include @replies in the list of retrieved tweets.
  -r, --retweets          Include retweets in the list of retrieved tweets.
  -s, --since-id TEXT     Retrieve tweets that follow this ID in the timeline.
                          Required, unless the ID has already been recorded in
                          the config path directory after a previous run of
                          Twempest.
  -k, --skip TEXT         Skip any rendered tweets that contain this regular
                          expression pattern.
  -V, --version           Show version and exit.
  -h, --help              Show this message and exit.

Sample Configuration

Contents of twempest.config.sample:

# Sample Twempest configuration file. See https://github.com/yukondude/Twempest
# for details. Save this to ~/.twempest/twempest.config as the default
# configuration whenever twempest is run, or save it somewhere convenient as
# twempest.config and reference it via the -c/--config-path command-line switch.

[twempest]
# Most twempest long-form command-line switches may be used here (excluding the
# leading double-dash). The obvious exceptions would include --config-path,
# --help, and --version, but go ahead and try them if you like. The commented-
# out defaults are shown below. See the --help output for details.

# Render the full text of the tweet.
# abbreviated=false

# Do not append to existing files.
# append=false

# Retrieve at most 200 tweets.
# count=200

# Retrieve tweets normally.
# dry-run=false

# Do not download image files.
# image-path=

# Do not download image files.
# image-url=

# Do not serialize the rendered tweets.
# pickle=false

# Do not suppress warning messages.
# quiet=false

# Render tweets to STDOUT.
# render-file=
# Because template expressions are allowed for this option, you can generate
# rendered file names using any of the tweet context variable contents. For
# example: render-file={{tweet.created_at|isodate}}-{{tweet.text|slugify}}.md
# might render to something like: 2017-02-14-be-my-valentine.md
# Use the slugify filter to eliminate any non-filesystem-safe characters from
# the tweet text.

# Write rendered tweets to the current directory.
# render-path=.
# Template expressions are also allowed for this option, so the directory path
# can be made to change based upon a tweet status variable.

# Exclude @replies from the list of retrieved tweets.
# replies=false

# Exclude retweets from the list of retrieved tweets.
# retweets=false

# Don't specify a most recent Twitter ID.
# since-id=
# Since this isn't specified (and normally wouldn't be in a config file), there
# must already be an ID recorded in the config path directory after a previous
# run of Twempest.

# Don't skip any tweets.
# skip=

[twitter]
# Visit https://apps.twitter.com/ to generate these keys, secrets, tokens, and
# token secrets. Secret tokens? Token keys? Secret secrets?
consumer_key=
consumer_secret=
access_token=
access_token_secret=

Sample Template

A simple template to render a tweet as Markdown text suitable for Jekyll posts (twempest.template.sample):

---
title: '{{ tweet.text|delink|truncate(80,False)|qescape }}'
author: '{{ tweet.user.name|qescape }}'
date: '{{ tweet.created_at }}'
tweet_id: {{ tweet.id }}
---
{{ tweet.text | relink("[{{ text }}]({{ url }})") | reimage("![{{ alt }}]({{ url }})", "\n\n") }}

[tweet](https://twitter.com/{{tweet.user.screen_name}}/status/{{ tweet.id }})

The rendered output of this template might look something like the following:

---
title: 'Ice fog 'boiling' up from the Yukon River.'
author: 'Dave Rogers'
date: '2016-12-06 12:12:36-08:00'
tweet_id: 806229878861201408
---
Ice fog 'boiling' up from the [#Yukon](https://twitter.com/hashtag/yukon) River.

![2016-12-06-ice-fog-boiling-up-from-the-yukon-river-0](/media/2016-12-06-ice-fog-boiling-up-from-the-yukon-river-0.jpg)

[tweet](https://twitter.com/yukondude/status/806229878861201408)

tweet Context Variable

See the Twitter API documentation for tweets for a list of all of the keys that can be found under the tweet context variable (a dictionary).

A couple of other keys are also available:

tweet.media[].original_media_url

The original value of the media_url key within the list of media items before any downloaded image URL rewriting took place.

tweet.media[].original_media_url_https

The original value of the media_url_https key within the list of media items before any downloaded image URL rewriting took place.

Template Filters

These are in addition to the built-in Jinja2 filters.

delink

Remove URLs and hashtag ‘#’ prefixes.

isodate

Format a date as YYYY-MM-DD.

qescape

Escape just single quote characters as HTML entities.

reimage(tag_format, delimiter=" ")

Remove image URLs and append them to the end (following the delimiter), using the template tag_format with variables alt and url to format each.

relink(tag_format)

Replace non-image URLs, hashtag, and user mention links, using the template tag_format with variables text and url to format each.

slugify

Transform the given text into a suitable file name that is also scrubbed of URLs and hashtags.

README.md generated December 10, 2018