Gnar Gear: Gnarly Python Apps

Part of Project Gnar:  base  •  gear  •  piste  •  off-piste  •  edge  •  powder  •  genesis  •  patrol

Get started with Project Gnar on  

Join Project Gnar on  

Support Project Gnar on  

Gnar Gear sets up a powerful Flask-based Python service with two lines of code:

from gnar_gear import GnarApp

...

GnarApp('my_gnarly_app', production=True, port=80).run()

Installation

pip3 install gnar-gear

Feature List

• Flask app with auto blueprint registration
• Flask WSGI Development Server
• Bjoern WSGI Production Server
  • Why Bjoern? Check out these benchmarks!
  • And also these benchmarks
• Postgres database connection via Postgres.py
• SES client connection via Boto 3
• JWT configuration via Flask-JWT-Extended
• Peer requests - HTTP requests to other microservices in the app
• External requests - Convenience wrapper around requests
• SQS Message Polling and Sending
• argon2_cffi.PasswordHasher instance
  • Argon2 was the winner of the 2015 Password Hashing Competition
• Logger configuration
• Error handler with traceback
• Overridable and extendable class-based design

Development Mode

• Flask WSGI server in fault-tolerant debug (watch & reload) mode
  • The module loader logs the stack trace of any modules that fail to load
  • If a module fails to load, the app continues to watch for file changes
• CORS "disabled" (Access-Control response headers set), so you don't need to use other means to circumvent CORS

Production Mode

• Bjoern WSGI server
• CORS "enabled" (Access-Control response headers are not set)

Requirements

Bjoern

Bjoern requires libev (high performance event loop)

• Install libev with brew install libev on Mac, or find your platform-specific installation command here

Application Structure

GnarApp expects to be instantiated in main.py at <top-level-module>/app, i.e. the minimum app folder structure is

+ <top-level-module>
    + app
        main.py
    __init__.py

It is recommended (not required) to place your apis in segregated folders under app and the tests in a test folder under the <top-level-module>, e.g.

+ <top-level-module>
    + app
        + admin
            apis.py
            constants.py
            services.py
        + user
            apis.py
            constants.py
            services.py
        __init__.py
        main.py
    + test
        constants.py
        test_app.py
    __init__.py

Blueprints

Each Flask Blueprint must be assigned to a global-level blueprint variable in its module, e.g.

from flask import Blueprint, jsonify


api_name = 'user'
url_prefix = '/{}'.format(api_name)

blueprint = Blueprint(api_name, __name__, url_prefix=url_prefix)
^^^^^^^^^

@blueprint.route('/get', methods=['GET'])
def user_get():
    return jsonify({'status': 'ok'})

By default, the GnarApp picks up every blueprint in an auto-scan of the application code.

Overview

The GnarApp class provides a highly configurable, feature-rich, production-ready Flask-based app.

Parameters

Args (required)

• name: The name of the application's top-level module
• production: Boolean flag indicating whether or not the build is in production mode
• port: The port to bind to the WSGI server

Kwargs (optional)

• env_prefix: Environment variable prefix (defaults to GNAR)
• log_level: Log level override
  • see configure_logger for log level overview
• blueprint_modules: List of modules to find Flask blueprints (default is auto-scan)
  • see configure_blueprints for blueprints overview
• sqs: List (or single dict) of SQS queues to poll
  • see configure_sqs_polls for SQS polling overview
• no_db: Boolean flag - specify True if the app does not need a Postgres connection
• no_jwt: Boolean flag - specify True if the app does not use JWT headers (i.e. non-api services)

Overridable Behavior

GnarApp.run simply calls a set of steps in the class. Here is an example of how to override any of the steps:

def postconfig():
    log.info('My Postconfig Step!')

ga = GnarApp('my_gnarly_app', production=True, port=80)
ga.postconfig = postconfig
ga.run()

Run Steps

The run steps rely on a set of environment variables which use a configurable prefix (i.e. the env_prefix parameter). The default env_prefix is GNAR. An example of a Gnar environment variable using a custom prefix is GnarApp( ..., env_prefix='MY_APP') and then instead of reading GNAR_LOG_LEVEL, the configure_logger step will read MY_APP_LOG_LEVEL.

preconfig

• No default behavior - provided as an optional initial step in the app configuration.

configure_flask

• Attaches a Flask instance to the Gnar app.

configure_logger

• Attaches the root logger to sys.stdout.
• Sets the logging level to the first defined:
  • log_level parameter
  • GNAR_<app name>_LOG_LEVEL environment variable, e.g. GNAR_MY_GNARLY_APP_LOG_LEVEL
  • GNAR_LOG_LEVEL environment variable
  • INFO
  • Reminder: Valid settings (in increasing order of severity) are DEBUG, INFO, WARNING, ERROR, CRITICAL
• Sets the log format to the first defined:
  • GNAR_LOG_FORMAT
  • '%(asctime)s %(levelname)-8s %(name)s:%(lineno)d %(message)s', e.g.:

  2018-07-09 15:41:46.420 INFO     gear.gnar_app:75   Logging at INFO
  

• Sets the log format default_msec_format to the first defined:
  • GNAR_LOG_FORMAT_MSEC
  • '%s.%03d' (e.g. .001)

configure_argon2

• Attaches an argon2_cffi.PasswordHasher instance to the Gnar app.

• Reads the following environment variables [TYPE: DEFAULT] to pass into the Argon2 instance:

  • GNAR_ARGON2_TIME_COST: [INT: 2] Number of iterations to perform
  • GNAR_ARGON2_MEMORY_COST: [INT: 512] Amount of memory (in KB) to use
  • GNAR_ARGON2_PARALLELISM: [INT: 2] Number of parallel threads (changes the resulting hash value)
  • GNAR_ARGON2_HASH_LEN: [INT: 16] Length of the hash in bytes
  • GNAR_ARGON2_SALT_LEN: [INT: 16] Length of random salt to be generated for each password in bytes
  • GNAR_ARGON2_ENCODING: [STR: 'utf-8'] Encoding to use when a string is passed into hash or verify

• Note from the docs:

  Only tweak these if you’ve determined using CLI that these defaults are too slow or too fast for your use case.

• To hash a password using Argon2:

  from <top-level-module>.main import app
  hash = app.generate_password_hash(<plain text password>)
  # OR
  hash = app.argon2.hash(<plain text password>)
  

  Note that this creates a randomly salted, memory-hard hash using the Argon2i algorithm.

• To validate a password with Argon2:

  from <top-level-module>.main import app
  is_valid = app.check_password_hash(<password hash from database>, <plain text password>)
  # OR
  is_valid = app.argon2.verify(<password hash from database>, <plain text password>)
  

  Note that app.argon2.verify raises an exception if the password is invalid whereas app.check_password_hash does not.

configure_database

• Creates a Postgres database connection and attaches it to the Gnar app
• Reads the following environment variables to set the host, dbname, user, password connection string parameters, respectively:
  • GNAR_PG_ENDPOINT
  • GNAR_PG_DATABASE
  • GNAR_PG_USERNAME
  • GNAR_PG_PASSWORD
• Note: The Postgres API primarily consists of run, one, and all

attach_instance

• Attaches the GnarApp instance to the app.main module. This enables easy access to the Gnar app from anywhere in the application using

  from <top-level-module>.main import app
  

• The GnarApp's runtime assets are db, argon2, flask, check_password_hash, generate_password_hash, and get_ses_client

• For example, to fetch one result (or None) from the database:

  app.db.one("SELECT * FROM foo WHERE bar='buz'")
  

configure_blueprints

• By default, GnarApp auto-scans every Python module under the app folder for blueprints.
• Each Flask Blueprint must be assigned to a global-level blueprint variable in its module.
• If you prefer to skip the auto-scan, you can provide a list (or single string) of blueprint modules.
  • Each item in the list of module names may use one of two formats:
    • Without a . in the module name: GnarApp will look for the module in <top-level-module>.app.<module name>.apis
    • With a . in the module: GnarApp will look for the module in <top-level-module>.app.<module name>

configure_errorhandler

• Defines a generic (Exception-level) Flask error handler which:
  • Logs the error message and its traceback (format_exec)
  • Returns a 200-level json response containing {"error": <error message>, "traceback": <traceback>}

configure_jwt

• Sets the Flask JWT_SECRET_KEY variable to the value of the GNAR_JWT_SECRET_KEY environment variable.
• Sets the Flask JWT_ACCESS_TOKEN_EXPIRES variable to the value of the GNAR_JWT_ACCESS_TOKEN_EXPIRES_MINUTES environment variable (default 15 mins).
• Attaches a JWTManager instance to the GnarApp.
• Defines functions for expired_token_loader, invalid_token_loader, and unauthorized_loader which return meaningful error messages as 200-level json responses containing {"error": <error message>}.

configure_sqs_polls

• Configures a set of polls to run at specified intervals to receive messages from SQS queues.

• Specified via the sqs kwarg as a list of dicts (or a single dict) with a set of required (*) and optional properties. See the AWS ReceiveMessage docs for more details on the optional properties:

  • * queue_name: The name of the SQS queue - the queue URL is retrieved from AWS with this name
  • * callback: The function to call with each received message. The message will be deleted from the queue unless the callback function returns False.
  • interval_seconds: The interval (in seconds) between receive message requests - default 60
  • attribute_names: A list of attributes to return with each message
  • max_number_of_messages: The maximum number of messages to return with each request - 1 to 10, default 1
  • message_attribute_names: Metadata to include with the message - more details
  • receive_request_attempt_id: The token used for deduplication of ReceiveMessage calls (FIFO queues only)
  • visibility_timeout: The duration (in seconds) that the received messages are hidden from subsequent retrieve requests
  • wait_time_seconds: The duration (in seconds) for which the call waits for a message to arrive in the queue before returning

• Each message received by the callback includes the following properties. See the AWS Message docs for more details on these properties:

  • Body: Message contents
  • Attributes: Map of requested attributes
  • MD5OfBody: MD5 Digest of the message body
  • MD5OfMessageAttributes: MD5 digest of the message attribute string
  • MessageAttributes: Map of custom message metadata

• Notes:

  • Messages are processed sequentially in the order they are received
  • The callback blocks message processing until it's complete
  • After all received messages are processed, the queue is immediately checked for new messages
  • Once the receive_message call returns no messages, a Timer with the specified interval_seconds starts the next loop

• Reads the following environment variables to set the region_name, aws_access_key_id, and aws_secret_access_key parameters of the boto3.client call, respectively:

  • GNAR_SQS_REGION_NAME
  • GNAR_SQS_ACCESS_KEY_ID
  • GNAR_SQS_SECRET_ACCESS_KEY

• Example:

  def receive_sqs_message(message):
    redis_connection.set(message['MessageId'], message['Body'], 3600)
  
  sqs = {'queue_name': 'gnar-queue', 'callback': receive_sqs_message}
  GnarApp('piste', production, port, sqs=sqs).run()
  

configure_after_request

• Adds a JWT Authorization header (Bearer token) to responses which received a valid JWT token in the request.
• In development mode, adds CORS headers to the response (so that you don't need to bother circumventing CORS).

postconfig

• No default behavior - provided as an optional initial step in the app configuration.

Runtime Functionality

peer

• Handles requests to other microservices in a Kubernetes app.

• Usage:

  from <top-level-module>.main import app
  peer = app.peer(<< service name >>)
  

• Requires environment variables in the form:

  • production: << SERVICE >>_SERVICE_PORT, e.g. PISTE_SERVICE_PORT
  • development: GNAR_<< SERVICE >>_SERVICE_PORT, e.g. GNAR_PISTE_SERVICE_PORT

• In production, the environment variable is automatically set by Kubernetes

• In development, the environment variable should point to another Gnar Gear app on localhost on a separate port

• The environment variable value must be in the form << ip address >>:<< port >>, e.g.

  export GNAR_PISTE_SERVICE_PORT='127.0.0.1:9401'
  

peer Methods

• Each peer method accepts all the kwargs of the underlying requests method in addition to an auto_auth property.

  • If auto_auth is True and the current route is protected, an Authorization header is added to the request.
  • Both microservices must use the same GNAR_JWT_SECRET_KEY for the authorization to work.

• Uses requests under the hood:

  • request(method, path, **kwargs): Constructs and sends a request
  • head(path, **kwargs): Sends a HEAD request
  • get(path, params=None, **kwargs): Sends a GET request
  • post(path, data=None, json=None, **kwargs): Sends a POST request
  • put(path, data=None, **kwargs): Sends a PUT request
  • patch(path, data=None, **kwargs): Sends a PATCH request
  • delete(path, **kwargs): Sends a DELETE request

• Example:

  from <top-level-module>.main import app
  response = app.peer('piste').post('login', {'email': 'ic3b3rg@gmail.com'})
  

external

• Convenience wrapper around requests

• Example:

  from <top-level-module>.main import app
  response = app.external.get('https://api.pwnedpasswords.com/range/5ce7d')
  

generate_password_hash / check_password_hash

• Convenience wrappers for app.argon2.hash and app.argon2.verify

• Usage:

  from <top-level-module>.main import app
  hash = app.generate_password_hash(<plain text password>)
  is_valid = app.check_password_hash(<password hash from database>, <plain text password>)
  

get_ses_client

• Exposed as runtime functionality (as opposed to creating the client at initialization) because AWS will close the client after a short period of time

• Returns an SES connection (using boto3)

• Reads the following environment variables to set the region_name, aws_access_key_id, and aws_secret_access_key parameters of the boto3.client call, respectively:

  • GNAR_SES_REGION_NAME
  • GNAR_SES_ACCESS_KEY_ID
  • GNAR_SES_SECRET_ACCESS_KEY

• Usage:

  from <top-level-module>.main import app
  app.get_ses_client().send_email( ... )
  

• See the Boto 3 Docs for the send_email request syntax.

send_sqs_message

• Sends a message to an SQS queue.

• Args (required):

  • queue_name: The name of the SQS queue - the queue URL is retrieved from AWS with this name
  • message_body: Message contents

• Kwargs (optional) - See the AWS SendMessage docs for more details on these properties:

  • delay_seconds: The length of time, in seconds, for which to delay a specific message
  • message_attributes: Metadata to include with the message
  • message_deduplication_id: The token used for deduplication of sent messages (FIFO queues only)
  • message_group_id: The tag that specifies that a message belongs to a specific message group (FIFO queues only)

• Returns a dict with the following attributes:

  • MessageId: Unique message identifier
  • MD5OfMessageAttributes: MD5 digest of the message attribute string
  • MD5OfMessageBody: MD5 digest of the message body
  • SequenceNumber: Unique, non-consecutive, 128 bit string (FIFO queues only)

• Reads the following environment variables to set the region_name, aws_access_key_id, and aws_secret_access_key parameters of the boto3.client call, respectively:

  • GNAR_SQS_REGION_NAME
  • GNAR_SQS_ACCESS_KEY_ID
  • GNAR_SQS_SECRET_ACCESS_KEY

• Example:

  from <top-level-module>.main import app
  message = app.send_sqs_message('gnar-queue', 'The gnarliest gear in the world 🤙')
  

Environment Variables

• The environment variables (with configurable prefix) used by GnarApp are:
  • GNAR_ARGON2_ENCODING
  • GNAR_ARGON2_HASH_LEN
  • GNAR_ARGON2_MEMORY_COST
  • GNAR_ARGON2_PARALLELISM
  • GNAR_ARGON2_SALT_LEN
  • GNAR_ARGON2_TIME_COST
  • GNAR_JWT_SECRET_KEY
  • GNAR_LOG_LEVEL
  • GNAR_PG_DATABASE
  • GNAR_PG_ENDPOINT
  • GNAR_PG_PASSWORD
  • GNAR_PG_USERNAME
  • GNAR_SES_ACCESS_KEY_ID
  • GNAR_SES_REGION_NAME
  • GNAR_SES_SECRET_ACCESS_KEY
  • GNAR_SQS_REGION_NAME
  • GNAR_SQS_ACCESS_KEY_ID
  • GNAR_SQS_SECRET_ACCESS_KEY
• See the relevant sections above for details

Made with by Brien Givens