sentry-tablestore

An extension for Sentry which support Alicloud Tablestore (OTS) as a node storage backend.

Objectives

Sentry provides an abstraction called ‘nodestore’ which is used for storing key/value blobs.

It's implemented by BigTable (a service of Google Cloud Platform) in sentry.io.

But for self-hosted Sentry, it's implemented by Django default. This means the large amounts of key-value data were stored in the SQL database.

It's horrible, it can lead to rapid growth in the size of SQL database, and make it difficult to clean up.

For more details from https://forum.sentry.io/t/postgres-nodestore-node-table-124gb/12753/3 .

Therefore, this project came into being. It provides a new solution in addition to Django Database or BigTable, that is Alicloud Tablestore (OTS).

Features

• Implement the TablestoreNodeStorage backend.
• Support migrating data from current nodestore backend to the new one.

Prerequisites

Sentry 21.9.0 or higher.

An Tablestore instance of Alibaba Cloud (International) or Aliyun (China).

Installation

To install, simply add sentry-tablestore to your requirements.txt for your Sentry environment (or pip install sentry-tablestore).

Configuration

To configure Sentry to use this module, set sentry_tablestore.nodestore.backend.TablestoreNodeStorage to your SENTRY_NODESTORE in your sentry.conf.py, like this:

from datetime import timedelta

SENTRY_NODESTORE = 'sentry_tablestore.nodestore.backend.TablestoreNodeStorage'
SENTRY_NODESTORE_OPTIONS = {
    # Get help from https://www.alibabacloud.com/help/tablestore/latest/python-sdk-initialization
    'end_point': '',
    'access_key_id': '',
    'access_key_secret': '',
}

Then, add any applicable configuration options. Depending on your environment, and especially if you are running Sentry in containers, you might consider using python-decouple so you can set these options via environment variables.

Example Configuration

from datetime import timedelta
SENTRY_NODESTORE = 'sentry_tablestore.nodestore.backend.TablestoreNodeStorage'
SENTRY_NODESTORE_OPTIONS = {
    # Auto clean data for 90 days from its creation
    'default_ttl': timedelta(days=90),
    'automatic_expiry': True,

    # Get help from https://www.alibabacloud.com/help/tablestore/latest/python-sdk-initialization
    'end_point': 'https://sentry-self-hosted.cn-shanghai.vpc.tablestore.aliyuncs.com',
    'access_key_id': 'LTAI****************LYhz',
    'access_key_secret': 'MKs5**********************6t0J',

    # A boolean whether to enable zlib-compression, or the string "zstd" to use zstd.
    # Use zstd compression to override the default zlib for more compact and more efficient
    'compression': 'zstd'
}

# Set log-level for debugging
# Make sure LOGGING.disable_existing_loggers is set to False (in sentry/conf/server.py)
import logging
logger = logging.getLogger('sentry_tablestore.kvstore.tablestore') # Logger of this project
logger.setLevel(logging.DEBUG)
logger = logging.getLogger('tablestore-client') # Logger of Alicloud SDK
logger.setLevel(logging.DEBUG)

Bootstrap

Execute the command to bootstrap tablestore (create or update the table):

sentry upgrade --with-nodestore