sqlogging

A logger that is based on Python's sqlite3 library. Log entries are stored in a SQLite table and can be accessed with SQL queries. It is inspired by the logging library, but does not stay strictly faithful to the API.

The strength of sqlogging isn't speed. It typically takes a few milliseconds to write a log entry (about 7 ms on my machine). But if that's not a blocker for you, the accessibility and flexibility of analysis it gives is a sheer delight. Skip json parsing and unwieldy pandas syntax. If you speak SQL you can make your log data dance for you.

Installation

pip install sqlite-logging

Usage

from sqlogging import logging

logger = logging.create_logger(name="test_logger", columns=["iter", "score"])
logger.info({"iter": 0, "score": .4})
logger.info({"iter": 1, "score": .1})
logger.info({"iter": 2, "score": .8})

result = logger.query(f"SELECT SUM(score) FROM {logger.name}")
print("sum of scores:", result[0][0])

logger.delete()

API

create_logger()

logging.create_logger(name="log", dir_name=".", level="info", columns=["ts", "data"])

For creating a new Logger from scratch. If you try to create a Logger by the same name as a pre-existing logger you'll get a sqlite3.OperationalError.

• Parameters
  • name(str) - The name of the Logger. This will be both the name of the name of the table and the name of the sqlite3 database file (<name>.db).
  • dir_name(str) - The directory in which the database file will be stored. If it doesn't already exist, it will be created.
  • level(str) - The logging severity level. Must be one of {'debug', 'info', 'warning', 'error', 'critical'} (case insensitive). Only log messages of equal or higher severity will be processed.
  • columns(List of str) - The names of the columns to be created in the sqlite database.
• Return type: Logger
• Raises:
  • ValueError: If level is not one of the 5 allowed levels.

open_logger()

logging.open_logger(name="log", dir_name=".", level="info")

For re-opening an existing Logger. If you try to open a Logger that doesn't already exist you'll get a RuntimeError.

• Parameters: as in create_logger()
• Return type: Logger

Logger

class logging.Logger(name, dir_name, level="info", columns=None, create=True)

• Parameters: as in create_logger()
  • create(bool) - Whether a new Logger should be created or an existing one re-opened.

close()

Close the connection to the logger database. Can be reopened later with logging.open_logger().

delete()

Close the connection to the database and delete the database file. Remove it from existence.

debug(data) \ info(data) \ warning(data) \ error(data) \ critical(data)

• Parameters
  • data(dict) - Write (at the specified severity level) a row into the sqlite db. The dictionary contains keys with the name of the column to be written, and values with the data element corresponding to that column. Any columns not included in the dict keys will be populated with NULL. (These will be None when queried and converted to Python.)

get_columns()

Returns a list of all column names.

• Return type: list of str

query(query_str)

Run a SQL query against the logger database. Here's a reference for the particular dialect of SQL. It's mostly standard stuff, but as with all SQL dialects can have some surprises, especially if you use some of the fancier features.

• Parameters:
  • query_str (str)
• Return type: list of tuple