data2notion 1.3.1

Synchronize your data with a Notion database

data2notion: export your data to Notion

This tool is intended for exporting various data from 3rd party systems into Notion. If you tool can export some records, then you can make this data available to Notion!

Installation

It requires Python 3.9+ and try not using too many dependencies.

to install it:

python3 install data2notion

This will install it and all plugins. You can now run it by typing data2notion in you terminal.

Changes

CHANGELOG.md

Goals

When working as a CTO, I had the need to expose data from various systems (Applications, AWS, SIEM, Active Directory, users, groups...) into Notion to generate automatic reports and select what I wanted to expose, so it made all systems transparent and open. This was extremelly precious to automatically generate reports for audits.

One of the advantages of the tool is that:

• it creates entries in Notion database if entries do not exists
• it modifies the various fields of the Notion database if they have been modified
• it cleanups the old records when they don't exist anymore

So, if you run this tool everyday, you have a Notion database in sync with the data from all your third party systems. And you can start commenting, taking decisions and documenting those entries within Notion: it gives you a centralized way to discuss topics about all of your IT, provides comments, reminders (eg: reming me about renewing this certificate in 2 years, check if the fix has been performed in 2 months...), and cleanup dead stuff automatically.

Initially, the tool was only dealing with CSV, but data2notion also deals with various formats and APIs, so you can integrate faster and better!

Features

data2notion supports plugins, so you can either work with JSON/CSV export or implement your own retrieval of data, for instance, from a 3rd party API.

Basic Usage

data2notion is build using commands. Each succession of command has its own help.

The first level is teh following:

data2notion --help
usage: data2notion [-h] [--version] [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--notion-log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}]
                   [--notion-rate-limit NOTION_RATE_LIMIT] [--statistics {console,disabled}] [--notion-token NOTION_TOKEN]
                   {plugins,write-to-notion} ...

Export some data into a notion database

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Set the default log level (default=WARNING)
  --notion-log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Set the log level for data2notion (default=INFO)
  --notion-rate-limit NOTION_RATE_LIMIT
                        Set the notion rate-limiter, by default {default_rate_limit} (3 requests/sec, 100 initial bucket size)
  --statistics {console,disabled}
                        Display Statistics when program ends
  --notion-token NOTION_TOKEN
                        Notion Token to use $NOTION_TOKEN by default

action to perform:
  action to perform

  {plugins,write-to-notion}
                        sub-command help
    write-to-notion     write to Notion Database

This is the level where you can specify using the command line the NOTION_TOKEN (or you can use the environemnt variable $NOTION_TOKEN instead).

You can also change the log-level (in case of issues) and enable/disable statistics.

First level of help

How to get the notion_database_id?

From official documentation, open the database in your brower (in a full page), the URL should then be https://notion.so/my_workspace/**668d797c76fa49349b05ad288df2d136**v=... => in such an example, 668d797c76fa49349b05ad288df2d136 would be the notion_database_id.

Be sure that you granted access rights to this database to the token you want to use in this app.

data2notion write-to-notion <notion_database_id> <plugin> --help

Will list the arguments that are specific to a plugin.

CSV plugin

Export CSV files arrays in Notion.

data2notion write-to-notion <notion_database_id> csv --help
usage: data2notion write-to-notion notion_database_id csv [-h] [--csv-dialect {excel,excel-tab,unix,excel_with_semi-colon}] csv_file

Write to Notion DB from a CSV file

positional arguments:
  csv_file              CSV file to inject in Notion, if '-' is set, read from stdin

options:
  -h, --help            show this help message and exit
  --csv-dialect {excel,excel-tab,unix,excel_with_semi-colon}
                        CSV Dialect, excel by default

JSON Plugin

Export some JSON arrays in Notion.

usage: data2notion write-to-notion <notion_database_id> json [-h] [--json-path PATH_IN_JSON] json_file

Write to Notion DB from a JSON file containing an array

positional arguments:
  json_file             JSON file to inject in Notion, if '-' is set, read from stdin

options:
  -h, --help            show this help message and exit
  --json-path PATH_IN_JSON
                        JSON path separated by dots to look for the array, example: calendar.appointments

Prometheus Plugin

Export metrics from prometheus (lastest values only) in Notion database.

Inspired by prom2csv, this plugin let you export your last prometheus metrics in Notion!

usage: data2notion [-h] [--version] [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--notion-log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--no-progress-bar] [--notion-rate-limit NOTION_RATE_LIMIT] [--statistics {console,disabled}]
                   [--notion-token NOTION_TOKEN] [--partition column_name=<regexp>]
                   {plugins,export-from-notion,write-to-notion} ...

Export some data into a notion database

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Set the default log level (default from $NOTION_LOG_LEVEL=WARNING)
  --notion-log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Set the log level for data2notion (default=INFO)
  --no-progress-bar     Disable the progress bar
  --notion-rate-limit NOTION_RATE_LIMIT
                        Set the notion rate-limiter, by default {default_rate_limit} (3 requests/sec, 100 initial bucket size)
  --statistics {console,disabled}
                        Display Statistics when program ends
  --notion-token NOTION_TOKEN
                        Notion Token to use $NOTION_TOKEN by default
  --partition column_name=<regexp>
                        Only synchronize records having a column matching given regexp

action to perform:
  action to perform

  {plugins,export-from-notion,write-to-notion}
                        sub-command help
    export-from-notion  export from Notion Database
    write-to-notion     write to Notion Database

Adding other plugins

Adding plugin can be done using environment variable $DATA2NOTION_ADDITIONAL_PLUGINS which support the following syntax python_module.python_file:PluginClass, thus to enable a new imaginary Bugzilla plugin, you might for instance export the variable:

export DATA2NOTION_ADDITIONAL_PLUGINS="bugzilla_api.bugzilla_plugin:BugzillPlugin"

Then, the plugin should be visible in:

data2notion plugins

and you might use it as any other plugin.

Writing new plugins

See DESIGN.md.