RR Connection Manager

A package to help you connect to all Registry databases

· Report Bug · Request Feature

About The Project

This package wraps pykeepass and sshtunnel along with various connection tools including SQLAlchemy, to allow an all in one solution for connecting via SSH to our servers without the need to store connection details locally

Prerequisites

You will need to ensure make sure you meet all the expected requirements for using pyscopg2

Getting Started

poetry add rr-connection-manager

or

pip install rr-connection-manager

Extras

This package allows for specific dependency setups inline with PEP 508 so that you can control which database connectors you want to work with.

pip install rr-connection-manager[pgsql]

Poetry allows for the same behaviour using extras but with it's own syntax

poetry add rr-connection-manager -E pgsql -E tunnel

There is a full list of possible options in the pyproject but the general patterns used are

Extra Name	Installed	Notes
tunnel	sshtunnel	Required if using the tunnel argument
keepass	pykeepass	Required if using a keepass file
sqlser	pyodbc	For only working with cursors
sqlser_sqla	pyodbc, sqlalchemy	For working with session and engines (SQL Server)
pgsql	sqlalchemy	For using PostgreSQL with SQLAlchemy (requires driver too)
pgsql_sqla	psycopg, sqlalchemy	Recommended for full PostgreSQL + SQLAlchemy support
mysql	mysql-connector-python	For using MySQL with raw connector
mysql_sqla	mysql-connector-python, sqlalchemy	For using MySQL with SQLAlchemy

Most of the database flavours have a standard name for just using cursors and one sufixed with _sqla for working with sqlalchemy

In the case where you wanted to use connection manager to install dependencies that allowed you to work with a postgres database using SQLAlchemy over and SSH connection using the details found in a keepass file you would use the following command

poetry install rr-connection-manager -E pgsql_sqla -E tunnel -E keepass

Connection Variables

The primary use case is for the connection variables to be taken from a keepass file. This will require some setup which includes running the RR Key Manager. Once you have done that connection manager should work out of the box.

Adding a conf.json file at the root of your project will override any attempt to connect to keepass. This is useful if you can't access the keepass file or you want use this on one of the servers. You can add connection details for as many servers as you like but the app_name field in the conf file must match the app_name variable past to the function to create your connection.

[
  {
    "db_id": "database_identifier",
    "db_host": "database_host",
    "db_port": "database_port",
    "db_name": "database_name",
    "db_user": "database_username",
    "db_password": "database_user_password",
    "tunnel_user": "tunnel_user",
    "tunnel_port": "tunnel_port",
    "local_port": "local_port"
  }
]

Not all of these are required for each connection. SQL Server connections for example only require DB_HOST and DATABASE.

(back to top)

Using a Tunnel

To connect to a database you need to use one of the predefined connection types. There are four possible connection type enums.

Enum	Notes
LOCAL	Connecting to a local database.
REMOTE	Connecting directly to a database on a remote server.
TUNNEL	Uses SSH to connect to a server and then connect locally on the server.
PROXY	Uses an SSH tunnel via a proxy server to connect to the DB server and then locally to the database.

When building the connection object you will need to import the ConnectionTypes. SO if we wanted to build a connection to a postgres database using an ssh tunnel and check to make sure we could connect we would do the following

from rr_connection_manager import PostgresConnection
from rr_connection_manager import ConnectionType

conn = PostgresConnection(db_id="db_identifier", conn_type=ConnectionType.TUNNEL)
conn.connection_check()

In cases where you want to tunnel through an app server to the database server you can use the proxy connection type.

from rr_connection_manager import PostgresConnection
from rr_connection_manager import ConnectionType

conn = PostgresConnection(db_id="db_identifier", conn_type=ConnectionType.PROXY)

SQL Server Connection

To create a SQL Server connection object

conn = SQLServerConnection(db_id="db_identifier")

From this you can choose to open a pyodbc cursor and use that to query the database

conn = SQLServerConnection(db_id="db_identifier")
cur = conn.cursor()
cur.execute("SELECT @@version")
info = cur.fetchone()
cur.close()
print(info)

You can pass in any extra config arguments you like when creating a cursor

conn = SQLServerConnection(db_id="db_identifier")
cur = conn.cursor(timeout=30, autocommit=True)

You can also create an untrusted connection to an SQL Server database whihc just means it will use a user name and password instead of using your windows credintials to authenticate.

conn = SQLServerConnection(db_id="db_identifier", trusted=False)

(back to top)

Postgres Connection

To create a Postgres connection object

conn = PostgresConnection(db_id="db_identifier")

From this you can choose to open a pyscopg2 cursor and use that to query the database

conn = PostgresConnection(db_id="db_identifier")
cur = conn.cursor()
cur.execute("SELECT version()")
info = cur.fetchone()
cur.close()
print(info)

You can pass in any extra config arguments you like when creating a cursor

conn = PostgresConnection(db_id="db_identifier")
cur = conn.cursor(connect_timeout=30, autocommit=True)

(back to top)

SQLite Connection

To create a SQLite connection object

conn = conn = SQLiteConnection(db_id="db_identifier")

From this you can choose to open a sqlite cursor and use that to query the database

conn = SQLiteConnection(db_id="db_identifier")
cur = conn.cursor()
cur.execute("SELECT sqlite_version()")
info = cur.fetchone()
cur.close()
print(info)

Redis Connection

To create a Redis connection object

conn = RedisConnection(db_id="db_identifier")

From this you can choose to open a redis cursor and use that to query the database

conn = RedisConnection(db_id="db_identifier")
cur = conn.cursor()
info = cur.ping()
cur.close()
print(info)

You can pass any extra options you wish to the cursor as you would when using the Redis package normally.

conn = RedisConnection(db_id="db_identifier")
cur = conn.cursor(charset="utf-8", decode_responses=True)

SQL Alchemy

All connection types also wrap sqlalchemy so you are able to access a session. This uses the standard set of arguments when creating the engine.

conn = PostgresConnection(db_id="db_identifier")
session = conn.session()

You can build the engine yourself which you allows you to pass arguments

conn = PostgresConnection(db_id="db_identifier")
eng = conn.engine(echo=True)
session = conn.session(eng)

A session maker is also accessible and allows you to pass arguments

conn = PostgresConnection(db_id="db_identifier")
Session = conn.session_maker(expire_on_commit=True)
session = Session()

You can combine both

conn = PostgresConnection(db_id="db_identifier")
eng = conn.engine(echo=True)
Session = conn.session_maker(eng, expire_on_commit=True)
session = Session()
check_message = session.execute(text("""select version()""")).first()
session.close()
conn.close()
print(check_message)

(back to top)

Connection check

To make testing the connection simple each class has a connection_check function that checks the version of the database it is connecting to. This uses the base packages not SQLAlchemy but it is assumed if they work so should SQLAlchemy.

from rr_connection_manager import PostgresConnection
from rr_connection_manager import SQLServerConnection

conn = PostgresConnection(db_id="db_identifier")
conn.connection_check()

conn = SQLServerConnection(db_id="db_identifier")
conn.connection_check()

Config Warning

To make it simple to create sanity checks before running the APi exposes a function named config_warning which prints out some information about the connection you have created. This works with any connection type Redis is used for demonstration purposes only.

conn = RedisConnection(db_id="db_identifier")
conn.config_warning()

Output

Warning: You are connecting using the following settings

Config Location: C:\Path\to\the\conf.json or keepass
Database ID: db_identifier
Host Address: some.host.somewhere.uk
Database Name: your_db
Database User: you

Using a Specific Local Port

⚠️ Warning: This is probably broken

To use a specific local port with your connection you can pass the local_port argument.

conn = PostgresConnection(db_id="db_identifier", tunnel=True, local_port=6100)

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

Contact

Renal Registry - @UKKidney - rrsystems@renalregistry.nhs.uk

Project Link: https://github.com/renalreg/rr-connection-manager

Acknowledgments

• Best-README-Template
• Psycopg2
• Pyodbc
• SQLAlchemy
• Pykeepass
• sshtunnel

(back to top)