conecta 0.0.8

Load data from SQL databases to Arrow

Conecta - SQL to Arrow made easy.

» Documentation | Releases | Issues | Repository | License

Conecta is a library designed to load data from SQL databases into Arrow with maximum speed and memory efficiency by leveraging zero-copy and true concurrency in Python.

from conecta import read_sql

table = read_sql(
    "postgres://user:password@localhost:5432/postgres",
    query="select * from lineitem",
    partition_number=4
)

This snippet will create a pool, launch 4 threads, send four different queries and stream the results back to arrow. The core is written in Rust.

Conecta integrates natively to the arrow ecosystem, we support several arrow libraries: pyarrow, arro3 and nanoarrow. Additionally, you can create dataframes like Polars or Pandas.

from conecta import read_sql
import polars as pl
import pandas as pd

table = read_sql(
    "postgres://user:password@localhost:5432/postgres",
    query="select * from lineitem",
    partition_number=4,
    
    # By default pyarrow is the arrow backend.
    results_backend='pyarrow'
)

# -- Polars --
# You could use results_backend='arro3' for a smaller
# installation setup.
df = pl.from_arrow(table)

# -- Pandas --
df = table.to_pandas()

Features:

• Connection pooling
• Real multithreading
• Client-based query partition
• Utilities like: sql bind parameters

Supported sources:

• Postgres: postgres_docs

How to install.

You can get started by running pip install conecta

Performance

name	captured_elapsed	peak_memory	function_time(s)	test_time(s)
conecta-4-lineitem_small	38.488903ms	2.520751MB	0.119171	0.284783
conecta-4-lineitem	1.75472146s	214.47775MB	1.79604	1.86721
connectorx-4-lineitem_small			0.494878	0.717556
connectorx-4-lineitem			2.15225	2.27106

Protocol is 'cursor'.

The most relevant metrics to compare are function_time and peak_memory

• captured_elapsed: The total time the data download and arrays creation.
• peak_memory: The total amount of memory when the data was downloaded.
• function_time: The time the function read_sql.
• test_time: The total amount of time the benchmark case took, taking into test setup.

Supported architectures

Conecta is compiled in almost all popular architectures.

Supported architectures:

• linux: x86_64, x86, aarch64, armv7 and ppc64le
• musllinux: x86_64, x86, aarch64, armv7
• windows: x64 and x86
• macos: x86_64 and aarch64

Unsupported architectures:

• linux: IBM s390x