conecta 0.0.6

Load data from SQL databases to Arrow

Conecta - SQL to Arrow made easy.

» Documentation | Releases | Issues | Repository | License

Conecta is a library designed to load data from SQL databases into Arrow with maximum speed and memory efficiency by leveraging zero-copy and true concurrency in Python.

from conecta import read_sql

table = read_sql(
    "postgres://user:password@localhost:5432/postgres",
    query="select * from lineitem",
    partition_number=4
)

This snippet will create a pool, launch 4 threads, send four different queries and stream the results back to arrow. The core is written in Rust.

Conecta integrates natively to the arrow ecosystem, we support several arrow libraries: pyarrow, arro3 and nanoarrow. Additionally, you can create dataframes like Polars or Pandas.

from conecta import read_sql
import polars as pl
import pandas as pd

table = read_sql(
    "postgres://user:password@localhost:5432/postgres",
    query="select * from lineitem",
    partition_number=4,
    
    # By default pyarrow is the arrow backend.
    results_backend='pyarrow'
)

# -- Polars --
# You could use results_backend='arro3' for a smaller
# installation setup.
df = pl.from_arrow(table)

# -- Pandas --
df = table.to_pandas()

Features:

• Connection pooling
• Real multithreading
• Client-based query partition
• Utilities like: sql bind parameters

Supported sources:

• Postgres: postgres_docs

How to install.

You can get started by running pip install conecta

Supported architectures

Conecta is compiled in almost all popular architectures.

Supported architectures:

• linux: x86_64, x86, aarch64, armv7 and ppc64le
• musllinux: x86_64, x86, aarch64, armv7
• windows: x64 and x86
• macos: x86_64 and aarch64

Unsupported architectures:

• linux: IBM s390x