Polynx

Polynx is a string-powered Polars expression engine with extended DataFrame and LazyFrame utilities. It brings the intuitive, concise string syntax of Pandas and SQL (such as query filtering, variable substitution, multi-statement evaluation, and conditional branching like case_when) directly to Polars while preserving Polars' blazing-fast execution and lazy query optimization.

Table of Contents

• Key Features
• Installation
• Quick Start
• String Expression Engine
  • Row Filtering with query()
  • Variable Substitution (@var)
  • Multi-Statement Assignments with wc()
  • Expression Evaluation with eval() and assign()
  • Conditional Logic: case_when(), select(), and where()
  • Date Arithmetic & mondf()
  • Horizontal Operations: max_horizontal() & min_horizontal()
  • Window Expressions & Method Chaining
  • Column Names with Spaces (Backticks)
• Extended DataFrame & LazyFrame Utilities
  • Deduplication with dd()
  • Convenient Sorting: dsort() & asort()
  • Pivoting with unstack()
  • Value & Unique Counts: vcnt() & ucnt()
  • String-Powered GroupBy with gb() & Subtotals
  • Extended Grouped Summary Statistics with describe()
  • Float Utilities: round() & cum_max()
  • Scalar Extraction & List Conversion: to_list(), max(), min()
  • Rename by List: rename()
  • Memory Footprint Inspection: size()
  • Quick Visualization with pplot()
  • Pandas-Style Merge: plx.merge()
• Extended Polars Expressions
  • Rolling Product: rolling_prod()
• First-Class LazyFrame Support
• Expression Parser Caching
• Custom UDF Registration
• Polars & Pandas Interoperability
• Contributing
• License

Key Features

• String Expression Engine: Write filtering, mathematical calculations, and column mutations using clean string expressions.
• Variable Substitution (@var): Reference external variables, lists, arrays, or dates directly within string expressions.
• Conditional Branching (case_when, select, where): Express complex conditional logic in string queries without verbose nested .when().then().otherwise() chains.
• Multi-Statement Assignments: Assign multiple columns sequentially separated by semicolons (;) in .wc() or .assign().
• String GroupBy Aggregations (gb): Run aggregate calculations using string expressions, with automatic subtotal rows via with_subtotal=True.
• Extended Statistical Summaries: Grouped descriptive statistics via .describe(group_keys=...).
• Convenience Utilities: Shorthand helpers for deduplication (dd), sorting (dsort/asort), pivoting (unstack), counting (vcnt/ucnt), and memory reporting (size).
• Extended Expressions: Window rolling product (rolling_prod) for Polars expressions.
• Full LazyFrame Support: All string methods and convenience methods work seamlessly on both eager DataFrame and lazy LazyFrame.
• Expression Caching & Diagnostics: High-performance LRU and hashing caching for parsed expressions, with automatic cache bypass for variable loops.
• Seamless Polars Compatibility: Fully wraps Polars; all standard Polars functions and methods (plx.concat, plx.col, etc.) are natively supported.

Installation

Via PyPI (Recommended)

pip install polynx

From Source

git clone https://github.com/LowellWinston/polynx.git
cd polynx
pip install .

Quick Start

import polynx as plx

# Create a Polynx DataFrame (wraps polars.DataFrame)
df = plx.DataFrame({
    'A': [1, 2, 3, 4],
    'B': ['abc', 'bc', 'aaa', None],
    'C': ['2023-01-01', '2021-01-01', '2009-11-01', '2000-11-11'],
    'E': [1.1, 2.1, 3.5, 0.0]
}).wc("C = C.str.to_date('%Y-%m-%d')")

# Filter using query with variable substitution
threshold = 2
result = df.query("A >= @threshold & B in ['bc', 'aaa']")
print(result)

Native Polars Namespace (df.plx.*)

You can also use Polynx methods directly on native Polars DataFrames and LazyFrames without creating a plx.DataFrame wrapper:

import polars as pl
import polynx  # registers the .plx namespace on Polars DataFrames and LazyFrames

df = pl.DataFrame({'A': [1, 2, 3], 'B': [10, 20, 30]})
result = df.plx.query("A > 1 & B >= 20").plx.assign("C = A + B")

String Expression Engine

Row Filtering with query()

df.query(query_str) provides an intuitive filtering interface similar to Pandas' .query(), powered by Polars:

# Chained comparison & logical operators
df.query("1 <= A < 4 & E > 1.0")

# Negation and null checks
df.query("not (B.is_null() & A.is_not_null())")

# Mathematical expressions in filters
df.query("A ** 2 - E > 1")

# String operations
df.query("B.str.contains('a|b')")
df.query("B.str.ends_with('c')")

# Date comparisons with strings
df.query("'2001-01-01' < C < '2020-01-01'")

# Membership testing
df.query("B in ['aaa', 'bc']")
df.query("B not in ['abc']")

Variable Substitution (@var)

Prefixing an identifier with @ pulls the variable dynamically from surrounding scopes:

min_val = 1
max_val = 3
target_categories = ['bc', 'aaa']
cutoff_date = '2010-01-01'

df.query("@min_val <= A <= @max_val & B in @target_categories & C > @cutoff_date")

Variables can be numbers, strings, lists, numpy arrays, or dates. Polynx automatically disables expression caching for expressions containing @ so loop variable updates are always respected:

tmp = plx.DataFrame()
for i in [10, 20]:
    print(tmp.wc("A = @i"))

Multi-Statement Assignments with wc()

The .wc() method extends Polars' with_columns() by supporting multi-statement assignment strings separated by semicolons (;):

# Multiple sequential assignments in a single call
df_new = df.wc("A_sum = A.sum().over('B'); E_mean = E.mean(); Ratio = A / E_mean")

# Mix string expressions and native Polars expressions
import polars as pl
df_new = df.wc(
    "Flag = A > 2",
    pl.col("E") * 100
)

Expression Evaluation with eval() and assign()

• df.eval(query_str, mode='select'): Evaluates an expression and returns only the calculated column(s).
• df.assign(query_str): Convenience shorthand for df.eval(query_str, mode='assign'), appending the calculated columns to the DataFrame.

# Return calculated column
df.eval("(A ** 2 - E) / (10 - A)")

# Append calculated column
df.assign("Score = A * 10 + E")

Conditional Logic: case_when(), select(), and where()

Polynx provides built-in conditional branching functions inside string expressions, eliminating deeply nested Polars .when().then() expressions:

case_when(conds, choices, default)

Unified conditional branching that handles both multiple conditions and single conditions:

# Multi-condition branching (conditions list, choices list, default)
df.wc("Tier = case_when([A < 2, B in ['bc']], ['Bronze', 'Silver'], 'Gold')")

# Single-condition branching (condition, choice, default)
df.wc("Is_High = case_when(A > 2, 1, 0)")

select(conds, choices, default)

Mirrors numpy.select for multi-condition branching:

df.wc("Category = select([A == 1, A == 2], ['First', 'Second'], 'Other')")

where(cond, choice, default)

Mirrors numpy.where for ternary branching:

df.wc("Status = where(A >= 3, 'High', 'Low')")

Date Arithmetic & mondf()

Calculate the month difference between two dates/datetimes with mondf(beg_date, end_date):

df.wc("Months_Diff = mondf(C, C.shift()); Prev_Date = C.shift()")

Horizontal Operations: max_horizontal() & min_horizontal()

Polars' horizontal aggregations are registered and directly usable inside string expressions:

df.wc("Row_Max = max_horizontal(A, E); Row_Min = min_horizontal(A, E)")

Window Expressions & Method Chaining

String expressions support Polars window operations via .over() (accepting single column names or lists of column names) and dynamic method chaining:

# Window aggregation over a column
df.wc("Group_Mean = A.mean().over('B')")

# Window aggregation over multiple columns
df.wc("Multi_Group_Mean = A.mean().over(['B', 'C'])")

# Method chaining
df.wc("Processed = B.str.to_uppercase().str.slice(0, 2)")

Column Names with Spaces (Backticks)

Reference columns with spaces or special characters using backticks (`):

df_special = plx.DataFrame({"First Name": ["Alice", "Bob"], "Score Value": [85, 92]})
df_special.query("`Score Value` > 90 & `First Name` == 'Bob'")

Extended DataFrame & LazyFrame Utilities

Polynx patches both DataFrame and LazyFrame with productivity helpers:

Deduplication with dd()

Deduplicates rows while maintaining order and keeping the first occurrence:

df.dd()  # Equivalent to df.unique(maintain_order=True, keep='first')

Convenient Sorting: dsort() & asort()

Quick descending and ascending sorting:

# Descending sort (defaults to all columns if by is omitted)
df.dsort('A')

# Ascending sort
df.asort('A')

Pivoting with unstack()

Pivots a DataFrame using the last two columns as the pivot key and value column, sorted by index:

df_piv = plx.DataFrame({
    'id': [1, 1, 2, 2],
    'attribute': ['height', 'weight', 'height', 'weight'],
    'value': [180, 75, 165, 60]
})

df_piv.unstack()

Value & Unique Counts: vcnt() & ucnt()

# Frequency count grouped by column(s), sorted descending by count
df.vcnt('B')

# Total distinct/unique count as a scalar
count = df.ucnt('A')

String-Powered GroupBy with gb() & Subtotals

Aggregate grouped data with string expressions. Enable with_subtotal=True to automatically append hierarchical subtotal rows and an 'All' total summary row:

# Group by with string aggregation
df.gb('B', "A.sum(); E.mean()")

# Group by with subtotal row ('All')
df.gb('B', "A.sum()", with_subtotal=True)

Extended Grouped Summary Statistics with describe()

Extends Polars' .describe() to support grouping by keys for selected columns:

# Computes count, null_count, mean, std, min, 25%, 50%, 75%, max grouped by 'B'
df.describe(group_keys='B', selected_columns=['A', 'E']).round(2)

Float Utilities: round() & cum_max()

• df.round(decimal=2): Rounds all floating-point columns across the DataFrame.
• df.cum_max(): Calculates the cumulative maximum for all floating-point columns.

df.round(2)
df.cum_max()

Scalar Extraction & List Conversion: to_list(), max(), min()

Extract column values quickly without verbose .get_column().to_list() calls:

# Extract column to a Python list (defaults to first column if col_name is omitted)
values = df.to_list('A')  # [1, 2, 3, 4]

# Extract scalar max and min values
max_val = df.max('A')    # 4
min_val = df.min('A')    # 1

Rename by List: rename()

In addition to Polars' standard dictionary rename syntax, Polynx allows renaming columns by supplying a list of new names matching the number of columns:

df.rename(['Col_1', 'Col_2', 'Col_3', 'Col_4'])

Memory Footprint Inspection: size()

Prints the estimated DataFrame memory footprint and optionally returns the size:

# Prints size to console
df.size(unit='kb')

# Return numeric value
size_in_mb = df.size(unit='mb', return_size=True)

Quick Visualization with pplot()

Provides pandas-style .plot() directly from a Polynx DataFrame, setting the first column as index:

df.select(['A', 'E']).pplot(kind='line')

Pandas-Style Merge: plx.merge()

Merges two DataFrames or LazyFrames with automatic suffix handling for overlapping column names:

left = plx.DataFrame({'id': [1, 2], 'val': [10, 20]})
right = plx.DataFrame({'id': [1, 2], 'val': [30, 40]})

merged = plx.merge(left, right, on='id', how='inner', suffixes=('_left', '_right'))

Extended Polars Expressions

Polynx patches Expr with additional mathematical operations:

Rolling Product: rolling_prod()

Computes the rolling product across a window (evaluated numerically via log transform and rolling sum):

df.select(plx.col('A').rolling_prod(window_size=2, min_samples=1))

First-Class LazyFrame Support

All Polynx methods are fully compatible with LazyFrame. Operations are deferred until .collect():

lazy_df = df.lazy()

result = (
    lazy_df
    .wc("A_doubled = A * 2")
    .query("A_doubled > 4")
    .collect()
)

# Convenient indexing on LazyFrames
lazy_df['A']                   # Select column
lazy_df[['A', 'B']]            # Select multiple columns
lazy_df[pl.col('A') > 2]       # Filter rows lazily

# Convert LazyFrame to Pandas directly (collects automatically)
pandas_df = lazy_df.to_pandas()

Expression Parser Caching

Polynx caches parsed Lark expression trees to eliminate parsing overhead on repeated queries.

Cache Modes

Configure caching behavior via plx.config:

• "raw" (default): In-memory dictionary cache keyed by expression string.
• "hash": SHA1-hashed cache for expressions.
• "custom_lru": LRU cache with configurable capacity.
• "none": Disables caching completely.

import polynx as plx

# Change cache mode
plx.config.set_cache_mode("custom_lru")
plx.config.set_cache_max_size(2000)

# Check active configuration
print(plx.config.get_cache_mode())       # 'custom_lru'
print(plx.config.get_cache_max_size())   # 2000

Cache Diagnostics & Management

# Inspect cached expressions
cache = plx.expr_parser.get_expr_cache()
cache_size = plx.expr_parser.get_expr_cache_size()

# Clear cache and reset statistics
plx.clear_all_expr_caches()
plx.expr_parser.reset_expr_cache_stats()

Note: Queries utilizing @var variable substitutions automatically bypass caching to prevent stale variable values during iterations.

Custom UDF Registration

Register custom Python functions to call them directly within string expressions:

import polynx as plx

# 1. Register a single function
def add_bonus(score, multiplier=1.1):
    return score * multiplier

plx.register_udf("add_bonus", add_bonus)

# Use inside string expressions
df = plx.DataFrame({'A': [10, 20, 30]})
df.wc("Bonus = add_bonus(A, 1.25)")

# 2. Register functions by name from a module
import math
plx.register_udfs_by_names(math, ['sin', 'cos', 'sqrt'])

Polars & Pandas Interoperability

Polynx is designed as a drop-in enhancement to Polars:

• Polars Function Inheritance: All Polars top-level functions (concat, col, lit, when, read_csv, read_parquet, etc.) are exposed directly through polynx with automatic wrapping and unwrapping:

  import polynx as plx
  df_combined = plx.concat([df, df])
  

• Conversion to Polars: Call .to_polars() on any DataFrame, LazyFrame, Series, or Expr to retrieve native Polars objects.
• Conversion to Pandas: Call .to_pandas() on any DataFrame, LazyFrame, or Series.

Contributing

We welcome contributions! If you'd like to help improve Polynx:

1. Fork the repository.
2. Create your feature branch (git checkout -b feature/amazing-feature).
3. Commit your changes (git commit -m 'Add amazing feature').
4. Push to the branch (git push origin feature/amazing-feature).
5. Open a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.