tablefile Package Tutorial (v1.1.0)

tablefile is a python package for reading, processing, and modifying tabular data files (separated by tabs, spaces, or any other delimiter) easily for analytical applications.

Installation

pip install tablefile

What's New in v1.1.0 (Changelog from v1.0.0)

1. Enhanced Delimiter Auto-detection:
   • Improved detection logic for whitespace, commas, semicolons, and pipes.
2. Improved Whitespace/Tab Handling:
   • Leading spaces and tabs are cleanly handled during validation and line parsing.
3. Empty Fields Preservation:
   • Preserves empty fields as '?' along with their original line endings for formatting consistency.

What's New in v1.0.0 (Changelog from v0.0.5)

1. New readlines() and readcols() APIs:
   • Replaced legacy cryptic parameter calls like f1.read('c/l') and f1.read('l/c') with dedicated, readable methods f1.readcols() and f1.readlines().
2. Line-Wise (Row-Wise) Statistics:
   • readlines(*operator) now accepts statistical operators ("av", "sd", "sm", "mx", "mn", etc.) and performs calculations row-wise (line-by-line) instead of column-wise.
3. Robust Missing-Column Handling (Padding):
   • If columns are missing or the file has uneven lines, the package pads them with "?" to avoid indexing exceptions, allowing mathematical calculations to proceed while letting users know some columns are missing.
4. Enhanced Separator Detection & Auto-splitting:
   • When no separator is specified (e.g. file("data.txt")), the package defaults to whitespace splitting (any combination of spaces and tabs). Even if a data separator is not explicitly given, the module is still expected to auto-detect the pattern and give correct results.
5. Direct File Modifying Slicing API:
   • Added support for index assignment (e.g., f1[lineNo, ColNo] = value) to replace any element, column (e.g., f1[:, ColNo] = col_values), or line (e.g., f1[lineNo, :] = line_values) in the file on disk, preserving comments, blank lines, and file delimiters (auto-detected).
6. Strict Type Preservation:
   • Elements are parsed preserving their exact types (int, float, and str). For example, integers in the file remain int when loaded, instead of being cast to float as in v0.0.5.
7. Clean Exception Handling:
   • Added exception handling. Common errors (missing file, out-of-bounds row/column index, invalid operator name) print clear explanations and exit normally with code 0 instead of throwing a Python traceback stack.

Quick Start Tutorial

1. Opening a File

To open a file, import the package and instantiate a file object.

from tablefile import *

# Open a file separated by tabs:
f1 = file("data.txt", "\t")

# Open a file without specifying a separator (even if the data separator is not given, the module is expected to give correct results):
f1 = file("data.txt")

2. Reading Data

You can read data row-wise (lines) or column-wise.

# Read lines (rows):
# Output is a list of lists representing each data line
lines = f1.readlines()
print(lines[0])     # Prints the first data row: e.g., [1.5, 2, 'abc']

# Read columns:
# Output is a list of lists representing each data column
cols = f1.readcols()
print(cols[0])      # Prints the first column: e.g., [1.5, 3.0, 5.0]

# Backward Compatibility:
# Calling read() without arguments behaves exactly as f1.readlines().
# Calling read("c/l") behaves exactly as f1.readcols().
lines = f1.read()
cols = f1.read("c/l")

3. Calculating Statistics

You can perform column-wise or line-wise statistical operations. The calculation ignores any strings, empty fields, or missing column values ("?").

Column-Wise Statistics (using readcols() or read()):

averages = f1.readcols("av")     # Column-wise averages
sums = f1.readcols("sm")         # Column-wise sums
stdev_pop = f1.readcols("sd")    # Column-wise population standard deviation
stdev_sam = f1.readcols("sds")   # Column-wise sample standard deviation
maximums = f1.readcols("mx")     # Column-wise maximum values
minimums = f1.readcols("mn")     # Column-wise minimum values

# Backward Compatibility syntax is also supported:
averages = f1.read("av")

Line-Wise (Row-Wise) Statistics (using readlines()):

line_averages = f1.readlines("av")  # Average value for each row
line_sums = f1.readlines("sm")      # Sum value for each row
line_std = f1.readlines("sd")       # Population standard deviation for each row
line_max = f1.readlines("mx")       # Maximum value for each row
line_min = f1.readlines("mn")       # Minimum value for each row

4. Modifying Data

To replace or insert values in the file on disk, use native Python indexing and slicing:

# Replace a single cell at 0-indexed row 2, column 5 with "new_val":
f1[2, 5] = "new_val"

# Replace an entire column with a list of new values:
f1[:, 1] = [10, 20, 30, 40]

# Replace an entire row with a list of new values:
f1[2, :] = [1.5, 2.5, "abc"]

Note: If the column index exceeds the current columns in that line, the package automatically pads the columns with "?" and writes the value, preserving the rest of the file layout (including comments and empty lines).

Operator Reference Sheet

Detailed Method Reference

readlines(*operator)

• Arguments: Optional string operator ("av", "sm", "sd", "sds", "mx", "mn", "l/c", "c/l").
• Return Type: List (List of lists representing rows, or list of line-wise values if statistics operator is used).
• Behavior: Auto-pads missing columns with "?". Preserves numeric/string types.

readcols(*operator)

• Arguments: Optional string operator ("av", "sm", "sd", "sds", "mx", "mn", "l/c", "c/l").
• Return Type: List (List of lists representing columns, or list of column-wise values if statistics operator is used).
• Behavior: Transposes rows to columns. Auto-pads missing columns with "?".

Indexing and Slicing (__setitem__)

• Syntax: f1[lineNo, ColNo] = value
• Arguments:
  • lineNo (int, slice, or ":"): 0-indexed data line number (ignores comment and empty lines), or a slice to target all lines. Supports negative indexes.
  • ColNo (int, slice, or ":"): 0-indexed column index, or a slice to target all columns. Supports negative indexes.
  • value (Any or List): The value(s) to write. If targeting a single cell, value must be a single element (not a list/tuple). If targeting a row or column, value can be a list of values.
• Return Type: None
• Behavior: Modifies the file on disk. Auto-pads columns with "?" if writing out of bounds. Prints an error and does not write if a list/tuple is assigned to a single cell index.

General Helpers (One-Dimensional Lists)

In addition to the file methods, standard functions are exported to compute metrics on any 1D list:

# Convert all numeric elements following a string expression
List_converted = convert(cols[0], '(x**2+sin(x))/2')

# Operations:
Value_sum = sm(cols[0])        # Summation
Value_av = av(cols[0])          # Average
Value_sd = sd(cols[0])          # Population StDev
Value_sd_sample = sds(cols[0])  # Sample StDev
Value_mx = mx(cols[0])          # Maximum
Value_mn = mn(cols[0])          # Minimum

(All standard functions ignore string values during the calculation.)