fid-tools 0.1.0

Functional Information Decomposition - analyze how inputs contribute to predicting outputs

Functional Information Decomposition (FID) Library - User Manual

Table of Contents

1. Overview
2. Installation & Setup
3. Core Concepts
4. Quick Start
5. Working with TPMs
6. FID Analysis
7. Handling Missing Data
8. Visualization
9. API Reference
10. Examples
11. Limitations & Warnings

---

Overview

FID (Functional Information Decomposition) is a library for analyzing how multiple input variables contribute to predicting an output, using information-theoretic concepts.

What It Does

• Quantifies independent information: How mucdh each input alone predicts the output
• Measures synergy: Information only available from joint inputs
• Computes solo synergy: How much each input contributes when combined with all others
• Handles incomplete data: Intelligently manages missing input patterns through probabilistic sampling

Key Features

• Build TPMs (Transition Probability Matrices) from raw data
• Compute FID decompositions on complete or incomplete data
• Sample completions using multiple strategies (Dirichlet, grid, deterministic, edges)
• Visualize results with scatter plots, clouds, and uncertainty bounds
• Flexible handling of categorical/discrete variables with arbitrary names and symbols

---

Installation & Setup

Prerequisites

• Python 3.7+
• NumPy, SciPy, Matplotlib

Install Dependencies

pip install -r requirements.txt

Import the Library

from fid import TPM, entropy, display_fid, plot_fid_clouds

---

Core Concepts

TPM (Transition Probability Matrix)

A matrix representing the relationship between inputs and an output:

• Rows: All possible combinations of input patterns (exhaustive enumeration)
• Columns: Output symbols/states
• Values: Probabilities P(Y|X) for conditional, or P(X,Y) for joint

RN_TPM (Row-Normalized TPM)

Each row sums to 1.0, representing conditional probability P(Y|X_i)

• Answers: "Given this input pattern, what's the probability of each output?"

MN_TPM (Matrix-Normalized TPM)

The entire matrix sums to 1.0, representing joint probability P(X,Y)

• Answers: "What's the probability of seeing this input-output pair?"
• Computed as: MN_TPM = RN_TPM / n_rows (assuming uniform input distribution)

Information-Theoretic Metrics

Entropy H(Y)

• Measure of output uncertainty
• Higher entropy = more uniform output distribution

Conditional Entropy H(Y|X)

• Average uncertainty in output given the inputs
• Lower is better (inputs explain more)

Mutual Information I(X;Y)

• How much knowing X reduces uncertainty about Y
• Equals: H(Y) - H(Y|X)

Independent Information

• How much a single input X_i predicts Y by itself

Synergy

• Information only available from the joint state of all inputs
• Equals: Total MI - Sum of Independents

Solo Synergy

• How much input X_i contributes when combined with all other inputs
• Accounts for interactions and non-linear effects

---

Quick Start

Example 1: Build and Analyze a Simple TPM

import numpy as np
from fid import TPM, display_fid

# Create data: one input perfectly predicts output, the other doesn't
input_A = ['a', 'a', 'b', 'b', 'a', 'b']
input_B = [0, 1, 0, 1, 0, 1]
output = ['X', 'Y', 'X', 'Y', 'X', 'Y']

# Build TPM from data
tpm = TPM.from_data(
    data=[input_A, input_B, output],
    input_names=['A', 'B'],
    output_name='Z'
)

# Display the TPM
tpm.display()

# Compute FID
fid_result = tpm.fid()

# Display results
display_fid(fid_result)

What to expect:

• Input B perfectly predicts output (B=0 always → X, B=1 always → Y)
• Input A does not predict output
• So: B has high independent information (~1.0 bits), A has none
• Synergy should be ~0.0 (no joint effects needed)

Example 2: Handle Incomplete Data

# Data with a missing input pattern (a,a)
input_1 = ['a', 'a', 'b', 'b']
input_2 = [0, 1, 0, 1]
output = ['X', 'Y', 'X', 'Y']

tpm = TPM.from_data(
    data=[input_1, input_2, output],
    input_names=['A', 'B'],
    output_name='Z'
)

# Check missing rows
print(f"Missing patterns: {len(tpm.missing_rows)}")

# Set a prior for missing pattern (a,0)
tpm.set_row(('a', 0), {'X': 0.8, 'Y': 0.2})

# Now compute FID on neutral (uniform) completion
result = tpm.neutral_completion_fid()
display_fid(result)

---

Working with TPMs

Creating a TPM

tpm = TPM.from_data(
    data=[input_1_seq, input_2_seq, output_seq],
    input_names=['Input1', 'Input2'],
    output_name='Output'
)

• Automatically builds all possible patterns
• Counts observed transitions to compute P(Y|X)
• Marks patterns never seen as "missing"

Inspecting a TPM

# Display the full TPM
tpm.display()

# Get input names
inputs = tpm.get_input_names()
print(inputs)  # ['A', 'B']

# Get symbols for each input
input_symbols = tpm.get_input_symbols()
print(input_symbols)  # {'A': ['a', 'b'], 'B': [0, 1]}

# Get output symbols
output_symbols = tpm.get_output_symbols()
print(output_symbols)  # ['X', 'Y']

# Check for missing rows
print(f"Missing: {len(tpm.missing_rows)} / {tpm.n_rows}")

Modifying a TPM

Add/Extend Output Symbols

# Add new output symbol
tpm.extend_symbols(['Z'])
tpm.display()  # New column automatically filled with 0.0 for complete rows

Set a Row (Prior)

Set a specific input pattern to a known probability distribution:

# Set pattern (a, 0) to probability {'X': 0.8, 'Y': 0.2}
tpm.set_row(('a', 0), {'X': 0.8, 'Y': 0.2})

# Internally, probabilities are normalized
# After refresh, RN_TPM[pattern_idx] = [0.8, 0.2]

Set a Row (Restriction)

Restrict which outputs are allowed for a pattern, but keep it incomplete:

# Pattern (a, 0) can only produce 'X' (but exact probability unknown)
tpm.set_row(('a', 0), ['X'])

# Row stays in missing_rows; FID requires all rows complete
# Useful for constraints in completion sampling

Clear a Row

# Reset pattern (a, 0) to unknown (all NaN)
tpm.clear_row(('a', 0))

Refresh After Changes

# Automatically called by set_row(), clear_row(), extend_symbols()
# But can be called manually:
tpm.refresh_TPMs()

# This recomputes:
# - RN_TPM normalization (each row sums to 1)
# - MN_TPM from RN_TPM (joint distribution)

---

FID Analysis

Single-Point FID

For a complete TPM (no missing rows), compute exact FID:

result = tpm.fid()

# Result keys:
# - 'total_information' (float): I(X;Y)
# - 'independents' (dict): {label -> float} for each input
# - 'synergy' (float): residual joint information
# - 'solo_synergies' (dict): {label -> float} for each input
# - 'output_entropy' (float): H(Y)
# - 'conditional_entropy' (float): H(Y|X)
# - 'input_labels' (list): formatted variable names

print(f"Total Info: {result['total_information']:.4f} bits")
print(f"Synergy: {result['synergy']:.4f} bits")
for label, val in result['independents'].items():
    print(f"  {label}: {val:.4f} bits")

Neutral Completion FID

For incomplete TPMs, fill missing rows uniformly and compute FID:

result = tpm.neutral_completion_fid()

# Returns same structure as fid()
# Missing rows filled with P(Y) = uniform over allowed outputs

---

Handling Missing Data

The library provides multiple strategies for sampling completions of missing rows.

Motivation

When data is sparse, not all input patterns are observed. FID requires a complete TPM. Different completion strategies yield different (but related) FID results. Sampling many completions gives uncertainty bounds.

Strategy 1: Dirichlet Cloud (Recommended)

Sample missing rows from a Dirichlet distribution (soft, probabilistic):

cloud = tpm.fid_cloud(
    method='dirichlet',
    n_samples=10000,
    alpha=1.0
)

# cloud keys:
# - 'total_information' (array of shape (n_samples,))
# - 'independents' (dict of arrays)
# - 'synergy' (array)
# - 'solo_synergies' (dict of arrays)
# - 'output_entropy', 'conditional_entropy' (arrays)
# - 'n_completions' (int): 10000

# Get summary statistics
import numpy as np
mean_synergy = np.mean(cloud['synergy'])
std_synergy = np.std(cloud['synergy'])
print(f"Synergy: {mean_synergy:.4f} ± {std_synergy:.4f}")

Alpha parameter:

• alpha = 1.0: Uniform (neutral, default)
• alpha < 1.0: Sparse distributions (peaked)
• alpha > 1.0: Uniform distributions

Strategy 2: Deterministic Cloud

Sample missing rows from discrete delta distributions (hard one-hot):

cloud = tpm.fid_deterministic(n_samples=5000)

# Each missing row is assigned to a single output with 100% probability
# Result: all outputs represented equally

Use when:

• You want extreme/boundary cases
• Grid is computationally feasible

Strategy 3: Grid-Based Cloud

Sample missing rows from a structured grid (balanced, parametric):

cloud = tpm.fid_grid(
    steps=5,      # discretization level
    n_samples=10000
)

# Automatically subsamples if grid is larger than n_samples
# Deterministic and reproducible

Use when:

• You want systematic coverage
• Comparability across runs

Strategy 4: Edges Cloud

Sample missing rows along interpolated paths between deterministic completions:

cloud = tpm.fid_edges(n_samples=5000)

# Samples midpoints between pairs of deterministic completions
# Explores the "surface" of the completion space

Use when:

• You want smooth transitions
• Interested in boundaries

Comparing Completions

# Compute multiple clouds
cloud_dirichlet = tpm.fid_cloud(n_samples=5000, alpha=1.0)
cloud_deterministic = tpm.fid_deterministic(n_samples=5000)

# Display with bounds
from fid import display_fid_with_bounds
neutral = tpm.neutral_completion_fid()
display_fid_with_bounds(neutral, [cloud_dirichlet, cloud_deterministic])

---

Visualization

Simple Scatter Plot

from fid import plot_fid_clouds

neutral = tpm.neutral_completion_fid()
cloud = tpm.fid_cloud(n_samples=10000)

plot_fid_clouds(
    base_fid=neutral,
    fid_clouds=[cloud],
    names=['Dirichlet'],
    colors=['blue'],
    alphas=[0.3],
    x_metric='synergy',
    y_metric='total_information',
    filename='plot1.png'
)

3D Scatter Plot

from fid import plot_fid_clouds_3d

plot_fid_clouds_3d(
    base_fid=neutral,
    fid_clouds=[cloud],
    names=['Dirichlet'],
    colors=['blue'],
    alphas=[0.3],
    x_metric='synergy',
    y_metric='total_information',
    z_metric='output_entropy',
    filename='plot_3d.png'
)

Relationship Grid (Advanced)

from fid import plot_fid_relationships

plot_fid_relationships(
    base_fid=neutral,
    fid_clouds=[cloud],
    names=['Dirichlet'],
    colors=['blue'],
    alphas=[0.3],
    y_axis='total_information',
    share_axis='common',  # all plots share axes
    filename='relationships.png'
)

This creates a matrix of scatter plots showing:

• Top row: Synergy and Total Information vs. y-axis
• Remaining rows (one per input):
  • Independent vs. y-axis
  • Solo Synergy vs. y-axis
  • Loss (Independent + Solo Synergy) vs. y-axis
  • Solo Synergy vs. Independent (2D relationship)

Retrieving Metrics from Clouds

from fid import get_metric

# Get a specific metric from a cloud
synergy_values = get_metric(cloud, 'synergy')
x0_independent = get_metric(cloud, 'X0_independent')
x1_solo_synergy = get_metric(cloud, 'X1_solo_synergy')

---

API Reference

TPM Class

Constructors

TPM.from_data(data, input_names=None, output_name='Y')

• Build TPM from sequences
• Args:
  • data (list): [input_1_seq, input_2_seq, ..., output_seq]
  • input_names (list, optional): Names for inputs
  • output_name (str): Name for output
• Returns: TPM instance

Display Methods

tpm.display()

• Print formatted TPM with all patterns and probabilities

tpm.get_input_names()

• Returns: List of input variable names

tpm.get_input_symbols()

• Returns: Dict mapping input names to symbol lists

tpm.get_output_symbols()

• Returns: List of output symbols

Modification Methods

tpm.set_row(pattern, symbols)

• Set prior or restriction for a row
• Args:
  • pattern (tuple): Input pattern using visible symbols
  • symbols (list or dict):
    • List: restriction to allowed outputs
    • Dict: prior probabilities {symbol -> weight}

tpm.clear_row(pattern)

• Reset row to missing (all NaN)

tpm.extend_symbols(new_symbols)

• Add new output symbols to TPM

tpm.refresh_TPMs()

• Recompute RN_TPM and MN_TPM after manual changes

Analysis Methods

tpm.fid()

• Compute FID for complete TPM
• Returns: FID result dict
• Raises: ValueError if missing rows exist

tpm.neutral_completion_fid()

• Compute FID with uniform completion of missing rows
• Returns: FID result dict

tpm.fid_cloud(method='dirichlet', n_samples=50000, alpha=1.0)

• Sample completions from Dirichlet distribution
• Returns: Cloud dict with arrays

tpm.fid_deterministic(n_samples)

• Sample deterministic completions
• Returns: Cloud dict with arrays

tpm.fid_grid(steps=None, n_samples=...)

• Sample from grid of probability compositions
• Returns: Cloud dict with arrays

tpm.fid_edges(n_samples)

• Sample interpolations between deterministic completions
• Returns: Cloud dict with arrays

Utility Methods

tpm.marginalize(indices_to_keep)

• Compute marginal distributions for subset of inputs
• Args: List of input indices
• Returns: Dict mapping partial patterns to output distributions

tpm.output_entropy()

• Returns: H(Y) as float

tpm.conditional_entropy()

• Returns: H(Y|X) as float

tpm.mutual_information()

• Returns: I(X;Y) as float

---

Module-Level Functions

entropy(probs)

• Compute entropy of probability distribution (vectorized)
• Args: NumPy array of shape (..., n)
• Returns: Array of shape (...)

display_fid(fid_result)

• Print formatted FID report

display_fid_with_bounds(base_fid, clouds)

• Print FID with uncertainty bounds from multiple completions

plot_fid_clouds(base_fid, fid_clouds, names, colors, alphas, x_metric, y_metric, ...)

• 2D scatter plot with base assumption overlay

plot_fid_clouds_3d(base_fid, fid_clouds, names, colors, alphas, x_metric, y_metric, z_metric, ...)

• 3D scatter plot

plot_fid_relationships(base_fid, fid_clouds, names, colors, alphas, y_axis, ...)

• Relationship grid (matrix of scatter plots)

get_metric(cloud, metric_name)

• Extract a metric from a cloud dict
• Handles special naming: X0_independent, X1_solo_synergy, etc.

---

Examples

Example 1: Boolean Function Analysis

from fid import TPM, display_fid

# XOR function: output = X1 XOR X2
x1 = [0, 0, 1, 1, 0, 0, 1, 1]
x2 = [0, 1, 0, 1, 0, 1, 0, 1]
y = [0, 1, 1, 0, 0, 1, 1, 0]

tpm = TPM.from_data([x1, x2, y], input_names=['X1', 'X2'], output_name='Y')
result = tpm.fid()
display_fid(result)

# Expected: High synergy, low independents (neither input alone predicts output)

Example 2: Redundant Inputs

# Two identical inputs predicting output (all 4 patterns observed)
x1 = [0, 0, 1, 1, 0, 0, 1, 1]
x2 = [0, 0, 1, 1, 0, 0, 1, 1]  # Same as x1
y = [0, 0, 1, 1, 0, 0, 1, 1]   # Same as x1 and x2

tpm = TPM.from_data([x1, x2, y], input_names=['X1', 'X2'], output_name='Y')
result = tpm.fid()

# Expected: Each input has ~1.0 bits independent info, very low synergy
# (both inputs are redundant—each alone fully predicts output)

Example 3: Sparse Data with Completion Sampling

# Only 2 out of 4 patterns observed
x1 = [0, 0, 1, 1]
x2 = [0, 1, 0, 1]
y = [0, 1, 1, 0]  # Missing patterns: (0,0) and (1,1)

tpm = TPM.from_data([x1, x2, y], input_names=['X1', 'X2'], output_name='Y')

# Get baseline
neutral = tpm.neutral_completion_fid()
print(f"Neutral synergy: {neutral['synergy']:.4f}")

# Sample completions
cloud = tpm.fid_cloud(n_samples=5000)

# Analyze uncertainty
import numpy as np
mean = np.mean(cloud['synergy'])
std = np.std(cloud['synergy'])
print(f"Cloud synergy: {mean:.4f} ± {std:.4f}")

Example 4: Multi-Input System with Visualization

from fid import TPM, plot_fid_relationships
import numpy as np

# 3 inputs, 2 outputs
x1 = np.random.choice(['a', 'b'], 100)
x2 = np.random.choice([0, 1], 100)
x3 = np.random.choice(['low', 'high'], 100)
y = np.random.choice(['yes', 'no'], 100)

tpm = TPM.from_data(
    [x1, x2, x3, y],
    input_names=['Feature1', 'Feature2', 'Feature3'],
    output_name='Outcome'
)

# Handle missing rows with prior knowledge
for (f1, f2, f3) in [('a', 0, 'low'), ('b', 1, 'high')]:
    if len(tpm.missing_rows) > 0:
        tpm.set_row((f1, f2, f3), {'yes': 0.7, 'no': 0.3})

neutral = tpm.neutral_completion_fid()
cloud = tpm.fid_cloud(n_samples=5000, alpha=2.0)

# Visualize relationships
plot_fid_relationships(
    base_fid=neutral,
    fid_clouds=[cloud],
    names=['Completion Cloud'],
    colors=['steelblue'],
    alphas=[0.2],
    y_axis='total_information',
    share_axis='common'
)

---

Limitations & Warnings

Combinatorial Explosion

⚠️ Critical Limitation: The library enumerates ALL possible input patterns upfront.

• 3 inputs × 3 values each = 27 patterns
• 5 inputs × 3 values each = 243 patterns
• 10 inputs × 3 values each = 59,049 patterns
• 20 inputs × 3 values each = 3.5 billion patterns ❌

Recommendation: For sparse data (>10 inputs or high cardinality), this library will struggle. Consider:

• Reducing dimensionality beforehand
• Using a sparse-friendly alternative for very high dimensions
• Accepting that dense enumeration is necessary for exact FID calculations

Missing Rows and Completeness

⚠️ FID requires all rows to be complete. Options:

1. fid() - Works only if no missing rows
2. neutral_completion_fid() - Fills missing rows uniformly
3. fid_cloud() etc. - Samples many completions and returns distributions

Choose based on your problem:

• All data observed? → Use fid()
• Sparse data with no prior? → Use fid_cloud() + neutral_completion_fid()
• Domain knowledge about missing patterns? → Use set_row() + neutral_completion_fid()

Normalization Assumptions

The library assumes uniform input distribution P(X) = 1/n_rows when computing MN_TPM.

If your observed data has non-uniform pattern frequencies, consider:

• Resampling or weighting your data upfront
• Using from_data() which properly counts observed patterns

Numerical Precision

• Small probabilities (< 1e-12) are clipped to avoid log(0)
• Very sparse distributions may have numerical issues
• Use np.clip() if you see NaN/inf in results

Reproducibility

• Sampling methods (fid_cloud, etc.) use random number generation
• Set np.random.seed() before calls if reproducibility is needed

np.random.seed(42)
cloud = tpm.fid_cloud(n_samples=5000)

Performance Considerations

• Bottleneck: Computing marginalizations for each FID sample
• For large n_samples (>100k): Memory usage grows ~10GB+
• Recommendation: Batch clouds or reduce n_samples if memory-constrained

---

FAQ

Q: What's the difference between fid() and neutral_completion_fid()?

A: fid() requires a complete TPM (raises error if missing rows). neutral_completion_fid() fills missing rows with uniform distributions and returns a single FID result.

Q: Can I compare FID results across different datasets?

A: Partially. FID values are relative to the datasets' entropies. Comparing synergy or independents directly is only valid if output entropies are similar.

Q: How do I know if my completion is "good"?

A: Sample many completions and check the bounds. Wide bounds = uncertain. Narrow bounds = robust. Compare across strategies (Dirichlet, deterministic, grid, edges) for consistency.

Q: What if I have ordered/continuous variables?

A: Discretize them first (bin into categories), then use the library.

Q: Can I export the TPM to CSV?

A: Use tpm.RN_TPM, tpm.input_info, and tpm.output_info to extract the data for export.

Q: How do I interpret "synergy" intuitively?

A: Synergy is the "magic" that only happens when inputs are jointly considered. Example: XOR (each input alone is useless; together they perfectly predict output).

---

Version & Author Notes

• Version: 2.0 (Retooled)
• Status: Active development
• Key Recent Fix: NaN handling in refresh_TPMs() — missing rows no longer contaminate MN_TPM

For issues or contributions, refer to the repository documentation.

---

Happy Information Decomposing! 📊