pandas-metaframe 0.0.1

A pandas DataFrame with DataFrame row and col!

MetaFrame

Metadata-aware DataFrames for pandas

MetaFrame is a lightweight layer on top of pandas for working with metadata-rich, hierarchical, and semantically structured datasets.

It is designed for situations where rows and columns are not just labels, but meaningful entities with structure, attributes, and relationships.

MetaFrame makes these tables easier to select, summarize, export, and—most importantly—reason about — while remaining fully pandas-compatible.

---

Why?

In pandas, it can becomes harder to work with:

• rows or columns that encode meaning across multiple levels
• metadata that lives “next to” the data instead of inside it
• selections that become unreadable
• exporting and re-importing structured MultiIndex tables

MetaFrame addresses these problems by treating structure as data.

---

When should I use MetaFrame?

Use MetaFrame if you work with:

• Scientific or experimental tables
• Excel files with semantic headers
• MultiIndex-heavy DataFrames
• Tables where rows and columns represent entities (samples, variables, runs)

If you only need flat, numeric DataFrames — pandas alone is perfect.

---

Installation

git clone https://gitlab.com/dwishsan/metaframe.git
cd metaframe
pip install .

or

pip install pandas-metaframe

---

Quick start

import metaframe as mf # or `import metaframe as pd`

df = mf.DataFrame(...)

# string-based filtering, query-like
df.gs["Sample:'1'", "ID:>2"]

# metadata-based selection
df.mfloc["Sample", "ID"]

# nice export
df.to_file("my_file.xlsx")

# easy reading, regardless of DataFrame dimensions
df = mf.DataFrame.from_file("my_file.xlsx")

# still pandas
df.groupby("Sample").mean()

---

The MetaFrame mental model

A metaframe.DataFrame is not a new data structure.

It:

• wraps a pandas.DataFrame
• preserves pandas behavior and performance expectations
• adds metadata-aware operations on top

But metaframe.DataFrame is not just a pandas.DataFrame with labels. It changes how you interact with the data.

It treats:

• rows as structured entities
• columns as structured entities
• metadata as first-class data

You can always access the underlying pandas DataFrame when needed.

---

Metadata as first-class data

Any DataFrame without MultiIndex and with a numeric index is a MetaData DataFrame, or MetaFrame.

MetaFrame can be attached to DataFrame:

• index/rows (MetaFrameRow, or MFR)
• columns (MetaFrameCol, or MFC)

From a DataFrame, MetaFrames are:

• queryable
• selectable
• settable

---

Selection & indexing

MetaFrame introduces semantic selectors:

Property	Purpose
q	General semantic selection (relying on pandas query)
gs	General semantic selection (relying on in-house Get-Strings)
mfloc	Metadata-aware label selection
mfiloc	Metadata-aware positional selection

These selectors are also settable, following the same rules as pandas loc & iloc.

---

Summaries & analytics

MetaFrame provides a structured summary system.

Summaries:

• range from basic to extensive
• can be customized
• integrate naturally with grouping operations

This makes exploratory analysis easier and more readable.

---

I/O & round-tripping

MetaFrame is designed for lossless export:

• Excel, with custom formats, extensive summaries and row + columns filters
• Structured text formats

---

Relationship to pandas

MetaFrame guarantees:

• pandas APIs remain available
• pandas methods behave as expected
• zero monkey-patching of pandas internals

You can drop back to pandas at any.

Documentation

📘 Wiki: conceptual guides and deep dives

📗 Cookbook: real-world patterns

📙 API reference: docstrings

Start with the Wiki if you’re new.

Status

MetaFrame is actively developed and used in real workflows.

The API is evolving, but core concepts are stable.

Contributions and feedback are welcome.

---

Full example

# -----------------------------------
#  Imports
# -----------------------------------
from numpy.random import seed, randn
import pandas as pd
import metaframe as mf
# or, to test it in pre-existing code
# import metaframe as pd

# -----------------------------------
#  Pandas DataFrame creation
# -----------------------------------

columns = pd.DataFrame(
    {
        'ID': [1, 2, 3],
        'Group': ['A', 'A', 'B'],
        'Name': ['UNK42', None, 'UNK37']
    }
)

index = pd.DataFrame(
    {
        'ID': [1, 2, 3, 4, 5],
        'Sample': ['1', '1', '2', '2', '2'],
        'Replicate': [1, 2, 1, 2, 3]
    }
)

seed(37)
df = pd.DataFrame(randn(5, 3), index=pd.MultiIndex.from_frame(index), columns=pd.MultiIndex.from_frame(columns))

# -----------------------------------
#  MetaFrame DataFrame creation
# -----------------------------------

df = mf.DataFrame(df)
df
"""
ID                          1         2         3
Group                       A         A         B
Name                    UNK42       nan     UNK37
ID Sample Replicate                              
1  1      1         -0.054464  0.674308  0.346647
2  1      2         -1.300346  1.518512  0.989824
3  2      1          0.277681 -0.448589  0.961966
4  2      2         -0.827579  0.534657  1.228386
5  2      3          0.519592 -0.063355 -0.034793
"""

# -----------------------------------
#  Get-strings
# -----------------------------------

df.gs["Sample:'1' or Replicate:>2", "Group:!B and Name:None"]
"""
ID                          2
Group                       A
Name                      nan
ID Sample Replicate          
1  1      1          0.674308
2  1      2          1.518512
5  2      3         -0.063355
"""

# -----------------------------------
#  Get-strings with RE
# -----------------------------------

df.gs[:, "Name:'.*7$'"]
"""
ID                          3
Group                       B
Name                    UNK37
ID Sample Replicate          
1  1      1          0.346647
2  1      2          0.989824
3  2      1          0.961966
4  2      2          1.228386
5  2      3         -0.034793
"""

# -----------------------------------
#  MetaFrame-based selection
# -----------------------------------

df.mfloc[
    ([1, 4, 2], ['ID', 'Sample']), 
    'Name'
]
"""
Name          UNK42       NaN     UNK37
ID Sample                              
2  1      -1.300346  1.518512  0.989824
5  2       0.519592 -0.063355 -0.034793
3  2       0.277681 -0.448589  0.961966
"""
df.mfiloc[
    (slice(None), 1), 
    ([0, 1],)
]
"""
ID             1         2
Group          A         A
Name       UNK42       nan
Sample                    
1      -0.054464  0.674308
1      -1.300346  1.518512
2       0.277681 -0.448589
2      -0.827579  0.534657
2       0.519592 -0.063355
"""

# -----------------------------------
#  MetaFrame indexing
# -----------------------------------

df.mfr = df.mfr.replace({'1': '3'})
df
"""
ID                          1         2         3
Group                       A         A         B
Name                    UNK42       nan     UNK37
ID Sample Replicate                              
1  3      1         -0.054464  0.674308  0.346647
2  3      2         -1.300346  1.518512  0.989824
3  2      1          0.277681 -0.448589  0.961966
4  2      2         -0.827579  0.534657  1.228386
5  2      3          0.519592 -0.063355 -0.034793
"""

df.mfc = df.mfc.merge(df.ms(axis=0))
df
"""
ID                          1         2         3   4   5
Group                       A         A         B nan nan
Name                    UNK42       nan     UNK37 nan nan
Sample                      3         3         2   2   2
Replicate                   1         2         1   2   3
ID Sample Replicate                                      
1  3      1         -0.054464  0.674308  0.346647 NaN NaN
2  3      2         -1.300346  1.518512  0.989824 NaN NaN
3  2      1          0.277681 -0.448589  0.961966 NaN NaN
4  2      2         -0.827579  0.534657  1.228386 NaN NaN
5  2      3          0.519592 -0.063355 -0.034793 NaN NaN
"""

# -----------------------------------
#  MetaFrame-based DataFrame set
# -----------------------------------

df.gs[:, "ID:4,5"] = [0, 1]
df
"""
ID                          1         2         3    4    5
Group                       A         A         B  nan  nan
Name                    UNK42       nan     UNK37  nan  nan
Sample                      3         3         2    2    2
Replicate                   1         2         1    2    3
ID Sample Replicate                                        
1  3      1         -0.054464  0.674308  0.346647  0.0  1.0
2  3      2         -1.300346  1.518512  0.989824  0.0  1.0
3  2      1          0.277681 -0.448589  0.961966  0.0  1.0
4  2      2         -0.827579  0.534657  1.228386  0.0  1.0
5  2      3          0.519592 -0.063355 -0.034793  0.0  1.0
"""

# -----------------------------------
#  Summaries
# -----------------------------------

df.summary_basic()
"""
           Rows  Columns  Cells
DataFrame     5        5     25
MSR           5        3     15
MSC           5        5     25
"""

df.summary_whole()
"""
                                           DataFrame
dtype   Mode     Metric        Type                 
all     Describe Num. elements count           25.00
                 NAs           count            0.00
                               %                0.00
Numeric Describe Num. elements count           25.00
                               %              100.00
                 mean          mean             0.37
                 std           std              0.68
                 min           min             -1.30
                 25%           percentile       0.00
                 50%           percentile       0.35
                 75%           percentile       1.00
                 max           max              1.52
                 sum           sum              9.32
        Custom   zeros         count            5.00
                               %               20.00
                 filled        count           14.00
                               %               56.00
"""

df.summary()
"""
ID                                              1       2       3       4       5
Sample                                          3       3       2       2       2
Replicate                                       1       2       1       2       3
dtype   Mode     Metric        Type                                              
all     Describe Num. elements count         5.00    5.00    5.00    5.00    5.00
                 NAs           count         0.00    0.00    0.00    0.00    0.00
                               %             0.00    0.00    0.00    0.00    0.00
Numeric Describe Num. elements count         5.00    5.00    5.00    5.00    5.00
                               %           100.00  100.00  100.00  100.00  100.00
                 mean          mean          0.39    0.44    0.36    0.39    0.28
                 std           std           0.45    1.12    0.62    0.83    0.47
                 min           min          -0.05   -1.30   -0.45   -0.83   -0.06
                 25%           percentile    0.00    0.00    0.00    0.00   -0.03
                 50%           percentile    0.35    0.99    0.28    0.53    0.00
                 75%           percentile    0.67    1.00    0.96    1.00    0.52
                 max           max           1.00    1.52    1.00    1.23    1.00
                 sum           sum           1.97    2.21    1.79    1.94    1.42
        Custom   zeros         count         1.00    1.00    1.00    1.00    1.00
                               %            20.00   20.00   20.00   20.00   20.00
                 filled        count         3.00    3.00    3.00    3.00    2.00
                               %            60.00   60.00   60.00   60.00   40.00
"""

df.groupby('Sample').apply(lambda x: x.summary())
"""
ID                                                     3       4       5       1       2
Sample                                                 2       2       2       3       3
Replicate                                              1       2       3       1       2
Sample dtype   Mode     Metric        Type                                              
2      all     Describe Num. elements count         5.00    5.00    5.00     NaN     NaN
                        NAs           count         0.00    0.00    0.00     NaN     NaN
                                      %             0.00    0.00    0.00     NaN     NaN
       Numeric Describe Num. elements count         5.00    5.00    5.00     NaN     NaN
                                      %           100.00  100.00  100.00     NaN     NaN
                        mean          mean          0.36    0.39    0.28     NaN     NaN
                        std           std           0.62    0.83    0.47     NaN     NaN
                        min           min          -0.45   -0.83   -0.06     NaN     NaN
                        25%           percentile    0.00    0.00   -0.03     NaN     NaN
                        50%           percentile    0.28    0.53    0.00     NaN     NaN
                        75%           percentile    0.96    1.00    0.52     NaN     NaN
                        max           max           1.00    1.23    1.00     NaN     NaN
                        sum           sum           1.79    1.94    1.42     NaN     NaN
               Custom   zeros         count         1.00    1.00    1.00     NaN     NaN
                                      %            20.00   20.00   20.00     NaN     NaN
                        filled        count         3.00    3.00    2.00     NaN     NaN
                                      %            60.00   60.00   40.00     NaN     NaN
3      all     Describe Num. elements count          NaN     NaN     NaN    5.00    5.00
                        NAs           count          NaN     NaN     NaN    0.00    0.00
                                      %              NaN     NaN     NaN    0.00    0.00
       Numeric Describe Num. elements count          NaN     NaN     NaN    5.00    5.00
                                      %              NaN     NaN     NaN  100.00  100.00
                        mean          mean           NaN     NaN     NaN    0.39    0.44
                        std           std            NaN     NaN     NaN    0.45    1.12
                        min           min            NaN     NaN     NaN   -0.05   -1.30
                        25%           percentile     NaN     NaN     NaN    0.00    0.00
                        50%           percentile     NaN     NaN     NaN    0.35    0.99
                        75%           percentile     NaN     NaN     NaN    0.67    1.00
                        max           max            NaN     NaN     NaN    1.00    1.52
                        sum           sum            NaN     NaN     NaN    1.97    2.21
               Custom   zeros         count          NaN     NaN     NaN    1.00    1.00
                                      %              NaN     NaN     NaN   20.00   20.00
                        filled        count          NaN     NaN     NaN    3.00    3.00
                                      %              NaN     NaN     NaN   60.00   60.00
"""

# -----------------------------------
#  Natural sorting & ordering
# -----------------------------------

df.mfc.natsort_values('Name', axis=1)
df.mfr.order_values({'Sample': [2, 3], 'Replicate': None})
df.auto_sort()

# -----------------------------------
#  I/O
# -----------------------------------

df.to_file('path/to/my/excel_file.xlsx')
df = mf.DataFrame.from_input('path/to/my/excel_file.xlsx')

License

MIT © Audric Cologne