No-Copy Operations on Immutable Data

Because all StaticFrame data is immutable, arrays can be safely shared between containers without defensive copies or complicated copy-on-write (CoW) management. Structural operations (relabeling, selecting columns, setting an index, concatenating) reuse the same underlying NumPy arrays and are often an order of magnitude (or more) faster than Pandas.

>>> f1 = ff.parse('s(10_000,1000)|v(int,int,str,float)')
>>> f2 = ff.parse('s(10_000,1000)|v(int,bool,bool,float)')
>>> df1, df2 = f1.to_pandas(), f2.to_pandas()
>>> compare('rename', lambda: f1.rename(index='foo'), lambda: df1.rename_axis('foo'), number=10000)
rename           StaticFrame     8.2 µs | Pandas   567.0 µs | 68.8x
>>> compare('set index', lambda: f1.set_index(0), lambda: df1.set_index(0, drop=False), number=2000)
set index        StaticFrame    64.7 µs | Pandas   596.1 µs | 9.2x
>>> compare('select columns', lambda: f1[[10, 50, 100, 500]], lambda: df1[[10, 50, 100, 500]], number=10000)
select columns   StaticFrame     5.5 µs | Pandas    69.0 µs | 12.6x
>>> compare('concat (axis 1)', lambda: sf.Frame.from_concat((f1, f2), axis=1, columns=sf.IndexAutoFactory), lambda: pd.concat((df1, df2), axis=1), number=2000)
concat (axis 1)  StaticFrame    62.3 µs | Pandas  1781.9 µs | 28.6x

Faster Computation

Version 5 extends this performance to commonly used group-by, reduce, pivot, join and related operations. A single one-million-row fixture serves these examples:

>>> f = (ff.parse('s(1_000_000,5)|v(int,int,float,float,float)').relabel(columns=('key', 'r', 'x', 'y', 'z')).assign['key'].apply(lambda s: 'g' + (s % 1000).astype('U4')).assign['r'].apply(lambda s: s % 100_000))
>>> df = f.to_pandas()
>>> compare('group-by', lambda: f.iter_group('key').reduce.from_label_map({'x': np.sum, 'y': np.sum}).to_frame(), lambda: df.groupby('key')[['x', 'y']].sum(), number=10)
group-by         StaticFrame    11.7 ms | Pandas    23.7 ms | 2.0x
>>> compare('reduce', lambda: f.iter_group('key').reduce.from_label_map({'x': np.sum, 'y': np.max}).to_frame(), lambda: df.groupby('key').agg({'x': 'sum', 'y': 'max'}), number=10)
reduce           StaticFrame    11.6 ms | Pandas    23.8 ms | 2.0x
>>> compare('pivot', lambda: f.pivot('key', data_fields='x', func=np.sum), lambda: df.pivot_table(index='key', values='x', aggfunc='sum'), number=10)
pivot            StaticFrame    13.2 ms | Pandas    22.7 ms | 1.7x
>>> compare('rank', lambda: f['r'].rank_mean(), lambda: df['r'].rank(method='average'), number=20)
rank             StaticFrame    18.5 ms | Pandas   108.5 ms | 5.9x
>>> compare('row-apply', lambda: f.iter_tuple(axis=1).apply(lambda t: t.x * 2 + t.y - t.z), lambda: df.apply(lambda t: t.x * 2 + t.y - t.z, axis=1), number=5)
row-apply        StaticFrame   485.8 ms | Pandas  5348.0 ms | 11.0x

For join, two frames are built sharing the same 500,000 integer keys, relabeled as strings:

>>> left = ff.parse('s(500_000,1)|v(float)|i(I,int)').relabel(columns=('lv',))
>>> right = ff.parse('s(500_000,1)|v(float)|i(I,int)').relabel(columns=('rv',))
>>> keys = 'k' + left.index.values.astype('U12')
>>> sf_left = left.relabel(index=keys).sort_index()
>>> sf_right = right.relabel(index=keys)
>>> df_left, df_right = sf_left.to_pandas(), sf_right.to_pandas()
>>> compare('join', lambda: sf_left.join_left(sf_right, left_depth_level=0, right_depth_level=0), lambda: df_left.join(df_right, how='left'), number=10)
join             StaticFrame    35.7 ms | Pandas    42.1 ms | 1.2x

For the complete performance suite (dozens of comparisons across construction, selection, iteration, grouping, and reduction) run python -m static_frame.profile --performance "*".