pandas-tfrecords 0.1.6

Quick start

pip install pandas-tfrecords

import pandas as pd
from pandas_tfrecords import pd2tf, tf2pd

df = pd.DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', 'c'], 'C': [[1, 2], [3, 4], [5, 6]]})

# local
pd2tf(df, './tfrecords')
my_df = tf2pd('./tfrecords')

# S3
pd2tf(df, 's3://my-bucket/tfrecords')
my_df = tf2pd('s3://my-bucket/tfrecords')

boolean support (v0.1.6+):

import pandas as pd
from pandas_tfrecords import pd2tf, tf2pd

df = pd.DataFrame({'A': [True, False, True], 'B': ['a', 'b', 'c'], 'C': [[1, False], [3, True], [5, False]]})

pd2tf(df, './tfrecords')

my_df = tf2pd('./tfrecords', schema={'A': bool, 'B': str, 'C': [int, bool]})

# or if needs 0/1 instead of True/False
my_df = tf2pd('./tfrecords', schema={'A': int, 'B': str, 'C': [int, int]})

# or just skip schema, int used by default if bool isn't specified
my_df = tf2pd('./tfrecords')

Converted types

bytes, str -> tf.string
int, np.integer, bool, np.bool_ -> tf.int64
float, np.floating -> tf.float32
list, np.ndarray of bytes, str, int, np.integer, bool, np.bool_, float, np.floating -> sequence of tf.string, tf.int64, tf.float32

tf.string -> bytes
tf.int64 -> int, bool
tf.float32 -> float
sequence of tf.string, tf.int64, tf.float32 -> list of bytes, int, bool, float

API

pandas_tfrecords.pandas_to_tfrecords(df, folder, compression_type='GZIP', compression_level=9, columns=None, max_mb=50)

Arguments:

• df - pandas dataframe. Please keep in mind above info about nested sequences.

• folder - folder to save tfrecords, local or S3. Please be sure that it doesn’t contain other files or folders, if you want to read from this folder then.

• compression_type='GZIP' - compression types: 'GZIP', 'ZLIB', None. If None not compressed.

• compression_level=9 - compression level 0…9.

• columns=None - list of columns to save, if None all columns will be saved.

• max_mb=50 - maximum size of uncompressed data to save in single file. If dataframe total size is bigger than this limit, then it will be splitted to several files. If None it isn’t limited and single file will be saved.

alias pandas_tfrecords.pd2tf

pandas_tfrecords.tfrecords_to_pandas(file_paths, schema=None, compression_type='auto', cast=True)

Arguments:

• file_paths - One or sequence of file paths or folders, local or S3, to read tfrecords from.

• schema=None - If None schema will be detected automatically. But you can specify which columns you want to read only. It should be a dict, which keys are column names and values are column data types: str (or bytes), int, float, and for sequences it should be wrapped to list: [str] (or [bytes]), [int], [float]. For example:

df = pd.DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', 'c'], 'C': [[1, 2], [3, 4], [5, 6]]})
print(df)
   A  B       C
0  1  a  [1, 2]
1  2  b  [3, 4]
2  3  c  [5, 6]

pd2tf(df, './tfrecords')
tf2pd('./tfrecords', schema={'A': int, 'C': [int]})
   A       C
0  1  [1, 2]
1  2  [3, 4]
2  3  [5, 6]

• compression_type='auto' - compression type: 'auto', 'GZIP', 'ZLIB', None.

• cast=True - if True it casts bytes data after converting from tf.string. It tries to cast it to int, float and str sequentially. If it’s not possible, otherwise keeps as is.

alias pandas_tfrecords.tf2pd