Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

utility for generating mock data sets as pandas dataframes

Project description

DFMock

pandas dataframe mocker for python3.

Why?

Sometimes you just need mock data that is already in pandas datatypes. There are lots of data gen tools out there, but most produce CSVs or some such that needs to be consumed by pandas; this adds a volitile point to your tests. What if your logic is correct, but the datatypes didn't import correctly from the CSV? So DFMock aims to generate mock dataframes where the datatypes are controlled.

Installation

Use pip

pip install dfmock

You can also get the source here

The API

using is simple. import into your module, select your column names and data types, set the number of columns, and generate your frame. from dfmock import DFMock

columns = { "hamburger":"string",
            "hot_dog":"integer",
            "shoelace":"timedelta"
          }
dfmock = DFMock(count=100, cols=columns)
dfmock.generate_dataframe()

my_mocked_dataframe = dfmock.dataframe

Pandas data types supported:

PANDAS TYPE DICT VALUE
object string
int64 int
float64 float
bool bool
datetime64 datetime
timedelta timedelta
category category

so to make a dataframe with column "banana" that is an int64 and "rockstar" that is a category type:

dfmock.cols = {"banana":"int","rockstar":"category"}

pretty simple.

Grow to Size

sometimes you need your dataset to be a certain memory size instead of row size. The grow_dataframe_to_size() allows you to grow a frame until it reaches or passes the given size (in MB). Need a 10MB dataframe?

dfmock.generate_dataframe()
dfmock.size
## returns 0.2 MB

dfmock.grow_dataframe_to_size(10)
dfmock.size
## returns ~10 MB

Grouping

NOTE: timedelta and category datatypes are not currently supported. Sometimes you need a column you can aggregate on with a given data type. For example, you may want 1M rows with one of 4 datetime values (maybe representing 4 observation reporting timestamps?). You can do this using grouping. a grouped column is declared by passing a dict as the data type value with the params for the grouped column as keys. Like this:

columns = {"amazing_grouped_column_with_3_values": { "option_count":3, "option_type":"string"}}

This will give you a column with only 3 distinct values and (nearly) equal distribution. If you need to control the distribution you can pass the histogram argument. This is useful if, for instance, you want a dataset with 4 datetime values and want one value for 50% of records, another for 30%, and the remaining values 20%. You would delcare this distribution like so:

columns = {"super_cool_grouped_column_with_histogram": {"option_count":4, "option_type":"datetime","histogram":(5,3,2,2,)}}

The integers you pass histogram need to add up to 10 with the same number of values as option count (ie. if option_count = 5, you need 5 values). This caps option count at 10 currently for histogram. Your desired row count must also be divisible by 10 - this keeps the math simple.

Contribution

We welcome pull requests!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dfmock, version 0.0.14
Filename, size File type Python version Upload date Hashes
Filename, size dfmock-0.0.14-py3-none-any.whl (5.7 kB) File type Wheel Python version py3 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page