dummy flat text/csv file generator, typical usage scenario can be load / stress / performance testing of file-processing data tools
Project description
dummy_file_generator
version 1.0.7
a dummy csv or flat text files generator written in Python 3.7
This tool is able to generate dummy csv or flat txt files based on the configuration settings you setup for your project(s).
How to install and run the tool as CLI
How to install and run the tool as an imported package
How to setup a new dummy file generator project
How to add a new source dataset for your project
How to install and run the tool as CLI
One common usage scenario can be load / stress / performance testing of file-processing data tools, allowing you to generate the files needed from a command line.
To install:
git clone https://github.com/datahappy1/dummy_file_generator c:\dummy_file_generator\
pip install -r requirements.txt
*You are strongly encouraged to use the Python virtual environment
To run:
The CLI tool needs these mandatory arguments defining:
- projectname
--projectname
or-pn
(mandatory argument, based on the projectname, the dummy file specific settings from /configurables/config.json file are loaded ), - absolutepath
--absolutepath
or-ap
(mandatory argument) defining the full output file path to the file you wish to generate
The CLI tool can consume these optional arguments defining:
- filesize
--filesize
or-fs
(optional argument) defining the desired size (in kBs) of the output file - rowcount
--rowcount
or-rc
(optional argument) defining the desired row count of the output file
*Note if you do NOT specify the filesize and do NOT specify the rowcount, the default row_count value ( set to 100 ) from
settings.py will be used ( or the value you provide in the default_rowcount
optional argument)
*these 5 optional arguments can be used to override values in configurables/settings.py
:
- logging_level
--logging_level
or-ll
(optional argument) defining the Python logging level - default_rowcount
--default_rowcount
or-drc
(optional argument) defining the rowcount fallback value when neither row_count,neither file_size set - file_encoding
--file_encoding
or-fen
(optional argument) defining the generated files encoding - file_line_ending
--file_line_ending
or-fle
(optional argument) defining the file line ending - csv_value_separator
--csv_value_separator
or-cvs
(optional argument) defining the .csv file value separator
*these arguments are typically needed when running the tool as an importable library, but you can use them to change the location of the source data files and the config json with the project setup even with this tool running as CLI:
- data_files_location
--data_files_location
or-dfl
(optional argument) defining the path to the source .txt data files - config_json_path
--config_json_path
or-cjp
(optional argument) defining the custom path to your config.json file
Example how to run :
- with the -fs argument to set the desired filesize of 256 kB :
cd c:\dummy_file_generator\dummy_file_generator
python c:\dummy_file_generator\dummy_file_generator\__main__.py -pn dummy1 -ap c:\myfiles\dummy1file.csv -fs 256
- with the -rc argument to set the desired rowcount of 1000 rows :
cd c:\dummy_file_generator\dummy_file_generator
python c:\dummy_file_generator\dummy_file_generator\__main__.py -pn dummy1 -ap c:\myfiles\dummy1file.csv -rc 1000
Pytest testing
Pytest unit and performance tests are also a part of this tool.
You can install Pytest with pip install pytest
To run tests:
cd c:\dummy_file_generator\dummy_file_generator
pytest
How to install and run the tool as an imported package
One common usage scenario can be load / stress / performance testing of file-processing data tools, where you can now generate dummy text files during the test fixtures / setup.
To install:
pip install dummy-file-generator
To run:
The dummy file generator imported package needs these mandatory arguments defining:
- projectname
--projectname
or-pn
(mandatory argument, based on the projectname, the dummy file specific settings from config.json file are loaded ), - absolutepath
--absolutepath
or-ap
(mandatory argument) defining the full output file path to the file you wish to generate
The dummy file generator imported package can consume these optional arguments defining:
- filesize
--filesize
or-fs
(optional argument) defining the desired size (in kBs) of the output file - rowcount
--rowcount
or-rc
(optional argument) defining the desired row count of the output file
*Note if you do NOT specify the filesize and do NOT specify the rowcount, the default row_count value ( set to 100 ) from
settings.py will be used ( or the value you provide in the default_rowcount
optional argument)
- data_files_location
--data_files_location
or-dfl
(optional argument) defining the path to the source .txt data files - config_json_path
--config_json_path
or-cjp
(optional argument) defining the custom path to your config.json file - logging_level
--logging_level
or-ll
(optional argument) defining the Python logging level - default_rowcount
--default_rowcount
or-drc
(optional argument) defining the rowcount fallback value when neither row_count,neither file_size set - file_encoding
--file_encoding
or-fen
(optional argument) defining the generated files encoding - file_line_ending
--file_line_ending
or-fle
(optional argument) defining the file line ending - csv_value_separator
--csv_value_separator
or-cvs
(optional argument) defining the .csv file value separator
Example how to run :
from dummy_file_generator import DummyFileGenerator as DFG, DummyFileGeneratorException
def generate_dummy_file(project_name, absolute_path, file_size,
data_files_location, config_json_path):
kwargs = {"project_name": project_name,
"absolute_path": absolute_path,
"file_size": file_size,
"logging_level": "INFO",
"data_files_location": data_files_location,
"config_json_path": config_json_path,
"file_encoding": "utf8",
"csv_value_separator": "%",
"file_line_ending": "\n"
}
obj = DFG(**kwargs)
try:
DFG.generate_file(obj)
except DummyFileGeneratorException as DFG_ERR:
raise(DFG_ERR)
generate_dummy_file(project_name="dummy1",
absolute_path="c:\myfiles\dummy1.csv",
file_size=1024,
data_files_location="c:\\dummy_file_generator\dummy_file_generator\data_files",
config_json_path="c:\\dummy_file_generator\dummy_file_generator\configurables\config.json"
)
How to setup a new dummy file generator project
You need to generate dummy files based on the content of the text files in your "data_files" folder, and these source text files need to be setup for instance like this:
This tool picks random item from each of the files configured for your project in config.json and uses these values to populate the data for "columns" for each written row.
- How to generate a .csv file
Let's say you need to generate a dummy .csv file containing 3 columns for Names, Dates and IDs. The "project" JSON object in your config.json would need to be setup like:
{
"project_name":"dummy1",
"file_type":"csv",
"header":true,
"columns":[
{
"column_name":"Name",
"datafile":"firstnames.txt"
},
{
"column_name":"Date",
"datafile":"dates.txt"
},
{
"column_name":"ID",
"datafile":"IDs.txt"
}
]
}
- How to generate a .txt flat file:
Let's say you need to generate a dummy .txt flat file containing 3 columns for Names, Dates and IDs with specific column lengths defined. The "project" JSON object in your config.json would need to be setup like:
{
"project_name":"dummy1",
"file_type":"flat",
"header":true,
"columns":[
{
"column_name":"Name",
"column_len":6,
"datafile":"firstnames.txt"
},
{
"column_name":"Date",
"column_len":10,
"datafile":"dates.txt"
},
{
"column_name":"ID",
"column_len":9,
"datafile":"IDs.txt"
}
]
}
How to add a new source dataset for your project
Whenever you need to add a new source .txt file in the data_files folder, just add it to your data_files
folder.
If running as a standalone CLI tool, the data_files folder is located here:
dummy_file_generator/data_files
When running as an imported package, the data_files folder is whereever you set it in
the argument data_files_location.
Now you can use this new data file in your project setup in config.json
file.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for dummy-file-generator-1.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50154addd523c1d2b8b2220026c668397808792a38327516d109d3f80901a563 |
|
MD5 | c9607bb3754f8802611da66decb3000b |
|
BLAKE2b-256 | 0cec41055fa88673ed9c52c5b77dbb0a04ff85125400927a1a8ed479c0bfa014 |