cmo-dataloader 0.1.0

Load a bulk of data from flat files or databases at once

Cmotions Dataloader

cmo_dataloader is a Python library created by The Analytics Lab, which is powered by Cmotions. This library combines our most used functionality when it comes to loading flat files or database tables into Python in bulk. Regarding flat files it also tries to take into account which separator, decimal sign and encoding to use when loading a file, this way the data is loaded automatically the best way possible, so you don't have to waste time fixing difficult bugs and errors. Nothing fancy, but very useful for our consultants. Mostly since this package is used in other packages, which makes our workflow simpler and more efficient.

Since we love to share what we do, why not also do that with our packages, that is why we've decided to make (almost) all of our packages open source, this way we hope to give back to the community that brings us so much. Enjoy!

Installation

Install cmo_dataloader using pip

pip install cmo-dataloader

If you want to be able to use the tutorial notebook or the code below, you also need to install dotenv using pip

pip install python-dotenv

Usage

import os
from dotenv import load_dotenv
from cmo_dataloader.resources import get_data_path
import cmo_dataloader as dl

load_dotenv()

# load flat files from a directory
dlc = dl.load_flat_file(get_data_path('formula_1'))
datafiles_dict = dlc.load()

# load tables from a SQL Server database
# using credentials stored in a .env file
ddu = dl.load_database(server=os.getenv("DB1_SERVER")
                       , db=os.getenv("DB1_DATABASE")
                       , server_name=os.getenv("DB1_SERVER_NAME")
                       , usr=os.getenv("DB1_USERNAME")
                       , pwd=os.getenv("DB1_PASSWORD")
                       , driver=os.getenv("DB1_DRIVER")
                       , port=os.getenv("DB1_PORT")
                       , tables=['dbo.mytable', 'dbo.anothertable'])
datafiles_dict = ddu.load()

# load tables from a SQL Server database
# using your Windows credentials
ddu = dl.load_database(server="localhost"
                       , db="mylocaldb"
                       , tables=['dbo.mytable', 'dbo.anothertable'])
datafiles_dict = ddu.load()

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to update tests as appropriate.

License

GNU General Public License v3.0

Contributors

Jeanine Schoonemann
Contact us