Data Science tools to ease access and use of data and models
Project description
Data Science Manager 👨💻
Data Science tools to ease access and use of data and models
Install
The easiest way to install scikit-learn is using pip
:
pip install dsmanager
or poetry
poetry add dsmanager
or conda
conda install dsmanager
Multiple sub dependencies are available depending on the needs:
pip install dsmanager[sharepoint] # Add Sharepoint source handling
pip install dsmanager[salesforce] # Add SalesForce source handling
pip install dsmanager[kaggle] # Add Kaggle source handling
pip install dsmanager[snowflake] # Add Snowflkae source handling
pip install dsmanager[mysql] # Add MySQL source handling
pip install dsmanager[pgsql] # Add PostgreSQL source handling
Usage
The DS Manager has 3 main components:
- A DataManager component
- A Controller component
- A Model component
DataManager
The DataManager allows to manage different types of data sources among which we can mention:
- Local (local and online files)
- Http (Http requests)
- Ftp (Ftp hosted files)
- Sql (Sql database tables)
- Sharepoint (Microsoft OneDrive files)
- SalesForce (SalesForce classes)
- Kaggle (Kaggle datasets)
from dsmanager import DataManager
dm = DataManager("data/metadata.json")
Development
Source code
You can check the latest sources with the command:
git clone https://gitlab.com/bigrayou/dsmanager
Source code
After installation, you can launch the test suite from outside the dsmanager directory (you will need to have pytest >= 5.3.1 installed):
pytest -v
Dependencies
The DSManager requires:
- dash >=2.7.1,<3.0.0
- explainerdashboard >=0.4.0,<0.5.0
- llvmlite >=0.39.1,<0.40.0
- numba >=0.56.4,<0.57.0
- numexpr >=2.8.4,<3.0.0
- numpy >=1.23.3,<2.0.0
- openpyxl >=3.0.10,<4.0.0
- pandas >=1.5.0,<2.0.0
- pickle-mixin >=1.0.2,<2.0.0
- python-dotenv >=0.21.0,<0.22.0
- requests >=2.28.1,<3.0.0
- scikit-learn >=1.2.0,<2.0.0
- setuptools >=65.6.3,<66.0.0
- sqlalchemy >=1.4.45,<2.0.0
- sweetviz >=2.1.4,<3.0.0
- tqdm >=4.64.1,<5.0.0
Optionnaly, the DSManager could require:
- azure-common >=1.1.28,<2.0.0
- azure-storage-blob >=12.14.1,<13.0.0
- azure-storage-common >=2.1.0,<3.0.0
- kaggle >=1.5.12,<2.0.0
- shareplum >=0.5.1,<0.6.0
- simple-salesforce >=1.12.2,<2.0.0
- snowflake-sqlalchemy >=1.4.4,<2.0.0
- psycopg2-binary >=2.9.5,<3.0.0
- mysqlclient >=2.1.1,<3.0.0
Author
👤 Rayane Amrouche
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dsmanager-1.0.13.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e9d4fb4fd921df89bf41b09631a074ab94c740ffb6a281532a9da2df934b253 |
|
MD5 | 7824c8f9fe5883e04805404f9e7391ee |
|
BLAKE2b-256 | de9e1713123fd0327c4ed00387d4eeb4a05ce6f267dbc25b99ad50f44a5d90bd |