dsws 0.14

Data Science Work Space for Jupyter & CDSW

Data Science WorkSpace (dsws)

DSWS is Python package providing conventions for configuration and use of existing python packages. Intent is to make disparate source data platforms constantly accessible in python ad-hoc and workflow analyses independent of data source or REPL/RunTime environment.

Main Features

DSWS offers no functionality beyond what’s in dependency libraries. Instead, it provides conventions to consistently and concisely use existing libraries. Conventions include:

• Configurations for data access are instantiated at the beginning of a session, specific to analysis.
• Configurations include not just source data connection, but also environmental preferences in the connection, like default database and resource queue.
• Common method syntax for connection and query.
• For ipython environments, include line and cell magic for executing sql.
• Consistent approach for profile startup scripts.
• Ability to transition to configured CLI from within python session.

Environments

DSWS originated to address connectivity complexity of connecting to popular hadoop distributions (HDP, CDH) from edge node. It now includes conventions to connect from cluster edge node virtual environment, local device virtual environment, cdsw, jupyter notebooks, or quickstart/sandbox. DSWS is intended to provide a consistent user experience within any of these environments.

Included Libraries & CLIs

Existing libraries, classes, configs, and type.

library / cli	default name	class	type	Incl	Exp	Roadmap
impyla	impyla	ws_impyla.py	conn	X
(Hive) beeline	hbl	ws_hbl.py	cli	X
impala-shell	imp	ws_imp.py	cli	X
phoenixdb	pheonix	ws_pheonix.py	conn			X
hbase (shell)	hbase	ws_hbase.py	cli			X
pyspark	spark (sql)	ws_spark.py	sess	X
dask-yarn	dask	ws_dask.py	sess			X
pandasql	ps	ws_ps.py				X
ibis	ibis	ws_ibis.py	conn			X
MySQL-python	mysql	ws_mysql.py	conn			X
pydruid	druid	ws_druid.py	conn			X