Web Client for Visualizing Pandas Objects
Project description
Getting Started
Setup/Activate your environment and install the egg
Python 3
# create a virtualenv, if you haven't already created one
$ python3 -m venv ~/pyenvs/dtale
$ source ~/pyenvs/dtale/bin/activate
# install dtale egg (important to use the "--upgrade" every time you install so it will grab the latest version)
$ pip install --upgrade dtale
Python 2
# create a virtualenv, if you haven't already created one
$ python -m virtualenv ~/pyenvs/dtale
$ source ~/pyenvs/dtale/bin/activate
# install dtale egg (important to use the "--upgrade" every time you install so it will grab the latest version)
$ pip install --upgrade dtale
Now you will have to ability to use D-Tale from the command-line or within a python-enabled terminal
Command-line
Base CLI options (run dtale --help to see all options available)
Prop |
Description |
---|---|
--host |
the name of the host you would like to use (most likely not needed since socket.gethostname() should figure this out) |
--port |
the port you would like to assign to your D-Tale instance |
--name |
an optional name you can assign to your D-Tale instance (this will be displayed in the <title> & Instances popup) |
--debug |
turn on Flask’s “debug” mode for your D-Tale instance |
--no-reaper |
flag to turn off auto-reaping subprocess (kill D-Tale instances after an hour of inactivity), good for long-running displays |
--open-browser |
flag to automatically open up your server’s default browser to your D-Tale instance |
Loading data from arctic
dtale --arctic-host mongodb://localhost:27027 --arctic-library jdoe.my_lib --arctic-node my_node --arctic-start 20130101 --arctic-end 20161231
Loading data from CSV
dtale --csv-path /home/jdoe/my_csv.csv --csv-parse_dates date
Loading data from a Custom loader - Using the DTALE_CLI_LOADERS environment variable, specify a path to a location containing some python modules - Any python module containing the global variables LOADER_KEY & LOADER_PROPS will be picked up as a custom loader - LOADER_KEY: the key that will be associated with your loader. By default you are given arctic & csv (if you use one of these are your key it will override these) - LOADER_PROPS: the individual props available to be specified. - For example, with arctic we have host, library, node, start & end. - If you leave this property as an empty list your loader will be treated as a flag. For example, instead of using all the arctic properties we would simply specify --arctic (this wouldn’t work well in arctic’s case since it depends on all those properties) - You will also need to specify a function with the following signature def find_loader(kwargs) which returns a function that returns a dataframe or None - Here is an example of a custom loader:
from dtale.cli.clickutils import get_loader_options
'''
IMPORTANT!!! This global variable is required for building any customized CLI loader.
When find loaders on startup it will search for any modules containing the global variable LOADER_KEY.
'''
LOADER_KEY = 'testdata'
LOADER_PROPS = ['rows', 'columns']
def test_data(rows, columns):
import pandas as pd
import numpy as np
import random
from past.utils import old_div
from pandas.tseries.offsets import Day
from dtale.utils import dict_merge
import string
now = pd.Timestamp(pd.Timestamp('now').date())
dates = pd.date_range(now - Day(364), now)
num_of_securities = max(old_div(rows, len(dates)), 1) # always have at least one security
securities = [
dict(security_id=100000 + sec_id, int_val=random.randint(1, 100000000000),
str_val=random.choice(string.ascii_letters) * 5)
for sec_id in range(num_of_securities)
]
data = pd.concat([
pd.DataFrame([dict_merge(dict(date=date), sd) for sd in securities])
for date in dates
], ignore_index=True)[['date', 'security_id', 'int_val', 'str_val']]
col_names = ['Col{}'.format(c) for c in range(columns)]
return pd.concat([data, pd.DataFrame(np.random.randn(len(data), columns), columns=col_names)], axis=1)
# IMPORTANT!!! This function is required for building any customized CLI loader.
def find_loader(kwargs):
test_data_opts = get_loader_options(LOADER_KEY, kwargs)
if len([f for f in test_data_opts.values() if f]):
def _testdata_loader():
return test_data(int(test_data_opts.get('rows', 1000500)), int(test_data_opts.get('columns', 96)))
return _testdata_loader
return None
In this example we simplying building a dataframe with some dummy data based on dimensions specified on the command-line: - --testdata-rows - --testdata-columns
Here’s how you would use this loader:
DTALE_CLI_LOADERS=./path_to_loaders bash -c 'dtale --testdata-rows 10 --testdata-columns 5'
Python Terminal
This comes courtesy of PyCharm Feel free to invoke python or ipython directly and use the commands in the screenshot above and it should work #####Additional functions available programatically
import dtale
import pandas as pd
df = pd.DataFrame([dict(a=1,b=2,c=3)])
# Assigning a reference to a running D-Tale process
d = dtale.show(df)
# Accessing data associated with D-Tale process
tmp = d.data.copy()
tmp['d'] = 4
# Altering data associated with D-Tale process
# FYI: this will clear any front-end settings you have at the time for this process (filter, sorts, formatting)
d.data = tmp
# Shutting down D-Tale process
d.kill()
# using Python's `webbrowser` package it will try and open your server's default browser to this process
d.open_browser()
# There is also some helpful metadata about the process
d._port # the process's port
d._url # the url to access the process
UI
Once you have kicked off your D-Tale session please copy & paste the link on the last line of output in your browser
The information in the upper right-hand corner is similar to saslook - lower-left => row count - upper-right => column count - clicking the triangle displays the menu of standard functions (click outside menu to close it)
Selecting/Deselecting Columns - to select a column, simply click on the column header (to deselect, click the column header again) - You’ll notice that the columns you’ve selected will display in the top of your browser
For Developers
Getting Started
Clone the code (git clone ssh://git@github.com:manahl/dtale.git), then start the backend server:
$ git clone ssh://git@github.com:manahl/dtale.git
# install the dependencies
$ python setup.py develop
# start the server
$ python dtale --csv-path /home/jdoe/my_csv.csv --csv-parse_dates date
You can also run dtale from PyDev directly.
You will also want to import javascript dependencies and build the source:
$ npm install
# 1) a persistent server that serves the latest JS:
$ npm run watch
# 2) or one-off build:
$ npm run build
Running tests
The usual npm test command works:
$ npm test
You can run individual test files:
$ TEST=static/__tests__/dtale/DataViewer-base-test.jsx npm run test-file
Linting
You can lint all the JS and CSS to confirm there’s nothing obviously wrong with it:
$ npm run lint -s
You can also lint individual JS files:
$ npm run lint-js-file -s -- static/dtale/DataViewer.jsx
Formatting JS
You can auto-format code as follows:
$ npm run format
Docker development
You can build python 27-3 & run D-Tale as follows:
$ yarn run build
$ docker-compose build dtale_2_7
$ docker run -it --network host dtale_2_7:latest
$ python
>>> import pandas as pd
>>> df = pd.DataFrame([dict(a=1,b=2,c=3)])
>>> import dtale
>>> dtale.show(df)
Then view your D-Tale instance in your browser using the link that gets printed
You can build python 36-1 & run D-Tale as follows:
$ yarn run build
$ docker-compose build dtale_3_6
$ docker run -it --network host dtale_3_6:latest
$ python
>>> import pandas as pd
>>> df = pd.DataFrame([dict(a=1,b=2,c=3)])
>>> import dtale
>>> dtale.show(df)
Then view your D-Tale instance in your browser using the link that gets printed
Documentation
Have a look at the detailed documentation.
Requirements
D-Tale works with:
Back-end
arctic
Flask
Flask-Caching
Flask-Compress
flasgger
Pandas
scipy
six
Front-end
react-virtualized
chart.js
Acknowledgements
D-Tale has been under active development at Man Numeric since 2019.
Original concept and implementation: Andrew Schonfeld
Contributors:
Youssef Habchi - title font
… and many others …
Contributions welcome!
License
D-Tale is licensed under the GNU LGPL v2.1. A copy of which is included in LICENSE
Changelog
1.0.0 (2019-09-06)
Initial public release
1.1.0 (2019-10-08)
IE support
Describe & About popups
Custom CLI support
1.1.1 (2019-10-23)
#13: fix for auto-detection of column widths for strings and floats
1.2.0 (2019-10-24)
1.3.0 (2019-10-29)
webbrowser integration (the ability to automatically open a webbrowser upon calling dtale.show())
flag for hiding the “Shutdown” button for long-running demos
“Instances” navigator popup for viewing all activate D-Tale instances for the current python process
1.3.1 (2019-10-29)
fix for incompatible str types when directly altering state of data in running D-Tale instance
1.3.2 (2019-11-05)
Bug fixes for:
display of histogram column information
reload of hidden “processes” input when loading instances data
correlations json failures on string conversion
1.3.3 (2019-11-05)
hotfix for failing test under certain versions of future package
1.3.4 (2019-11-07)
updated correlation calculation to use numpy.corrcoef for performance purposes
github rebranding from manahl -> man-group
1.3.5 (2019-11-07)
Bug fixes for:
duplicate loading of histogram data
string serialization failing when mixing future.str & str in scatter function
1.3.6 (2019-11-08)
Bug fixes for:
choose between pandas.corr & numpy.corrcoef depending on presence of NaNs
hide timeseries correlations when date columns only contain one day
1.3.7 (2019-11-12)
1.4.0 (2019-11-19)
Correlations Pearson Matrix filters
“name” display in title tab
“Heat Map” toggle
dropped unused “Flask-Caching” requirement
1.4.1 (2019-11-20)
#32: unpin jsonschema by moving flasgger to extras_require
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for dtale-1.4.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 171dbf838f72153abde06f5486009fc6599547bd2eb89ea65df5863568317dc6 |
|
MD5 | 6066ca49719310137b657bc49e61c550 |
|
BLAKE2b-256 | cb379c799a4f991ce2e0025065ce9edbb81332f78a9cc249fe71c9490d13446e |