Skip to main content

Web Client for Visualizing Pandas Objects

Project description

image0

Live Demo


CircleCI PyPI ReadTheDocs codecov Downloads

Getting Started

Setup/Activate your environment and install the egg

Python 3

# create a virtualenv, if you haven't already created one
$ python3 -m venv ~/pyenvs/dtale
$ source ~/pyenvs/dtale/bin/activate

# install dtale egg (important to use the "--upgrade" every time you install so it will grab the latest version)
$ pip install --upgrade dtale

Python 2

# create a virtualenv, if you haven't already created one
$ python -m virtualenv ~/pyenvs/dtale
$ source ~/pyenvs/dtale/bin/activate

# install dtale egg (important to use the "--upgrade" every time you install so it will grab the latest version)
$ pip install --upgrade dtale

Now you will have to ability to use D-Tale from the command-line or within a python-enabled terminal

Command-line

Base CLI options (run dtale --help to see all options available)

Prop

Description

--host

the name of the host you would like to use (most likely not needed since socket.gethostname() should figure this out)

--port

the port you would like to assign to your D-Tale instance

--debug

turn on Flask’s “debug” mode for your D-Tale instance

--no-reaper

flag to turn off auto-reaping subprocess (kill D-Tale instances after an hour of inactivity), good for long-running displays

--open-browser

flag to automatically open up your server’s default browser to your D-Tale instance

Loading data from arctic

dtale --arctic-host mongodb://localhost:27027 --arctic-library jdoe.my_lib --arctic-node my_node --arctic-start 20130101 --arctic-end 20161231

Loading data from CSV

dtale --csv-path /home/jdoe/my_csv.csv --csv-parse_dates date

Loading data from a Custom loader - Using the DTALE_CLI_LOADERS environment variable, specify a path to a location containing some python modules - Any python module containing the global variables LOADER_KEY & LOADER_PROPS will be picked up as a custom loader - LOADER_KEY: the key that will be associated with your loader. By default you are given arctic & csv (if you use one of these are your key it will override these) - LOADER_PROPS: the individual props available to be specified. - For example, with arctic we have host, library, node, start & end. - If you leave this property as an empty list your loader will be treated as a flag. For example, instead of using all the arctic properties we would simply specify --arctic (this wouldn’t work well in arctic’s case since it depends on all those properties) - You will also need to specify a function with the following signature def find_loader(kwargs) which returns a function that returns a dataframe or None - Here is an example of a custom loader:

from dtale.cli.clickutils import get_loader_options

'''
  IMPORTANT!!! This global variable is required for building any customized CLI loader.
  When find loaders on startup it will search for any modules containing the global variable LOADER_KEY.
'''
LOADER_KEY = 'testdata'
LOADER_PROPS = ['rows', 'columns']


def test_data(rows, columns):
    import pandas as pd
    import numpy as np
    import random
    from past.utils import old_div
    from pandas.tseries.offsets import Day
    from dtale.utils import dict_merge
    import string

    now = pd.Timestamp(pd.Timestamp('now').date())
    dates = pd.date_range(now - Day(364), now)
    num_of_securities = max(old_div(rows, len(dates)), 1)  # always have at least one security
    securities = [
        dict(security_id=100000 + sec_id, int_val=random.randint(1, 100000000000),
             str_val=random.choice(string.ascii_letters) * 5)
        for sec_id in range(num_of_securities)
    ]
    data = pd.concat([
        pd.DataFrame([dict_merge(dict(date=date), sd) for sd in securities])
        for date in dates
    ], ignore_index=True)[['date', 'security_id', 'int_val', 'str_val']]

    col_names = ['Col{}'.format(c) for c in range(columns)]
    return pd.concat([data, pd.DataFrame(np.random.randn(len(data), columns), columns=col_names)], axis=1)


# IMPORTANT!!! This function is required for building any customized CLI loader.
def find_loader(kwargs):
    test_data_opts = get_loader_options(LOADER_KEY, kwargs)
    if len([f for f in test_data_opts.values() if f]):
        def _testdata_loader():
            return test_data(int(test_data_opts.get('rows', 1000500)), int(test_data_opts.get('columns', 96)))

        return _testdata_loader
    return None

In this example we simplying building a dataframe with some dummy data based on dimensions specified on the command-line: - --testdata-rows - --testdata-columns

Here’s how you would use this loader:

DTALE_CLI_LOADERS=./path_to_loaders bash -c 'dtale --testdata-rows 10 --testdata-columns 5'

Python Terminal

This comes courtesy of PyCharm image6 Feel free to invoke python or ipython directly and use the commands in the screenshot above and it should work #####Additional functions available programatically

import dtale
import pandas as pd

df = pd.DataFrame([dict(a=1,b=2,c=3)])

# Assigning a reference to a running D-Tale process
d = dtale.show(df)

# Accessing data associated with D-Tale process
tmp = d.data.copy()
tmp['d'] = 4

# Altering data associated with D-Tale process
# FYI: this will clear any front-end settings you have at the time for this process (filter, sorts, formatting)
d.data = tmp

# Shutting down D-Tale process
d.kill()

# using Python's `webbrowser` package it will try and open your server's default browser to this process
d.open_browser()

# There is also some helpful metadata about the process
d._port  # the process's port
d._url  # the url to access the process

UI

Once you have kicked off your D-Tale session please copy & paste the link on the last line of output in your browser image7

The information in the upper right-hand corner is similar to saslook image8 - lower-left => row count - upper-right => column count - clicking the triangle displays the menu of standard functions (click outside menu to close it) image9

Selecting/Deselecting Columns - to select a column, simply click on the column header (to deselect, click the column header again) - You’ll notice that the columns you’ve selected will display in the top of your browser image10

For Developers

Getting Started

Clone the code (git clone ssh://git@github.com:manahl/dtale.git), then start the backend server:

$ git clone ssh://git@github.com:manahl/dtale.git
# install the dependencies
$ python setup.py develop
# start the server
$ python dtale --csv-path /home/jdoe/my_csv.csv --csv-parse_dates date

You can also run dtale from PyDev directly.

You will also want to import javascript dependencies and build the source:

$ npm install
# 1) a persistent server that serves the latest JS:
$ npm run watch
# 2) or one-off build:
$ npm run build

Running tests

The usual npm test command works:

$ npm test

You can run individual test files:

$ TEST=static/__tests__/dtale/DataViewer-base-test.jsx npm run test-file

Linting

You can lint all the JS and CSS to confirm there’s nothing obviously wrong with it:

$ npm run lint -s

You can also lint individual JS files:

$ npm run lint-js-file -s -- static/dtale/DataViewer.jsx

Formatting JS

You can auto-format code as follows:

$ npm run format

Docker development

You can build python 27-3 & run D-Tale as follows:

$ yarn run build
$ docker-compose build dtale_2_7
$ docker run -it --network host dtale_2_7:latest
$ python
>>> import pandas as pd
>>> df = pd.DataFrame([dict(a=1,b=2,c=3)])
>>> import dtale
>>> dtale.show(df)

Then view your D-Tale instance in your browser using the link that gets printed

You can build python 36-1 & run D-Tale as follows:

$ yarn run build
$ docker-compose build dtale_3_6
$ docker run -it --network host dtale_3_6:latest
$ python
>>> import pandas as pd
>>> df = pd.DataFrame([dict(a=1,b=2,c=3)])
>>> import dtale
>>> dtale.show(df)

Then view your D-Tale instance in your browser using the link that gets printed

Documentation

Have a look at the detailed documentation.

Requirements

D-Tale works with:

  • Back-end

    • arctic

    • Flask

    • Flask-Caching

    • Flask-Compress

    • flasgger

    • Pandas

    • scipy

    • six

  • Front-end

    • react-virtualized

    • chart.js

Acknowledgements

D-Tale has been under active development at Man Numeric since 2019.

Original concept and implementation: Andrew Schonfeld

Contributors:

Contributions welcome!

License

D-Tale is licensed under the GNU LGPL v2.1. A copy of which is included in LICENSE

Changelog

1.0.0 (2019-09-06)

  • Initial public release

1.1.0 (2019-10-08)

  • IE support

  • Describe & About popups

  • Custom CLI support

1.1.1 (2019-10-23)

  • #13: fix for auto-detection of column widths for strings and floats

1.2.0 (2019-10-24)

  • #20: fix for data being overriden with each new instance

  • #21: fix for displaying timestamps if they exist

  • calling show() now returns an object which can alter the state of a process

    • accessing/altering state through the data property

    • shutting down a process using the kill() function

1.3.0 (2019-10-29)

  • webbrowser integration (the ability to automatically open a webbrowser upon calling dtale.show())

  • flag for hiding the “Shutdown” button for long-running demos

  • “Instances” navigator popup for viewing all activate D-Tale instances for the current python process

1.3.1 (2019-10-29)

  • fix for incompatible str types when directly altering state of data in running D-Tale instance

1.3.2 (2019-11-05)

  • Bug fixes for:

    • display of histogram column information

    • reload of hidden “processes” input when loading instances data

    • correlations json failures on string conversion

1.3.3 (2019-11-05)

  • hotfix for failing test under certain versions of future package

1.3.4 (2019-11-07)

  • updated correlation calculation to use numpy.corrcoef for performance purposes

  • github rebranding from manahl -> man-group

1.3.5 (2019-11-07)

  • Bug fixes for:

    • duplicate loading of histogram data

    • string serialization failing when mixing future.str & str in scatter function

1.3.6 (2019-11-08)

  • Bug fixes for:

    • choose between pandas.corr & numpy.corrcoef depending on presence of NaNs

    • hide timeseries correlations when date columns only contain one day

1.3.7 (2019-11-12)

  • Bug fixes for:

    • #28: “Instances” menu option will now be displayed by default

    • #29: add hints to how users can navigate the correlations popup

    • add “unicode” as a string classification for column width calculation

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dtale-1.3.7.tar.gz (4.3 MB view hashes)

Uploaded Source

Built Distributions

dtale-1.3.7-py3.6.egg (4.4 MB view hashes)

Uploaded Source

dtale-1.3.7-py2.py3-none-any.whl (4.4 MB view hashes)

Uploaded Python 2 Python 3

dtale-1.3.7-py2.7.egg (4.4 MB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page