Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Interactions between Dask and XGBoost

Project Description

Distributed training with XGBoost and Dask.distributed

This repository enables you to perform distributed training with XGBoost on Dask.array and Dask.dataframe collections.

pip install dask-xgboost

Example

from dask.distributed import Client
client = Client('scheduler-address:8786')  # connect to cluster

import dask.dataframe as dd
df = dd.read_csv('...')  # use dask.dataframe to load and
df_train = ...           # preprocess data
labels_train = ...

import dask_xgboost as dxgb
params = {'objective': 'binary:logistic', ...}  # use normal xgboost params
bst = dxgb.train(client, params, df_train, labels_train)

>>> bst  # Get back normal XGBoost result
<xgboost.core.Booster at ... >

predictions = dxgb.predict(client, bsg, data_test)

How this works

For more information on using Dask.dataframe for preprocessing see the Dask.dataframe documentation.

Once you have created suitable data and labels we are ready for distributed training with XGBoost. Every Dask worker sets up an XGBoost slave and gives them enough information to find each other. Then Dask workers hand their in-memory Pandas dataframes to XGBoost (one Dask dataframe is just many Pandas dataframes spread around the memory of many machines). XGBoost handles distributed training on its own without Dask interference. XGBoost then hands back a single xgboost.Booster result object.

Larger Example

For a more serious example see

History

Conversation during development happened at dmlc/xgboost #2032

Release History

Release History

This version
History Node

0.1.5

History Node

0.1.3

History Node

0.1.2

History Node

0.1.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
dask-xgboost-0.1.5.tar.gz (11.5 kB) Copy SHA256 Checksum SHA256 Source Nov 17, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting