MARS: a tensor-based unified framework for large-scale data computation.
Project description
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and many other libraries.
Installation
Mars is easy to install by
pip install pymars
Installation for Developers
When you want to contribute code to Mars, you can follow the instructions below to install Mars for development:
git clone https://github.com/mars-project/mars.git
cd mars
pip install -e ".[dev]"
More details about installing Mars can be found at installation section in Mars document.
Architecture Overview
Getting Started
Starting a new runtime locally via:
>>> import mars
>>> mars.new_session()
Or connecting to a Mars cluster which is already initialized.
>>> import mars
>>> mars.new_session('http://<web_ip>:<ui_port>')
Mars Tensor
Mars tensor provides a familiar interface like Numpy.
Numpy |
Mars tensor |
|
|
|
|
Mars can leverage multiple cores, even on a laptop, and could be even faster for a distributed setting.
Mars DataFrame
Mars DataFrame provides a familiar interface like pandas.
Pandas |
Mars DataFrame |
|
|
|
|
Mars Learn
Mars learn provides a familiar interface like scikit-learn.
Scikit-learn |
Mars learn |
|
|
Mars learn also integrates with many libraries:
Mars remote
Mars remote allows users to execute functions in parallel.
Vanilla function calls |
Mars remote |
|
|
|
|
DASK on Mars
Refer to DASK on Mars for more information.
Eager Mode
Mars supports eager mode which makes it friendly for developing and easy to debug.
Users can enable the eager mode by options, set options at the beginning of the program or console session.
>>> from mars.config import options
>>> options.eager_mode = True
Or use a context.
>>> from mars.config import option_context
>>> with option_context() as options:
>>> options.eager_mode = True
>>> # the eager mode is on only for the with statement
>>> ...
If eager mode is on, tensor, DataFrame etc will be executed immediately by default session once it is created.
>>> import mars.tensor as mt
>>> import mars.dataframe as md
>>> from mars.config import options
>>> options.eager_mode = True
>>> t = mt.arange(6).reshape((2, 3))
>>> t
array([[0, 1, 2],
[3, 4, 5]])
>>> df = md.DataFrame(t)
>>> df.sum()
0 3
1 5
2 7
dtype: int64
Mars on Ray
Mars also has deep integration with Ray and can run on Ray efficiently and interact with the large ecosystem of machine learning and distributed systems built on top of the core Ray.
Starting a new Mars on Ray runtime locally via:
import ray
ray.init()
import mars
mars.new_ray_session(worker_num=2)
import mars.tensor as mt
mt.random.RandomState(0).rand(1000_0000, 5).sum().execute()
Or connecting to a Mars on Ray runtime which is already initialized.
import mars
mars.new_ray_session('http://<web_ip>:<ui_port>')
# perform computation
Interact with Ray Dataset:
import mars.tensor as mt
import mars.dataframe as md
df = md.DataFrame(
mt.random.rand(1000_0000, 4),
columns=list('abcd'))
# Convert mars dataframe to ray dataset
ds = md.to_ray_dataset(df)
print(ds.schema(), ds.count())
ds.filter(lambda row: row["a"] > 0.5).show(5)
# Convert ray dataset to mars dataframe
df2 = md.read_ray_dataset(ds)
print(df2.head(5).execute())
Refer to Mars on Ray for more information.
Easy to scale in and scale out
Mars can scale in to a single machine, and scale out to a cluster with thousands of machines. It’s fairly simple to migrate from a single machine to a cluster to process more data or gain a better performance.
Bare Metal Deployment
Mars is easy to scale out to a cluster by starting different components of mars distributed runtime on different machines in the cluster.
A node can be selected as supervisor which integrated a web service, leaving other nodes as workers. The supervisor can be started with the following command:
mars-supervisor -h <host_name> -p <supervisor_port> -w <web_port>
Workers can be started with the following command:
mars-worker -h <host_name> -p <worker_port> -s <supervisor_endpoint>
After all mars processes are started, users can run
>>> sess = new_session('http://<web_ip>:<ui_port>')
>>> # perform computation
Kubernetes Deployment
Refer to Run on Kubernetes for more information.
Yarn Deployment
Refer to Run on Yarn for more information.
Getting involved
Read development guide.
Join the mailing list: send an email to mars-dev@googlegroups.com.
Please report bugs by submitting a GitHub issue.
Submit contributions using pull requests.
Thank you in advance for your contributions!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pymars-0.9.0a2-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6cdbe322cc5a13fb11f097b5836f393964fee62b4759babd00db0a30d1deff5a |
|
MD5 | f2b4f5400cc7fd89224ea4218d4c3672 |
|
BLAKE2b-256 | a0c7f48207609142f772e670a8ad941c636f5fb4218f4734efa36c8cb87bc264 |
Hashes for pymars-0.9.0a2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cdcd5b48e98c27b2aa9fb1e7ffb823418db648f87cff70e9ff9f0fdfcbd89810 |
|
MD5 | 743ee15b84df6fe3a2cb3bf874f73a86 |
|
BLAKE2b-256 | 2907e582d9a9b2bd9d3a8ea4a2adfdf25f9dda2d3ddf4537f21f259318356104 |
Hashes for pymars-0.9.0a2-cp310-cp310-macosx_11_0_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aee4a757bfe11e4a2f27668822aa96c8efbeb76c4111f88071ee2adcf469a1f7 |
|
MD5 | fe1df5664c482b173911aa7ba0d2412f |
|
BLAKE2b-256 | b93156caf23685c05a77c667b3040dbd6c05aaaa7e4b430fcdc5ae35823c8f02 |
Hashes for pymars-0.9.0a2-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd00765777c91ae0d301e514f37e6ffb61ec5959e570d19bad1fc79036164216 |
|
MD5 | aafc3a3a673066599bd07dfb9c8bb886 |
|
BLAKE2b-256 | cb68cdd1560cab249192b482526b972196bff6214d2d334c93441f8c773e7062 |
Hashes for pymars-0.9.0a2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f567808195f0647f5592a5afa8223ead6d21d90b73230b7de0d3900e3e3fd2a |
|
MD5 | 2f1ff92a41c52a001f243873dc743008 |
|
BLAKE2b-256 | 91e34074a2e5a63d45df1913273805b5019681322716ab227f1e5b1d5d73eaf7 |
Hashes for pymars-0.9.0a2-cp39-cp39-macosx_11_0_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f00f194f981353fc52c44cb557018f21a9c2de3b5af201730b96917b9b0aaa4e |
|
MD5 | aeaf50468d4df4f1479453265c79b796 |
|
BLAKE2b-256 | 634f269130fbcd730a312025c89a4e016d63fcae2d2b903985b869010ae314da |
Hashes for pymars-0.9.0a2-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c2bbe510fa8b133bcc4dfc3dc344f02f1afb55c5affbb98524c2f0663e2313c |
|
MD5 | 7dbd000360432726f2873ac7a7e7a2b0 |
|
BLAKE2b-256 | cbec0b148caabc066ac1b11c9ae3b37ed29e59a587c60902c868b95e583c312a |
Hashes for pymars-0.9.0a2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1170cd5eea6f50d97ea983b1b6970c8be21ae0ab82f6c3518f5f5c58d73b3681 |
|
MD5 | c39af24cf39e9bb8e11b58d58a42c7fa |
|
BLAKE2b-256 | 9a7eaaa779401ff85fea7adcbbbad13fa5c31551ee5c26af481a75dc6751a3f6 |
Hashes for pymars-0.9.0a2-cp38-cp38-macosx_11_0_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 049104be433bc9a889887285761e1651a837ac07aee9e44fc6a14c5e33c69940 |
|
MD5 | e714cd909556779863f1487029f5e991 |
|
BLAKE2b-256 | b90b3126cfa3a63232d87abee8f221bc2a72f7251240a2fc33b3c30f20f16abb |
Hashes for pymars-0.9.0a2-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ebb2b8c10cc75021504b582c819b91703e44664f0eb3e1e6393bb64bf28be906 |
|
MD5 | cf702adcc3bc333a0585ba19dd197912 |
|
BLAKE2b-256 | 4b0b56606c99458882d1453539004332e1e976a0028b73cd769f0db47b5fc913 |
Hashes for pymars-0.9.0a2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af27cdc5df2cb1b263bed5416e696b06cffbe345ff7a1ef54bee08f3757ddac7 |
|
MD5 | 16aa6da45b89e758b7bfb4678230f7b8 |
|
BLAKE2b-256 | 1820104438f5ca470f8742cd76376c927cd438b0b4f5a3ba5b44cc97b5a4c741 |
Hashes for pymars-0.9.0a2-cp37-cp37m-macosx_11_0_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17710b877e799bf46df199a4731fa579b5fac97e2cb9126a09b34f7f7b4ead44 |
|
MD5 | dc8eb57e56b3bcb306e96e031314cfa6 |
|
BLAKE2b-256 | f4cc03c0191f38b33e447b2e6bd02c97b975c0fadd31d538657a4ea6c1b77d23 |