MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.
Project description
What Is MindSpore
MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios. MindSpore is designed to provide development experience with friendly design and efficient execution for the data scientists and algorithmic engineers, native support for Ascend AI processor, and software hardware co-optimization. At the meantime MindSpore as a global AI open source community, aims to further advance the development and enrichment of the AI software/hardware application ecosystem.
For more details please check out our Architecture Guide.
Automatic Differentiation
There are currently three automatic differentiation techniques in mainstream deep learning frameworks:
- Conversion based on static compute graph: Convert the network into a static data flow graph at compile time, then turn the chain rule into a data flow graph to implement automatic differentiation.
- Conversion based on dynamic compute graph: Record the operation trajectory of the network during forward execution in an operator overloaded manner, then apply the chain rule to the dynamically generated data flow graph to implement automatic differentiation.
- Conversion based on source code: This technology is evolving from the functional programming framework and performs automatic differential transformation on the intermediate expression (the expression form of the program during the compilation process) in the form of just-in-time compilation (JIT), supporting complex control flow scenarios, higher-order functions and closures.
TensorFlow adopted static calculation diagrams in the early days, whereas PyTorch used dynamic calculation diagrams. Static maps can utilize static compilation technology to optimize network performance, however, building a network or debugging it is very complicated. The use of dynamic graphics is very convenient, but it is difficult to achieve extreme optimization in performance.
But MindSpore finds another way, automatic differentiation based on source code conversion. On the one hand, it supports automatic differentiation of automatic control flow, so it is quite convenient to build models like PyTorch. On the other hand, MindSpore can perform static compilation optimization on neural networks to achieve great performance.
The implementation of MindSpore automatic differentiation can be understood as the symbolic differentiation of the program itself. Because MindSpore IR is a functional intermediate expression, it has an intuitive correspondence with the composite function in basic algebra. The derivation formula of the composite function composed of arbitrary basic functions can be derived. Each primitive operation in MindSpore IR can correspond to the basic functions in basic algebra, which can build more complex flow control.
Automatic Parallel
The goal of MindSpore automatic parallel is to build a training method that combines data parallelism, model parallelism, and hybrid parallelism. It can automatically select a least cost model splitting strategy to achieve automatic distributed parallel training.
At present, MindSpore uses a fine-grained parallel strategy of splitting operators, that is, each operator in the figure is splitted into a cluster to complete parallel operations. The splitting strategy during this period may be very complicated, but as a developer advocating Pythonic, you don't need to care about the underlying implementation, as long as the top-level API compute is efficient.
Installation
Binaries
MindSpore offers build options across multiple backends:
Hardware Platform | Operating System | Status |
---|---|---|
Ascend910 | Ubuntu-x86 | ✔️ |
EulerOS-x86 | ✔️ | |
EulerOS-aarch64 | ✔️ | |
GPU CUDA 9.2 | Ubuntu-x86 | ✔️ |
GPU CUDA 10.1 | Ubuntu-x86 | ✔️ |
CPU | Ubuntu-x86 | ✔️ |
Windows-x86 | ✔️ |
For installation using pip
, take CPU
and Ubuntu-x86
build version as an example:
-
Download whl from MindSpore download page, and install the package.
pip install https://ms-release.obs.cn-north-4.myhuaweicloud.com/0.2.0-alpha/MindSpore/cpu/x86_ubuntu/mindspore-0.2.0-cp37-cp37m-linux_x86_64.whl
-
Run the following command to verify the install.
import numpy as np import mindspore.context as context import mindspore.nn as nn from mindspore import Tensor from mindspore.ops import operations as P context.set_context(mode=context.GRAPH_MODE, device_target="CPU") class Mul(nn.Cell): def __init__(self): super(Mul, self).__init__() self.mul = P.Mul() def construct(self, x, y): return self.mul(x, y) x = Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) y = Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) mul = Mul() print(mul(x, y))
[ 4. 10. 18.]
From Source
Docker Image
MindSpore docker image is hosted on Docker Hub, currently the containerized build options are supported as follows:
Hardware Platform | Docker Image Repository | Tag | Description |
---|---|---|---|
CPU | mindspore/mindspore-cpu |
x.y.z |
Production environment with pre-installed MindSpore x.y.z CPU release. |
devel |
Development environment provided to build MindSpore (with CPU backend) from the source, refer to https://www.mindspore.cn/install/en for installation details. |
||
runtime |
Runtime environment provided to install MindSpore binary package with CPU backend. |
||
GPU | mindspore/mindspore-gpu |
x.y.z |
Production environment with pre-installed MindSpore x.y.z GPU release. |
devel |
Development environment provided to build MindSpore (with GPU CUDA10.1 backend) from the source, refer to https://www.mindspore.cn/install/en for installation details. |
||
runtime |
Runtime environment provided to install MindSpore binary package with GPU CUDA10.1 backend. |
||
Ascend | — | — | Coming soon. |
NOTICE: For GPU
devel
docker image, it's NOT suggested to directly install the whl package after building from the source, instead we strongly RECOMMEND you transfer and install the whl package inside GPUruntime
docker image.
-
CPU
For
CPU
backend, you can directly pull and run the latest stable image using the below command:docker pull mindspore/mindspore-cpu:0.2.0-alpha docker run -it mindspore/mindspore-cpu:0.2.0-alpha /bin/bash
-
GPU
For
GPU
backend, please make sure thenvidia-container-toolkit
has been installed in advance, here are some install guidelines forUbuntu
users:DISTRIBUTION=$(. /etc/os-release; echo $ID$VERSION_ID) curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add - curl -s -L https://nvidia.github.io/nvidia-docker/$DISTRIBUTION/nvidia-docker.list | tee /etc/apt/sources.list.d/nvidia-docker.list sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit nvidia-docker2 sudo systemctl restart docker
Then you can pull and run the latest stable image using the below command:
docker pull mindspore/mindspore-gpu:0.2.0-alpha docker run -it --runtime=nvidia --privileged=true mindspore/mindspore-gpu:0.2.0-alpha /bin/bash
To test if the docker image works, please execute the python code below and check the output:
import numpy as np import mindspore.context as context from mindspore import Tensor from mindspore.ops import functional as F context.set_context(device_target="GPU") x = Tensor(np.ones([1,3,3,4]).astype(np.float32)) y = Tensor(np.ones([1,3,3,4]).astype(np.float32)) print(F.tensor_add(x, y))
[[[ 2. 2. 2. 2.], [ 2. 2. 2. 2.], [ 2. 2. 2. 2.]], [[ 2. 2. 2. 2.], [ 2. 2. 2. 2.], [ 2. 2. 2. 2.]], [[ 2. 2. 2. 2.], [ 2. 2. 2. 2.], [ 2. 2. 2. 2.]]]
If you want to learn more about the building process of MindSpore docker images, please check out docker repo for the details.
Quickstart
See the Quick Start to implement the image classification.
Docs
More details about installation guide, tutorials and APIs, please see the User Documentation.
Community
Governance
Check out how MindSpore Open Governance works.
Communication
- MindSpore Slack - Communication platform for developers.
- IRC channel at
#mindspore
(only for meeting minutes logging purpose) - Video Conferencing: https://meet.jit.si
- Mailing-list: https://mailweb.mindspore.cn/postorius/lists
Contributing
Welcome contributions. See our Contributor Wiki for more details.
Release Notes
The release notes, see our RELEASE.
License
Release 0.2.0-alpha
Major Features and Improvements
Ascend 910 Training and Inference Framework
-
New models
- MobileNetV2: Inverted Residuals and Linear Bottlenecks.
- ResNet101: Deep Residual Learning for Image Recognition.
-
Frontend and User Interface
- Support for all python comparison operators.
- Support for math operators **,//,%. Support for other python operators like and/or/not/is/is not/ in/ not in.
- Support for the gradients of function with variable arguments.
- Support for tensor indexing assignment for certain indexing type.
- Support for dynamic learning rate.
- User interfaces change log
- DepthwiseConv2dNative, DepthwiseConv2dNativeBackpropFilter, DepthwiseConv2dNativeBackpropInput(!424)
- ReLU6, ReLU6Grad(!224)
- GeneratorDataset(!183)
- VOCDataset(!477)
- MindDataset, PKSampler(!514)
- map(!506)
- Conv(!226)
- Adam(!253)
- _set_fusion_strategy_by_idx, _set_fusion_strategy_by_size(!189)
- CheckpointConfig(!122)
- Constant(!54)
-
Executor and Performance Optimization
- Support parallel execution of data prefetching and forward/backward computing.
- Support parallel execution of gradient aggregation and forward/backward computing in distributed training scenarios.
- Support operator fusion optimization.
- Optimize compilation process and improve the performance.
-
Data processing, augmentation, and save format
- Support multi-process of GeneratorDataset/PyFunc for high performance
- Support variable batchsize
- Support new Dataset operators, such as filter,skip,take,TextLineDataset
Other Hardware Support
- GPU platform
- Use dynamic memory pool by default on GPU.
- Support parallel execution of computation and communication.
- Support continuous address allocation by memory pool.
- CPU platform
- Support for windows 10 OS.
Bugfixes
- Models
- Fix mixed precision bug for VGG16 model (!629).
- Python API
- Executor
- GPU platform
- Data processing
Contributors
Thanks goes to these wonderful people:
Alexey_Shevlyakov, Cathy, Chong, Hoai, Jonathan, Junhan, JunhanHu, Peilin, SanjayChan, StrawNoBerry, VectorSL, Wei, WeibiaoYu, Xiaoda, Yanjun, YuJianfeng, ZPaC, Zhang, ZhangQinghua, ZiruiWu, amongo, anthonyaje, anzhengqi, biffex, caifubi, candanzg, caojian05, casgj, cathwong, ch-l, chang, changzherui, chenfei, chengang, chenhaozhe, chenjianping, chentingting, chenzomi, chujinjin, dengwentao, dinghao, fanglei, fary86, flywind, gaojing, geekun, gengdongjie, ghzl, gong, gongchen, gukecai, guohongzilong, guozhijian, gziyan, h.farahat, hesham, huangdongrun, huanghui, jiangzhiwen, jinyaohui, jjfeing, jojobugfree, jonathan_yan, jonyguo, jzw, kingfo, kisnwang, laiyongqiang, leonwanghui, lianliguang, lichen, lichenever, limingqi107, liubuyu, liuxiao, liyong, liyong126, lizhenyu, lupengcheng, lvliang, maoweiyong, ms_yan, mxm, ougongchang, panfengfeng, panyifeng, pengyanjun, penn, qianlong, seatea, simson, suteng, thlinh, vlne-v1, wangchengke, wanghua, wangnan39, wangqiuliang, wenchunjiang, wenkai, wukesong, xiefangqi, xulei, yanghaitao, yanghaoran, yangjie159, yangzhenzhang, yankai10, yanzhenxiang2020, yao_yf, yoonlee666, zhangbuxue, zhangz0911gm, zhangzheng, zhaojichen, zhaoting, zhaozhenlong, zhongligeng, zhoufeng, zhousiyi, zjun, zyli2020, yuhuijun, limingqi107, lizhenyu, chenweifeng.
Contributions of any kind are welcome!
Release 0.1.0-alpha
Main Features
Ascend 910 Training and Inference Framework
- Recommended OS: Ubuntu 16.04 (or later) or EulerOS 2.5 or EulerOS 2.8
- Python version: 3.7.5
- Preset models
- ResNet-50: residual structure-based convolutional neural network (CNN) for image classification, which is widely used.
- AlexNet: classic CNN for image classification, achieving historical results in ImageNet LSVRC-2012.
- LeNet: classic CNN for image classification, which was proposed by Yann LeCun.
- VGG16: classic CNN for image classification, which was proposed by Oxford Visual Geometry Group.
- YoloV3: real-time object detection network.
- NEZHA: BERT-based Chinese pre-training network produced by Huawei Noah's Ark Laboratory.
- Execution modes
- Graph mode: provides graph optimization methods such as memory overcommitment, IR fusion, and buffer fusion to achieve optimal execution performance.
- PyNative mode: single-step execution mode, facilitating process debugging.
- Debugging capability and methods
- Save CheckPoints and Summary data during training.
- Support asynchronous printing.
- Dump the computing data.
- Support profiling analysis of the execution process performance.
- Distributed execution
- Support AllReduce, AllGather, and BroadCast collective communication.
- AllReduce data parallel: Each device obtains different training data, which accelerates the overall training process.
- Collective communication-based layerwise parallel: Models are divided and allocated to different devices to solve the problem of insufficient memory for large model processing and improve the training speed.
- Automatic parallel mode: The better data and model parallel mode can be predicted based on the cost model. It is recommended that this mode be used on ResNet series networks.
- Automatic differentiation
- Implement automatic differentiation based on Source to Source.
- Support distributed scenarios and automatic insertion of reverse communication operators.
- Data processing, augmentation, and save format
- Load common datasets such as ImageNet, MNIST, CIFAR-10, and CIFAR-100.
- Support common data loading pipeline operations, such as shuffle, repeat, batch, map, and sampler.
- Provide basic operator libraries to cover common CV scenarios.
- Support users to customize Python data augmentation operators through the Pyfunc mechanism.
- Support the access of user-defined datasets through the GeneratorDataset mechanism.
- Provide the MindSpore data format, data aggregation and storage, random access example, data partition, efficient parallel read, user-defined index, and dataset search.
- Convert user datasets to the MindSpore data format.
- After data processing and augmentation, provide training applications in feed and graph modes.
- FP32/16 mixed precision computation, supporting automatic and manual configuration
- Provide common operators such as nn, math, and array, which can be customized.
Inference Deployment
- Deploy models in MindSpore format on the Ascend 310 platform for inference.
- Save models in ONNX format.
- Support saving models in LITE format and running models based on the lightweight inference framework.
- Recommended OS: Android 4.3 or later
- Supported network type: LeNet
- Provide the generalization operators generated by TVM and operators generated after specific networks are tuned.
Other Hardware Support
- GPU platform training
- Recommended OS: Ubuntu 16.04
- CUDA version: 9.2 or 10.1
- CuDNN version: 7.6 or later
- Python version: 3.7.5
- NCCL version: 2.4.8-1
- OpenMPI version: 3.1.5
- Supported models: AlexNet, LeNet, and LSTM
- Supported datasets: MNIST and CIFAR-10
- Support data parallel.
- CPU platform training
- Recommended OS: Ubuntu 16.04
- Python version: 3.7.5
- Supported model: LeNet
- Supported dataset: MNIST
- Provide only the stand-alone operation version.
Peripherals and Tools
- [MindSpore Official Website] (https://www.mindspore.cn/)
- [MindInsight Visualization Debugging and Optimization] (https://gitee.com/mindspore/mindinsight)
- [MindArmour Model Security Hardening Package] (https://gitee.com/mindspore/mindarmour)
- [GraphEngine Computational Graph Engine] (https://gitee.com/mindspore/graphengine)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for mindspore-0.2.0-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d0fb1f368904e1a0cd2c3242f1e43338efecab40af61fb66002247ee2cb96e7 |
|
MD5 | d2562fa56a5aaa831439dd0c18ceeee4 |
|
BLAKE2b-256 | cb360dbc7f9edc97d1a0b65a559ddb3e95082b7fc2b2b47e7ba164d444630497 |
Hashes for mindspore-0.2.0-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1380a9752c38f04ca098235dc51f5cee16a6985d9a4c659ac298d79a80640f6a |
|
MD5 | 85b5c93f441a5f60e91c098f0488da9d |
|
BLAKE2b-256 | 151282ce07817925a8dd18dcc2f068313ec62575d74b83744793d6ec4b655646 |