Paperspace Python
Project description
Gradient CLI
Get started: Create Account • Install CLI • Tutorials • Docs
Resources: Website • Blog • Support • Contact Sales
Gradient is an an end-to-end MLOps platform that enables individuals and organizations to quickly develop, train, and deploy Deep Learning models. The Gradient software stack runs on any infrastructure e.g. AWS, GCP, on-premise and low-cost Paperspace GPUs. Leverage automatic versioning, distributed training, built-in graphs & metrics, hyperparameter search, GradientCI, 1-click Jupyter Notebooks, our Python SDK, and more.
Key components:
- Notebooks: 1-click Jupyter Notebooks.
- Experiments: Run experiments from a web interface, CLI, SDK, or GradientCI bot.
- Models: Store, analyze, and version models.
- Inference: Deploy models as API endpoints.
Gradient supports any ML/DL framework (TensorFlow, PyTorch, XGBoost, etc).
See releasenotes.md for details on the current release, as well as release history.
Getting Started
-
Make sure you have a Paperspace account set up. Go to http://paperspace.com to register.
-
Use pip, pipenv, or conda to install the gradient package, e.g.:
pip install -U gradient
To install/update prerelease (Alpha/Beta) version version of gradient, use:
pip install -U --pre gradient
-
Enable autocomplete:
Add following to your
.bashrc
(or.zshrc
) to enable autocomplete anytime you activate your shell. If gradient was installed in a virtual environment, the following has to be added to theactivate
script:eval "$(_GRADIENT_COMPLETE=source gradient)"
Alternatively, you can create activation script by:
(_GRADIENT_COMPLETE=source gradient) > ~/paperspace_complete.sh
and then add
. ~/paperspace_complete.sh
to your.bashrc
,.zshrc
oractivate
script.More: https://click.palletsprojects.com/en/7.x/bashcomplete/
-
Set your api key by executing the following:
gradient apiKey <your-api-key-here>
Note: your api key is cached in ~/.paperspace/config.json
You can remove your cached api key by executing:
gradient logout
-
Use gradient to run a python script remotely:
gradient run myscript.py --name <your-new-job-name>
The script will be run on the Paperspace job cluster node, and its output will be logged locally. Note: this will archive and upload your entire current working directory to our server if
--workspace
was not provided
Create/create and start experiment
To create new experiment use:
gradient experiments create [type] [--options]
The two available experiment types are singlenode
and multinode
.
To create and immediately start new experiment use:
gradient experiments run [type] [--options]
For a full list of available commands run gradient experiments --help
.
Note that some options are required to create new experiment.
Running a python script by name
You can run a Python script on a Paperspace server from the command line as follows:
gradient run myscript.py --name my_new_job
You can also provide additional jobs options on the command line:
gradient run myscript.py --name my_new_job --projectId myproject --machineType P5000 \
--container paperspace/tensorflow-python`
Note: this functionality is deprecated and will not be available in future releases
See the Gradient docs for the full list of jobs create options that can be specified.
Basic Run Scenarios
-
Run a python script remotely:
gradient run <python_script.py> [args]
Example:
gradient run myscript.py a b c
-
Run a python module remotely using the
-m
option:gradient run -m <module_path> [args]
Example:
gradient run -m pip --version
-
Run a python command remotely using the
-c
option:gradient run -c "python_statement;..."
Example:
gradient run -c "import os; print(os.getcwd())"
-
Run an executable or shell command remotely using the
--shell
option:gradient run --shell "<executable or shell command>"
Example:
gradient run --shell "ls -al"
Run Options
The <script>
option is a python script or path to a python module. The script or module will be uploaded if it exists on the local file system.
Other scriptargs
can be provided after the python script or module path. You can use the -
option to suppress interpretation of the list of script args as gradient run
options.
The -m <module path>
option runs the specified library module as a script. This is equivalent to the -m
option of the python
executable. Further paperspace run option processing is disabled after the -m
option.
The -c "python_statement;..."
option runs the specified python statements. This is equivalent to the -c
option of the python
executable. Further paperspace run option processing is disabled after the -c
option.
The -
option disables further run command option processing and passes the remaining arguments to the script specified. This allows you to pass arguments to your script that might otherwise conflict with run command options or jobs create options.
The --command "shell cmd"
option is used to run an arbitrary executable or shell command inside the container. Note: the executable or shell command must already be available inside the container image, or be copied over using the --workspace
option.
Job Options
The --workspace
option allows you to specify a workspace file or directory to upload, or a git repo link to download and merge with the container. For example, to upload the current directory along with a script file run:
gradient run myscript.py --name my_new_job --workspace workspace-dir
Note: --workspace
defaults to .
See the Paperspae API jobs create documentation for more details on the --workspace
option and related options.
The --ignoreFiles "<file-or-dir>,..."
option can be used specify a simple comma separated list of files and directories to ignore for the workspace upload:
gradient run myscript.py --name my_new_job --workspace . --ignoreFiles "hello.py,paperspace"
The following files and directories are ignored by default: .git
, .gitignore
, __pycache__
.
Other jobs create
options can be specified, such as --machineType <machine-type>
, --container <container-image-reference>
, and --projectId <project-id>
.
See the Paperspae API jobs create documentation for a complete description of these options.
Default Container
If no container
option is specified when using paperspace run <script.py>
or the paperspace.run()
function the default container image used is paperspace/tensorflow-python
on Docker Hub. This container has the tensorflow-gpu libraries installed for both python2 and python3, as well as several other popular packages, including numpy, scipy, scikit-learn, pandas, Pillow and matplotlib.
It is based off the Google docker image gcr.io/tensorflow/tensorflow:1.5.0-gpu
with the addition of support for python3, pip3, and pipenv.
A Dockerfile for building this container image is here.
Other examples
See the scripts in the tests
folder for other examples.
Authentication options
-
Set the package paperspace.config option in your python code:
gradient.config.PAPERSPACE_API_KEY = '1qks1hKsU7e1k...'
-
Set the PAPERSPACE_API_KEY environment variable:
(on linux/mac:)
export PAPERSPACE_API_KEY=1qks1hKsU7e1k...
(on windows:)
set PAPERSPACE_API_KEY=1qks1hKsU7e1k...
Note: the above methods take precedence over use of the cached api key in
~/.paperspace/config.json
-
Set the key in the
~/.paperspace/config.json
file from the command line by running:gradient apiKey 1qks1hKsU7e1k...
Generating documentation from docstrings
-
Install dev dependencies
When using pipenv:
pipenv install --dev
Otherwise:
pip install ".[dev]"
-
Automatically extract docstrings with
sphinx-apidoc -f -o source gradient
-
[OPTIONAL] To create HTML version from .rst files
make html
The html version with basic search functionality should appear in
build
directory -
[OPTIONAL] To generate github pages
make gh-pages
Note: Sphinx offers many more documentation formats via external plugins
For more info visit Sphinx web page
Contributing
Want to contribute? Contact us at hello@paperspace.com
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for gradient-0.3.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 896f4ca09390f64d9a2ced7cfca856d2a55420822b546056628e3c7138a99a49 |
|
MD5 | 638cc5fe412db8f31eb609557acf4b73 |
|
BLAKE2b-256 | 4f0d4a2269a7e011f098a118faa072d76d57656412dd0e2a0ae976a0891964f3 |