cli and sdk to manage Jobs in Databricks
Project description
fish-databricks-jobs
fish-databricks-jobs is cli and python sdk to manage Jobs for Databricks. e.g assign permissions to multiple jobs. User can filter jobs by job name or tags.
The functionality of current databricks-cli(v0.17.4) is limited on the jobs
api. e.g it can not assign job permission.
With fish-databricks-jobs
, you can assign job permission, e.g:
to assign group mygroup
with permission can_manage
to job by filter 843966944901662
(job_id)
$ fish-databricks-jobs permission-assign mygroup --type group --level can_manage --filter 843966944901662
Installation
Python Version >= 3.7
$ pip install --upgrade fish-databricks-jobs
$ fish-databricks-jobs --version
Version: 0.6.8
Usage
permission-assign
$ fish-databricks-jobs permission-assign -h
Usage: fish-databricks-jobs permission-assign [OPTIONS] NAME
Assign permission to user
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * name TEXT User name, group name or serive principal id. Who will receive the permisssion. [default: None] │
│ [required] │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --type -t [user|principal|group] Permission receiver type. [default: user] │
│ * --level -l [can_view|can_manage|can_manage_run] Permission level. [default: None] [required] │
│ --filter -f TEXT filter jobs, case insensitively. [default: None] │
│ --profile -p TEXT profile name in ~/.databrickscfg [default: DEFAULT] │
│ --force Attempt to assign permission without prompting for │
│ confirmation. **Use this flag with caution** │
│ --help -h Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
use as sdk to list jobs
in databricks' notebook
%pip install fish-databricks-jobs
from fish_databricks_jobs.services.jobs import JobsService, Job
host = f'https://{spark.conf.get("spark.databricks.workspaceUrl")}'
token = 'exampletokenc0e27d8b91fd8c0144f0a23'
job_list = JobsService(host, token).list()
df = spark.createDataFrame(job_list)
display(df)
Config authentication
fish-databricks-jobs uses same config file as databricks-cli
. e.g.~/.databrickscfg
for macOS. Similar for Windows.
[DEFAULT]
host = https://example.cloud.databricks.com
token = exampletokenc0e27d8b91fd8c0144f0a23
Developer
Setup
- Checkout code
- install
poetry
, e.g: in macOS or Windows-cmder
curl -sSL https://install.python-poetry.org | python -
- config poetry
poetry config virtualenvs.in-project true
- install dependencies python libraries in the venv
poetry install
poetry shell
- in pycharm, Add New Interpreter -> Poetry Environment ...
- Verifiy
(fish-databricks-jobs-py3.8) (base) username@macbook:~/projects/fish-databricks-jobs [main]
$ python ./fish_databricks_jobs/cli.py --version
Version: 0.7.6
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for fish_databricks_jobs-0.7.18.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f436b736da456392f246ac124206c0b7e88db748504a72d749a89f9519c94a28 |
|
MD5 | 5243fd623ea2d2a9da7b93320ac400d7 |
|
BLAKE2b-256 | e7fe245f934b7eb8f970bee62f67a1561868c6f644ae81bccbe068c901f5e456 |
Close
Hashes for fish_databricks_jobs-0.7.18-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48c9c5d85ece2471d0e5f6ea70dc6bc3f0cb5f13540ff1e5a92d808660849319 |
|
MD5 | db57a1d7401358c1a161cbb36e694a6e |
|
BLAKE2b-256 | 47fbc0a8462664591cb41f83f4ea21af4d2642fc0ed550ba0ec75c9bc2cbc14d |