Skip to main content

Base PySpark application for running Merlin prediction batch job

Project description

Merlin Batch Predictor

Merlin Batch Predictor is a PySpark application for running batch prediction job in Merlin system.

Usage

The application accept a yaml file for configuring source, model, and sink of the prediction job. The schema of the configuration file is described by the proto file. An example of the config file is as follow.

kind: PredictionJob
version: v1
name: integration-test
bigquerySource:
  table: "project.dataset.table_iris"
  features:
    - sepal_length
    - sepal_width
    - petal_length
    - petal_width
model:
  type: PYFUNC_V2
  uri: gs://bucket-name/e2e/artifacts/model
  result:
    type: DOUBLE
bigquerySink:
  table: "project.dataset.table_iris_result"
  result_column: "prediction"
  save_mode: OVERWRITE
  options:
    project: "project"
    temporaryGcsBucket: "bucket-name"

The above prediction job specification will read data from bigquery-public-data:samples.shakespeare Bigquery table, run prediction using a PYFUNC_V2 model located at gs://bucket-name/mlflow/6/2c3703fbbf9f4866b26e4cf91641f02c/artifacts/model GCS bucket, and write the result to another bigquery table project.dataset.table.

To start the application locally you need:

  • Set GOOGLE_APPLICATION_CREDENTIALS environment variable and point it to the service account which has following privileges:
    1. Storage Writer for the temporaryGcsBucket
    2. Storage Object Writer for temporaryGcsBucket
    3. BigQuery Job User
    4. BigQuery Read Session User
    5. BigQuery Data Reader from the source dataset
    6. BigQuery Data Editor for the destination dataset

Then you can invoke

python main.py --job-name <job-name> --spec-path <path-to-spec-yaml> --local

In mac OS you need to set OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES python main.py --job-name <job-name> --spec-path <path-to-spec-yaml> --local

For example

OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES python main.py --job-name iris-prediction --spec-path sample/sample_1.yaml --local

Development

Requirements

Setup Dev Dependencies

make setup

Run all test

You need to set GOOGLE_APPLICATION_CREDENTIALS and point it to service account file which has following privileges:

  1. BigQuery Job User
  2. BigQuery Read Session User
  3. BigQuery Data Editor for dataset project:dataset
  4. Storage Writer for bucket-name bucket
  5. Storage Object Writer for bucket-name bucket
make test

Run only unit test

make unit-test

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

merlin_batch_predictor-0.46.0.tar.gz (41.1 kB view details)

Uploaded Source

Built Distribution

merlin_batch_predictor-0.46.0-py3-none-any.whl (15.9 kB view details)

Uploaded Python 3

File details

Details for the file merlin_batch_predictor-0.46.0.tar.gz.

File metadata

File hashes

Hashes for merlin_batch_predictor-0.46.0.tar.gz
Algorithm Hash digest
SHA256 0d7baec0a2f791ed8277b2ae620bc45fb3df23b3cb62c8b50c754cdc3959c7de
MD5 ccba1ddb9b453003ebe631785cd9caca
BLAKE2b-256 3fe97e16ca351e7433779b758480add791b51159f1cad48224eeb4da2963260e

See more details on using hashes here.

File details

Details for the file merlin_batch_predictor-0.46.0-py3-none-any.whl.

File metadata

File hashes

Hashes for merlin_batch_predictor-0.46.0-py3-none-any.whl
Algorithm Hash digest
SHA256 4d456974efc8f3baa5c0cdb9ad47fb77b6237a2a8e531a0ed7579e294f43655f
MD5 ed9f9ea8a0529313f69b2c2f7e35933a
BLAKE2b-256 b2fba142fd68ce060a48fd161028be14ee6b6551af13bea6e56d7060feb61901

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page