Clarifai PySpark Python SDK
Project description
ClarifaiPySpark
Introduction
This readme provides overview of the Software Development Kit (SDK) under development for integrating Clarifai with Databricks. The primary use case for this SDK is to facilitate the interaction between Databricks and Clarifai for tasks related to uploading client datasets, annotating data, and exporting and storing annotations in Spark DataFrames or Delta tables.
The initial use case for this SDK revolves around three main objectives:
Uploading Client Datasets into Clarifai App:
The SDK should enable the seamless upload of datasets into the Clarifai application, simplifying the process of data transfer from Databricks to Clarifai.
Annotate the Data:
It should provide features for data annotation, making it easier for users to add labels and metadata to their datasets within the Clarifai platform.
Export Annotations to Spark DataFrames/Delta Tables:
The SDK should offer functionality to export annotations and store them in Spark DataFrames or Delta tables, facilitating further data analysis within Databricks.
Requirements:
- Databricks : Runtime 13.3 or later
- Clarifai :
pip install clarifai
- Create your Clarifai account
- Follow the instructions to get your own Clarifai PAT
- Protocol Buffers : version 4.24.2
pip install protobuf==4.24.2
Setup:
Install the package and initialize the clarifaipyspark class to begin.
pip install clarifai-pyspark
Getting Started:
from clarifaipyspark.client import ClarifaiPySpark
Create a Clarifai-PySpark client object to connect to your app on Clarifai. You can also choose the dataset or create one in your clarifai app to upload the data.
claps_obj = ClarifaiPySpark(user_id=USER_ID, app_id=APP_ID, pat=CLARIFAI_PAT)
dataset_obj = claps_obj.dataset(dataset_id=DATASET_ID)
Examples:
Checkout these notebooks for various operations you can perform using clarifai-pyspark SDK.
If you want to enhance your AI journey with workflows and leveraging custom models (programmatically) our Clarifai SDK might be good place to start with. Please refer below resources for further references.
- Docs - Clarifai Docs
- Explore our community page - Clarifai Community
- Fork and contribute to our SDK here !
- Reach out to us on socials
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for clarifai_pyspark-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 39abace7d009b4dcac5ab5e90897d9106b269a814e09fb84fc0b76fa1e36bbb8 |
|
MD5 | 478fddffb15d12c16699603d817413ac |
|
BLAKE2b-256 | 345ddd1b22256368bf22511df740347f9f16237641f2e4642b9fc00abb800b56 |