SageMakerStudioDataEngineeringSessions
SageMaker Unified Studio Data Engineering Sessions
This pacakge depends on SageMaker Unified Studio environment, if you are using SageMaker Unified Studio, see AWS Doc for guidance.
This package contains functionality to support SageMaker Unified Studio connecting to various AWS Compute including EMR/EMR Serverless/Glue/Redshift etc.
It is utilizing ipython magics and AWS DataZone Connections to achieve the following features.
Features
- Connect to remote compute
- Execute Spark code in remote compute in Python/Scala
- Execute SQL queries in remote compute
- Send local variables to remote compute
How to setup
If you are using SageMaker Unifed Studio, you can skip this part, SageMaker Unifed Studio already set up the package.
This package contains various Jupyter Magics to achieve its functionality.
To load these magics, make sure you have iPython config file generated. If not, you could run ipython profile create, then a file with path ~/.ipython/profile_default/ipython_config.py should be generated
Then you will need to add the following line in the end of that config file
c.InteractiveShellApp.extensions.extend(['sagemaker_studio_dataengineering_sessions.sagemaker_connection_magic'])
Once that is finished, you could restart the ipython kernel and run %help to see a list of supported magics
Interactive vs background session
This packages uses SM_INPUT_NOTEBOOK_NAME environment variable to determine if the execution is through interactive or background session. See sagemaker_studio_dataengineering_sessions/sagemaker_database_session_manager/redshift/redshift_session.py file for usage.
Examples
To connect to remote compute, a DataZone Connection is required, you could create it via CreateConnection API, Let's say there's an existing connection called project.spark.
Supported Connection Type:
- IAM
- SPARK
- REDSHIFT
- ATHENA
Connect to remote compute and Execute Spark Code in Python
The following example will connect to AWS Glue Interactive session and run the spark code in Glue.
%%pyspark project.spark
import sys
import boto3
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
args = getResolvedOptions(sys.argv, ["redshift_url", "redshift_iam_role", "redshift_tempdir","redshift_jdbc_iam_url"])
print(f"{args}")
sc = SparkContext.getOrCreate()
spark = SparkSession(sc)
df = spark.read.csv(f"s3://sagemaker-example-files-prod-{boto3.session.Session().region_name}/datasets/tabular/dirty-titanic/", header=True)
df.show(5, truncate=False)
df.printSchema()
df.createOrReplaceTempView("df_sql_tempview")
Execute Spark Code in Scala
The following example will connect to AWS Glue Interactive session and run the spark code in Scala.
%%scalaspark project.spark
val dfScala = spark.sql("SELECT count(0) FROM df_sql_tempview")
dfScala.show()
Execute SQL query in remote compute
The following example will connect to AWS Glue Interactive session and run the spark code in Scala.
%%sql project.redshift
select current_user()
Some other helpful magics
%help - list available magics and related information
%send_to_remote - send local variable to remote compute
%%configure - configure spark application config in remote compute
1 maintainer
Metadata
1 maintainer
Release files for sagemaker-studio-dataengineering-sessions 1.3.27
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sagemaker_studio_dataengineering_sessions-1.3.27.tar.gz | 275.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sagemaker_studio_dataengineering_sessions-1.3.27-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 614.8 kB
Release files / sagemaker_studio_dataengineering_sessions-1.3.27.tar.gz
| Download URL | sagemaker_studio_dataengineering_sessions-1.3.27.tar.gz |
|---|---|
| Size | 275.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
df5d69aa2756564d7375c52efaee8f419e0ac256fb086a501ca2e4bec32f8d55
|
|
BLAKE2b-256 checksum How to use checksums |
f1b4bfb4c8c2587a517749ca607637533b2fbc25bb9ab74b8f781afebb5cdff4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.12
|
Release files / sagemaker_studio_dataengineering_sessions-1.3.27-py3-none-any.whl
| Download URL | sagemaker_studio_dataengineering_sessions-1.3.27-py3-none-any.whl |
|---|---|
| Size | 339.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4be729d871d828d9d0829ef07efedd1b9a585e5058c0371d2071b203e1c82262
|
|
BLAKE2b-256 checksum How to use checksums |
ab93d4931c0d6355f4dc0fdd38207f20bfd643a3d4671b88408a5202e2b8bb4d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.12
|
