CloudDuck is a CDK construct for simple and easy-to-use analysis environment for S3 data, featuring DuckDB with built-in authentication.
By simply deploying the Construct, you can launch a SaaS that provides an analytics dashboard like the one shown below. User authentication for access is implemented using Cognito, ensuring that only authorized users can log in.
Detailed information
- In Japanese: duckdbを用いたS3データ分析SaaSをCDKコンストラクトとして提供しました
- In english: CDK construct for a SaaS that performs data analysis on S3 using DuckDB
Table of Contents
Use Cases
- When you want to request data analysis on S3 using DuckDB but prefer not to issue S3 access credentials to the analysts.
- When you want to minimize the costs incurred from downloading large amounts of S3 data to local storage.
Architecture
Installation
npm i cloud-duck
Setup
Deploy
You can deploy the CloudDuck with the following code in the CDK stack.
import { CloudDuck } from 'cloud-duck';
import { Size } from 'aws-cdk-lib';
import * as cognito from 'aws-cdk-lib/aws-cognito';
declare const logBucket: s3.IBucket;
new CloudDuck(this, 'CloudDuck', {
// The S3 bucket to analyze
// CloudDuck can access to all of the buckets in the account by default.
// If you want to restrict the access, you can use the targetBuckets property.
targetBuckets: [logBucket],
// The memory size of the Lambda function
// Default: 1024 MB
memory: Size.mebibytes(1024),
// You can customize the Cognito User Pool
// For example, you can force the user to use MFA.
userPoolPlpos: {
mfa: cognito.Mfa.REQUIRED,
mfaSecondFactor: {
sms: false,
otp: true,
},
},
});
Add user to the Cognito User Pool
Add a user to the Cognito User Pool with the following command.
aws cognito-idp admin-create-user \
--user-pool-id "us-east-1_XXXXX" \
--username "naonao@example.com" \
--user-attributes Name=email,Value="naonao@example.com" Name=email_verified,Value=true \
--message-action SUPPRESS \
--temporary-password Password1!
You can also add a user via the AWS Management Console.
Access
Access to the CloudDuck with the cloudfront URL.
❯ npx cdk deploy
...
AwsStack.CloudDuckDistributionUrl84FC8296 = https://dosjykpv096qr.cloudfront.net
Stack ARN:
arn:aws:cloudformation:us-east-1:123456789012:stack/AwsStack/dd0960c0-b3d5-11ef-bcfc-12cf7722116f
✨ Total time: 73.59s
Enter the username and password.
When you log in at the first time, you need to change the password.
Play with the CloudDuck!
Usage
Query
You can query the S3 data with SQL.
SELECT * FROM read_csv_auto('s3://your-bucket-name/your-file.csv');
SELECT * FROM parquet_scan('s3://your-bucket-name/your-file.parquet');
Ofcourse, you can store the result as a new table.
CREATE TABLE new_table AS SELECT * FROM read_csv_auto('s3://your-bucket-name/your-file.csv');
Detail usage of DuckDB is available at DuckDB Documentation.
Persistence
All query results are persisted in individual DuckDB files for each user. Therefore, you can freely save your query results without worrying about affecting other users.
Note
CloudDuck is still under development. Updates may include breaking changes. If you encounter any bugs, please report them via issues.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file cloud_duck-0.0.28.tar.gz.
File metadata
- Download URL: cloud_duck-0.0.28.tar.gz
- Upload date:
- Size: 54.0 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.14.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
bbd350be338a3cb670107d90b6ef7fed3ce3b62af64da7b02ccc25667afd8985
|
|
| MD5 |
775971c2f15bf5807a22ddbc4b68cc84
|
|
| BLAKE2b-256 |
159f1d8dfb68f706276d3567acf4abffe08cf6fa0d960b14e3295b3a4ef3a22c
|
File details
Details for the file cloud_duck-0.0.28-py3-none-any.whl.
File metadata
- Download URL: cloud_duck-0.0.28-py3-none-any.whl
- Upload date:
- Size: 54.0 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.14.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
edb4bdae51e49b925a421149d808a4e80ae2f5df6c484d57bc750a8ac97f53ed
|
|
| MD5 |
09e67b9aeea6f159ba4d87173c0584ca
|
|
| BLAKE2b-256 |
2cf68433aa089dc7c0d39ad4995a621ebb2656e8fac1299da240147e1df9d374
|
