aws-kinesisfirehose-s3-and-kinesisanalytics module

---

Stability: Experimental

All classes are under active development and subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

Reference Documentation:	https://docs.aws.amazon.com/solutions/latest/constructs/

Language	Package
Python	`aws_solutions_constructs.aws_kinesisfirehose_s3_and_kinesisanalytics`
Typescript	`@aws-solutions-constructs/aws-kinesisfirehose-s3-and-kinesisanalytics`
Java	`software.amazon.awsconstructs.services.kinesisfirehoses3kinesisanalytics`

Overview

This AWS Solutions Construct implements an Amazon Kinesis Firehose delivery stream connected to an Amazon S3 bucket, and an Amazon Kinesis Analytics application.

Here is a minimal deployable pattern definition:

Typescript

import { Construct } from 'constructs';
import { Stack, StackProps } from 'aws-cdk-lib';
import { KinesisFirehoseToAnalyticsAndS3 } from '@aws-solutions-constructs/aws-kinesisfirehose-s3-and-kinesisanalytics';

new KinesisFirehoseToAnalyticsAndS3(this, 'FirehoseToS3AndAnalyticsPattern', {
  kinesisAnalyticsProps: {
    inputs: [{
      inputSchema: {
        recordColumns: [{
          name: 'ticker_symbol',
          sqlType: 'VARCHAR(4)',
          mapping: '$.ticker_symbol'
        }, {
          name: 'sector',
          sqlType: 'VARCHAR(16)',
          mapping: '$.sector'
        }, {
          name: 'change',
          sqlType: 'REAL',
          mapping: '$.change'
        }, {
          name: 'price',
          sqlType: 'REAL',
          mapping: '$.price'
        }],
        recordFormat: {
          recordFormatType: 'JSON'
        },
        recordEncoding: 'UTF-8'
      },
      namePrefix: 'SOURCE_SQL_STREAM'
    }]
  }
});

Python

from aws_solutions_constructs.aws_kinesis_firehose_s3_kinesis_analytics import KinesisFirehoseToAnalyticsAndS3
from aws_cdk import (
    aws_kinesisanalytics as kinesisanalytics,
    Stack
)
from constructs import Construct

KinesisFirehoseToAnalyticsAndS3(self, 'FirehoseToS3AndAnalyticsPattern',
                                kinesis_analytics_props=kinesisanalytics.CfnApplicationProps(
                                    inputs=[kinesisanalytics.CfnApplication.InputProperty(
                                        input_schema=kinesisanalytics.CfnApplication.InputSchemaProperty(
                                            record_columns=[kinesisanalytics.CfnApplication.RecordColumnProperty(
                                                name='ticker_symbol',
                                                sql_type='VARCHAR(4)',
                                                mapping='$.ticker_symbol'
                                            ), kinesisanalytics.CfnApplication.RecordColumnProperty(
                                                name='sector',
                                                sql_type='VARCHAR(16)',
                                                mapping='$.sector'
                                            ), kinesisanalytics.CfnApplication.RecordColumnProperty(
                                                name='change',
                                                sql_type='REAL',
                                                mapping='$.change'
                                            ), kinesisanalytics.CfnApplication.RecordColumnProperty(
                                                name='price',
                                                sql_type='REAL',
                                                mapping='$.price'
                                            )],
                                            record_format=kinesisanalytics.CfnApplication.RecordFormatProperty(
                                                record_format_type='JSON'
                                            ),
                                            record_encoding='UTF-8'
                                        ),
                                        name_prefix='SOURCE_SQL_STREAM'
                                    )]
                                )
                                )

Java

import software.constructs.Construct;
import java.util.List;

import software.amazon.awscdk.Stack;
import software.amazon.awscdk.StackProps;
import software.amazon.awscdk.services.kinesisanalytics.*;
import software.amazon.awscdk.services.kinesisanalytics.CfnApplication.*;
import software.amazon.awsconstructs.services.kinesisfirehoses3kinesisanalytics.*;

new KinesisFirehoseToAnalyticsAndS3(this, "FirehoseToS3AndAnalyticsPattern",
        new KinesisFirehoseToAnalyticsAndS3Props.Builder()
                .kinesisAnalyticsProps(new CfnApplicationProps.Builder()
                        .inputs(List.of(new InputProperty.Builder()
                                .inputSchema(new InputSchemaProperty.Builder()
                                        .recordColumns(List.of(
                                                new RecordColumnProperty.Builder()
                                                        .name("ticker_symbol")
                                                        .sqlType("VARCHAR(4)")
                                                        .mapping("$.ticker_symbol")
                                                        .build(),
                                                new RecordColumnProperty.Builder()
                                                        .name("sector")
                                                        .sqlType("VARCHAR(16)")
                                                        .mapping("$.sector")
                                                        .build(),
                                                new RecordColumnProperty.Builder()
                                                        .name("change")
                                                        .sqlType("REAL")
                                                        .mapping("$.change")
                                                        .build(),
                                                new RecordColumnProperty.Builder()
                                                        .name("price")
                                                        .sqlType("REAL")
                                                        .mapping("$.price")
                                                        .build()))
                                        .recordFormat(new RecordFormatProperty.Builder()
                                                .recordFormatType("JSON")
                                                .build())
                                        .recordEncoding("UTF-8")
                                        .build())
                                .namePrefix("SOURCE_SQL_STREAM")
                                .build()))
                        .build())
                .build());

Pattern Construct Props

Name	Type	Description
kinesisFirehoseProps?	`kinesisFirehose.CfnDeliveryStreamProps`	Optional user-provided props to override the default props for the Kinesis Firehose delivery stream.
kinesisAnalyticsProps?	`kinesisAnalytics.CfnApplicationProps`	Optional user-provided props to override the default props for the Kinesis Analytics application.
existingBucketObj?	`s3.IBucket`	Existing instance of S3 Bucket object. If this is provided, then also providing bucketProps is an error.
bucketProps?	`s3.BucketProps`	User provided props to override the default props for the S3 Bucket.
logGroupProps?	`logs.LogGroupProps`	User provided props to override the default props for for the CloudWatchLogs LogGroup.
loggingBucketProps?	`s3.BucketProps`	Optional user provided props to override the default props for the S3 Logging Bucket.
logS3AccessLogs?	boolean	Whether to turn on Access Logging for the S3 bucket. Creates an S3 bucket with associated storage costs for the logs. Enabling Access Logging is a best practice. default - true

Pattern Properties

Name	Type	Description
kinesisAnalytics	`kinesisAnalytics.CfnApplication`	Returns an instance of the Kinesis Analytics application created by the pattern.
kinesisFirehose	`kinesisFirehose.CfnDeliveryStream`	Returns an instance of the Kinesis Firehose delivery stream created by the pattern.
kinesisFirehoseRole	`iam.Role`	Returns an instance of the iam.Role created by the construct for Kinesis Data Firehose delivery stream.
kinesisFirehoseLogGroup	`logs.LogGroup`	Returns an instance of the LogGroup created by the construct for Kinesis Data Firehose delivery stream
s3Bucket?	`s3.Bucket`	Returns an instance of the S3 bucket created by the pattern.
s3LoggingBucket?	`s3.Bucket`	Returns an instance of s3.Bucket created by the construct as the logging bucket for the primary bucket.
s3BucketInterface	`s3.IBucket`	Returns an instance of s3.IBucket created by the construct.

Default settings

Out of the box implementation of the Construct without any override will set the following defaults:

Amazon Kinesis Firehose

Enable CloudWatch logging for Kinesis Firehose
Configure least privilege access IAM role for Amazon Kinesis Firehose

Amazon S3 Bucket

Configure Access logging for S3 Bucket
Enable server-side encryption for S3 Bucket using AWS managed KMS Key
Enforce encryption of data in transit
Turn on the versioning for S3 Bucket
Don't allow public access for S3 Bucket
Retain the S3 Bucket when deleting the CloudFormation stack
Applies Lifecycle rule to move noncurrent object versions to Glacier storage after 90 days

Amazon Kinesis Data Analytics

Configure least privilege access IAM role for Amazon Kinesis Analytics

Architecture

Architecture Diagram

Release files for aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics 2.45.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics 2.45.0
File	Size	Uploaded
aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics-2.45.0.tar.gz	156.5 kB	Oct 14, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics 2.45.0
File	Interpreter	ABI	Platform	Reset
aws_solutions_constructs.aws_kinesis_firehose_s3_kinesis_analytics-2.45.0-py3-none-any.whl 156.0 kB Oct 14, 2023	Python 3	none	any	Details

Total release size: 312.5 kB

Release files / aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics-2.45.0.tar.gz

Download URL	aws-solutions-constructs.aws-kinesis-firehose-s3-kinesis-analytics-2.45.0.tar.gz
Size	156.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`d0e74a2c3b4e992c9859f30080564d800418fbf844182445b96b61c34976e3c5`
BLAKE2b-256 checksum How to use checksums	`d59d7dc7ac8373dfd18525a527f20d784c78f3ba11f72fc48628c02b34ea0ed2`
Upload date	Oct 14, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.12`

Release files / aws_solutions_constructs.aws_kinesis_firehose_s3_kinesis_analytics-2.45.0-py3-none-any.whl

Download URL	aws_solutions_constructs.aws_kinesis_firehose_s3_kinesis_analytics-2.45.0-py3-none-any.whl
Size	156.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`ae509cb2f986c8956febfc7bdce78e9d7eaa563d5d04f1a02f3440092cad837e`
BLAKE2b-256 checksum How to use checksums	`d0700e2ed86d44114ad19013853b199b60ae1eb84d02bc970201ed3ae554c136`
Upload date	Oct 14, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.12`