lambda-kinesis-to-es

baikonur-oss/lambda-kinesis-to-es/aws

Terraform Module HCL AWS

Terraform module and Lambda for Kinesis to Elasticsearch Service logs transfer

Install

module "lambda-kinesis-to-es" {

source = "baikonur-oss/lambda-kinesis-to-es/aws"

version = "2.1.0"

}

plain text: /constructs/tfmod-baikonur-oss-lambda-kinesis-to-es-aws/install.txt

⭐ Source on GitHub 📦 Registry page

README

Amazon Kinesis to Elasticsearch Service log transfer Terraform module Terraform module and Lambda for saving JSON log records from Kinesis Data Streams to Elasticsearch Service. !terraform v0.12.x ![Language grade: Python](https://lgtm.com/projects/g/baikonur-oss/terraform-aws-lambda-kinesis-to-es/context:python) Prerequisites 1. Records in Kinesis stream must be valid JSON data. Non-JSON data will be saved with unknown prefix. 1. gzipped JSON, CloudWatch Logs subscription filters log format are supported. 2. Logs without either of necessary keys listed below will be saved as unknown as well. 2. JSON data must have the following keys (key names are modifiable via variables): 1. log_type: Log type identifier. Elasticsearch indices will be created daily, one per log type: %index_name_prefix%

Inputs (23)

Name	Type	Description	Default
batch_size	string	Maximum number of records passed for a single Lambda invocation	required
index_name_prefix	string	Prefix for Elasticsearch indices names	required
kinesis_stream_arn	string	Source Kinesis Data Streams stream name	required
elasticsearch_host	string	Elasticsearch Service endpoint (without https://)	required
elasticsearch_arn	string	Elasticsearch Service ARN	required
failed_log_s3_bucket	string	S3 bucket name for saving failed logs (ES API errors etc.)	required
lambda_package_url	string	Lambda package URL (see Usage in README)	required
name	string	Resource name	required
failed_log_s3_prefix	string	Path prefix for failed logs	required
timezone	string	tz database timezone name (e.g. Asia/Tokyo)	`"UTC"`
runtime	string	Lambda Function runtime	`"python3.7"`
log_id_field	string	Key name for unique log ID	`"log_id"`
log_type_unknown_prefix	string	Log type prefix for logs without log type field	`"unknown"`
starting_position	string	Kinesis ShardIterator type (see: https://docs.aws.amazon.com/kinesis/latest/APIR	`"TRIM_HORIZON"`
log_type_field_whitelist	list(string)	Log type whitelist (if empty, all types will be processed)	`[]`
tracing_mode	string	X-Ray tracing mode (see: https://docs.aws.amazon.com/lambda/latest/dg/API_Tracin	`"PassThrough"`
memory	string	Lambda Function memory in megabytes	`256`
handler	string	Lambda Function handler (entrypoint)	`"main.handler"`
log_type_field	string	Key name for log type	`"log_type"`
log_timestamp_field	string	Key name for log timestamp	`"time"`
timeout	string	Lambda Function timeout in seconds	`60`
tags	map(string)	Tags for Lambda Function	`{}`
log_retention_in_days	string	Lambda Function log retention in days	`30`