How to Upload Terraform Files to S3 and Use
Tabular array of contents
- Events structures
- S3 event structure
- SQS event structure
- Demo Lambda function code
- Terraform lawmaking
- Project structure
- Provider configuration
- Variables
- S3 and SQS
- Lambda function
- Deployment
- Testing
- Cleaning up
- Summary
- Related articles
1 of the common integration patterns used by modern enterprises is exchanging files using S3 buckets. Files might be uploaded to the S3 bucket from various sources by humans or automated systems using HTTPS, SFTP, or SCP protocols. As soon as the file uploaded, at that place's a demand for future file processing. In this article, nosotros'll depict how this integration works and how to deploy it using Terraform.
Here'southward how the compages diagram for this integration looks like.
Events structures
The integration is very easy to understand as soon as you tin see the event messages sent from i service no another equally soon as the file uploaded to the S3 bucket.
Jumping ahead, S3 result is transferred within SQS message torso and that's how AWS Lambda office code can get access in that location (will be shown below).
S3 event structure
S3 generated events are widely used and well covered over the internet.
Here's how this event structure looks similar:
{ "Records":[ { "eventVersion":"2.1", "eventSource":"aws:s3", "awsRegion":"us-east-1", "eventTime":"2021-04-26T23:31:08.107Z", "eventName":"ObjectCreated:Put", "userIdentity":{ "principalId":"AWS:012850762433:admin" }, "requestParameters":{ "sourceIPAddress":"108.41.58.86" }, "responseElements":{ "x-amz-request-id":"YP7DR0F7H7R1GN1S", "x-amz-id-ii":"WYvnoGQrVxe2LfV6yr/sDsZXj/QDL0vD02WQYn9zXg3jX2iKfq83omTmcOcIiuSUk4dTmRRDrhdNNzffoi8AeSBN7RHs2ab0" }, "s3":{ "s3SchemaVersion":"1.0", "configurationId":"tf-s3-queue-20210426224851886600000002", "bucket":{ "proper noun":"amaksimov-s3-sqs-demo-bucket", "ownerIdentity":{ "principalId":"A1W385KKD8Q319" }, "arn":"arn:aws:s3:::amaksimov-s3-sqs-demo-bucket" }, "object":{ "key":"iv.+Beginner%27s+Guide+to+AWS+Step+functions+-+HelloWorld+Example.png", "size":9714, "eTag":"b21c122beffd36c0f0caabc4dbd8b16d", "sequencer":"0060874D3FC2FA681D" } } } ] } The S3 issue consist of a list of records, each of which describes the object within S3 saucepan.
The nigh normally used fields are:
-
issue['Records'][*]['s3']['saucepan']['proper noun']– gives us a saucepan name where the file been uploaded -
event['Records'][*]['s3']['object']['central']– gives us a file proper name and location within the S3 bucket
In addition to the file upload events, S3 sends test effect to test which has the following structure:
{ "Service":"Amazon S3", "Event":"s3:TestEvent", "Time":"2021-04-27T13:57:03.224Z", "Bucket":"amaksimov-s3-sqs-demo-saucepan", "RequestId":"MDSYJ6FFMMZ75MJ8", "HostId":"bydBlxgzo+XD8x1szLD+YfeaN8DUtNoxEHsMDySKd1wuX1PKvuYx4h/Iw8uUM1wx/uImu1On5sI=" } We'll need to skip this issue in our Lambda function code to avoid error messages.
SQS event construction
SQS events are widely used too and they have the similar structure.
{ "Records":[ { "messageId":"581db230-9853-4be3-a1fe-72c9a5b3e4d4", "receiptHandle":"AQEBAwV4m8sSkn5jDd1k/GBLco1znfiv+xT0KTRZdEhQE7clWhAcFlVusMR07RQsBo5ImrlIDafWwdzfX+ZqsuRQPGWE0CcsR6ga8yQTTtG6N1CpWuotJ69Ef55XILtkOMKS+7HR3Ek1oigests3bmx5eCj0QlsRR56qSpj0o1yOOLktLsUehPPTEmWmWXGGPoTc2GayxbnL6lCheolswgiMdE2u0qmbaKV6Ek3E4PyvPfzkOx8XGXIurYJCkFMGcpi0sWrus1WO+dzbm5NtOL9n8qAzjxaMyMyV+nXvy+EO1QCLu2CuX0/rhKfjoq0+txWm8tNVb27VKbwsRKrU12odmV9mbULuvKDU55CqNOMF+LZl8zdZzceegvK2wgfA8KjMmpJ5wQVWo0S8WqVpcJCKSJYhoh/XzqGde+1gQ957YR8=", "body":"{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"u.s.-due east-1\",\"eventTime\":\"2021-04-26T23:25:17.884Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AWS:012850762433:admin\"},\"requestParameters\":{\"sourceIPAddress\":\"108.41.58.86\"},\"responseElements\":{\"x-amz-request-id\":\"74CMGJPKH3HA1G87\",\"x-amz-id-2\":\"c52dEWNgb6rNUs7MNY20ArZHLgtNFiRJIhREfnNAnlLsXHotTUvLS7InfWnkniuawxPgTlkOkTKZICwIgsbfdHDZKQvL0LcV\"},\"s3\":{\"s3SchemaVersion\":\"i.0\",\"configurationId\":\"tf-s3-queue-20210426224851886600000002\",\"bucket\":{\"name\":\"amaksimov-s3-sqs-demo-bucket\",\"ownerIdentity\":{\"principalId\":\"A1W385KKD8Q319\"},\"arn\":\"arn:aws:s3:::amaksimov-s3-sqs-demo-bucket\"},\"object\":{\"central\":\"6.+Beginner%27s+Guide+to+AWS+Pace+functions+-+AWS+HelloWorld+example.png\",\"size\":458757,\"eTag\":\"e1148e80d0798b0e23502cbdae1fef58\",\"sequencer\":\"0060874BE06812C89A\"}}}]}", "attributes":{ "ApproximateReceiveCount":"1", "SentTimestamp":"1619479521272", "SenderId":"AIDAJHIPRHEMV73VRJEBU", "ApproximateFirstReceiveTimestamp":"1619479521279" }, "messageAttributes":{ }, "md5OfBody":"7195d8d0f011fac4dc115b59d3e86797", "eventSource":"aws:sqs", "eventSourceARN":"arn:aws:sqs:us-east-one:012850762433:amaksimov-s3-event-notification-queue", "awsRegion":"us-east-ane" } ] } SQS consequence consist of a list of records, each of which stand for a message grabbed from the SQS queue.
The nearly unremarkably field hither is:
-
upshot['Records'][*]['body']– this field contains the text body of the SQS bulletin
Demo Lambda function code
Equally soon as know events structures for both events, nosotros tin can easily write a demo Lambda office, which tin can process those events.
Here's how the lawmaking looks like (index.py):
#!/usr/bin/env python3 import logging import json LOGGER = logging.getLogger() LOGGER.setLevel(logging.INFO) def handler(event, context): try: LOGGER.info('SQS EVENT: %s', event) for sqs_rec in event['Records']: s3_event = json.loads(sqs_rec['body']) LOGGER.info('S3 EVENT: %s', s3_event) # Skipping S3 test event if 'Event' in s3_event.keys() and s3_event['Event'] == 's3:TestEvent': interruption for s3_rec in s3_event['Records']: LOGGER.info('Bucket proper noun: %s', s3_rec['s3']['bucket']['proper name']) LOGGER.info('Object key: %s', s3_rec['s3']['object']['key']) except Exception as exception: LOGGER.error('Exception: %s', exception) enhance This Lambda office code processes SQS events.
For every message from the received outcome the lawmaking extracting the S3 message from the SQS message body field.
Nosotros're checking if we received the test effect and skipping it.
Finally, the code logs events, S3 bucket name, and uploaded S3 object key.
This minimal working lawmaking is useful if you'd like to quickly get started using it in your own project.
Terraform code
As soon as we are having a Lambda role and understanding how all the pieces need to be ties together, we tin can declare everything in Terraform.
Project structure
Hither's the project structure, which we'll use for our projection:
. ├── lambda │ └── index.py ├── principal.tf ├── provider.tf ├── s3.tf └── variables.tf ane directory, v files Provider configuration
Elementary AWS provider configuration is declared in provider.tf file:
# AWS provider configuration provider "aws" { region = var.region } Variables
We parameterize our Terraform code with the following variables defined in variables.tf file:
variable "region" { default = "us-east-ane" description = "AWS Region to deploy to" } variable "app_env" { default = "amaksimov" description = "Common prefix for all Terraform created resources" } S3 and SQS
Declaration and integration of the S3 bucket and SQS has been put to the s3.tf file:
# SQS queue resource "aws_sqs_queue" "queue" { name = "${var.app_env}-s3-event-notification-queue" policy = <<POLICY { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Primary": "*", "Activeness": "sqs:SendMessage", "Resource": "arn:aws:sqs:*:*:${var.app_env}-s3-outcome-notification-queue", "Condition": { "ArnEquals": { "aws:SourceArn": "${aws_s3_bucket.bucket.arn}" } } } ] } POLICY } # S3 bucket resources "aws_s3_bucket" "bucket" { bucket = "${var.app_env}-s3-sqs-demo-saucepan" } # S3 consequence filter resource "aws_s3_bucket_notification" "bucket_notification" { bucket = aws_s3_bucket.bucket.id queue { queue_arn = aws_sqs_queue.queue.arn events = ["s3:ObjectCreated:*"] } } # Event source from SQS resource "aws_lambda_event_source_mapping" "event_source_mapping" { event_source_arn = aws_sqs_queue.queue.arn enabled = true function_name = aws_lambda_function.sqs_processor.arn batch_size = i } Lambda function
We've put Lambda function related resources and Lambda role CloudWatch Log Group to the master.tf file:
# Data resource to archive Lambda part code data "archive_file" "lambda_zip" { source_dir = "${path.module}/lambda/" output_path = "${path.module}/lambda.zip" type = "zip" } # Lambda function policy resource "aws_iam_policy" "lambda_policy" { name = "${var.app_env}-lambda-policy" description = "${var.app_env}-lambda-policy" policy = <<EOF { "Version": "2012-ten-17", "Statement": [ { "Action": [ "s3:GetObject", "s3:PutObject" ], "Effect": "Allow", "Resource": "${aws_s3_bucket.bucket.arn}" }, { "Activity": [ "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes" ], "Issue": "Allow", "Resource": "${aws_sqs_queue.queue.arn}" }, { "Activeness": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Effect": "Allow", "Resources": "*" } ] } EOF } # Lambda function role resources "aws_iam_role" "iam_for_terraform_lambda" { name = "${var.app_env}-lambda-role" assume_role_policy = <<EOF { "Version": "2012-ten-17", "Statement": [ { "Activeness": "sts:AssumeRole", "Master": { "Service": "lambda.amazonaws.com" }, "Upshot": "Allow" } ] } EOF } # Role to Policy zipper resource "aws_iam_role_policy_attachment" "terraform_lambda_iam_policy_basic_execution" { role = aws_iam_role.iam_for_terraform_lambda.id policy_arn = aws_iam_policy.lambda_policy.arn } # Lambda office declaration resources "aws_lambda_function" "sqs_processor" { filename = "lambda.cypher" source_code_hash = data.archive_file.lambda_zip.output_base64sha256 function_name = "${var.app_env}-lambda" role = aws_iam_role.iam_for_terraform_lambda.arn handler = "index.handler" runtime = "python3.8" } # CloudWatch Log Group for the Lambda function resource "aws_cloudwatch_log_group" "lambda_loggroup" { name = "/aws/lambda/${aws_lambda_function.sqs_processor.function_name}" retention_in_days = 14 } Deployment
As soon every bit we created all Terraform projection files, nosotros tin deploy this integration using the following commands:
terraform init terraform utilise -auto-approve Testing
Afterward finishing the deployment, nosotros can upload any file to the created S3 bucket and see the post-obit CloudWatch messages in identify.
Cleaning up
To clean upward everything, you need to delete all the uploaded files from the S3 saucepan and so execute the following Terraform control:
terraform destroy -auto-approve Summary
In this article, we've created a widely used integration building block that consists of an S3 bucket, SQS queue, and Lambda function. The purpose of this building block is to requite you a guarantee of processing every S3 upload event with the Lambda function in a Serverless way.
We hope, that you found this article useful. If then, please, assistance us to spread it to the globe!
Related manufactures
I'm a passionate Cloud Infrastructure Builder with more than fifteen years of experience in It.
Any of my posts represent my personal experience and stance nearly the topic.
Source: https://hands-on.cloud/using-terraform-to-deploy-s3-sqs-lambda-integration/
Post a Comment for "How to Upload Terraform Files to S3 and Use"