Easy Event-Driven Workflows With AWS Lambda V2 Featuring Serverless Framework
Prefect makes it easy to do hard work.
March 8, 2022
urllib, but the Prefect Client keeps our code tidy, legible, and easy to modify.
decrypt_parameterdecrypts secrets from the AWS System Manager Parameter Store.
runfunction takes an S3 PUT event and extracts
Keyfrom the payload.
get_memory_requiredinterrogates the S3 API for information about the size of the file that was uploaded.
trigger_flow_runkicks off a Prefect flow run with a custom name derived from the S3 key. We pass the file size derived from
RunConfigin order to set a memory request on the Kubernetes job that will be running our flow.
version_group_idwhich is the unique unarchived flow within this version group will be scheduled to run. This input can be used as a stable API for running flows that are regularly updated.
serverless.ymlfile. Further, serverless has their own concept of plugins. Most relevant to us is the
serverless-python-requirementsplugin, which makes the packaging Python requirements painless. The file is broken up into arrays from top to bottom:
providerdefines platform specific configurations. In our case, we have Lambda runtime settings and some IAM role configurations.
functionsdefines any number of Lambda functions as well as some incredibly useful ancillary options:
handlerspecifies which function Lambda is going to start our function.
layersallows us to take advantage of the work that our
serverless-python-requirementsplugin does and package the Python requirements separately from our code, which means that we can still edit code in the Lambda IDE.
eventsis one of my favorites, through it we can automatically subscribe our lambda to AWS events. In our example, we are subscribing our function to specific events from an S3 bucket.
serverless deployand watch:
- Spins up a Docker container to package all of the aforementioned Python requirements.
- Creates CloudFormation templates from our YAML file.
- Zips up our code and dependencies.
- Uploads everything to S3.
- Join our Slack community for ad-hoc questions.
- Follow us on Twitter for updates.
- Visit us on GitHub to open issues and pull requests.
- Check out our Discourse knowledge base.