
Easy Event-Driven Workflows With AWS Lambda V2 Featuring Serverless Framework
Prefect makes it easy to do hard work.
March 8, 2022
requests
or urllib
, but the Prefect Client keeps our code tidy, legible, and easy to modify.decrypt_parameter
decrypts secrets from the AWS System Manager Parameter Store.- Our
run
function takes an S3 PUT event and extractsBucket
andKey
from the payload. get_memory_required
interrogates the S3 API for information about the size of the file that was uploaded.trigger_flow_run
kicks off a Prefect flow run with a custom name derived from the S3 key. We pass the file size derived fromget_memory_required
to theKubernetesRun
RunConfig
in order to set a memory request on the Kubernetes job that will be running our flow.
version_group_id
which is the unique unarchived flow within this version group will be scheduled to run. This input can be used as a stable API for running flows that are regularly updated.serverless.yml
file. Further, serverless has their own concept of plugins. Most relevant to us is the serverless-python-requirements
plugin, which makes the packaging Python requirements painless. The file is broken up into arrays from top to bottom:provider
defines platform specific configurations. In our case, we have Lambda runtime settings and some IAM role configurations.functions
defines any number of Lambda functions as well as some incredibly useful ancillary options:handler
specifies which function Lambda is going to start our function.layers
allows us to take advantage of the work that ourserverless-python-requirements
plugin does and package the Python requirements separately from our code, which means that we can still edit code in the Lambda IDE.events
is one of my favorites, through it we can automatically subscribe our lambda to AWS events. In our example, we are subscribing our function to specific events from an S3 bucket.
serverless deploy
and watch:serverless:
- Spins up a Docker container to package all of the aforementioned Python requirements.
- Creates CloudFormation templates from our YAML file.
- Zips up our code and dependencies.
- Uploads everything to S3.
- Join our Slack community for ad-hoc questions.
- Follow us on Twitter for updates.
- Visit us on GitHub to open issues and pull requests.
- Check out our Discourse knowledge base.