A new Prefect 2.0 Collection, prefect-gcp, was recently released. It consists of prebuilt Prefect integrations to easily interact with Google Cloud Platform within Prefect flows.
prefect-gcpcurrently supports features from a variety of the Google Cloud Platform services, including Cloud Storage, Secret Manager, and BigQuery.
There are now a total of six collections available for community use (at the time of writing); all available collections are shown in the Prefect Collections Catalog.
We welcome any user feedback, questions, and contributions!
At Prefect, we want to make it straightforward to orchestrate your data stack. We want to reduce the friction you experience when creating dataflows—friction like spending time writing boilerplate code to reliably download blobs from Google Cloud Storage. With those ideas in mind, we developed
prefect-gcp so that users can easily and quickly interact with a variety of the Google Cloud Platform services within Prefect flows, no sweat!
For example, grab a blob from a Google Cloud Storage bucket in under 10 lines of code:
Want to know a secret 😉? I suppose the secret is obvious since it’s so easy to read important secrets from the Google Cloud Secret Manager and incorporate your existing secrets in a flow with just a line of code!
Or if secrets aren’t your thing, perhaps posing queries is? If so, BigQuery can be integrated into your flow effortlessly to enable simple reading and writing of data during your flows to tackle your organization’s biggest input-queries! (Inquiries? 😄)
These three code snippets demonstrate how effortless it is to integrate Google Cloud Platform services within Prefect 2.0 flows! For more examples and all the integrations available in
prefect-gcp check out the docs page!
Note that the Collections maintained by Prefect are all open source. That means you can help build
prefect-gcp, or any other Collections. Really! Whether that’s asking usage questions, providing user experience feedback, submitting bug reports or feature requests, or opening pull requests to resolve the aforementioned bugs or features, they’re all welcome ways of contributing from our perspective! Hope to collaborate with you soon!