How to automate Google Cloud Storage to Bigquery load
If you have a process to upload files in GCS, you can automate the Google cloud storage to BigQuery Load using the below methods
Solution:
Here are a few ways:
- You can use a Google Cloud Function (Supports Python, Go, Node.js) with a Cloud Storage Trigger is the best and optimal solution
- You can always set up an app engine instance with a CRON job that runs a python script every day
- You can use a dedicated compute engine with a CRON job that runs a python script every day
In all the above three methods you can use PubSub queue to notify stakeholders on the successful load completion
Bozhack-miller
posted on 15 Sep 18Enjoy great content like this and a lot more !
Signup for a free account to write a post / comment / upvote posts. Its simple and takes less than 5 seconds
Post Comment