15 Sep 18 · Bozhack-miller ·       Add to Favorites   Report

How to automate Google Cloud Storage to Bigquery load

If you have a process to upload files in GCS, you can automate the Google cloud storage to BigQuery Load using the below methods


Here are a few ways:

  • You can use a Google Cloud Function (Supports Python, Go, Node.js) with a Cloud Storage Trigger is the best and optimal solution
  • You can always set up an app engine instance with a CRON job that runs a python script every day
  • You can use a dedicated compute engine with a CRON job that runs a python script every day

In all the above three methods you can use PubSub queue to notify stakeholders on the successful load completion


posted on 15 Sep 18

Read great educational content like this and a lot more !

Members get free exclusive access to content, new courses, and discounts. Signup for a free account to write a post / comment / upvote posts. Creating an account takes less than 5 seconds and you can start earning badges & points too