15 September 2018 · Bozhack-miller ·       Add to Favorites   Report

How to automate Google Cloud Storage to Bigquery load

If you have a process to upload files in GCS, you can automate the Google cloud storage to BigQuery Load using the below methods

Solution:

Here are a few ways:

  • You can use a Google Cloud Function (Supports Python, Go, Node.js) with a Cloud Storage Trigger is the best and optimal solution
  • You can always set up an app engine instance with a CRON job that runs a python script every day
  • You can use a dedicated compute engine with a CRON job that runs a python script every day

In all the above three methods you can use PubSub queue to notify stakeholders on the successful load completion

Bozhack-miller

posted on 15 September 2018

Read great educational content like this and a lot more !

Members get free exclusive access to content, new courses, and discounts. Signup for a free account to write a post / comment / upvote posts. Creating an account takes less than 5 seconds and you can start earning badges & points too

Copied