BigQuery - How to find the error stream?
While using the bq command to load data into Bigquery, you will often have reject/error records based on bad data in the columns or even mismatch in the number of columns.
The Error message usually looks like the below:
Error While reading data, error message: CSV table encountered too many errors, giving up, Rows: 3894; errors:1 Please look into the error stream for more details
Where to find the error stream?
Run the following command in the terminal:
bq --format=prettyjson show -j <JobID>
It returns a JSON with more details. In my case it was:
"message": "Error while reading data, error message: Could not parse '16.66666666666667' as int for field Course_Percentage (position 46) starting at location 1717164"
Read great educational content like this and a lot more !
Members get free exclusive access to content, new courses, and discounts. Signup for a free account to write a post / comment / upvote posts. Creating an account takes less than 5 seconds and you can start earning badges & points too