We have a python fastapi RESTAPI backing a frontend and want to allow users to upload files to our gcp instance (esp via a frontend drag and drop but also via an API call). I want to use the GCP URL based file upload to a Google Cloud Storage bucket but I want to do file validation on these csv files first (some simple file checking like csv headers are what we expect etc, file type is csv etc.). These files will likely be around 1 GB or more and if possible, I don't want to load the whole file into memory. I couldn't find a way to insert a kind of validation function into the GCS upload. In which case is the way to go just:
- allow users to upload to the GCS generated URL
- Then programmatically read the newly uploaded file from the GCS bucket and do file validation on the csv header in memory.
- Delete the file from the GCS bucket if the file is not valid
Is there a way to do this that avoids the upload entirely in the case of invalid files?