Fix large remote file uploads #3739
Conversation
Apache expects a response to the request within 60 seconds (ProxyTimeout). Avoid that by responding with a transfer for the upload, which the browser will query the status of instead.
| dest_remote: remote, | ||
| tempfile: tempfile | ||
| ) | ||
| transfer.tap(&:perform_later) |
There was a problem hiding this comment.
digging through RcloneUtil.rclone_with_progress I wonder what happens when rclone moveto fails? I'm guessing Rack has some way to delete the file on failure. Do we have such a safety mechanism?
There was a problem hiding this comment.
It's something we can defer - it's just an edge case we likely need to take care of if Rack isn't going to guarantee it.
There was a problem hiding this comment.
tempfile here is a Tempfile, so it is cleaned up on garbage collection (which is why I pass the whole thing to RemoteTransfer) or when the Ruby interpreter exits.
I pushed a commit to explicitly clean up (close) the file on transfer errors, so now it should always be cleaned up instantly either when the transfer completes or fails.
Prevents the Tempfile from staying around until garbage collection on error.
* Handle remote uploads as file transfers Apache expects a response to the request within 60 seconds (ProxyTimeout). Avoid that by responding with a transfer for the upload, which the browser will query the status of instead. * Explicitly close RemoteTransfer tempfile Prevents the Tempfile from staying around until garbage collection on error.
* Handle remote uploads as file transfers Apache expects a response to the request within 60 seconds (ProxyTimeout). Avoid that by responding with a transfer for the upload, which the browser will query the status of instead. * Explicitly close RemoteTransfer tempfile Prevents the Tempfile from staying around until garbage collection on error.
Uploading large files to remote filesystems has had problems for a long time (#2250), so I spent some time trying to find a solution and this was the result. I attempted to find some alternative solutions as Jeff commented on that issue, but was not able to find any.
This PR fixes #2250 by making the actual upload from the Rails temporary file to the remote filesystem happen in an Active Job instead of during the 60 second period (Apache
ProxyTimeout) before Apache will return an error to the browser.I was able to reuse most of the existing code for file transfers. The back end for uploads now sends a JSON response with a transfer if one was created, i.e. a transfer from
/tmp/RackMultipart...tos3remote:/mybucket/myobject. The front end then queries the status of that transfer.