Upload to AWS S3 from Android: S3 Integration & Customizations for large files
A n android app that allows teachers to record video lectures and share with students on a centralized platform ran into a problem this ongoing pandemic, all of a sudden upload to S3 bucket started failing posterior to debugging we found this case is with files large(>1 GB) in size. No surprise that teachers working from home had started recording longer videos.
To handle this issue I tweaked some of the AWS S3 properties which were found while going through multiple AWS docs, in this read I’ve documented all those steps that worked for me in achieving this. Hope this helps you as well.
AWS Gradle Dependency
let’s start with Gradle dependency for AWS:
Initialize AWS S3 Client:
AWS S3Client can be initialized in the following ways:
1)with cognitoCredentialsProvider & Region
2)or with one other property ClientConfiguration()
using ClientConfiguration() object many properties of s3Client can be customized such as connectionTimeout, socketTimeout, maxErrorRetry, maxConnections, protocol, userAgent, trustManager, curlLogging, and other various proxy related properties. here, I have used connectionTimeout, socketTimeout, and maxErrorRetry. The default value of both connectionTimeout & socketTimeout is 15*1000 millis you should try with different values based on your region and conclude what works best for you. for start 50*1000 sounds a sufficient RTT value for any AWS hosted region.
AWS Transfer Utility:
TransferUtility is a high-level AWS API for handling uploads/downloads. It automatically handles multipart uploads, if the file is bigger than 5MB in size then it will be uploaded in multiple parts. The maximum allowed parts in a multipart upload are 10000 & the minimum part size is 5MB. Earlier with TransferManager API multiPart had to be configured manually.
In the case of large files, do configure appropriate values for connectionTimeout & socketTimeout while initializing the AWS S3 Client, as I have noticed transfer failing for large files with low RTT value.
also, you can set #threads in the pool for transfers. different values can be set for different NetworkConnectionType. to do that call transferUtilityOptions() on transferUtility’s builder method.
Transfer observer & Transfer listener:
calling upload on TransferUtility object returns TransferObserver object, you can observe changes in transferState, transfer progress by setting a TransferListener to the observer and overriding below methods:
onStateChanged(): invoked when there is any change in TransferState, TransferState.COMPLETED: This indicates a completed transfer. TransferState.FAILED: This indicates a failed transfer but you can try resuming the transfer here check the code snippet below.
onProgressChanged(): reports change in bytes sent, update UI/Notification here to show progress. this callback is on the main thread.
onError(): indicates a failed transfer, invoked with an exception (HTTPTimeout/SocketTimeout/etc.). check exception & stack trace for details
That’s how Amazon Web Services Simple Storage Service(AWS S3) is used with Android. Let me share one more pro tip with you: if you are working with large files and constantly getting SocketTimeout exception /SocketClosed error, put socketTimeout = 0 while initializing ClientConfiguration. This will set your socketTimeout to infinity and may solve your issue. Clap if you found this helpful.
Let me know if you have any questions/doubts by commenting below and I’ll try answering them.
Thanks for reading... Happy coding mate :)
