Low upload speeds with S3¶

S3 can be used as primary storage or as external storage, where there are different implementations that are used for the upload of files. Files are uploaded in chunks before being reassembled in a final request. If you have S3 as primary storage the files are uploaded in chunks directly to S3 without additional overhead. With external storage this currently means that each chunk gets uploaded to the s3 bucket, before then downloading all chunks again for reassembly and uploading the result, creating a fairly large overhead while uploading large files. Most recommendations can be applied to both primary as well as external S3 storage and this article outlines the recommendations.

Solution¶