Upload API limits and serverless environments
Hi there 👋
At Apideck we're building a Unified API for file storage providers such as Box, OneDrive and Dropbox. Our solution runs on AWS Lambda, and Lambda has a request body limit of 6MB. Other serverless providers such as Google Cloud have similar limits.
For small files (<6MB) we use the direct upload API, no issues there. For >6MB files we have to use upload sessions, as Lambda can only handle 6MB per request. Unfortunately we can't use the Box upload sessions API because it only supports files larger than 20MB. In addition, file chunks in a Box upload session are always 8MB, which is also above the lambda limit.
Because of these limits we can't support uploading >6MB files to Box at the moment. We already support a number of other providers such as Google Drive, OneDrive and Dropbox.
Could these limits be removed or lowered? I imagine other serverless users run into the same issue. Alternatively, is it possible to change these limits on a per-customer basis?
https://docs.apideck.com/apis/file-storage/reference#tag/Upload-Sessions
https://docs.apideck.com/apis/file-storage/reference#operation/filesUpload
-
Official comment
Hi,
Thanks for reaching out. I will need to talk some folks internally and report back. Have you worked with anyone from Box in the past?
Thanks,
Alex, Box Developer Advocate
Comment actions -
Hi Sand Box
I am the product manager for Uploads & Downloads.
Thanks for your request and I don't think we have a workaround available today.We will evaluate if we can lower the minimum to 5MB to support this use-case but in the meantime have you reached out to AWS to see if they can support larger payload with Lambda?
Jacques
-
Hi Jacques,
We have reached out to AWS but the payload quota of 6MB cannot be changed, unfortunately. https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html
When using API gateway with Lambda, the payload is base64 encoded, increasing the size by ~33%. This further lowers the limit from 6 to ~4.5MB. https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings.html
This is the standard AWS approach to publish an API with Lambda functions. Would it be possible to lower the minimum to 4MB instead of 5MB?
Thanks for your consideration,
Team Apideck
Please sign in to leave a comment.
Comments
6 comments