How Do I Solve the Out of Memory Error Reported by curl?

Symptom

If the --data-binary parameter is used with curl to transfer large files or large amounts of data, all the files and data will be completely loaded to memory before the request is initiated. When the data size exceeds a certain value, an out of memory (OOM) error is reported.

Example: curl -sS -L -w '%{http_code}' -X PUT --data-binary '@/root/example.tar.gz' -H 'Content-Type: application/octet-stream' 'https://example.com/example'
Output:
curl: option --data-binary: out of memory
curl: try 'curl --help' or 'curl --manual' for more information

Possible Cause

In curl 7.73, the bugfix curl: make file2memory use dynbuf was added to use dynamic buffer (dynbuf) instead of custom reallocation code, for unified memory management and improved reliability. In this version, MAX_FILE2MEMORY was introduced and set to 1 GB. When the --data-binary parameter is used for data transmission, if more than 1 GB is transmitted, an OOM error is reported. In curl 8.10.0, the value was set to 16 GB.

MAX_FILE2MEMORY was introduced because there was no upper limit on the size of files or data loaded to the memory. When you use the --data-binary parameter to transfer a large file (for example, a video or compressed package of several GBs), curl will attempt to load the entire file to the memory, which can cause the OOM error. In this case, it is not just that curl crashes. Core system resources may be preempted, affecting other services. The risk is further amplified especially when curl requests are batch processed on the server.

Solution

Preferentially use the -T parameter to transfer large files. (Files are uploaded in streaming mode, which means they are read and sent block by block. The files do not need to be fully loaded to the memory at once.)