Uploading 350MB worth of zips with nodejs multer and getting 408 error

JavaScript
1

I have 4 CPUs and 4GB of memory on my .toml, which I assume is the RAM and not disk? I’ve seen many topics similar to this, but the best answer has been just to scale up. Should I scale further? since 180MB seems to upload more reliably though it takes about 5 minutes. Are there other caps I’m unaware of?

2

That is likely to be RAM, yes. A GB size for disk would be specified as a volume. But do add your TOML file to your post if you want an accurate answer.

Uploading 350MB worth of zips with nodejs multer and getting 408 error

408 is a timeout, so adding more RAM isn’t likely to fix this. Are you uploading from your localhost to a Node app on your Fly app, or are you uploading from your Fly app to elsewhere?

On the surface of it, I should think you could do what you’re doing in a smaller RAM allocation. Uploads over HTTP should be chunked regardless of whether you’re sending or receiving in your Fly app. However it would be better if you could supply more detail, so readers can give a more certain answer.

5

This is in my .toml The app name is the only thing not included. I am trying to upload zip files with my fly app (nodejs based) onto the server to run python operations in the backend. The fly app deployed and trying to use the site to upload files. Sorry for the late reply. I usually only have time once a week to work on this.

primary_region = 'dfw'

[build]


[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '4gb'
  cpu_kind = 'shared'
  cpus = 4
6

It sounds like your Fly app is Node based, and you have another server that is largely Python based. You are doing an upload operation from the Node server to the Python server.

You mention you’re using Multer in Node. This JavaScript library transfers files via multipart/form-data.

However I am not convinced you are running out of RAM on the sender/Node end. May I confirm that you’re getting the 408 from the Python server? If so you will need to find why this timeout happened. Is the upload starting at all on the Node end? What time elapses before the 408 is received? Is any part of the upload received on the Python end?

7

Sorry for misleading about the python. It’s only a Node server that spins off a python child process when needed. No extra server handled. So the upload is happening on the Node server and giving a 408. I’ve actually been able to upload 350MB at a coffee shop in a little less than 2 minutes, but those same 350MB times out after 5 minutes at the home network which has 7MB/S

[error][PU02] could not complete HTTP request to instance: user body write aborted: early end, expected 52262358 more bytes

I see that it’s mainly a timeout issue with upload speeds. So far the timeouts have been at 5min and 15 seconds. So trying to find where that setting is at to change to at least 10minutes. I tried

req.socket.setTimeout(10 * 60 * 1000);

But it didn’t do anything.

8

Not sure if it’s due to docker or a different preset

9

OK, so your home connection is slow or unreliable.

Maybe this will help: javascript - Node.js connection reset on file upload with multer - Stack Overflow

10

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.