[solved] machines API returned an error: "rate limit exceeded" (in proxy logs)

Questions / Help
1

VM ID e7840d9c290d83 / instance 01GMQH5BA9WDSB3B34GCH9XTYJ is no longer starting today when trying to establish a TCP connection to port 22.

The proxy logs show this:

2023-03-01T12:59:18.865 proxy[e7840d9c290d83] cdg [info] Starting machine

2023-03-01T12:59:19.001 proxy[e7840d9c290d83] cdg [info] Starting machine

2023-03-01T12:59:19.118 proxy[e7840d9c290d83] cdg [info] Starting machine

2023-03-01T12:59:19.304 proxy[e7840d9c290d83] cdg [info] Starting machine

2023-03-01T12:59:19.441 proxy[e7840d9c290d83] cdg [info] Starting machine

2023-03-01T12:59:19.693 proxy[e7840d9c290d83] cdg [info] Starting machine

2023-03-01T12:59:19.693 proxy[e7840d9c290d83] cdg [error] machines API returned an error: "rate limit exceeded"

(many times in a row).

I almost never start/stop this machine manually: the proxy wakes it up on incoming TCP connections, and if it’s idle for 600s, the main process exits itself.

Any idea what’s going on / how I can fix it?

2

Tried again 10 minutes later and it started! Sounds like the proxy might be spamming “start machine” requests a bit too often if there’s a lot of connections coming in?

I’ll mark this as solved.

3

We’ve been having some capacity issues and the problem is worse with volumes. A stopped machine does not presently reserve its resources. That means a start is not guaranteed to work on a machine if the server has already allocated all its memory.

We’re working on the start spam :slight_smile:, however the real reason this worked later is because I rescheduled instances that don’t have volumes to other hosts (we have capacity in CDG, but the particular server you’re on was crowded).

1 Like
4

Thanks so much for unblocking me! I didn’t want to have to turn my other computer to finish this :sunglasses:

5

fwiw I am experiencing this again today;

image
image1214Ă—830 243 KB

6

Is this something the team is aware of @jerome?

7

I’m just having this tonight.

8

Hey! Any progress in this?
I’ve had this multiple times and it makes the logs a bit messy :sweat_smile: