If a machine reaches it’s hard limit as configured in the fly.toml, it won’t accept any more connections, and if there’s no machines that aren’t at the hard limit then the proxy will return an error (typically 502 HTTP error I think). If your machine happens to crash because it’s out of memory then sometimes you’ll receive an email from fly. In either case, the recommendation would be to setup your own monitoring that can alert you to when your app is hitting it’s limits.
You can use either the fly CLI to scale up or the machines API, though typically you’ll only use the CLI. There isn’t currently a way to scale up from the dashboard.
You should look at the max number of machines as the limit for how much you’re willing to spend. Are you willing to spend $100/month or $10,000/month? If your app suddenly gets a lot of traffic are you willing to pay for that extra traffic?
With that in mind, another option to consider is to over-provision your machines for expected spikes. So you could provision 25 machines with autostart/autostop meaning that when your traffic is low most of the machines will be stopped and you won’t be paying for them, and then when you have a spike those extra machines will automatically start up and service the traffic which is when you’d start paying for them until the stop again.