Unable to deploy Elixir app, no feedback in output or logs

[jerome]

@OldhamMade This is very odd. We’re up as far as I can tell and apps are running.

What’s the old and new apps’ names? I’ll take a look.

[OldhamMade]

Info shared privately.

[jerome]

Quick postmortem: This was an issue with any apps with a statics section. We had misconfigured servers (whole regions) where statics could not be deployed.

This has now been resolved.

Our usual fallbacks didn’t work because there were no servers available to serve statics.

[jsierles]

Thanks! Might it be useful to declare certain features as optional during a deployment, so we don’t end up in this catch-22 situation? The statics error only showed up randomly, by the way, not on every deployment.

[OldhamMade]

Thanks.

It would be great if we could get more log output to help with these things. Also, implementation of these options would be super useful. All 3 are available with Heroku, for example.

[jerome]

@jsierles

Thanks! Might it be useful to declare certain features as optional during a deployment, so we don’t end up in this catch-22 situation? The statics error only showed up randomly, by the way, not on every deployment.

Maybe, yes.

This actually started happening on a few servers starting on June 2nd. It got worse more recently and a change was made last night that effectively removed the configuration from most of our fleet.

@OldhamMade

It would be great if we could get more log output to help with these things.

Apps without a release command would’ve seen a more significant (but not helpful for you) error. We need to show better error logs for release commands. Adding that to our backlog.

@OldhamMade

Also, implementation of these options would be super useful. All 3 are available with Heroku, for example.

These are good ideas!

@OldhamMade

• A toggle for “maintenance mode” to display a static HTML file while the toggle is active. I could switch that on and provide a notice to our users.

Absolutely. This is definitely something on the short-medium term roadmap. I might be able to whip that up fairly quickly in the coming weeks.

@OldhamMade

• A “failover” option when an app is “dead” or non-responsive, to either show a static HTML page or proxy to another site. Again, I could provide a notice to our users.

This one is a bit harder, but it could be part of a redesign of the “routing” layer we’ve been thinking about.

@OldhamMade

• A manual rollback option. Something like flyctl deploy --rollback v28 , or being able to click a Rollback to here button in the “Activity” dashboard screen next to a “green” build. Being able to revert to a previously running state without a deploy would really help when deploys aren’t working as expected.

Yes, we need to add that. We’ve been thinking about it for a while.

[OldhamMade]

The “maintenance mode” toggle is a good catch-all for a lot of these issues, if it can be implemented quickly. Could be as “simple” as a toggle button and <textarea/> in the “Settings” screen for an app.

Heroku do this by proxying through to another URL set as an environment variable, which would also be adequate – in the case where a URL isn’t set, they show their own Heroku-branded page. Fly could do it this way too, if that is an easier option.

As long as the toggle can be triggered from both the dashboard and flyctl; there have been occasions where throwing that toggle from a mobile phone has been a bit of a “life-saver”!