App fails to deploy with no error logs

[Zak]

Hi,

I’ve just started trying to use fly.io but so far it’s been quite buggy. Right now the problem I’m having is that my app isn’t deploying - if I try deploy from my computer CLI, it times out after 5-10 minutes:

➜  go git:(main) fly deploy --config fly.backend.toml
==> Verifying app config
Validating fly.backend.toml
✓ Configuration is valid
--> Verified app config
WARN POSTHOG_PERSONAL_API_KEY may be a potentially sensitive environment variable. Consider setting it as a secret, and removing it from the [env] section: https://fly.io/docs/apps/secrets/

==> Building image
==> Building image
Error: failed to fetch an image or build from source: error building: deadline_exceeded: context deadline exceeded

And whenever it deploys from the UI, it times out and gives no logs, errors or context:

This is my current fly.backend.toml (located at go/fly.backend.toml). The app is located at go/services/backend/Dockerfile:

app = 'monorepo-zucgtw'
primary_region = 'iad'

[build]
dockerfile = "services/backend/Dockerfile"
context = "."

[env]
HTTP_PORT = '80'
ENV_FLAG = 'prod'
POSTHOG_PERSONAL_API_KEY = 'phc_d5SD2NEXlxYpGf7lB12ItWadB0rqmGm3hPotqXr5lop'

[http_service]
internal_port = 80
force_https = true
auto_stop_machines = 'stop'
auto_start_machines = true
min_machines_running = 0
processes = ['app']

[[http_service.checks]]
grace_period = "10s"
interval = "15s"
method = "GET"
timeout = "5s"
path = "/alive"

[[vm]]
memory = '256mb'
cpu_kind = 'shared'
cpus = 1

If anyone could offer any assistance in seeing the failure logs or fixing the problem, that’d be great - this is my first time using fly.io. Thanks

Note that the posthog key is not a secret, so I haven’t exposed anything I didn’t intend to.

[JeanJ]

Also experiencing similar issues. Our backend (LAX region) is completely down. There’s no way to create new machines.

error creating a new machine: failed to launch VM: request returned non-2xx status: 504: upstream request timeout

[johanaa]

Have the same problem in ARN region. First deployment made it to fly and failed, (the instance says its stopped but i can still reach and use the server). Now i am not getting past the “waiting for depot builder…” on flyctl deploy

[harold1]

Also down in AMS - cannot deploy either using Depot or remote builders.

[ishands]

I am also facing the same issue. I am in BOM.

Trying to bypass depot with FLY_REMOTE_BUILDER_REGION=sin fly deploy --depot=false

fails with the following error:

WARN Failed to start remote builder heartbeat: failed to launch VM: request returned non-2xx status: 504: upstream request timeout

==> Building image
WARN Failed to start remote builder heartbeat: Validation failed: Name has already been taken

Error: failed to fetch an image or build from source: Validation failed: Name has already been taken (Request ID: 01KBMQK6YVC4X3AYQNRTJSJC98-bom) (Trace ID: 33dd9cfc560a5e8725853d0f724159bc

[hilja]

Same, stuck at “Waiting for depot builder…” at ams.

[vitorcrovador]

Same here.

[greg]

They’ve now opened an issue to investigate this

Issues with App Deploys + Machine API

[Zak]

I guess I’m unlucky this is my first time using fly.io and it’s down

[hilja]

Works now, woop woop

[Zak]

Ok - I’m not sure if it’s died again or if my issue is different (likely since this is my first time).

It appears any build triggered by a webhook from GitHub fails to build. Doing fly deploy locally from the same location as I have set in Settings -> Current Working Directory deploys the application fine.

It’s annoying because there’s no logs for me to look at, nor reason it tells me why it’s failed to build (are other parts of the platform designed this badly?)…

[lubien]

Hi @Zak apologies for the lack of logs. Can you email me your deployment URL so I can take a look? lubien[at]fly.io

[Zak]

Sent

[lubien]

@Zak Email received. Here’s the log:

Config file go/fly.backend.toml does not exist

Since your CWD is already go/ your can change your .toml file to just fly.backend.toml

You’re correct, the settings should have been clearer too and should have validated this before you even was able to save.

And there’s the missing log from UI.

I’ll take a look at both now, thanks. Feel free to add more feedback in our forum too, preferrably in a new topic since this one is related to something else.

[Zak]

Thanks for the response.

If I change it to just fly.backend.toml it says the file doesn’t exist (both if i try run the deployment with that value previously set, and if I try update it in the settings). The file was also in the location specified at the time of that error.

Current repo + settings:

I can only trigger the deployment again once I revert the the Config Path back to go/fly.backend.toml, at which point it fails like previously seen (assume still the same error, have emailed the deployment link again just in case it’s changed).

[lubien]

Hey @Zak Im literally shipping a UX change for this that will also fix the validation

Now our form will properly prepend the CWD to the fly.toml path. Can you give it a try?