# Zero downtime deployment

**URL:** <https://community.fly.io/t/zero-downtime-deployment/7971>\
**Category:** General\
**Created:** [October 18, 2022, 6:48pm UTC](https://community.fly.io/t/zero-downtime-deployment/7971 "2022-10-18T18:48:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![stayallive](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/stayallive/32/15016_2.png) [@stayallive](https://community.fly.io/u/stayallive)\
**Post date:** [October 18, 2022, 6:48pm UTC](https://community.fly.io/t/zero-downtime-deployment/7971/1 "2022-10-18T18:48:06Z")

</div>

When reading these threads it seems there were some issues with deployment causing intermittent donwtime:

> [@how do fly.io deploys work?](https://community.fly.io/t/how-do-fly-io-deploys-work/3378):
>
> I’m running into a problem where usually I deploy my app, I have a little bit of downtime. This seems like it must be because I’m configuring my healthchecks wrong, but I couldn’t find an explanation of how [fly.io](http://fly.io) deploys move traffic between the old version and the new version. Is there an explanation somewhere I can read so that I can understand exactly what’s happening?

> [@Is downtime expected post app deploy?](https://community.fly.io/t/is-downtime-expected-post-app-deploy/3884):
>
> When I deploy a simple Go app (based on the example go app), I see the health checks pass. But when I try and visit the browser, the browser hangs for a long time. After a few minutes, the request works. This happens every deployment. Once things start to work, we are good. But it seems that the new instance is not up and accepting traffic before the old? Sometimes the first request is valid and is stood up in IRA. But the second request hangs while it decides I should be in ORD. kill\_signa…

However trying it today it seems no better ☹ Am I missing something?

I’ve used this guide to get a simple Go app: [Run a Go App · Fly Docs](https://fly.io/docs/languages-and-frameworks/golang/)

When deploying (using default, `rolling` or `canary` strategy) I get spotty service for about 30-60s (depends on the strategy how bad and long it is).

Is zero downtime deployment possible currently?

It’s pretty hard deploying anything production worthy if downtime is expected on every deployment.

I noticed this also for another application where I have very aggresive health checks and large grace periods but it always results in downtime. I assumed that was because I configured it wrong so testing with the Go app seemed like a good way to ensure it was Fly and not me.

This is a log of me sending a request every second:

```auto
HTTP/2 200
date: Tue, 18 Oct 2022 18:34:55 GMT
content-length: 137
content-type: text/html; charset=utf-8
server: Fly/af1875645 (2022-10-18)
via: 2 fly.io
fly-request-id: 01GFP614XR2P7Z12EAES7HHS07-ams

curl: (28) Operation timed out after 803 milliseconds with 0 bytes received
curl: (28) Operation timed out after 803 milliseconds with 0 bytes received
curl: (28) Operation timed out after 806 milliseconds with 0 bytes received
curl: (28) Operation timed out after 800 milliseconds with 0 bytes received
curl: (28) Operation timed out after 804 milliseconds with 0 bytes received
curl: (28) Operation timed out after 804 milliseconds with 0 bytes received
curl: (28) Operation timed out after 806 milliseconds with 0 bytes received
curl: (28) Operation timed out after 803 milliseconds with 0 bytes received
curl: (28) Operation timed out after 802 milliseconds with 0 bytes received
curl: (28) Operation timed out after 800 milliseconds with 0 bytes received
curl: (28) Operation timed out after 805 milliseconds with 0 bytes received
curl: (28) Operation timed out after 802 milliseconds with 0 bytes received
curl: (28) Operation timed out after 806 milliseconds with 0 bytes received
curl: (28) Operation timed out after 802 milliseconds with 0 bytes received
curl: (28) Operation timed out after 803 milliseconds with 0 bytes received
curl: (28) Operation timed out after 800 milliseconds with 0 bytes received

HTTP/2 200
date: Tue, 18 Oct 2022 18:35:26 GMT
content-length: 138
content-type: text/html; charset=utf-8
server: Fly/af1875645 (2022-10-18)
via: 2 fly.io
fly-request-id: 01GFP622J88D0VZ8HE2JWTWHZ3-ams

```

There are ~30 seconds between the request on the previous version and the next one, however it takes until `18:35:55` (so a full minute) before service is stable again with intermittent timeouts.

---

<div class="post-metadata">

**Author:** ![mobilevet](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mobilevet/32/5728_2.png) [@mobilevet](https://community.fly.io/u/mobilevet)\
**Post date:** [October 18, 2022, 7:10pm UTC](https://community.fly.io/t/zero-downtime-deployment/7971/2 "2022-10-18T19:10:55Z")

</div>

I was just about to post something similar. I see the original box running, but as soon as I start a deployment my calls fail. They don’t succeed again until well after the new box is up and running.

 ![Screen Shot 2022-10-18 at 3.08.48 PM](https://global.discourse-cdn.com/fly1/original/3X/d/1/d10cbfeaf97e88c1d338fcda9a042c9fa122b9ac.png)

---

<div class="post-metadata">

**Author:** ![ignoramous](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@ignoramous](https://community.fly.io/u/ignoramous)\
**Post date:** [October 18, 2022, 10:39pm UTC](https://community.fly.io/t/zero-downtime-deployment/7971/3 "2022-10-18T22:39:09Z")

</div>

> [@stayallive](#):
>
> There are ~30 seconds between the request on the previous version and the next one, however it takes until `18:35:55` (so a full minute) before service is stable again with intermittent timeouts.

May be this post has some _more_ insights (even if it could be outdated): [High availability on Fly.io - #3 by kurt](https://community.fly.io/t/high-availability-on-fly-io/3448/3)

> [@stayallive](#):
>
> Is zero downtime deployment possible currently?

Only if one is running _multi-region_ Fly apps, does `fly deploy --strategy rolling <other-args>` prevent outages from client PoV ([ref](https://community.fly.io/t/liveview-app-unavailable-for-2-minutes-after-deployment/3753/14))?

I am compelled to point out that, [Machine app](https://community.fly.io/t/were-launching-fly-machines-today/5305) instances get [fixed `6pn` (private) IPs](https://community.fly.io/t/leftover-dns-records/6319/4) across deploys (unlike regular Fly apps unless they [‘anchor’ themselves to a host](https://community.fly.io/t/can-an-instance-have-a-persistent-network-identity/5045/7)), and so, these are likely to not be subject to deploy-induced downtime. If you can, you should move to Machines (after evaluating it for [a round or two](https://community.fly.io/t/fly-instance-id-header-alternative-for-websockets/7536/5)).

---

<div class="post-metadata">

**Author:** ![stayallive](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/stayallive/32/15016_2.png) [@stayallive](https://community.fly.io/u/stayallive)\
**Post date:** [October 19, 2022, 10:07am UTC](https://community.fly.io/t/zero-downtime-deployment/7971/4 "2022-10-19T10:07:20Z")

</div>

Interesting, thanks for your detailed reply, it seems like v1 apps can’t guarantee any high availabilty / zero downtime because the backhaul updates too slow to make this happen if I’m reading between the lines a little.

I must admit that Machines sound like they are supposed to be v2 apps at some point and could be treated like that although I have a hard time figuring out how to deploy the example as a v2/machines app instead of a v1/nomad app.

Maybe I’m just not as smart as I’m thinking or I’m missing some doc page but I would have expected a way to set I want to use machines in my fly.toml but haven’t found any docs explaining this well yet and all examples seem to be v1 examples still. Reading the machines doc ([Machines · Fly Docs](https://fly.io/docs/reference/machines/)) gives me more questions than answers to be honest because it mostly tells you how to do things manually.

Is there some doc I’m missing?

---

<div class="post-metadata">

**Author:** ![mobilevet](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mobilevet/32/5728_2.png) [@mobilevet](https://community.fly.io/u/mobilevet)\
**Post date:** [October 19, 2022, 12:55pm UTC](https://community.fly.io/t/zero-downtime-deployment/7971/5 "2022-10-19T12:55:41Z")

</div>

@stayallive This morning when I pushed some new code and then ran `flyctl deploy` the rolling update worked as I expected in that the first instance stayed up until the second was ready.

Interestingly, changing an ENV did NOT work this way (which is what I was doing yesterday when I first noticed this. In that case, the old instance is terminated immediately and there is downtime before the second is available to handle incoming requests.

---

<div class="post-metadata">

**Author:** ![ignoramous](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@ignoramous](https://community.fly.io/u/ignoramous)\
**Post date:** [October 19, 2022, 1:58pm UTC](https://community.fly.io/t/zero-downtime-deployment/7971/6 "2022-10-19T13:58:50Z")

</div>

> [@mobilevet](#):
>
> Interestingly, changing an ENV did NOT work this way

You mean `fly secrets set ...`? That’s interesting.

* * *

> [@stayallive](#):
>
> it seems like v1 apps can’t guarantee any high availabilty / zero downtime because the backhaul updates too slow to make this happen if I’m reading between the lines a little.

Wait for Fly engs to confirm this one way or the other. We need zero down-time deploys ourselves and so I use Machines but I haven’t yet moved _all_ our prod traffic to Fly. To my untrained eye, Machines might provide this, but I haven’t yet put ours under any duress, yet.

> [@stayallive](#):
>
> Maybe I’m just not as smart as I’m thinking or I’m missing some doc page

Join me in writing a Haiku, won’t you: [Write us a haiku and we'll put it on the login screen - #13 by ignoramous](https://community.fly.io/t/write-us-a-haiku-and-well-put-it-on-the-login-screen/5924/13) (:

> [@stayallive](#):
>
> I would have expected a way to set I want to use machines in my fly.toml but haven’t found any docs explaining this well yet and all examples seem to be v1 examples still.

```bash
# create a machines app
fly apps create <app-name> --machines

# then run it
fly m run
  . \
  # services.port and services.internal_port with handlers go here
  -p <443:8080/tcp:tls?> \
  --dockerfile </path/to/dockerfile?>
  --memory <256?> \
  --region <iad?> \
  --name <uniq-name?>
  -a <app-name?>

```

After a Machine is `run` once, you can `clone` it [to other regions](https://community.fly.io/t/flyctl-machine-clone-is-defunct/7265/3):

```bash
# list machines
fly m list -a <app-name>

# note down alloc-id from above, and clone like so:
fly m clone <machine-alloc-id> --config fly.toml --name <uniq-name> --region <aws>

```

You can also use `fly deploy` (once the app’s created) to deploy to _all_ machines (but it [is rough](https://community.fly.io/t/flyctl-deploy-does-not-honour-all-entries-in-config-for-machine-apps/7269) at its edges). I fixed a few issues with `fly deploy` and Machines, but [I maintain a local-fork of it](https://community.fly.io/t/fly-deploy-for-machines-app-memory-is-reset-after-each-deploy/7689/4).

See also: [Ephemeral fly machine? - #2 by ignoramous](https://community.fly.io/t/ephemeral-fly-machine/6965/2)

Ref [this Fly blog post](https://archive.is/FhqLv#prepare-the-fly-machines-app).

---

<div class="post-metadata">

**Author:** ![mobilevet](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mobilevet/32/5728_2.png) [@mobilevet](https://community.fly.io/u/mobilevet)\
**Post date:** [October 19, 2022, 5:45pm UTC](https://community.fly.io/t/zero-downtime-deployment/7971/7 "2022-10-19T17:45:47Z")

</div>

> [@ignoramous](#):
>
> You mean `fly secrets set ...`? That’s interesting.

Yep.