New Instance Automatically Restarted and Lost All Contents

hello · August 31, 2022, 8:15am

I deployed my Ghost v5 to fly.io today.

All works smoothly but after a hour, suddenly all contents are lost (happened twice).

Contents include: admin account (it guided me to setup the admin account again), posts (published posts no longer exists)

My question: is there any mechanism that Fly.io restart / clear / reset volumes or instance? (for free plan?)

NODE_ENVIRONMENT is dev, the ghost uses sqlite to save data in this way

my fly.toml

$ cat fly.toml 
# fly.toml file generated for myblog on 2022-08-31T11:39:30+08:00

app = "myblog"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  image = "ghost:latest"

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[mounts]
  destination = "/var/lib/ghost/content"
  source = "ghost_data"

[[services]]
  http_checks = []
  internal_port = 2368
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

$ fly checks list
Health Checks for myblog
  NAME                             | STATUS  | ALLOCATION | REGION | TYPE | LAST UPDATED | OUTPUT                                     
-----------------------------------*---------*------------*--------*------*--------------*--------------------------------------------
  33f5840f97ca75deac29e8c58906e54e | passing | 5b566303   | hkg    | TCP  | 21m36s ago   | TCP connect {This is the IPV4}:2368: Success[✓]  
                                   |         |            |        |      |              |                                            
                                   |         |            |        |      |              |

flyctl logs 
2022-08-31T07:41:14Z app[5b566303] hkg [info][2022-08-31 07:41:14] INFO "GET /members/api/member/" 204 32ms
2022-08-31T07:41:14Z app[5b566303] hkg [info][2022-08-31 07:41:14] INFO "GET /ghost/api/content/settings/?key=61110f2254b42fe57a8813fdb2&limit=all" 200 208ms
2022-08-31T07:41:14Z app[5b566303] hkg [info][2022-08-31 07:41:14] INFO "GET /ghost/api/content/newsletters/?key=61110f2254b42fe57a8813fdb2&limit=all" 200 28ms
2022-08-31T07:41:14Z app[5b566303] hkg [info][2022-08-31 07:41:14] INFO "GET /ghost/api/content/tiers/?key=61110f2254b42fe57a8813fdb2&limit=all&include=monthly_price,yearly_price,benefits" 200 33ms
2022-08-31T07:41:18Z app[5b566303] hkg [info][2022-08-31 07:41:18] ERROR "PUT /ghost/api/admin/custom_theme_settings/" 403 205ms
2022-08-31T07:41:18Z app[5b566303] hkg [info]
2022-08-31T07:41:18Z app[5b566303] hkg [info]Authorization failed
2022-08-31T07:41:18Z app[5b566303] hkg [info]"Unable to determine the authenticated user or integration. Check that cookies are being passed through if using session authentication."
2022-08-31T07:41:18Z app[5b566303] hkg [info]Error ID:
2022-08-31T07:41:18Z app[5b566303] hkg [info]    4c9d7600-2900-11ed-88a8-7d633d9579d7
2022-08-31T07:41:18Z app[5b566303] hkg [info]----------------------------------------
2022-08-31T07:41:18Z app[5b566303] hkg [info]NoPermissionError: Authorization failed
2022-08-31T07:41:18Z app[5b566303] hkg [info]    at authorizeAdminApi (/var/lib/ghost/versions/5.12.0/core/server/services/auth/authorize.js:33:25)
2022-08-31T07:41:18Z app[5b566303] hkg [info]    at Layer.handle [as handle_request] (/var/lib/ghost/versions/5.12.0/node_modules/express/lib/router/layer.js:95:5)
2022-08-31T07:41:18Z app[5b566303] hkg [info]    at next (/var/lib/ghost/versions/5.12.0/node_modules/express/lib/router/route.js:144:13)
2022-08-31T07:41:18Z app[5b566303] hkg [info]    at authenticate (/var/lib/ghost/versions/5.12.0/core/server/services/auth/session/middleware.js:28:13)
2022-08-31T07:41:18Z app[5b566303] hkg [info]    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-08-31T07:41:18Z app[5b566303] hkg [info]

lubien · August 31, 2022, 1:48pm

There’s none.

In fact, I run my blog on ghost with SQLite here but I use NODE_ENV=production, is it possible your development setup has some sort of wipe-out mechanism?

hello · August 31, 2022, 3:02pm

If fly.io does not have such limit, I believe it is related to the configurations of Ghost or some health check?

I tested that, whenever I restart my instance. The SQLite data wiped out and I also found server logs of initializing the database (creating the tables).

I am not sure which part I missed. Could you suggest me the correct image of Ghost & fly.toml?

Thank you!

lubien · August 31, 2022, 3:42pm

Here’s how I did mine:

Go to a folder, run this

$ fly apps create

Follow the instructions above. Remeber the app name.

Create a fly.toml like this:

[build]
image = "ghost"

[mount]
source = "ghost_content"
destination = "/var/lib/ghost/content"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
  url="https://YOURAPPNAME.fly.dev"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
http_checks   = []
internal_port = 2368
processes     = [ "app" ]
protocol      = "tcp"
script_checks = []

  [services.concurrency]
  hard_limit = 500
  soft_limit = 250
  type       = "requests"

  [[services.ports]]
  handlers = [ "http" ]
  port     = 80

  [[services.ports]]
  handlers = [ "tls", "http" ]
  port     = 443

  [[services.tcp_checks]]
  grace_period  = "1s"
  interval      = "15s"
  restart_limit = 6
  timeout       = "2s"

Create a volume then deploy:

$ fly volumes create ghost_content
$ fly deploy

As you can see I directly use the official ghost image rather than a Dockerfile so I did not customize it a lot.

Hopefully this can help you.

hello · September 1, 2022, 3:51am

Thank you @lubien. I tried the latest official ghost image with

[build]
image = "ghost"

There is a breaking change in the latest version that Ghost is no longer support SQLite(although i see there is a trick way to use it).

I am not sure if fly.io provides us to deploy a free MySQL instance?

Thanks again!

lubien · September 1, 2022, 11:00am

Thanks for letting me know, that’s a to me.

You can either run a MySQL image on a shared-cpu 256MB and it would be free (assuming you have at most 3 of those VMs on all apps) or opt for this amazing MySQL service: PlanetScale Pricing

Curiositry · September 27, 2022, 1:36am

@lubien Is it possible to run an up-to-date MySQL install with the 256MB of RAM the free tier offers? I have tweaked MySQL 8 for minimal memory usage in every way I know of, and still get the dreaded out of memory error on the free tier.

It would be fantastic if there was a way to get MySQL 8 running on the free tier, so that people could spin up a low-traffic Wordpress or Ghost blog!

patfly · September 27, 2022, 1:24pm

+1 on being able to run MySQL8 on the free tier. I ran into the same issue trying to get Ghost 5+ hosted in fly.io
PlanetScale not an option for Ghost as it uses foreign key constraints, a feature PlanetScale does not support.

fideloper-fly · September 27, 2022, 2:49pm

Hi!

So MySQL8 isn’t going to run on a low-ram environment - it OOM’s during it’s initialization phase (first run of MySQL). I’m not sure if it would also throw OOM errors during normal operation (haven’t gotten that far!)

However one thing I want to experiment with (and you can as well!) is setting up swap in the VM.

This would best be done in the ENTRYPOINT script.

It looks like you can add *.sh scripts to /docker-entrypoint-initdb.d and those will run automatically. However:

Those run after other initialization, so I’m not sure we’d even reach these scripts for mysql 8 with 256m ram
Those only run on the first-run, when adding to the mysql data directory. Subsequent restarts (deploys) would not run that script, so SWAP wouldn’t always get configured.

This means we need to setup a custom ENTRYPOINT script that adds some swap, and THEN runs the MySQL default entrypoint stuff (being sure to pass it any commands/flags, etc).

Doing this means we’d need our own Dockerfile that uses mysql8 as the base image.

The custom entrypoint could setup swap and then call the MySQL entrypoint (/usr/local/bin/docker-entrypoint.sh)

Haven’t tested this, but the custom entrypoint script could look something like this:

#!/bin/bash


fallocate -l 512M /swapfile
chmod 0600 /swapfile
mkswap /swapfile
echo 10 > /proc/sys/vm/swappiness
swapon /swapfile

`/usr/local/bin/docker-entrypoint.sh "$@"

Random reference:

Swap reference (we do it in the Rails launcher): flyctl/fly.rake at master · superfly/flyctl · GitHub

Mysql 8 entrypoint: mysql/docker-entrypoint.sh at master · docker-library/mysql · GitHub

fideloper-fly · September 27, 2022, 7:40pm

OK, I got some help from Will on the Fly team, and without swap, we got this working.

(Sidenote: I still like the idea of enabling swap, but I haven’t played with that personally just yet).

There’s 2 things to care about:

The init process needs to not OOM
Regular usage shouldn’t take up too much RAM.

What we did to get MySQL 8 running in the free tier is:

Disable the performance schema (helps with point 1 above)
Reduced the Innodb buffer pool size from the default 128M down to 64M (GENERALLY you set this to be some large percentage of available ram, but we reduced it here. YMMV and maybe you don’t even need to. Depends on the database load).

I haven’t tried out setting swap yet, I still think that’s a good idea.

Here’s what the fly.toml file looks like, which is based on following this guide + adjusting it a bit (and obviously not scaling the server up to 2G ram):

app = "some-tiny-mysql8-app"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  image = "mysql:8"

[env]
  MYSQL_DATABASE = "some_db"
  MYSQL_USER = "some_user"

[experimental]
  cmd = [
    "--default-authentication-plugin", 
    "mysql_native_password", 
    "--datadir", "/data/mysql", 
    "--performance-schema=OFF", 
    "--innodb-buffer-pool-size", "64M"
]

[mounts]
  destination = "/data"
  source = "mymysqldata"

The key points there are the last 2 flags used (altho pay attention to the differing “array” syntax used):

"--performance-schema=OFF"
"--innodb-buffer-pool-size", "64M"

Curiositry · September 28, 2022, 4:11am

@fideloper-fly That did it. THANK YOU Chris! You really made my day.

I had a different syntax for the performance schema, which is probably what was wrong. I haven’t tested it with any traffic yet, but it seems to be happy idling along with ~5MB of free memory

patfly · September 28, 2022, 9:56am

@Curiositry Exciting! Any open repo with your setup?

fideloper-fly · September 28, 2022, 12:49pm

If you’re just looking for the MySQL setup, I updated the docs here to mention the settings to stay on the free tier: Use a MySQL Database · Fly Docs

Curiositry · September 28, 2022, 5:54pm

Tutorial and one-step script on the way

Curiositry · September 28, 2022, 6:57pm

Here’s my script. Let me know if you run into any issues!

patfly · September 28, 2022, 7:05pm

Thanks @Curiositry and @fideloper-fly and Will!

hello · September 29, 2022, 2:43am

Thank you so much!!! will definitely try again!!!

hello · October 7, 2022, 7:35am

Just tested the copy&paste script, MySQL & Ghost Blog are successfully installed.

In my fly environment, I need to replace the following line
sed -i 's/internal_port = 8080/internal_port = 2368/g' fly.toml
into the:
sed -i '' 's/internal_port = 8080/internal_port = 2368/g' fly.toml

igf · October 10, 2022, 7:35pm

if you still wish to run SQLite in Ghost 5,
add to fly.toml

[env]
  database__client = "sqlite3"
  database__connection__filename = "/var/lib/ghost/content/data/ghost.db"

anatoly · September 7, 2023, 9:38pm

Currently, ghost only supports MySQL 8 in production. Running ghost with SQLite is possible in development mode. If you intentionally want to run ghost in dev mode, set env NODE_ENV=development for your ghost application. More on ghost’s supported databases: Supported databases in production for self-hosting Ghost - Ghost Developers