Running Discourse on Fly

Discourse is a popular forum platform (it also powers community.fly.io) and runs on Postgres, Redis, and Sidekiq — all of which can run on Fly.

We’ll use the built-in Postgres system and fly-apps/redis to handle persistence, volumes for data, and run the Discourse web server and Sidekiq worker on the same instance using the hivemind.

This will be a single-region single-instance deployment — Discourse requires a data volume for uploads, and while we can use S3 or a virtual filesystem that’s not covered in this guide. Attaching a data volume on Fly anchors the instance to the region and physical host that the disk is on.

Let’s create the database first with fly pg create — I’ll calling this database discourse-db and choose my organization and region. Once the database has been created, I see the following message:

Creating postgres cluster discourse-db in organization personal
Postgres cluster discourse-db created
  Username:    postgres
  Password:    babec71dc5d2fc42a04ae47d591b87b56239b232c6bbb2ad
  Hostname:    discourse-db.internal
  Proxy Port:  5432
  PG Port: 5433
Save your credentials in a secure place, you won't be able to see them again!
...

I can now connect to the main Postgres database by forming the credentials into a URL like postgres://postgres:babec71dc5d2fc42a04ae47d591b87b56239b232c6bbb2ad@discourse-db.internal:5432, but we don’t actually want to use the main Postgres DB — we’ll create a Discourse specific DB soon using fly pg attach.

Moving on to Redis, let’s create an app using fly apps create and call it discourse-redis. We also need to create a volume for Redis to persist data in:

flyctl volumes create discourse_redis_disk --region maa -a discourse-redis

We can now create a fly.toml based on the contents of https://github.com/fly-apps/redis/blob/main/fly.toml — and since this isn’t the main app in my working directory I’ll rename it to redis-fly.toml, and update it with the correct app and volume names:

app = "discourse-redis"

[[mounts]]
  destination = "/data"
  source = "discourse_redis_disk"

[build]
  image = "flyio/redis:6.2.6"

We also need to set a Redis password (before the first deployment) with

fly secrets set REDIS_PASSWORD=OHSOSECRET -a discourse-redis

and then deploy specifically from the redis-fly.toml file using

fly deploy -c redis-fly.toml

With our dependencies all set up, we can now get to deploying Discourse itself. Bitnami has a fully packaged Discourse container available, and by default two instances of the same image are run with different CMDs — one for the web server and one to run Sidekiq. These two containers would need access to the same data volume because they operate on the same directory, but this means we can’t run them separately on Fly — if they need access to the same data volume they’ll need to be running on the same instance.

To make this happen we’ll use an idea described on the guide to running multiple processes, and use the hivemind process manager to run both the web server and Sidekiq simultaneously. We’ll extend docker.io/bitnami/discourse:2 a bit with our own Dockerfile.

FROM docker.io/bitnami/discourse:2

RUN curl -L https://github.com/DarthSim/hivemind/releases/download/v1.0.6/hivemind-v1.0.6-linux-amd64.gz -o hivemind.gz \
  && gunzip hivemind.gz \
  && mv hivemind /usr/local/bin

COPY Procfile Procfile
RUN chmod +x /usr/local/bin/hivemind

ENTRYPOINT []
CMD ["/usr/local/bin/hivemind"]

We’ll need to create the Procfile referenced there as well. Since Bitnami uses an ENTRYPOINT to properly bootstrap each script, we’ve cleared in the Dockerfile and we’ll use it in the Procfile instead:

web: /opt/bitnami/scripts/discourse/entrypoint.sh /opt/bitnami/scripts/discourse/run.sh
sidekiq: /opt/bitnami/scripts/discourse/entrypoint.sh /opt/bitnami/scripts/discourse-sidekiq/run.sh

Let’s now create the app with fly apps create — I’ll call it discourse. And we’ll set up a simple fly.toml with the internal port set at 3000 (that’s what this image uses):

app = "discourse"

[[services]]
  internal_port = 3000  
  protocol = "tcp"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

[[mounts]]
  source      = "discourse_data"
  destination = "/bitnami/discourse"

And add the data volume we’re referencing here with

fly volumes create discourse_data --region maa

Let’s now tell Fly to set up an application-specific database on the main DB instance with

fly pg attach --app discourse --postgres-app discourse-db

which should give us

Postgres cluster discourse-db is now attached to discourse
The following secret was added to discourse:
  DATABASE_URL=postgres://discourse_g4k3rxgloy80qznj:800548c36945dace5916a07b9d574a99@discourse-db.internal:5432/discourse?sslmode=disable

That’s everything we need to configure and launch our Discourse app. Let’s set up the following environment variables / secrets that we know are required based on https://github.com/bitnami/bitnami-docker-discourse/blob/master/README.md#user-and-site-configuration

DISCOURSE_HOST=discourse.fly.dev
DISCOURSE_DATABASE_HOST=discourse-db.internal
DISCOURSE_DATABASE_PORT_NUMBER=5432
DISCOURSE_DATABASE_USER=discourse_g4k3rxgloy80qznj
DISCOURSE_DATABASE_PASSWORD=800548c36945dace5916a07b9d574a99
DISCOURSE_DATABASE_NAME=discourse
DISCOURSE_REDIS_HOST=discourse-redis.internal
DISCOURSE_REDIS_PORT_NUMBER=6379
DISCOURSE_REDIS_PASSWORD=OHSOSECRET

I’ll put these in a .env file and set them all at one shot with

cat .env | fly secrets import

We’ll also need to give Discourse a little more RAM than the default 256MB, and since we’re running the web server and Sidekiq together let’s give it a safe 2GB with fly scale memory 2048.

We can then deploy with fly deploy, and watch the logs in another tab with fly logs… and that’s it! Almost. Discourse also needs an SMTP server to do anything useful and some site configuration to look normal, but we have it running now, and can add these options as part of setting up the forum.

6 Likes

What is the likelihood that you can follow up with the typical hosting costs of a setup like this?

2 Likes

How much is it?

1 Like

Thanks, @sudhir.j, for the very helpful tutorial!

I was able to get Discourse running on Fly by following it, with a few changes, since it’s been two years:

… had to be changed to:

fly pg attach discourse-db --app discourse

And the curl was not installed, so the line in the Dockerfile:

RUN curl -L https://github.com/DarthSim/hivemind/releases/download/v1.0.6/hivemind-v1.0.6-linux-amd64.gz -o hivemind.gz \
  && gunzip hivemind.gz \
  && mv hivemind /usr/local/bin

… had to be changed to:

RUN apt-get -y update; apt-get -y install curl && curl -L https://github.com/DarthSim/hivemind/releases/download/v1.0.6/hivemind-v1.0.6-linux-amd64.gz -o hivemind.gz \
  && gunzip hivemind.gz \
  && mv hivemind /usr/local/bin

(Note: you might get a lot of instance refused connection. is your app listening on 0.0.0.0:3000? errors while Discourse is combing it’s hair and slowly compiling assets. Be patient and don’t panic: for me, they went away after a while, and Discourse became accessible.)

I then successfully upgraded to Discourse 3 latest by changing the first line of the Dockerfile from:

FROM docker.io/bitnami/discourse:2

… to:

FROM docker.io/bitnami/discourse:latest

And to my surprise, nothing broke. :joy:

However, I’m still trying to get the configuration working with the Bitnami image, coming from the world of ./setup-discourse… could you elucidate on the advantages and disadvantages of starting with the Bitnami image, rather than the base Discourse docker image?

2 Likes

This is a nice write-up! Thanks for sharing the steps.

1 Like