Best plan/structure for a multi tenant solution

Questions / Help
1

Hi everyone.

I do have a single image that will be deployed by diverses fly.toml files (one per tenant). Every tenant app will use a postgres database.

So what is the best plan and/or suggestions for this scenario?

Thank you all!

2

If each of your tenants need their own fly.toml configs, you either have a choice to deploy them as separate apps, or deploy them as raw “Machines” (1, 2) – as opposed to using Apps v1 or Apps v2 – with their own fly.toml configs. Note that Fly apps (and Machines) in the same org can discover and talk to each other over 6pn and so, you may want to carve out one 6pn network namespace per app (3) or, per cluster of Machines belonging to one tenant, if that’s even possible.

Also, billing might get tricky as you’d have to keep track of resources consumed by your tenants yourself (at least until Fly complete their billing API implementation, if they haven’t already; 4).

All that said, there are others (5) doing multi-tenant on Fly (I don’t), may be they’ll chime in.

1 Like
3

Do you have more you can share about your use-case? I think that will be quite instructive as to how you approach this. For example, fly.toml is a configuration file that does not necessarily have a one-to-one relationship to a Fly app: in my own multi-environment setup, I have a single fly.toml which does not contain an app identifier and then I run it against different apps with any divergence in Environment configuration handled via secrets. Maintaining different configurations for different applications is certainly doable, but it will introduce challenges – if you can have consistent configuration across all apps, that’ll make management much easier.

If your goal is, for example, to provide a highly secure dedicated instance of some software for each of your clients with the ability to scale for each client independently, there will be different ideal approaches when compared to a scenario in which you are pursuing this approach because the application you’re deploying is only available in a single-tenant setup.

For example, if it’s out of necessity because the software is single tenancy then you could probably run just one database server and give each client their own database on that server, whereas if it’s for security and clients expect their own database server, you’d be forced to run app + database for each client.

2 Likes
4

Thank you for the valuable info!.

Right now the customer is not giving feedback about requirements for this part.

The goal is:
-Provide a multi tenant app under Flyio infrastructure
-It should be easy to mantain in every phase(building, deploy, keeping track of health, etc.)
-If I can get it done with just one toml file vs multiple, the better. The same for apps, vms. One database server vs multiple… I would go for one database server.

You get the idea.

I just want to sell my service for many clients but using the same image so they all get the same features when I deploy, but at the same time that I can bring a fast running application.

(Can you share how do you implement one fly.toml file and many secrets variables for tenancy? how do you map users to the corresponding tenant database when login the app? )

Thank you again!

5

The answer to this depends mostly on: what are you trying to solve for by splitting customers up into their own VMs? Will they have direct access to these VMs, and be able to execute custom code?

1 Like
6

Thank you.

My goal is provide the same application to many customers.

Which plan or structure do you suggest?

7

When you’re interacting with Fly you can provide an app name explicitly for each command, e.g:

fly deploy --app example-app-customer-1

In practice, you could do something like:

  1. Create a generic fly.toml which does not include the app parameter
  2. Run all fly commands with the addition of the app flag, e.g: fly deploy --app example-app-customer-1
    a. note: sometimes you may need to use the FLY_APP Environment Variable instead

The caveat is that some Fly commands try to be helpful and write back to the fly.toml: if you’re using git then you can just discard the changes automatically but if you’re not using git then you’ll need to manually remove app from fly.toml after each command.

I use Taskfile for managing my task definitions, so the following is an example of the Taskfile configuration I use for managing this workflow:

version: "3"

silent: true

dotenv:
  - ".env.task"
  - ".env.example.task"

env:
  FLY_APP: "my-example-app-{{.USER}}"
  FLY_APP_NAME: "my-example-app-{{.USER}}"

  deploy:
    desc: Deploy your copy of the application to Fly
    cmds:
      - task: launch-fly-app
      - task: deploy-fly-app
      - task: open-fly-app

  launch-fly-app:
    cmds:
      - fly launch --copy-config --name $FLY_APP
      - git checkout fly.toml
    status:
      - test -f fly.toml
      - fly info
  deploy-fly-app:
    cmds:
      - fly deploy --env BUILD_ID=$(date +%s)
    preconditions:
      - sh: fly info
        msg: "Fly app cannot be deployed without first being launched"
  print-fly-name:
    cmds:
      - echo $FLY_APP
  open-fly-app:
    cmds:
      - fly open

I’m using the account name of the person running the command ({{.USER}}) to dynamically generate the app name but you could pass it in explicitly if you like. The preconditions are helpful to determine if a new application needs to be created or if it’s just a deploy.

Regarding Environment Variables and secrets: Secrets are “just” Environment Variables that are managed securely by Fly and passed into your application at deploy time rather than from the fly.toml so you can treat the Environment Variables in your fly.toml as sensible defaults (for all clients) and then on a per-client basis you can set the value as a secret to overwrite the default.

For example, if your application requires a database to interact with that is available as the DATABASE_URL Environment Variable, you could deploy a “default” database (which is readonly) and then put that value into your fly.toml. Then, for each client, when you launch their Fly application, you add a secret containing their actual database URL, e.g:

[env]
DATABASE_URL = "postgres://databases.example.com:5432/default"

Then when you launch an application for a new client, you’d do something like:

fly secrets set DATABASE_URL=postgres://username:password@databases.example.com:5432/client-database

So the only variance between each of your clients is their secrets, they’re all being deployed from the same fly.toml.

Another consideration is that while Fly can build your application for you, and does do that by default, your use-case is one where that behaviour is probably not ideal. Instead, you can specify the exact image version in your fly.toml and handle the build process separately from your deployment.

Regarding the database server, I’d recommend against Fly for that component. Instead, go with a managed database provider like Google Cloud or AWS. The only knowledge each Fly app should have of its database is through the Environment Variable containing the URL for it. If you experience any meaningful amount of volume (e.g: you’ve got hundreds of Fly apps running, all with their own database on a single database server) you’ll need to be able to evolve/scale your approach, and that’s where managed database providers really excel.

Anyway, that’s a quick summary!

3 Likes
8

Wao! thank you brother for so much detailed guide.

This I will follow and let you know how it goes for me. But I feel it’s a good one and it can be improved any way.

Keep the hard work and God bless you.