Timeline for more & updated documentation

,
1

Hey guys I was wondering when y’all are planning to update documentation on many of the amazing features you have. For example I think it’d be super helpful to have a further in depth guide for A Rails/Sidekiq/Redis stack because of how often it’s used. I know that you have something like this already, but at the end, it just says “Go to redis guide” and " Multi-Process Apps guide", good luck.

There’s some really cool features such as running processes within the same container which is great for rails btw by adding

[processes]
  web = "bundle exec puma -C config/puma.rb"
  worker = "bundle exec sidekiq"

I cannot find anything of this sort in the docs, it’s so cool though!

Anyways, I really want you guys to succeed, what you have is really great! However in my opinion, as a newcomer, it’s really intimidating to get things going and customized after you hit fly launch. Maybe it’s just me though.

Anyways, let me know if there’s any way I can help in this process, and when we can expect some docs with your new awesome features that really should be highlighted and not easter egged in the forum haha!

Thanks for the amazing platform

3 Likes
2

Thanks for raising this!

The short answer - we’re developing a whole new runtime and orchestration layer to replace some of the undocumented features. The new ones will work better and probably have a different interface.

That said, it sounds like the current version of the multi-process feature is known-enough to be worth documenting. Especially because we’ve led you astray - these processes do not run in in the same VM. They run as separate ‘process groups’, each in its own VM with the same contents but different commands.

Besides this feature, what else did you find missing from docs?

Also, FYI, our docs repo is open for PRs, if you feel adventurous!

2 Likes
3

Thanks so much for repsonding!

Okay that makes a lot of sense when it comes to process groups. I think it’d be really insightful if there was a bit more visual separation than just filters on the metrics and the scaling. For example I think it might be a bit more clear visually if you separated the scaling part

image
image1050×587 48.6 KB

As for documentation:

  • How to migrate a postgresql instance to fly (I wrote this one but it’s pretty janky cause I was having a really hard time migrating a rails instance with what I’ve found)
  • How to make read replications from an outside database. I know there is this but no specific guide
    • Would love to have an example databases hosted by other major players like AWS/DO/Heroku & Render (for me :wink: )
  • How to backup databases
  • How to use snapshots

To be clear my confidence in self-hosting a database is super low, so I’d rather have a managed database solution that I can call in and replicate on fly instead of having to handle me messing up my database somehow on here. Hence me requesting these things. I’m fine with downing my server every once and a while but I’m not fine with losing data (obviously haha)

  • In-depth Rails deployment guide with the application of most common libraries ie: Sidekiq
    • I think also having things like this for Django and others would be really helpful too
  • Managing processes, how to modify your docker file, and what to know about them.
    • Example: I don’t know if I should leave my docker file alone or not because if you see in the example above I have 3 processes running, but I actually should just have web & worker running. I think this is being called from docker, but it brings me peace of mind to know for a fact that I’m doing things right than discovering them on my own that I should delete the docker run file, as compared to “I hope I’m right about this, but it’s working” haha
    • Sidenote: I think it’d be very helpful to know what my upcoming bill will be because right now it says 0 for the month and I don’t think that’s correct.
  • More information on health checks like how to create them, examples of health checks
  • More indepth information on implementing multistage build targets that doesn’t just link out to the docker site (if this is still going to be a thing). Maybe implementing a rails/sidekiq approach depending on what you guys think best practice is
  • While the how to read fly.toml line by line is good, it only shows what’s there. I think having a fly.toml mastery guide or something would be super helpful to know everything that one can do in there.

Alright, that’s all I can think of for now :slight_smile:

I’d contribute but I don’t feel like I know enough, one day when I become a DevOps pro I can pitch in but I’m not even confident in how I do things now.

1 Like
4

I realise one needs to overhaul entire systems sometimes to meet quality goals, but hopefully Joel isn’t proven right.

Echoing OP’s sentiments: Machines is one of the most elegant API for “serverless” out there (tall order given GCP Cloud Run and Cloudflare Workers are no slouches)! If more stuff like Machines is forthcoming, then the wait would be all the more worth it.

2 Likes
5

I am also eager to see more documentation. As a hobbyist, it’s intimidating to hand over payment information to a service lacking robust, detailed walkthroughs. You don’t know if you’re going to get halfway through a deploy only to be told to draw the rest of the owl.

I successfully migrated my tiny site off a VPS, but a few hours into deployment I almost quit fly entirely because I wasn’t confident I could launch without official guidance.

2 Likes
6

Morzaram,

Do you mind sharing how you got Rails Redis and Sidekiq running together.? I’ve tried to use multiple processes as you showed but I can never get the worker to come up. It always dies as unhealty.

I created a separate Redis server but I need to get Sidekiq working.

INFO -- : [bd854e49-1919-4f81-8ae2-c689731b38db] [ActiveJob] Enqueued OrdersCreateApiJob (Job ID: f773c6f8-f2ef-4ae0-a5d5-2b3ae1f35c41) to Sidekiq(async_api) with arguments: {:order_id=>"f8d2c522-918e-4037-996d-faxxxxxxxxxx"

I didn’t have this kind of trouble on Render and I so agree with you on the docs here.

Thanks in advance for any help.

8

Okay so be sure that the redis internal url was Set on the main secret.

This was the toml for the redis

# fly.toml file generated for climate-coolers-redis on 2022-05-30T12:59:48+02:00 
  
 app = "climate-coolers-redis" 
  
 kill_signal = "SIGINT" 
 kill_timeout = 5 
 processes = [] 
  
 [build] 
   image = "flyio/redis:6.2.6" 
  
 [env] 
  
 [experimental] 
   allowed_public_ports = [] 
   auto_rollback = true 
  
 [[mounts]] 
   destination = "/data" 
   source = "redis_server"

This is the toml for the main instance

# fly.toml file generated for climate-coolers-2 on 2022-06-01T13:15:38+02:00 
  
 app = "climate-coolers" 
  
 kill_signal = "SIGINT" 
 kill_timeout = 5 
  
 [deploy] 
   release_command = "bundle exec rails db:migrate" 
  
 [env] 
  
 [experimental] 
   allowed_public_ports = [] 
   auto_rollback = true 
  
 [processes] 
   web = "bundle exec puma -C config/puma.rb" 
   worker = "bundle exec sidekiq" 
  
 [[services]] 
   http_checks = [] 
   internal_port = 8080 
   processes = ["web"] 
   protocol = "tcp" 
   script_checks = [] 
  
   [services.concurrency] 
     hard_limit = 25 
     soft_limit = 20 
     type = "connections" 
  
   [[services.ports]] 
     force_https = true 
     handlers = ["http"] 
     port = 80 
  
   [[services.ports]] 
     handlers = ["tls", "http"] 
     port = 443 
  
   [[services.tcp_checks]] 
     grace_period = "1s" 
     interval = "15s" 
     restart_limit = 0 
     timeout = "2s"

Of course change the app name. Hope this helps! If you have errors and can’t solve it, post it here and I might be able to help.

9

Thanks! That is almost exactly what I had but somehow mine worker always came up unhealthy, this morning it didn’t. Yeah! I just need to get my regions set and I should be ok.

2022-07-08T12:31:28.902 app[18f65e75] mad [info] 2022-07-08T12:31:28.902Z pid=515 tid=mh7 WARN: Your Redis network connection is performing extremely poorly.

2022-07-08T12:31:28.902 app[18f65e75] mad [info] Last RTT readings were [150642, 151262, 150861, 150947, 156946], ideally these should be < 1000.

2022-07-08T12:31:28.902 app[18f65e75] mad [info] Ensure Redis is running in the same AZ or datacenter as Sidekiq.

2022-07-08T12:31:28.902 app[18f65e75] mad [info] If these values are close to 100,000, that means your Sidekiq process may be

2022-07-08T12:31:28.902 app[18f65e75] mad [info] CPU-saturated; reduce your concurrency and/or see https://github.com/mperham/sidekiq/discussions/5039

Thanks!

1 Like