Run SQS on Fly.io

Show & Tell
1

Hello! For the past few months I’ve been working on building a message queue and I want to share it with everyone!

It’s called “SmoothMQ” - because it’s so smooth and easy to get started. It is written in Go, deploys as a single binary, and uses SQLite to store messages.

Most interestingly, it implements the SQS API. That means you can connect all of your existing SQS code and private background workers.

It has a lot of features that improve upon SQS itself. For example, you can schedule messages into the future. You can also use the dashboard to search for a specific message. Rate limiting and better dead-letter queues are coming soon!

Here is the code: https://github.com/poundifdef/smoothmq

The other interesting thing is that it exposes prometheus metrics. This means you could use the fly-autoscaler to automatically scale workers based on queue depth.

I would like to be able to work on this full-time. So I am offering a paid hosted version on Fly. I’d love your feedback - what you would want to see in a queue hosted on Fly? If you would be willing to have a zoom call with me then I’m also happy to throw in some credits too.

Finally, you can deploy it yourself! Here is a fly.toml to get you started. (But really, it would be swell if you wanted to support my excellent open-source work.)

Any and all feedback and suggestions welcome!

4 Likes
2

Hey this is fantastic! I don’t currently have a use case for this now but likely in the near future.
Some quick questions:

  1. Does it support FIFO?
  2. How does it handle scaling? Does it automatically scale as you scale up fly instances?
  3. Does is handle polling similar to SQS (eg short/long polling)
3

It does not support FIFO. Auto-scaling is in the works - but I have more dev and testing to do before I’m ready to launch it as it is a much bigger implementation. That said, the current iteration can currently manage thousands of messages per second and can scale vertically.

Short polling yes, long-polling no (in progress.) That said, there is no limit on the number of messages you can fetch in a single request (whereas SQS has a limit of 10.)

1 Like
4

Just an update!

Live demo. You can now demo this, without any signup, on a real server, hosted on fly. Do your worst (and I’d love the feedback if/when you break it.)

https://www.smoothmq.com/demo

Long polling. Long polling now works.