Pay-as-you-go pricing for Redis databases

Fresh Produce
1

Good day, Redis lovers!

We’ve brought pay-as-you-go pricing to our Upstash Redis extension. This means you only pay for what you use, instead a fixed price per database, just like the rest of the Fly.io platform.

Check the Upstash pricing page for details on how requests and storage are priced.

For those of you running databases already, nothing will change right now. However, you can update your databases to this new plan at any time. The fixed price plans are no longer available.

Note that Upstash will retire the fixed price plans on January 1st, 2024. Meanwhile, they will email you periodically with your usage to help you best choose a plan going forward. You can also track your usage in the Upstash web console.

Also - some of you noticed that tools like Sidekiq and BullMQ poll Redis aggressivelym increasing your Redis request count. Upstash does not count empty Redis responses, so you don’t need to worry about these tools incurring unfair charges.

11 Likes
2

Yay! This is wonderful news!

Sadly, this isn’t quite true. LiteFS Cloud has a $5/mo minimum. Would be great if you added a low/bursty pay-as-you-go option for that like Upstash just did for Redis.

1 Like
3

Just FYI: The page transition from the “Visit the Upstash dashboard” in the WebUI (https://fly.io/redis/[NAME]/usage) is broken; the transition from the CLI (fly redis dashboard [ORG NAME]) still works.

1 Like
4

Is there a way to see the actual usage that we will be billed for?

Upstash started limiting us on a free tier since we kept hitting 10k request per day. Which shouldn’t happen if they do not count the empty responses.

We upgraded to pay-as-you go and it seems that BullMQ is generating ~10k request per hour. Still haven’t hit 100k requests to see if we are billed for them or not. If we are then having to pay ~15$ just to keep an idle Redis instance alive seems absurd.

EDIT: Looks like we are indeed billed for the whatever empty traffic BullMQ is generating:

upstash_bullmq
upstash_bullmq1472×1323 61.4 KB

Our implementation should be pretty standard (ioredis with bullmq) and the graphs above don’t have any actual data (we tested the integration on Thursday and since then haven’t added a single job to the queue).
At this stage it looks like Upstash is not a suitable provider for us, which is a shame since I really liked the ability to deploy and manage the Redis instance via flyctl.

1 Like
5

I’m Mehmet from Upstash team.

We are not counting empty requests/responses but unfortunately not all requests from Bullmq are considered empty. Our definition of empty is, any read command returning null (or empty array etc) and any write command not modifying the state.
Even when idle, Bullmq sends some write commands, especially EVALSHA.

I did a test just now, with a single Bullmq queue and a single worker, in 5 minutes period Bullmq sent 445 requests and 335 of them identified as empty and not counted in quota.

"DailyProcessedCommands": 445,
"DailyWriteCommands": 184,
"DailyReadCommands": 261,
"DailyEmptyCommands": 335,
6

Hello @sander29

The commands count you see (145K) includes empty commands. It might be misleading.

We will show (total ops count - empty commands) in the console so you will see the exact number.

I have checked internally your databases’ %70 commands are empty. So they are not billed.

7

I guess this is the main problem - with the help of your replies and having a bit more data the situation becomes somewhat easier to understand.

Yeah, we can actually see that from the dashboard - at the moment there are 600k commands (out of which 30k were generated under free tier) and we are billed for ~190k commands = ~33% of the total.

The ratio seems to be in the same ballpark as to what we are seeing. Although we are seeing ~2x more commands:

upstash_bullmq_2
upstash_bullmq_21021×884 29.5 KB

Not really sure what is causing the increase though - ideally it should remain the same and not increase 40% overnight.

Either way, assuming it does not keep increasing then I guess it’s not that bad - at ~0.2$ per day per worker it is still much cheaper than any alternative we found.

Thank you for looking into it!

8

How can one do it?

9

Hey! Where can I check the number of non-empty requests? The new pricing is kicking in so we would like to have a estimation for the billing too. Not sure if I’m missing out anything from the dashboard. Thanks in advance! :slight_smile:

10

You can use flyctl redis update <yourdb> and select the pay-as-you-go plan.

11

Thank you!
I’m not able to select replica in fra region despite my redis databases are all deployed in fra. Is it expected?

❯ fly redis list
NAME      	ORG     	PLAN	EVICTION	PRIMARY REGION	READ REGIONS 
adopt     	personal	Free	Disabled	fra           	            	
donate1024	personal	200M	Disabled	fra           	            	
blogigram 	personal	200M	Disabled	fra           	            	

❯ fly redis update donate1024
Some regions require a paid plan (maa).
See https://fly.io/plans to set up a plan.

? Choose replica regions, or unselect to remove replica regions:  [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]
> [ ]  Amsterdam, Netherlands (ams)
  [ ]  Stockholm, Sweden (arn)
  [ ]  Paris, France (cdg)
  [ ]  Denver, Colorado (US) (den)
  [ ]  Dallas, Texas (US) (dfw)
  [ ]  Secaucus, NJ (US) (ewr)
  [ ]  Guadalajara, Mexico (gdl)
  [ ]  Sao Paulo, Brazil (gru)
  [ ]  Hong Kong, Hong Kong (hkg)
  [ ]  Ashburn, Virginia (US) (iad)
  [ ]  Johannesburg, South Africa (jnb)
  [ ]  Los Angeles, California (US) (lax)
  [ ]  London, United Kingdom (lhr)
  [ ]  Madrid, Spain (mad)
  [ ]  Miami, Florida (US) (mia)

There is no fra in the list.

12

Replicas are for reducing read latency in regions other than the primary. Within a single region, reads are already balanced across multiple instances. Does this help?

1 Like
13

Yeah it does, for some reason I thought that selector was intended for primary :man_facepalming:

14

I have a couple of Rails apps that only use Redis for Sidekiq jobs (not many of them either). With the new pricing, I am going to be paying an extra $30-$40 per month for a service used to be free.

My usage stats:

  • 1.8 million commands per day (around 30 per minute?!)
  • 309mb bandwidth
  • 33.34kb storage

Does anyone know how I can reduce the number of commands? I don’t know why there are so many commands when even on the most active day there are only around 100 Sidekiq jobs scheduled. Most days there are only around 10-20 jobs.

Any help would be appreciated.

1 Like
15

For anyone wondering if you can still use Sidekiq with the Upstash free tier… you can’t.

Apart from regular Redis calls directly related to running background jobs, Sidekiq has to make other calls to maintain its functionality.:

  • Heartbeat thread makes lots of Redis calls every 5 seconds (or 10 seconds since Sidekiq v7.0.0) [1]
  • While checking for queues with jobs to process, Sidekiq makes 1 blocking call per worker every 2 seconds [2]

Try to connect to Redis-CLI on your dev machine and use MONITOR command to see how many calls really Sidekiq does. [3]
With only 1 active worker, without scheduling any job it takes less than an hour for Sidekiq to reach the Upstash limit.

You can try using Sidekiq alternatives that don’t require Redis:

  1. Solid Queue (MySQL or PostgreSQL) [4]
  2. Lite Stack (SQLite) [5]
  3. Delayed Job (PostgreSQL and more) [6]

Or just use another Redis provider or simply pay Upstash for their services.

[1] sidekiq/lib/sidekiq/fetch.rb at v7.2.0 · sidekiq/sidekiq · GitHub
[2] sidekiq/lib/sidekiq/launcher.rb at v7.2.0 · sidekiq/sidekiq · GitHub
[3] MONITOR | Redis
[4] GitHub - basecamp/solid_queue: Database-backed Active Job backend
[5] GitHub - oldmoe/litestack
[6] GitHub - collectiveidea/delayed_job: Database based asynchronous priority queue system -- Extracted from Shopify

3 Likes
16

I have pet project using Sidekiq for both simple scheduled jobs and “cron” like ones. Since the latest update there were no ways to use free tier redis because of the number of requests Sidekiq makes to it.

I have successfuly migrated ActiveJob from Sidekiq to DelayedJob (that uses DB).
Final stack: delayed_job_active_record + delayed_job_web + delayed_cron_job. It covered all the functionality I was using with the previous setup: UI to manage jobs, simple scheduled jobs and cron scheduled jobs

17

I fed up with Upstash timeouts and ridiculous pricing (50% of my Fly bill accounted for two upstash instances) and reverted back to self-hosting.


primary_region = "fra"

[metrics]
  port = 9091
  path = "/metrics"

[build]
  image = "flyio/redis:6.2.6"

[[vm]]
  cpu_kind = "shared"
  cpus = 1
  memory_mb = 256

[[services]]
  internal_port = 6379
  protocol = "tcp"

That’s it, dead simple, works like a charm.
For more details: GitHub - fly-apps/redis: Launch a Redis server on Fly

5 Likes