Load testing recommendations and shipping v1

So my site is getting close to finally shipping: https://kent.dev

When it’s officially launched, it’ll replace https://kentcdodds.com which had 150k unique visitors and 500k page views in the last 30 days. When my new site is launched, I expect a LOT more traffic, especially soon after the launch.

I want my site to be lightning fast, which is why I’ve got a multi-region deployment of my node app and postgres and redis clusters.

I’m wondering what folks here suggest for load testing before I launch. Just so I can fix any issues before the stress of “everyone’s watching my site fall on its face” :sweat_smile:

I was thinking about just making a script with a bunch of curls and distributing that to some friends all over the world and hitting it hard at the same time. But part of me feels like that might trigger Fly’s DDoS protections or something. Any tips are helpful!

@jsierles used k6s for his recent article: Run Ordinary Rails Apps Globally · Fly

The test setup is here: GitHub - jsierles/fly-k6: Run k6 from multiple regions

You don’t actually need volumes like his example test used, you can force one instance per region with fly scale count <x> --max-per-region 1 now.

It’s definitely important to test this from a bunch of different places!

Thanks!

I think I understand most of what’s going on. Just to be clear, what’s happening here is @jsierles is deploying this to fly and running the tests on fly itself? Seems like you’d want it run on real people’s machines :thinking: Otherwise wouldn’t it appear to be faster than it would for real users?

I guess what I’m mostly testing is whether my app can survive a lot of concurrent requests, so measuring the speed probably doesn’t matter as much…

So I could probably fork this and deploy it to hit my own app. Looking at it I’m not sure what actually kicks off the test. Does it happen automatically on deploy? Does fly just know to run the run.sh file?

Yes you have it right. To get end user performance, you’ll want to run outside Fly. To get app scale/performance, running on Fly is just fine .

I think he ran run_local.sh manually from his local machine, it seems to just connect over SSH and fire off the run command in each VM: fly-k6/run_local.sh at main · jsierles/fly-k6 · GitHub

Cool. Makes sense. I think I’ve got everything set up and running. When I try to run run_local.sh, I get this output and I’m not sure where to go from here:

$ ./run_local.sh
Warning: Permanently added 'fdaa:0:23df:a7b:12de:0:3540:2' (ED25519) to the list of known hosts.
Warning: Permanently added 'fdaa:0:23df:a7b:aa3:0:3541:2' (ED25519) to the list of known hosts.
Warning: Permanently added 'fdaa:0:23df:a7b:6b:0:3543:2' (ED25519) to the list of known hosts.
root@fdaa:0:23df:a7b:12de:0:3540:2: Permission denied (publickey).
Warning: Permanently added 'fdaa:0:23df:a7b:acc:0:3542:2' (ED25519) to the list of known hosts.
root@fdaa:0:23df:a7b:aa3:0:3541:2: Permission denied (publickey).
Warning: Permanently added 'fdaa:0:23df:a7b:b92:0:3544:2' (ED25519) to the list of known hosts.
root@fdaa:0:23df:a7b:6b:0:3543:2: Permission denied (publickey).
root@fdaa:0:23df:a7b:acc:0:3542:2: Permission denied (publickey).
root@fdaa:0:23df:a7b:b92:0:3544:2: Permission denied (publickey).

And my fly logs show:

2021-08-20T19:54:37.994383072Z app[067bdf9c] dfw [info] 2021/08/20 19:54:37 unexpected error: [ssh: no auth passed yet, ssh: normal key pairs not accepted]
2021-08-20T19:54:38.701534789Z app[eb072068] ams [info] 2021/08/20 19:54:38 unexpected error: [ssh: no auth passed yet, ssh: normal key pairs not accepted]
2021-08-20T19:54:38.954910801Z app[5a768c37] scl [info] 2021/08/20 19:54:38 unexpected error: [ssh: no auth passed yet, ssh: normal key pairs not accepted]
2021-08-20T19:54:39.405915965Z app[df8db981] hkg [info] 2021/08/20 19:54:39 unexpected error: [ssh: no auth passed yet, ssh: normal key pairs not accepted]
2021-08-20T19:54:40.262577503Z app[0bd15895] syd [info] 2021/08/20 19:54:40 unexpected error: [ssh: no auth passed yet, ssh: normal key pairs not accepted]

(Oh, and note for anyone reading in the future, I also had to have wireguard running (I followed these instructions)

It looks like he set it up to SSH directly. You need to run this, I believe, to get an SSH key for your local machine:

flyctl ssh issue -a

You may need to run an SSH agent on your machine for this to work. I’m pretty sure I already had one running on MacOS, so it might be included for you already.

Hey! Kurt is right here - you need the ssh agent running to be able to ssh to instances. This is all a bit of a hack. I cooked it up to demonstrate a way to run a command globally.

You might just want to use k6 directly from a VM, by hand, as its full output is much more informative than the value extracted by the script.

k6 also has great docs - check it out if you want to run some more involved scenarios.