Upstash Kafka available in public beta

Fresh Produce
1

Last month we announced private beta availability of Upstash Kafka.

I’m happy to announce that Kafka clusters are open in public beta to all Fly.io customers in the iad and fra regions. Check out our docs to get started.

We want your feedback! If you’re interested in deploying in a different region, or if you have an interesting use case for Kafka, let us know.

3 Likes
2

Hey Joshua, I was playing around with the Redis and Kafka connections- I got Upstash’s Redis working great but I cannot get the Kafka one to work. I only get an EOF error. I’m using Golang for reference and copy-pasting the code inside my project (I’ll paste here for reference).

func Producer(topic string) {
  mechanism, _ := scram.Mechanism(scram.SHA256, "********", "********")
  w := kafka.Writer{
    Addr:  kafka.TCP("*********"),
    Topic: topic,
    Transport: &kafka.Transport{
      SASL: mechanism,
      TLS:  &tls.Config{},
    },
  }
  w.WriteMessages(context.Background(), kafka.Message{Value: []byte("MESSAGE")})
  w.Close()
}

I also noticed that on Upstash, there’s an option when viewing the code to reveal secrets- but the “username” provided at the top and the “username” provided when viewing the code with revealed secrets are not the same. I’m not sure which one to use but I tried both with no luck.

I’m not sure if it’s connecting to the kafka cluster at all even actually.

Any help would be appreciated! Thank you!

EDIT: Also I cannot change the budget on the fly upstash integration one but I can on an upstash only created one.

3

Also just to confirm, I went to Upstash directly, made a Kafka cluster, plugged in the username/password/address and it immediately worked with no configuration.

The username mismatch is weird, the top one gives a user friendly name with many dashes (like “cobra-kai-name”) while the reveal code one gives one that is random digits (and like 30 digits long). I think the random digits one is the real username- since the direct Upstash kafka cluster only displays the random digits one.

4

is there better latency when using upstash kafka from fly.io offering than compared to direct ? I am using kafka and redis and the latencies are quite high

5

Both Upstash Kafka and Redis run on Fly.io infrastructure, next to your apps. Low latency access from Fly.io apps is the primary reason you’d want to use them!

6

Hmm, that’s no good. What region are you deploying in? Can you send in a support ticket for this? That will get more eyes on the problem.

1 Like
7

ok thanks, so is it correct to asumeif I spin a kafka in FRA and have a app in LAX the request will route over fly.io network and not public network ? As currerntly I have a upstash kafka in FRA and app in LAX and the request is going over the public internet and the latency is somewhat horrendous for this reason I gave up on their Redis

8

Requests will route over the Fly.io network, encrypted, but latency will be similar. You can’t beat the speed of light!

Upstash Redis on Fly.io runs in all Fly.io regions, so you can provision right next to your app, for example in LAX.

Kafka is still limited to a few regions, but if you’re interested in a particular one, let us know.