How I could effectly deploy Readpanda brokers and Redpanda Console?

Questions / Help
, ,
1

So i’m very new to the fly.io ecosystem even that I already know to to deploy my elixir/phoenix apps effectly. For now i’ll give a workshop on developing async web APIs with phoenix and a message queue and I chose redpanda as it is easy to set up locally and have less work that trying to build a zookeeper server.

However I don’t even know how to start from deploying the redpanda on fly.io… All examples I find is from a docker-compose.yml or manual setup on a VPS server, which isn’t the case for fly.io.

Here’s a docker-compose that I use to set up redpanda, and redpanda console:

version: '3.7'
services:
  redpanda:
    image: docker.redpanda.com/vectorized/redpanda:v23.1.2
    command:
      - redpanda start
      - --smp 1
      - --overprovisioned
      - --node-id 0
      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
      - --pandaproxy-addr 0.0.0.0:8082
      - --advertise-pandaproxy-addr localhost:8082
    ports:
      - 8081:8081
      - 8082:8082
      - 9092:9092
      - 9644:9644
      - 29092:29092

  console:
    image: docker.redpanda.com/vectorized/console:latest
    entrypoint: /bin/sh
    command: -c "echo \"$$CONSOLE_CONFIG_FILE\" > /tmp/config.yml; /app/console"
    environment:
      CONFIG_FILEPATH: /tmp/config.yml
      CONSOLE_CONFIG_FILE: |
        kafka:
          brokers: ["redpanda:29092"]
          schemaRegistry:
            enabled: true
            urls: ["http://redpanda:8081"]
        redpanda:
          adminApi:
            enabled: true
            urls: ["http://redpanda:9644"]
        connect:
          enabled: true
          clusters:
            - name: local-connect-cluster
              url: http://connect:8083
    ports:
      - 8080:8080
    depends_on:
      - redpanda

  connect:
    image: docker.cloudsmith.io/redpanda/connectors/connectors:624ff9e
    hostname: connect
    container_name: connect
    depends_on:
      - redpanda
    ports:
      - "8083:8083"
    environment:
      KAFKA_CONNECT_CONFIGURATION: |
        offset.storage.topic=docker-connect-offsets
        value.converter=org.apache.kafka.connect.json.JsonConverter
        config.storage.topic=docker-connect-configs
        key.converter=org.apache.kafka.connect.json.JsonConverter
        group.id=compose-connect-group
        status.storage.topic=docker-connect-status
        config.storage.replication.factor=1
        offset.storage.replication.factor=1
        status.storage.replication.factor=1
      KAFKA_CONNECT_METRICS_ENABLED: "false"
      KAFKA_CONNECT_BOOTSTRAP_SERVERS: redpanda:29092
      KAFKA_GC_LOG_ENABLED: "false"
      KAFKA_HEAP_OPTS: -Xms128M

How can I generate a fly.toml from this docker-compose? I would need to split it into 3 different applications? Also, How I could reach the redpanda brokers then on external application on fly.io? Also notice that docker-compose uses the development mode of redpanda, so it would need to start on production mode inside fly.io.

Sorry if it a begginners question, but I really don’t know how to start from.

2

Ok I build this Dockerfile:

FROM docker.redpanda.com/vectorized/redpanda:v23.1.2

EXPOSE 8081 8082 9092 9644 29092

CMD ["redpanda", "start", "--smp 0", "--memory 1G", "--reserve-memory ", "--overprovisioned", "--node-id 0", "--kafka-addr FLY://[::1]:9092", "--advertise-kafka-addr FLY://redpanda-scti.internal:9092", "--pandaproxy-addr FLY://redpanda-scti.internal:8082", "--advertise-pandaproxy-addr FLY://redpanda-scti.internal:8082"]

which generated this fly.toml:

app = "redpanda-scti"
primary_region = "gig"

[build]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[mounts]
  destination = "/var/lib/redpanda/data"
  source = "redpanda_poc"

[http_service]
  internal_port = 8081
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ["app"]

However I’m getting this error:

Could not initialize seastar: boost::wrapexcept<boost::bad_lexical_cast> (bad lexical cast: source type value could not be interpreted as target)
   INFO Main child exited normally with code: 1
   INFO Starting clean up.
   INFO Umounting /dev/vdb from /var/lib/redpanda/data
  ERROR error umounting /var/lib/redpanda/data: EBUSY: Device or resource busy, retrying in a bit
  ERROR error umounting /var/lib/redpanda/data: EBUSY: Device or resource busy, retrying in a bit
  ERROR error umounting /var/lib/redpanda/data: EBUSY: Device or resource busy, retrying in a bit
  ERROR error umounting /var/lib/redpanda/data: EBUSY: Device or resource busy, retrying in a bit
   WARN hallpass exited, pid: 314, status: signal: 15 (SIGTERM)
3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

4

For what it’s worth, I know the platform intimately and couldn’t get redpanda deployed either. It was too hard to figure out what it wanted / expected.

I couldn’t get a cluster formed either, which makes me wonder if IPv6 is supported.