We’d like to connect Airbyte to a Fly Postgres instance that only has private networking enabled.
Airbyte can use an SSH tunnel, but how would we set that up on the Fly side?
I could create a Bastion app, but credentials from fly ssh issue are limited to 72 hours.
Another idea (not a little effort) would be to peer Fly to GCP and have a bastion running there?
Am I missing an easy trick here?
There are a lot of options available, including using .internal URIs.
Check this page out -
I am not 100% sure the best path for you, but I would imagine if you have Postgres and your Airbyte instance on fly, you should be able to use the default internal networking features without needing a SSH tunnel.
If the Bastion is a passthrough, and Airbyte is hosted somewhere else, I am not 100% sure the best way.
I played with self hosting Airbyte last week, and its so massive (like 7 GBs on disk).
I really liked this product over Airbyte. Though it has less connections, it seems like a better architecture. I would consider looking at it honestly, because it has a step in the middle that gives you more resilience and access to modify/transform the data after the fact.
Thanks for the ideas! In the end just setup a Fly worker pushing parquet snapshots out of the Fly network…
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.