Is Fly a good fit for running data ELT pipelines?
I want to self host Dagster for orchestration. Then I pull data from multiple sources and store this in a DuckLake on object storage.
Once a month, I need to parse a large XML to parquet. This is CPU bound. Would it be possible to spin up a fly machine for an hour once a month and do the calcultions?
Can I consider fly to be basically be serverless compute that I can spin up for bursty workloads?
Does anyone have experience with using Fly for data engineering work?