I believe Fly defaults to a new release when secrets are changed. Arguably that shouldn’t include a secret being added as that could not affect existing code (which would not know about that secret even being there) but it would (potentially) affect a running app if a secret is updated or deleted.
As regards taking the app offline, I think that is an unintended consequence of any deploy/release (currently), rather than due to secrets specifically. The current solution is to have an app with at least three instances. See e.g: