Problem: Using fly-replay with JSON format to route requests between apps in the same org. The JSON payload appears to be generated correctly in logs, but Fly’s proxy returns [PA03] 'fly-replay' response returned by app was invalid: replay was malformed: Invalid JSON: expected value at line 1 column 1.
Response Flow
- Browser requests
https://{main-app}/machines/{project-id} - Main app generates fly-replay JSON response with path transformation
- Fly proxy should replay request to target app (
{main-app}-{project-id}) at path/ - Target machine runs http service on ports
80:8080
Code Implementation
def code(conn, %{"project_id" => project_id}) do
project = Projects.get_project_by_id!(project_id)
app_name = Projects.cloud_app_name(project)
replay_config = %{
app: app_name, # app-1234abcd (exists in fly)
instance: project.machine_id, # 12314abcdefg (exists in fly)
state: project.machine_key, # 1234abcdefghi1234
transform: %{
path: "/"
}
}
conn
|> put_resp_header("content-type", "application/vnd.fly.replay+json")
|> json(replay_config)
end
Logs
Successful JSON generation:
20:26:37.083 [info] fly-replay config: %{state: "smzv5snfd0z0p", instance: "6839740b6d5ed8", transform: %{path: "/"}, app: "app-8453e84d"}
20:26:37.084 [info] Encoded JSON body: {"state":"smzv5snfd0z0p","instance":"6839740b6d5ed8","transform":{"path":"/"},"app":"app-8453e84d"}
20:26:37.084 [info] Sent 200 in 61ms
Fly proxy error immediately after:
[PA03] 'fly-replay' response returned by app was invalid: replay was malformed: Invalid JSON: expected value at line 1 column 1
Request headers (Elixir formatted)
Request headers: [
{"pragma", "no-cache"},
{"cache-control", "no-cache"},
{"sec-ch-ua", "\"Chromium\";v=\"139\", \"Not;A=Brand\";v=\"99\""},
{"sec-ch-ua-mobile", "?0"},
{"sec-ch-ua-platform", "\"macOS\""},
{"dnt", "1"},
{"x-request-start", "t=1755989309202546"},
{"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"},
{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"},
{"sec-fetch-site", "same-origin"},
{"sec-fetch-mode", "navigate"},
{"sec-fetch-dest", "iframe"},
{"referer", "https://{main-app}.fly.dev/projects/8453e84d-3b96-4697-a2fb-3afe303f68dc"},
{"accept-encoding", "gzip, deflate, br, zstd"},
{"accept-language", "en-US,en;q=0.9"},
{"cookie", "..."},
{"priority", "u=0, i"},
{"host", "{main-app}.fly.dev"},
{"fly-client-ip", "{redacted-ip}"},
{"x-forwarded-for", "{redacted-ip}, {redacted-ip}"},
{"fly-forwarded-proto", "https"},
{"x-forwarded-proto", "https"},
{"fly-forwarded-ssl", "on"},
{"x-forwarded-ssl", "on"},
{"fly-forwarded-port", "443"},
{"x-forwarded-port", "443"},
{"fly-region", "dfw"},
{"fly-request-id", "01K3CHWXRJTD3TTYR32RJEZMCD-dfw"},
{"via", "2 fly.io, 2 fly.io"}
]
What Works vs What Doesn’t
Works: Direct curl requests work perfectly and get proper 302 redirects to http service on target machine
curl -v https://{main-app}/machines/{project-id}
# Returns: HTTP/2 302, location: ./
Doesn’t Work: Browser requests, directly navigating, or trying to embed in an iframe, get 502 Bad Gateway due to the malformed JSON error.
Questions
- Why would the same response work for curl but fail for browser requests?
- Could Phoenix middleware be interfering with the response body after
json()? - Is there something about the request headers (cookies, user-agent, etc.) that affects fly-replay JSON parsing?
