Hi all, so I’m running into an issue where enabling a specific import in my code is causing smoke checks to fail, and therefore the overall Fly deployment fails. Both building the docker image, and pushing it succeed, but I’m getting error Smoke checks for 683d4e4b753258 failed: the app appears to be crashing in the last step where the console outputs:
Updating existing machines in 'foo-service' with rolling strategy
I’ve done a fair amount of debugging, and I was able to pinpoint the issue to a single import in my code. At the beginning I thought this was a capacity-related issue, but even after scaling memory and cpus, the failure persists. Below is the full output, has anyone seen this before?
2024-05-09T13:58:58Z runner[683d4e4b753258] mia [info]Machine started in 516ms
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info]Traceback (most recent call last):
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/bin/uvicorn", line 8, in <module>
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] sys.exit(main())
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] return self.main(*args, **kwargs)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1078, in main
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] rv = self.invoke(ctx)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] return ctx.invoke(self.callback, **ctx.params)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] return __callback(*args, **kwargs)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 409, in main
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] run(
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 575, in run
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] server.run()
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 65, in run
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] return asyncio.run(self.serve(sockets=sockets))
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] return loop.run_until_complete(main)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 69, in serve
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] await self._serve(sockets)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 76, in _serve
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] config.load()
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/config.py", line 433, in load
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] self.loaded_app = import_from_string(self.app)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 19, in import_from_string
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] module = importlib.import_module(module_str)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] return _bootstrap._gcd_import(name[level:], package, level)
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap_external>", line 850, in exec_module
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/app/main.py", line 7, in <module>
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] from deepgram import DeepgramClient, PrerecordedOptions
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/deepgram/__init__.py", line 9, in <module>
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] from .client import Deepgram, DeepgramClient
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/deepgram/client.py", line 253
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] match self.parent:
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info] ^
2024-05-09T13:58:59Z app[683d4e4b753258] mia [info]SyntaxError: invalid syntax
2024-05-09T13:59:00Z app[683d4e4b753258] mia [info] INFO Main child exited normally with code: 1
2024-05-09T13:59:00Z app[683d4e4b753258] mia [info] INFO Starting clean up.
2024-05-09T13:59:00Z app[683d4e4b753258] mia [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-05-09T13:59:00Z app[683d4e4b753258] mia [info][ 2.398792] reboot: Restarting system
2024-05-09T13:59:00Z runner[683d4e4b753258] mia [info]machine did not have a restart policy, defaulting to restart
2024-05-09T13:59:02Z app[683d4e4b753258] mia [info][ 0.062250] PCI: Fatal: No config space access function found
2024-05-09T13:59:02Z app[683d4e4b753258] mia [info] INFO Starting init (commit: d772ddd9)...
2024-05-09T13:59:02Z app[683d4e4b753258] mia [info] INFO Preparing to run: `uvicorn main:app --host 0.0.0.0 --port 8080` as root
2024-05-09T13:59:02Z app[683d4e4b753258] mia [info] INFO [fly api proxy] listening at /.fly/api
2024-05-09T13:59:02Z app[683d4e4b753258] mia [info]2024/05/09 13:59:02 INFO SSH listening listen_address=[fdaa:9:38d5:a7b:189:e9a9:8ca8:2]:22 dns_server=[fdaa::3]:53
2024-05-09T13:59:02Z runner[683d4e4b753258] mia [info]Machine started in 516ms
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info]Traceback (most recent call last):
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/bin/uvicorn", line 8, in <module>
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] sys.exit(main())
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] return self.main(*args, **kwargs)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1078, in main
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] rv = self.invoke(ctx)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] return ctx.invoke(self.callback, **ctx.params)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] return __callback(*args, **kwargs)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 409, in main
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] run(
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 575, in run
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] server.run()
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 65, in run
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] return asyncio.run(self.serve(sockets=sockets))
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] return loop.run_until_complete(main)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 69, in serve
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] await self._serve(sockets)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 76, in _serve
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] config.load()
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/config.py", line 433, in load
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] self.loaded_app = import_from_string(self.app)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 19, in import_from_string
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] module = importlib.import_module(module_str)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] return _bootstrap._gcd_import(name[level:], package, level)
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap_external>", line 850, in exec_module
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/app/main.py", line 7, in <module>
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] from deepgram import DeepgramClient, PrerecordedOptions
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/deepgram/__init__.py", line 9, in <module>
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] from .client import Deepgram, DeepgramClient
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/deepgram/client.py", line 253
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] match self.parent:
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info] ^
2024-05-09T13:59:03Z app[683d4e4b753258] mia [info]SyntaxError: invalid syntax
2024-05-09T13:59:04Z app[683d4e4b753258] mia [info] INFO Main child exited normally with code: 1
2024-05-09T13:59:04Z app[683d4e4b753258] mia [info][ 2.395297] reboot: Restarting system
2024-05-09T13:59:04Z runner[683d4e4b753258] mia [info]machine did not have a restart policy, defaulting to restart
2024-05-09T13:59:11Z app[683d4e4b753258] mia [info][ 0.059565] PCI: Fatal: No config space access function found
2024-05-09T13:59:11Z app[683d4e4b753258] mia [info] INFO Starting init (commit: d772ddd9)...
2024-05-09T13:59:11Z app[683d4e4b753258] mia [info] INFO Preparing to run: `uvicorn main:app --host 0.0.0.0 --port 8080` as root
2024-05-09T13:59:11Z app[683d4e4b753258] mia [info] INFO [fly api proxy] listening at /.fly/api
2024-05-09T13:59:11Z app[683d4e4b753258] mia [info]2024/05/09 13:59:11 INFO SSH listening listen_address=[fdaa:9:38d5:a7b:189:e9a9:8ca8:2]:22 dns_server=[fdaa::3]:53
2024-05-09T13:59:11Z runner[683d4e4b753258] mia [info]Machine started in 492ms
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info]Traceback (most recent call last):
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/bin/uvicorn", line 8, in <module>
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] sys.exit(main())
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] return self.main(*args, **kwargs)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1078, in main
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] rv = self.invoke(ctx)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] return ctx.invoke(self.callback, **ctx.params)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] return __callback(*args, **kwargs)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 409, in main
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] run(
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/main.py", line 575, in run
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] server.run()
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 65, in run
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] return asyncio.run(self.serve(sockets=sockets))
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] return loop.run_until_complete(main)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 69, in serve
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] await self._serve(sockets)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 76, in _serve
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] config.load()
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/config.py", line 433, in load
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] self.loaded_app = import_from_string(self.app)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 19, in import_from_string
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] module = importlib.import_module(module_str)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] return _bootstrap._gcd_import(name[level:], package, level)
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap_external>", line 850, in exec_module
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/app/main.py", line 7, in <module>
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] from deepgram import DeepgramClient, PrerecordedOptions
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/deepgram/__init__.py", line 9, in <module>
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] from .client import Deepgram, DeepgramClient
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] File "/usr/local/lib/python3.9/site-packages/deepgram/client.py", line 253
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] match self.parent:
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info] ^
2024-05-09T13:59:12Z app[683d4e4b753258] mia [info]SyntaxError: invalid syntax
2024-05-09T13:59:13Z app[683d4e4b753258] mia [info] INFO Main child exited normally with code: 1
2024-05-09T13:59:13Z app[683d4e4b753258] mia [info] INFO Starting clean up.
2024-05-09T13:59:13Z app[683d4e4b753258] mia [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-05-09T13:59:13Z app[683d4e4b753258] mia [info][ 2.376093] reboot: Restarting system
2024-05-09T13:59:14Z runner[683d4e4b753258] mia [info]machine did not have a restart policy, defaulting to restart