No longer able to chmod a file in my app

1

I’m trying to deploy a new version of my app. On startup, my code tries to chmod a supporting utility that I ship with the application. But I’m now getting an error:

    File "/workspace/webmonaco.py", line 15, in <module>
      os.chmod(monaco_bin, st.st_mode | stat.S_IEXEC)
  PermissionError: [Errno 1] Operation not permitted: '/workspace/bin/linux/monaco'
  [2024-04-12 18:29:37 +0000] [321] [INFO] Worker exiting (pid: 321)
  [2024-04-12 18:29:37 +0000] [305] [ERROR] Worker (pid:321) exited with code 3
  [2024-04-12 18:29:37 +0000] [305] [ERROR] Shutting down: Master
  [2024-04-12 18:29:37 +0000] [305] [ERROR] Reason: Worker failed to boot.

I haven’t changed anything in my app configuration or the code (the new deployment is just publishing a newer version of the supporting utility). And this approach has been working for a long time now.

Has anything changed at the fly end that might have caused this?

2

It’s possible this is due to our recent change of making the rootfs read-only. Can you provide the app name or any machine ID(s)?

3

Ouch, yes that would do it. App is webmonaco, machine id 91857509c91268.

So what should I do if I want to include a Linux executable in my app? I’m developing on Windows, so I can’t set the executable bit locally before deployment. And it seems a bit overkill to create a new volume for this 20M file…

4

Is this something that can be done in the Dockerfile?

For now, I’ve rolled back the change so new deploys should work while we discuss internally how to best handle the change.

5

Ok, looking at the machine config we used, it did NOT use our new rootfs feature so it’s very likely this is something else.

6

I’ve been doing some investigating. I set the app up from the tutorial for building a Python application, so I don’t have a dockerfile, I just have a directory with Python code in it, requirements.txt and Procfile files. The docs say that the launch directory is copied across (which is something I should look into - that probably means there’s a bunch of testing and deployment code getting copied over, not just the app!) but as I say, because I’m on Windows rather than Unix, the executable won’t have the executable bit set - which is why I’m doing that on app startup.

7

Gotcha. Looking at the history of the app and it’s machine, looks like the last deploy before today was on 2024-01-22. Does that sound right? It’s also using buildpacks which are known to have random breaking changes at times.

8

Yes, it does.

I’ve just done a new deploy, removing the chmod call in the app, and then gone in manually via fly ssh console -s and executed the chmod from the console. That appears to have worked, but it’s a bit messy.

The buildpacks I’m using are picked up automatically with the auto-detection of a Python application, as I said. Is there a better way I should be building the app? I’m not too keen on things “known to have random breaking changes”… I do very little work on this app, and “deploy just works even if I can’t recall what I did last time” is an important priority for me :slight_smile: When I set this up, I simply followed the “building a flask app” tutorial (I was in a hurry, I was moving from Heroku as they had discontinued their free tier and I was looking to get something up quickly). I’m happy to switch to something more reliable if that’s the right thing to do.

9

@pf_moore which version of flyctl are you using?

There was breakage related to Buildpacks changing file permissions in the image to non-writable by the user running the service, though that was fixed in flyctl 0.2.26. It’s still possible your /workspace/bin/linux/monaco file is non-writable by the user running the web service. One thing you could do to diagnose this is have your webmonaco.py script do a time.sleep(3600) or similar before attempting the os.chmod operation. This will cause the machine to start and stay up (although understandably it won’t really do anything). You can then do:

fly ssh console

to get a shell session in the machine. Check the permissions for the file:

ls -la /workspace/bin/linux/monaco

And check which user your service is running as:

ps wwuxa | grep webmonaco.py

(Note I’m doing this blind, you may need to adapt this a bit and explore the machine on your own to see whether the user is mismatched or not).

Please let us know what you find !

  • Daniel
10

A quick follow-up to @roadmr is it looks like you are using flyctl "0.2.23" based on the machine config, so his suggestions should get you unblocked.

11

Ah. I was using flyctl 0.2.23 (I hadn’t spotted there had been an update). Updating flyctl has fixed the issue. Sorry for not spotting that!

I’m looking into a Dockerfile based build, just to give me some more control over what’s deployed. With that I can ensure the executable bit is set in advance and avoid the whole chmod thing altogether. Plus, I can test what will be deployed locally, which has some advantages as well. I think I have a working version, but I’m not sure how to switch my app to use it. I can use fly launch to build a new app, but I’m not sure how to update an existing app. Can I just edit the new app, changing the name in fly.toml, and deploy, or do I need to do anything more than that?

12

Hi @pf_moore,

Switching to a Dockerfile, if you do have the Dockerfile already, should just require removing the [build] section from your fly.toml, including the buildpack designation. You can also remove the Procfile as it won’t be needed with a Dockerfile-based deploy. The next fly deploy will pick up the Dockerfile and do a Docker-based image build.

(by the way, buildpacks under the hood end up spitting out a Docker image anyway! They just do so without a Dockerfile).

If you run into trouble with the Dockerfile, the community can help - though in that case I would advocate creating a new forum thread as “I need help getting my Python app Dockerized” is rather orthogonal to “buildpack-based builds are read-only” :slight_smile:

Regards,

  • Daniel
13

Awesome. Thanks so much for your help, it’s really appreciated.

14

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.