Git commands in fly ssh console work but not inside a node process running in the same machine

Hi there, I have a machine running a simple node server that runs Git operations on the underlying file system (via Fly Volumes). There is currently difference in behaviour when running Git commands when I run it directly in shell via fly ssh console and when the node process runs the exact same command.

In the shell, I have no issues running git clone [URL] [PATH IN MOUNTED VOLUME] - the folder is created with all refs and content populated.

When the node server executes the exact same command, it creates the folder with a .git folder but everything else isn’t there. I’ve dug a bit more and found that the git configs in .git/config is set up correctly but the refs are not fetched - i.e. when I do git branch there are no branches in that folder etc.

Other context:

  • I’ve ruled out SSH/access issues separately
  • The repository in question is not empty
  • The node process is being run as the same user as the one in the shell

I’ve tried checking the env variables available in the shell vs in the node process (via process.env) and I noticed some differences in what’s made available but the differences don’t look like they’ll affect git commands.

I’ve been investigating this for some time and honestly I’m not sure if this is the right place to ask because it might not be an issue with Fly machines. Any other possible areas to look will be greatly appreciated!

I definitely feel for you. I’ve been in similar situations myself, and it looks like you have explored and potentially ruled out all the right things.

The next step I personally would take would be to create a node CLI program that either mimics or directly calls the actions the node server makes and try running it from fly ssh console.

Thanks for the suggestion! Since I spent more than a day on this and I didn’t have much more time to spare to debug this I ended up moving the deployment to a plain Hetzner instance just to unblock this part. I’ll probably want to try this again down the road because it’s just easier to manage scaling Fly machines than a bare machine :sweat_smile:

could you (relatively) easily test this outside of Fly by running the node server locally? that would help you narrow this down a notch

Yup I’ve done that on my Macbook Air M1 prior to deploying - plain node server running natively running git commands via child_process and also via execa and simple-git. All worked fine.

Also tried this in a Docker container running on ubuntu-24.04 with the target directories for git operations mounted in from the host machine:

docker run -d \
    --name proxy_$NEW_PORT \
    -p $NEW_PORT:3001 \
    --restart unless-stopped \
    --env-file .env \
    -v HOST_DIR:/mnt \
    IMAGE:TAG

where the node server was running git operations targeting /mnt.

For Fly, I also ruled out whether it’s a volume issue by running the commands via ssh console targeting directories that weren’t mounted by Fly Volume (and same thing with node server). Running via console worked by not via the node server on a non-mounted directory.

To elaborate a bit further, I’m quite certain that what was failing was the git fetch part of the git clone because:

  1. The folder was created
  2. There was a .git folder initialized
  3. The .git/config file showed that the remotes were set to the right place
    But these were missing:
  • No remote refs
  • No local branches
  • No local files in the directory

What’s interesting is that I could access these directories afterward via console and run git fetch, git pull and it’d be populated. But trying to do the same via the node process would always fail at the same part.

Even when I tried to hack around it by manually setting up the git repo - git init > set remote url > fetch > checkout … etc. It wasn’t able to fetch remote refs and thus subsequent steps would fail.

Feels like I’ve exhausted all the options I could think of.

So after writing all that out, I setup another repo with a bare minimum node server running Hono with 1 endpoint to clone and it looks like it worked. I tried with public/private repos. So I think we can conclude it’s not related to Fly.

I think I’ll probably have to strip down the current server and slowly add things back to see what’s breaking git in the fly machine. Also, thanks for the prompt replies everyone, appreciate it!

1 Like

In the end the issue had nothing to do with Fly etc but thought it’d be good to close the loop on this in case anyone comes across this thread in the future.

There is a gap between a repository being freshly created in GitHub and its contents/refs being available for git clone. If you try to git clone too soon after creating a repository, there’s a good chance it’ll clone an empty one with no refs despite the remote origin existing. How this was not caught out in dev is still something we haven’t figured out, but the solution was to loop and check that the remote branch existed in the newly created repository with git ls-remote before trying to clone it.