I run a web scraper so the regions of my deployments are important, basically the websites that I scrape behave differently if being requested from the UK, US, AU etc.
When I deploy in the SYD (Sydney) region it behaves fine as if the requests are coming from Australia. When I deploy in the LHR(London) region however, it behaves as if the requests are coming from the US.
Is there something I’m doing wrong or is this an issue on fly.io’s end?
Hi,
I guess the first thing to check would be that your vms are where you expect. In this case, when you deploy to lhr. Running fly status should show that. Since an app has a region pool and while vms should be evenly distributed, they may not be.
Assuming your vm is indeed listed as being in e.g lhr when you say … “behaves as if the requests are coming from the US” what exactly differs?
For example the outgoing IP the app uses may appear to be in the US because it will be registered to Fly, which is in the US. Depending on how the geo-location is done. But the server (host) would be in the UK.
Hi Greg,
Thanks for your response. I have set up the vms so they are deployed in lhr and lhr is the only region in the region pool. This is confirmed when I run fly status, the region is displayed as lhr.
What differs is that the websites I’m scraping from believe that the requests are coming from a US computer, which means that the data I receive is skewed towards US customers rather than the UK customers I’m attempting to cater for.
Is there a way I can check these outgoing IPs? Or change them so that they appear as if they’re from the UK?
Ah, ok so the vm is in the UK.
So the problem seems to be that the websites being accessed think the IP is in the US. That is trickier to fix since I’m assuming they are using some kind of geo-location service to figure out what country an IP is in, and that is returning the wrong answer.
If you randomly look up a Fly IP, it geo-locates to the US. Take a look at e.g 109.105.216.0 IP Address Details - IPinfo.io and scroll down. As that’s where they are based. Not necessarily where the server is.
You can’t specify the outgoing IP for a Fly app to use (to force one you know geo-locates to the UK, in your case).
However you can see what it probably is by making a request from your app vm to some other server/website you control. A curl/wget/ping … whatever. Since you can then see in your access logs for [other site] what IP that request came from. And hence the IP your Fly app used for that request at least. Of course that won’t be the same for every site. Armed with such an IP, you can look it up yourself out of interest in a public geo-location site to see where that service thinks the IP is located in the world.