Getting error "We're sorry, but this service is not available in your location" when downloading content from firestore

Questions / Help
1

It used to work fine. It just started today the app couldn’t download content from firestore database, even though the link is fine.

Here is an example link: https://firebasestorage.googleapis.com/v0/b/bookforward.appspot.com/o/product%2Ftemp%2Fattachments%2F720972000?alt=media&token=8d604063-3702-463f-82d6-84a3f749409d

irb(main):003:0> r = Faraday.get("https://firebasestorage.googleapis.com/v0/b/bookforward.appspot
.com/o/product%2Ftemp%2Fattachments%2F720972000?alt=media&token=8d604063-3702-463f-82d6-84a3f7494
09d")
=>
#<Faraday::Response:0x00007fc3e40d6678
...
irb(main):004:0> r
=>
#<Faraday::Response:0x00007fc3e40d6678
 @env=
  #<struct Faraday::Env
   method=:get,
   request_body=nil,
   url=
    #<URI::HTTPS https://firebasestorage.googleapis.com/v0/b/bookforward.appspot.com/o/product%2Ftemp%2Fattachments%2F720972000?alt=media&token=8d604063-3702-463f-82d6-84a3f749409d>,
   request=
    #<struct Faraday::RequestOptions
     params_encoder=nil,
     proxy=nil,
     bind=nil,
     timeout=nil,
     open_timeout=nil,
     read_timeout=nil,
     write_timeout=nil,
     boundary=nil,
     oauth=nil,
     context=nil,
     on_data=nil>,
   request_headers={"User-Agent"=>"Faraday v1.10.2"},
   ssl=
    #<struct Faraday::SSLOptions
     verify=true,
     ca_file=nil,
     ca_path=nil,
     verify_mode=nil,
     cert_store=nil,
     client_cert=nil,
     client_key=nil,
     certificate=nil,
     private_key=nil,
     verify_depth=nil,
     version=nil,
     min_version=nil,
     max_version=nil>,
   parallel_manager=nil,
   params=nil,
   response=#<Faraday::Response:0x00007fc3e40d6678 ...>,
   response_headers=
    {"x-guploader-uploadid"=>
      "ADPycdvC4IRoBDglfrNwoMAXAR0tt1J7l5cA9eFRTfh_IDGfhh87Z1YsFNq-DDLL5x3P0hbbZvSDxDLSz-kkKyHtUqyGHowWM7vV",
     "content-type"=>"application/xml; charset=UTF-8",
     "content-length"=>"193",
     "date"=>"Fri, 16 Dec 2022 13:03:08 GMT",
     "expires"=>"Fri, 16 Dec 2022 13:03:08 GMT",
     "cache-control"=>"private, max-age=0",
     "server"=>"UploadServer",
     "alt-svc"=>
      "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""},
   status=403,
   reason_phrase="Forbidden",
   response_body=
    "<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>We're sorry, but this service is not available in your location</Details></Error>">,
 @on_complete_callbacks=[]>
irb(main):005:0>

What could be the issue and how can I fix it?

2

There are line breaks in the URL you pasted. Is that just a quirk of the IRB session or are those actual linebreaks?

If Firestore is returning an error like that, there’s nothing we can do to fix it. You’d need to ask Google what’s going on there.

3

The line break shouldn’t be a problem, just a copy and paste from terminal. Is it a bit too soon to rule out the problem from fly? Several reasons:

  • At the same time, making the same call on local machine did work
  • I restarted the instance and it started working afterwards
  • It did work normally, just started to happen a few days ago.

Could it be the fly instance is deployed from a different region, or the outbound ip is not in the correct region and hence Google rejected the request?

4

It could be something like that, yes, but Google would have to document what makes that error happen.

Many of our IPs are owned by a company in the US, if they’re doing weird IP filtering to try and region lock something, that might be a check they run. That’s different than where the IP came from though. When apps run in Sydney, they connect to things from Sydney, even though we own that IP. There’s nothing between your VM and Google.

5

Some updates on this topic: I’ve created a support ticket with Firebase, and awaiting for their meaningful response. In the meantime, when I got the issue, I tried to get the outbound IP and it was detected as in Singapore.

Another issue just happened today, I have a feeling that it’s relevant, where the sidekiq app couldn’t connect to the database ActiveRecord::NoDatabaseError, whereas the db is totally fine and the web server app works just fine.

What could be the reason?

Edit: looks like that same box couldn’t make the request, scaled it up to 2 boxes and then scaled down to 1 fixed the issue