Proxy CONNECT issues

Hi!

I’m setting up a Squid proxy (and spent 6 hours before that trying to setup an Nginx one, to no avail).

Dockerfile
FROM yegor256/squid-proxy
COPY squid.conf /etc/squid/squid.conf
squid.conf
# See https://www.pks.mpg.de/~mueller/docs/suse10.2/html/opensuse-manual_en/manual/sec.squid.configfile.html

http_port 3128
http_access allow all

# ! Commented out just to see if it works without any auth.
# To make it ask for HTTP Basic Authorization
# auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwd
# auth_param basic children 5
# auth_param basic realm proxy
# auth_param basic credentialsttl 2 hours
# auth_param basic casesensitive on
# acl auth proxy_auth REQUIRED
# http_access allow auth

# To disable all logging
access_log none
cache_log /dev/null

# To disable caching
cache deny all
cache_mem 8 MB
cache_dir null /tmp

# To make it anonymous
forwarded_for off
request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Cache-Control allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access User-Agent allow all
request_header_access Cookie allow all
request_header_access All deny all
fly.toml
app = <REDACTED>
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 3128
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

It seems to work locally, but once when I’m curl-ing the Fly app, it just errors out with 500 status code:

curl attempt
❯ curl https://google.com -v -x https://<REDACTED>.fly.dev

*   Trying 137.66.34.131:443...
* Connected to <REDACTED>.fly.dev (<REDACTED>) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server accepted to use http/1.1
* Proxy certificate:
*  subject: CN=*.fly.dev
*  start date: Jan 22 23:19:23 2023 GMT
*  expire date: Apr 22 23:19:22 2023 GMT
*  subjectAltName: host "<REDACTED>.fly.dev" matched cert's "*.fly.dev"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* allocate connect buffer!
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.79.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 500 Internal Server Error
< server: Fly/d0c3ef57 (2023-01-26)
< mime-version: 1.0
< date: Wed, 01 Feb 2023 17:22:07 GMT
< content-type: text/html;charset=utf-8
< content-length: 4332
< x-squid-error: ERR_CANNOT_FORWARD 0
< vary: Accept-Language
< content-language: en
< via: 1.1 fly.io
< fly-request-id: 01GR7001CE41BXM85V2YRMA7MY-waw
<
* Received HTTP code 500 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 500 from proxy after CONNECT

I’ve spent too much time on this already and I’m starting to think that there may be a problem on Fly’s end. Does Fly block outgoing CONNECT requests? And what’s happening?

Turns out, if you are trying to build a HTTP proxy, you need to remove handlers from [[services.ports]] in your fly.toml file, since you essentially want your proxy to handle all the connections.