[Protocol] NGINX doesn't support HTTP/1.1 on h2c ports

[zhangzhhz]

Hi,

I have trojan running at 443, redirecting to 80 which Nginx runs at. Nginx reverse proxies notebook.DOMAIN.TLD to http://localhost:6600 which runs a jupyter notebook.

I can connect to notebook.DOMAIN.TLD and see a list of notebooks and other files, but opening a notebook will give "Connection failed" error:

A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.

Firefox console shows:

Starting WebSockets: wss://notebook.DOMAIN.TLD/api/kernels/4439066a-d496-49c8-af22-f472ca7e019c kernel.js:462
Firefox can’t establish a connection to the server at wss://notebook.DOMAIN.TLD/api/kernels/4439066a-d496-49c8-af22-f472ca7e019c/channels?session_id=f3f30c1c9a904f09840e44b7974f4e42. kernel.js:464
Kernel: kernel_disconnected (4439066a-d496-49c8-af22-f472ca7e019c) kernel.js:106
WebSocket connection failed:  wss://notebook.DOMAIN.TLD/api/kernels/4439066a-d496-49c8-af22-f472ca7e019c true kernel.js:545
Connection lost, reconnecting in 8 seconds.

There is no problem if I stop trojan and run Nginx at 443 directly.

Can someone give me some pointers?

Thanks.

My Nginx server config for notebook.DOMAIN.TLD is as follows:

map $http_upgrade $connection_upgrade {
  default Upgrade;
  '' close;
}

server {
        listen 80 http2; # forwarded by trojan-gfw
        listen [::]:80 http2; # forwarded by trojan-gfw
        #listen 443 ssl http2;
        #listen [::]:443 ssl http2;

        root /home/xxx/yyy/zzz;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        server_name notebook.DOMAIN.TLD;

        location / {
            proxy_pass http://localhost:6600;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $http_host;
            proxy_http_version 1.1;
            proxy_redirect off;
            proxy_buffering off;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_read_timeout 86400;
        }

        #gzip on;
        #gzip_comp_level 3;
        #gzip_types text/plain text/css image/*;

        ssl_certificate     /home/zzce/.jupyter/fullchain.pem;
        ssl_certificate_key /home/zzce/.jupyter/privkey.pem;
        ssl_ciphers         EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH;
        ssl_prefer_server_ciphers on;
        ssl_protocols       TLSv1.1 TLSv1.2 TLSv1.3;
        add_header Strict-Transport-Security max-age=15768000;
}

[GreaterFire]

@zhangzhhz Can I see your trojan server config? (redact as needed)

[GreaterFire]

Actually, I would suggest not use http/2 but listen for http/1.1 in your nginx (and change the alpn option in trojan server config accordingly).

[zhangzhhz]

@GreaterFire, thanks, it does work now after removing http2 and h2 from Nginx and trojan.

I thought http2 was supported in majority of cases now (that's why I added it in trojan configuration in the first place). Can trojan have better support for http2?

[GreaterFire]

I'll try to look into this.

[GreaterFire]

It's NGINX's issue actually (https://trac.nginx.org/nginx/ticket/816). It supports both 1.1 and 2 simultaneously on its 443 port, but it doesn't support plain 1.1 and 2 simultaneously on its 80 port.

[GreaterFire]

This is a valuable problem. It points out a distinction between NGINX behind trojan and the standalone NGINX itself when HTTP/2 is present. As HTTP/2 gets more prevalent, we cannot ignore this issue, because this distinction can be used to precisely identify the existence of trojan in front of NGINX.

I'll repeat the problem of NGINX here. If we have a standalone NGINX listen on 443 with both http2 and ssl, this port actually supports HTTP/1.1, because NGINX will adjust to the lower version based on the ALPN negotiation result. However, if we let NGINX listen on 80 with only http2 and let trojan do the SSL termination, NGINX has no idea what the version of HTTP is (because the ALPN negotiation is done in trojan) and will assume HTTP/2 no matter what. If the protocol is, in fact, HTTP/1.1, NGINX will simply ignore that and respond with an HTTP/2 message.

[GreaterFire]

After a closer look at the NGINX bug tracker (https://trac.nginx.org/nginx/ticket/816), it seems that NGINX is against the standard. An HTTP/2 port should be backwards compatible with HTTP/1.1.

[GreaterFire]

See also https://trac.nginx.org/nginx/ticket/808

[GreaterFire]

The problem lies here (https://github.com/nginx/nginx/blob/9e07862d6e9d37041a42bbfa34dd1f56ed547505/src/http/ngx_http_request.c#L324-L331) where NGINX will choose to use HTTP/2 unilaterally if the config says so. Too hard to patch this.

[mem0rz]

easy to solve this problem.
just config like this
-----trojan-------------------
"remote_addr": "127.0.0.1",
"remote_port": 81,
------nginx------------
server{
listen 80;
listen 127.0.0.1:81 http2;

[GreaterFire]

@cybmp3 This doesn't solve the problem when you have to serve HTTP/1.1 over the h2c port because the client browser doesn't support h2.

[mem0rz]

@cybmp3 This doesn't solve the problem when you have to serve HTTP/1.1 over the h2c port because the client browser doesn't support h2.

Yes.If you browser http://example.com you will get a normal website(http1.1).
if you browser https://example.com Trojan(with h2 config) will proxy to 127.0.0.1:81(HTTP2 without ssl)

[WillyPillow]

I mentioned it in #171, but I wonder if it is reasonable to mitigate this by redirecting to different remote_ports given different ALPNs. I can try to cook up a PR in the coming days if you guys are okay with this.

[GreaterFire]

@WillyPillow This will break the current config format and will be more confusing to configure. I recommend we put it on hold for now.

[WillyPillow]

@GreaterFire

Technically one should be able to preserve backwards compatibility by, say, adding a new config entry that maps ALPNs to port numbers, and using the port specified in remote_port only if the ALPN does not exist in the map. But yeah, I agree that it would be confusing.

[GreaterFire]

@WillyPillow Good idea. I'll try to design this scheme.