Support for HTTP/1.0

Support for HTTP/1.0

[Joe McIlvain]

We work on an IoT-oriented web service that uses Unicorn.  One of our
requirements is to support HTTP/1.0 for low-complexity devices.

We've noticed that HTTP/1.0 requests to Unicorn always get HTTP/1.1
responses, which is invalid behaviour for the HTTP/1.0 protocol.

Sure enough, looking through the Unicorn source I see that the
"HTTP/1.1" protocol is hard-coded in the response writing logic:
https://github.com/defunkt/unicorn/blob/a72d2e7fbd13a6bfe64b79ae361c17ea568d4867/lib/unicorn/http_response.rb#L30

When behind our nginx/haproxy frontend, this behaviour is a little
more sneaky.  For "short" response payloads, the proxy will override
the response and always (correctly) use HTTP/1.0.  However, for "long"
response payloads, the content encoding is "chunked" and the proxy
will use "HTTP/1.1"

You can actually see this bug in action in the GitHub API (a prominent
web service using Unicorn).  If you send `curl -v --http1.0
https://api.github.com/gists/public`, you will get an HTTP/1.1 chunked
response, which an HTTP/1.0 client cannot handle.

I've experimented with using puma instead of unicorn, and it behaves
correctly in this respect (it responds to HTTP/1.0 requests with
HTTP/1.0 responses).  However we'd like to keep using unicorn if
possible.

Does Unicorn intend to support HTTP/1.0?

Re: Support for HTTP/1.0

[Eric Wong]

Joe McIlvain <joe.eli.mac@gmail.com> wrote:
> We work on an IoT-oriented web service that uses Unicorn.  One of our
> requirements is to support HTTP/1.0 for low-complexity devices.
> 
> We've noticed that HTTP/1.0 requests to Unicorn always get HTTP/1.1
> responses, which is invalid behaviour for the HTTP/1.0 protocol.
> 
> Sure enough, looking through the Unicorn source I see that the
> "HTTP/1.1" protocol is hard-coded in the response writing logic:
> https://github.com/defunkt/unicorn/blob/a72d2e7fbd13a6bfe64b79ae361c17ea568d4867/lib/unicorn/http_response.rb#L30

Right, it's certainly faster to avoid having an extra hash
lookup to get the correct string in the response.

Ruby performance ought to be better nowadays than it was when
the code was written, however; so maybe we can use up some
extra cycles in the interest of correctness.

> When behind our nginx/haproxy frontend, this behaviour is a little
> more sneaky.  For "short" response payloads, the proxy will override
> the response and always (correctly) use HTTP/1.0.  However, for "long"
> response payloads, the content encoding is "chunked" and the proxy
> will use "HTTP/1.1"

Interesting, that sounds like a bug or misconfiguration in nginx
or haproxy if it's giving chunked responses to 1.0 clients.

Is this nginx->haproxy->unicorn  or  haproxy->nginx->unicorn?

Are persistent connections from nginx->unicorn enabled?
(I suggest keeping it disabled, the default as far as I recall)

What else can you share about your nginx/haproxy version and
configuration?

> You can actually see this bug in action in the GitHub API (a prominent
> web service using Unicorn).  If you send `curl -v --http1.0
> https://api.github.com/gists/public`, you will get an HTTP/1.1 chunked
> response, which an HTTP/1.0 client cannot handle.

I can't seem to reproduce the chunked response.

It says "HTTP/1.1", but it either has Content-Length set (if
requested w/o gzip) or just closes the connection if requested
with compression.  Even with the Content-Length set, the
connection is still closed, so should be easily handled by
1.0 parsers.

> I've experimented with using puma instead of unicorn, and it behaves
> correctly in this respect (it responds to HTTP/1.0 requests with
> HTTP/1.0 responses).  However we'd like to keep using unicorn if
> possible.

Puma was meant for real-world HTTP clients, whereas unicorn is
mainly designed to talk to nginx, so correctness was less of a
priority.

> Does Unicorn intend to support HTTP/1.0?

We can consider it; but this is the first I recall hearing about
this after all these years.

Can you please share more info about your nginx/haproxy so
we can poke around?  The proxies should really be massaging
the response into something the client can handle...

Strangely, I remember going out of my way to support headerless
"HTTP/0.9" responses :>

Anyways thanks for bringing this up and I look forward
to having more info about your proxy setup.

Re: Support for HTTP/1.0

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> Is this nginx->haproxy->unicorn  or  haproxy->nginx->unicorn?
>
> Are persistent connections from nginx->unicorn enabled?
> (I suggest keeping it disabled, the default as far as I recall)
> 
> What else can you share about your nginx/haproxy version and
> configuration?

(out-of-order)

Ping?   With my nginx (1.6.2 on Debian jessie) -> unicorn config on
I can confirm nginx is sending 1.0 requests to backends, which
ought to prevent Rack::Chunked from chunking, at least...

Haven't looked at haproxy in a while... :x

> Joe McIlvain <joe.eli.mac@gmail.com> wrote:
> > Sure enough, looking through the Unicorn source I see that the
> > "HTTP/1.1" protocol is hard-coded in the response writing logic:
> > https://github.com/defunkt/unicorn/blob/a72d2e7fbd13a6bfe64b79ae361c17ea568d4867/lib/unicorn/http_response.rb#L30
> 
> Right, it's certainly faster to avoid having an extra hash
> lookup to get the correct string in the response.

There's also several places where we assume "HTTP/1.1"
that would involve deeper changes than merely taking
the version string: the wacky[1] check_client_connection
+ response_start_sent logic

So more work would be necessary to respond with HTTP/1.0...

But I've been thinking about ways to cleanup and micro-optimize
that a bit more, anyways...


[1] https://bogomips.org/unicorn-public/?q=s:%22combating+nginx+499%22
    (weirdest. feature. ever.)