A couple of days ago I read this toot (German) over on Mastodon which starts with “IPv6 is hard”.
No, it’s not. It’s different.
I’ve run across this multiple times: There is an A and an AAAA record for an FQDN, but the web server is only reachable via IPv4. You can easily test this with curl.
$ curl -4 https://github.com -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 273k 0 273k 0 0 3417k 0 --:--:-- --:--:-- --:--:-- 3553k
$ curl -6 https://github.com -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Couldn't connect to serverWhen using IPv4, 273k is ‘saved’ to /dev/null. However, using IPv6 we get an error message: “Couldn’t connect to server”.
Note that I’m using GitHub here because it is a well-known example of not offering IPv6 at the time of writing, and it probably will have not IPv6 any time soon.
So in the case of the toot mentioned above, there is an AAAA for the FQDN. But the connection doesn’t work, which the normal end user doesn’t notice. This is because there is a browser feature called Happy Eyeballs (HE). Basically, a browser tries both protocols and uses the faster one. In the case of non-working IPv6, IPv4 is always faster.
HE can have some funny side effects. In a project, a connection to a development web server sometimes worked and sometimes didn’t — the users got a ‘403 Forbidden’ page. The solution was quite simple. The customer used a split VPN tunnel. IPv4 was routed via the VPN tunnel and those IPv4 addresses were allowed in the web server access list. IPv6 was routed via the normal Internet connection and those addresses weren’t allowed. Sometimes IPv4 over the tunnel was better, sometimes IPv6 from the provider was better. HE cares about TCP connections, not HTTP response codes.
The paragraph above led to some questions: Why didn’t HE work? Well, it did work. Sometimes the IPv4 connection was better than the IPv6 connection, and sometimes IPv6 was better than IPv4. The TCP connection worked, and that is what counts for Happy Eyeballs.
The web server had an access list that allowed certain addresses (IPv4 and IPv6) and rejected others with an HTTP 403 error (Forbidden). HE does not care about the HTTP status code.
I had another case where I couldn’t use my online banking. HE decided that IPv6 was better than IPv4. The initial TCP connection worked fine, but during the TLS handshake, packets became too large. Since, at least for some people, ICMPv6 is considered evil, the bank broke Path MTU Discovery (PMTUD). I had to configure radvd to announce a smaller MTU (Maximum Transmission Unit) on the interface. The default for Ethernet is 1,500 bytes, but I had to lower it to 1,492 bytes (1,500 – 8 bytes for PPPoE).
HE does work, but it hides several problems and sometimes leads to other issues that are hard to debug and are probably most often answered with, ‘Turn off IPv6’. However, if it didn’t hide a lot of problems, we would probably have far fewer IPv6-enabled websites.
But back to the original problem. In another toot, we learn that a support ticket was opened four weeks ago and that traceroute was prohibited. However, there is always the option to use the -T option for traceroute or tcptraceroute, which uses TCP instead of UDP (the default for standard traceroute) or ICMP (used by Windows tracert).
When you run traceroute -T for both IPv4 and IPv6, you’ll see that, in this case, IPv6 ends one hop earlier than IPv4. Alternatively, the last hop might loop back to itself.
This could mean that there is a missing firewall rule allowing traffic to the server or there is a routing issue on the firewall.
But we can learn much more from this:
- If the browser didn’t do Happy Eyeballs, the support hotline would have exploded
- There is no external monitoring, at least not for IPv6
- There is no automation, or at least no complete automation
So please, if you do IPv6 — take it seriously. If you don’t take it seriously,
don’t do IPv6. That leads to people thinking that IPv6 is hard and cannot be done.
Jens Link is a freelance IT consultant with interests in DNS, switching / routing, IPv6, automation, and monitoring.
Adapted from the original posts on Jens’ Techlog.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.