I still have a problem on this server with its IPv6 configuration.

The server has a NIC with a public IP on the WAN router's LAN, and a private IP on the WiFi router's LAN. The public IP allows Internet traffic to find the web and mail and other servers. The private IP allows me to maintain it from the inside, and gives it access to other servers that aren't on the Internet. That all works great, except now the IPv6 configuration.

Unlike IPv4, there aren't really any private IPs on IPv6. I mean, there is the fc00::/7 range, which is like IPv4's 10.0.0.0/8 or 192.168.0.0/16 we're all familiar with from our NAT WAN addresses. These should only be used on IPv6 networks not intended for connecting to the Internet, and unlike the IPv4 cousins, they don't work as a private network behind a public IP in a NAT situation. I suppose one could (and maybe someone has) make an IPv6 NAT behave, but IPv6 was (in part) created to make NAT unnecessary.

I've digressed.

So, my previous attempt at routing an IPv6 /64 out of a /48 that I'd been given wasn't working in my router. So I changed to allow each of my routers to have their own IPv6 range. The public servers will use the WAN /64, and the LAN workstations will use the WiFi /64. The routing between them will happen at my tunnelbroker.net servers, since my ISP doesn't do native. The other way I could have done it was to have my WiFi router use its IPv6 pass-through ability to make its LAN nodes get IPv6 addresses from the its WAN side. I did get this working, but then the firewall on the router got tricker, so I scrapped that idea.

Small aside, there, I did have to use the router command line to add the firewall rules to the IPv6 firewall, because its web UI didn't allow them to be created. In researching that, I also found a third-party intrusion detection that works on my routers, not unlike fail2ban or CrowdSec on my servers. I also found that my iptables script that slurps IP ranges from ipdeny.com and sets up wholesale blocs also works! I added that to the router's custom scripts, so any time the firewall does something (like start or get edited in the UI), the tables get updated and the firewall rules block all that attention from around the world. Again, because I front my domains with Cloudflare (I have a whitelist for their servers), there's really no need for many other servers to try to access web ports. E-mail is a little trickier, because technically it could come from anywhere, and I hate that a real person using a real server in one of those denied places might not be able to e-mail me, but they can use my e-mail form in a pinch.

Aside aside...

So now my WAN router, and its LAN nodes, have IPv6 addresses from one /64 network, and my WiFi router, and its LAN nodes, have IPv6 addresses from another /64 network. This all works great, until a node with a WiFi address tries to hit a WAN address on a node that participates in both networks, like these servers do. Then I run into the same multi-home routing problem that I started trying to fix a couple weeks ago.

It seems like the server is getting traffic from a WiFi router LAN address to its WAN router LAN address, but then it wants to respond through its WiFi router LAN address, which isn't how the Internet works.

Maybe I can diagram this, obfuscated a little with short IPv6 addresses.

  INTERNET -- [ WAN (fd01::1/64) ] -- [ WiFi (fd02::1/64) ]
                     |                    |            |
          [ (fd01::99/64) SERVER (fd02::99/64) ]  [ (fd02::42/64) DEVICE ]

Hey, I used the private IPv6 range to obfuscate!

Maybe a little lost in the ASCII art, but imagine the WAN router LAN has the address of fd01::1, and serves the fd01::/64 network. It doesn't have a WAN side IPv6 address because it's being tunneled through IPv4. Technically the tunnel is leveraging that the WiFi router has a public IPv4 address on the WAN router LAN that is used to tunnel the IPv6 traffic to the Internet. 

Briefly, how the tunnel works.

When the a device on the network makes an IPv6 request of something on the Internet, that traffic is sent by the router to the tunnel's IPv4 server node, and they deposit it on the IPv6 Internet. I'm kind of glossing over the details, because it's technically the same Internet, just packets with different headers. Since the network I'm attached to can only work with IPv4, the tunnel is used to wrap all this stuff and get it shuttled around; think of them like an addressed envelope inside an addressed envelope, because that's really what they are. When the IPv4 envelope reaches the tunnel server, the IPv6 envelope is removed and resubmitted to the network. Except for that bit of IPv4 enveloping, the rest of the router and systems behave the same as they would for a native IPv6 network.

Because it's also using a tunnel, the WiFi router isn't actually participating in the WAN IPv6 network, but instead leverages its public IPv4 address, and appears as an IPv6 network peer to the WAN. The WiFi router LAN has the address of fd02::1, and serves the fd02::/64 network.

The server participates in both the WAN and WiFi router networks. On its "server side" it has the IPv6 address of fd01::99, and on the "private side" it has the IPv6 address of fd02::99. Note the subtle difference in the prefix; the node address is the same to make it easy to go "ah, server."

Finally, consider a device on the WiFi LAN, like a tablet or laptop, that receives the address of fd02::42.

Wen an IPv6 request is going to somewhere other than my server, like google.com or apple.com, once it leaves my tunnel, their server receives it (directly or through their tunnels, or however their ISPs have integrated them), deals with it, and sends responses back to the tunnel server, which sends it back through the tunnel, and finally it gets injected back on my IPv6 network.

When that IPv6 request is my own server, it leaves my WiFi fd02::/64 network to the tunnel broker via IPv4. The tunnel broker puts it back on the IPv6 network, which then gets tunneled back through the broker over IPv4 to my WAN fd01::/64 network to get injected in its IPv6 network to my fd01::99 node. The server then does what it does and tries to send it back.

Here's where I think the breakdown happens. The server knows it received a request "from fd02::42 to fd01::99," but it also sees it has an address in the same network, fd02::99. I've configured the server's network routing rules so that it should remember the interface through which the traffic arrived, and respond through the same interface, basically sending a response back "from fd01::99 to fd02::42," the opposite way it arrived. Instead what I think happens is it sees it's participating in the originating network, so it thinks the better way is to send the response "from fd02::99 to fd02::42," which probably happens, but since the device isn't expecting the traffic, it kindly rejects the envelope.

For my own remembering, and for anyone who's interested and happens to find this on the Internet, this is what the server's configuration looks like (obfuscated, of course):

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses:
      - "192.168.1.99/24"
      dhcp6: false
    eth1:
      addresses:
      - "10.0.0.99/29"
      - "fd01::99/64"
      dhcp6: false
      routes:
      - to: "default"
        via: "fd02::1"
      - to: "default"
        via: "10.0.0.1"
      routing-policy:
      - table: 4
        from: "10.0.0.0/24"
      - table: 6
        from: "fd01::/64"

Here eth0 is the private interface and eth1 is the public interface. The routing-policy bits should take care of the "if it arrived from this network, send it out on this interface" configuration, and the default routes, in the absence of other routes, should also encourage the packets to exit via the eth1 interface.

Important to note that eth0 has been instructed to not use DHCP for the IPv6 addresses, nor has one been configured, yet it still ends up with one, which also adds to my configuration pickle. There is no configuration for fd02::/64 in there.

Mystery one is "why is there an IPv6 address on eth0 at all?" The DHCP shouldn't be happening, and there isn't one manually configured.

Mystery two is "why doesn't the routing-policy work?" I've tried to manually assign the address, and given new routing table information, but that doesn't work, either.

I can get it to work post-boot by using the Webmin tool to disable IPv6 on eth0. This is a fine work-around, and once done does allow the traffic to flow as expected, but it isn't maintainable as there are blips and bits that reload the configuration or reacquire a DHCP address when I'm not looking.

Oh, something to remember for later. The site used to confirm the "private IPv6" thing, with a bit of discussion on addressing a private/test IPv6 network running with a more thorough /64 range, https://simpledns.plus/private-ipv6 has a brief discussion, and visiting the page gives an actual subnet that one could safely use in a lab, or for Docker, or whatever. Of course, the one with all zeroes in by example is valid, and should work, but the spec does say they should be unique, so whether a "random" tool like this or a made up string is used, at least the first 64 bits should be filled with more than zeroes.

I do tend to use zeroes for my static addresses, taking the prefix for my /64 and just filling in the last bit after the :: like in the examples. Technically this address could be used by DHCP as it generates the second half of the address, but the likelihood of that being mostly zeroes is literally 1 out of a 64-bit number. Pretty good odds it isn't going to happen. Leveraging this unlikely event of lots of zeroes gives a huge pool to use for static addresses. I've toyed with using "IPv6 address words" to fill the gaps, too, but it's easier to type :: than funny words.