I've made some progress by trying to think ahead of what's gone awry. I think the Firewalla is getting in its own way.

I reset the Firewalla and app on my iPad. I connected the Firewalla to my ISP, and connected my WiFi router to it. I turned off the WiFi on my phone and enabled its hotspot. I connected my iPad to my phone's hotspot. I started the process on the Firewalla app on my iPad, scanning the photo on my phone when prompted. This bit always works. The Firewalla ran through the installation and delivered a functioning configuration.

I started on the LAN, addressing it to my static IP subnet. I turned off IPv6, as my ISP doesn't support it, and it doesn't seem Firewalla does the 6in4 tunnel workaround that I use otherwise. Unlike before, I set the DHCP to only use the two addresses that weren't assigned to the devices. I noticed previously that there's no way to add a reservation until a device requests an IP, but since my devices are statically assigned, they don't make the request, so I figured this would avoid collisions. That all went well.

Before diving into changing the WAN, which seems to break the world, I poked at a few other things.

I connected my laptop to one of the LAN ports on the Firewalla, and it got one of the two addresses. I used the app's SSH configuration to get a password and was able to SSH into the Firewalla. Not wanting to do that again, I created a .ssh/authorized_keys file and added my laptop's public key. I disconnected and reconnected without a password! In previous pokes around, I've found that there are not only logs in the /var/log folder, but also in the ~/logs folder. I started a tail -F /var/log/* logs/* to watch the system do its thing.

I checked to make sure the laptop was able to reach the Internet. I quick hit https://whatismyipaddress.com/ and was rewarded with the same address that the Firewalla reported, so NAT was working. I also hit the Firewalla's LAN IP an was rewarded with the status page showing all success.

I paired an additional device, adding my phone. This worked fine. From this point on I kept the phone at the box's "home" screen, watching the network performance, flows, blocks, and devices change while I did the rest. I hoped that if there is something weird in the app that edits the WAN, maybe the other will stay connected. I didn't think it would be, but I hoped.

Then I added an inbound rule that would allow my servers to behave. I created a group that I named "static" that included the connected WiFi router, and made a "to and from Internet" rule that allowed all traffic to devices in the group. I then connected the other server network cables. I was able to ping from the SSH session to the servers, and they were all seen successfully. This added the devices to the list in the app, so I added them to the static group. This should allow the servers to bypass any other rules on the server. They've all got their own firewalls, so I'm fine with this for now. I'll probably refine the rules so that only the necessary ports (HTTP/HTTPS to both and all the mail ports to the one) are allowed, preventing port scanning from finding anything I don't intend to serve.

I poked at and enabled DNS-over-HTTPS, too. The app configuration makes it seem like it'll only apply to the LAN, even though I selected "all devices."

So far, so good. Both the iPad and iPhone apps were showing the same data, and the log files were flowing without apparent errors. I poked around the app a little on both devices and everything looked the same.

I decided it's time to try the hard thing.

I made note of the DNS server provided by the DHCP. When looking at the logs, I noticed it would sometimes complain that some host names wouldn't resolve. I don't believe there are any blocks on my ISP, but I figured since this was working with the DNS configured, it should continue to work in the future.

Before diving in, I turned off the source NAT on the LAN. I want the traffic to route, not NAT. Plus this is one of the things that fails to change after I change the WAN addresses.

I dove into changing the WAN addressing. I switched from DHCP to static. I provided the IP address, DNS, and MAC address, and turned off IPv6. I used one DNS server provided by DHCP, and one specified with my static IP configuration. I had previously used 1.1.1.1 an 8.8.8.8 as they're easy to remember, but the error messages mentioned previously complained using those, so I thought if they're blocked, avoid them, yeah? I also changed the connection test targets. I put the IP address of my immediate upstream, and left the 1.1.1.1 and 9.9.9.9 that were in there, and also changed the DNS target name to that of my ISP.

I crossed my fingers and waited for it to finish.

Before the app stopped waiting for the spinner to fade, a whole bunch of things happened at about the same time.

The laptop logs started showing errors. I thought it might disconnect instead, but better than the other routers it just affected the one interface. Good ol' Ubuntu.

The phone and iPad started popping alerts that the network monitors noticed the network services had come back. Clearly the routing was working, and the WAN IP had correctly changed.

The Firewalla app on my phone stopped updating, though. I was looking at the performance for the last 60 minutes, and it had stopped scrolling a few minutes before. I waited and checked, and even as I write this it shows that last snapshot. The other numbers also haven't changed.

Both the phone and iPad app alerted that the Firewalla was offline. I hit the "diagnose" button (more of a link), and it showed the right IP configuration, and both the ping and DNS tests passed. Before too long, the alert faded and the network showed that it was connected.

There were still errors flying through the logs. Some of the things that worked before, weren't working any more.

I ran through some checks on the servers, and they seem to behave just fine. Mail and web traffic are flowing, and all of my other quick and normal checks pass.

I disconnected the laptop from the Ethernet cable and reconnected it to the WiFi. It behaved as expected. I reconnected the SSH to the Firewalla through the WiFi, and that worked, too. Checking the IP address at http://ifconfig.me gave me the WiFi WAN address (on the Firewalla LAN), as expected. And still the logs show some errors.

It's a little weird, because the logs show things like this:

2025-05-07 14:14:07 ERROR DestIPFoundHook: Failed to load intel, err: Error: HTTP failed after 1 attempt(s) POST https://firewalla.encipher.io/bone/api/v3/intel/host/*/check

ETIMEDOUT

When I check for the hostname, it works. When I try to ping, it fails, and when I try to hit that URL with CURL, it also fails. I can't even get the IP check to work:

pi@Firewalla:~ (Firewalla) $ curl http://ifconfig.me
curl: (28) Failed to connect to ifconfig.me port 80 after 130202 ms: Connection timed out
pi@Firewalla:~ (Firewalla) $ ping ifconfig.me
PING ifconfig.me (34.160.111.145) 56(84) bytes of data.
^C
--- ifconfig.me ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3072ms

It seems like something happened when changing the WAN IP address that is causing the Firewalla to block the outbound traffic.

I rebooted the Firewalla from the command line (as sometimes when it gets in this state the app can't get it done), and it came up fine, but has the same problems. I had hoped that the reboot might clear some of the "abuse" bits it might have found while running, or cleared some cached configuration to allow it to find the right IP addresses. It did restart and maintain the configuration (as near as I can tell), and all the servers and the rest continue to work.

The firewall rules are obnoxiously long and complex, but I bet somewhere in there is something that tries to allow the DHCP address, but isn't allowing the static address. I experienced this success when I first tried configuring the Firewalla when connected to the LAN; on returning it to the LAN, it began to behave. I had tried once to revert the ISP connection from static back to DHCP, but that never took; it happens a lot that I can't change the network configuration when it gets stuck in this state.

One of the failures looks to be trying to register a WAN address change, but it fails, I think for the same kind of blocking rule that the pings and curls fail.

I settled in for some lunch and will poke to see what I can find. I think I'm going to leave the Firewalla in place, though, and see if I can't fix it with my Ubuntu skills.