Not Firewalla's Problem, But Sending It Back
I still don't have resolution from the ISP about the blocked traffic originating from the router IP, but I decided to return the Firewalla regardless.
It's a nice box, and the features seem very full. In the experiments where I had little or no advanced configuration, the router worked right out of the box. Just booting it, plugged into the WAN and a device, and running through the installer let the device perform without any touches. I think it'd be great if a person had a need for it.
The more I poked at it, the less I was excited for trying to do advanced things.
First, it touts that it runs Ubuntu. It does report Ubuntu when it boots and if you ask, but so much of the configuration is stored in custom locations, many of the expected default tools are missing, and it really relies on a small set of proprietary apps to do its job. That last bit is fine, and makes sense, but they seem to do all the configuration and control from local and internet data sources that aren't the usual file-based suspects. Additionally, it warns not to update the packages to avoid breaking things, and that's necessary to add things that aren't installed from the beginning. So it really isn't running Ubuntu the way I would think a system would to be comfortable saying that.
The configuration thing warrants being mentioned again. I believe a big part of the trouble I had was that the router expects to be able to reach the internet to read configuration and deal with metrics. It frequently logs failures trying to GET and POST to various amazonaws.com locations. There isn't anything in the documentation about this, and it seems like a big security risk for the sake of some convenience. Sure, it's probably all encrypted before its stored, hopefully with keys unique to and only on the router, but that's not certain. That it can't work with its own configuration app if the router can't reach the Internet combined with the proprietary configuration bits makes it an uncomfortable situation for something I'd be responsible for maintaining.
I to rescind anything that seems like blaming the router for doing something wrong with the static IP. I thought it might have been something to do with that remote configuration, but it seems more because the router can't send or read things from the Internet. That block seems to be something awry at my ISP, as I've now run the same ping and curl commands on my old router as I found failing on the Firewalla. They also fail. I've got a ticket open with my ISP, escalated to an SME, but I haven't heard back since the escalation note.
I've been holding my breath, considering how to wait it out and make it work. It's a clever little box, and the app is polished and shows lots of potential. But I can't get past the use of the cloud, the non-standard configuration inside, and that it can't work with its own app if the Internet is somehow unavailable. Probably the most critical time to need the app would be when the Internet is unavailable, and without transparency and utility for the configuration, it's a deal-breaker for me.
Oh, and it doesn't support IPv6 via a 6in4 tunnel, or other tunnels. It only supports static IPv6 or DHCPv6, which is unfortunate. My ISP doesn't support IPv6 beyond not blocking the tunnels. So that's a bit of a deal-breaker, too.
I requested and received an RMA. I'm sad to see it go, but it'll be better for me in the future. Maybe I'll revisit them if they change their functioning, but since it works out-of-the-box for all the DHCP-to-NAT users, they probably won't make many changes.
I do have my eye on a different router. I'm scouring the documentation on OPNsense, which comes pre-installed on that one. It's based on FreeBSD (not sure why not NetBSD, but we'll find out), which I'm familiar with. It supports VLANs, routing public IPs, and IPv6 in a variety of configurations. The box itself has a straight-up Intel CPU (like the Firewalla), and LINUX-supported NICs, but with expandable DIMM and a pair of M.2 SDD connectors, as well as HDMI and USB for keys (which the Firewalla also had), so it can be used however I'd like. Worst case, I dump the OPNSense and set up everything by hand, like I did before when I had my server initially do the routing.
I'm really dismayed at how hard it's been to find a consumer or "prosumer" wired router without WiFi. Almost all of those with WiFi expect to be serving a NAT LAN, even those that offer VLAN support. It's a big step from this level of router to the professional or enterprise routers, which can also require a but of infrastructure just to support the routers.
Hopefully the little box I'm looking at can work. So far the OPNsense looks pretty, and should be able to do what I need it to do.