content/posts/new-homenet-architecture/index.md

@@ -0,0 +1,241 @@
+---
+title: "New Home Network Architecture"
+date: "2025-11-20T07:30:00-07:00"
+description: "The architecture of my current home network"
+summary: "I decided to rearchitect my home network, and this is the result."
+---
+
+Almost two years ago, I decided to undertake another major overhaul of my home
+network. It is the latest step in the long evolution of my personal systems,
+and is now in a state that I'm fairly happy with.
+
+## History
+
+I've been networking my computers together for decades. The first example that
+could reasonably be called a network was around 1996, where I connected my
+Amiga 1200 and Amiga 3000T together over a null-modem serial cable and ran IP
+between them. I eventually got some Ethernet hardware and ran mostly simple
+flat networks for a while.
+
+### Orthodoxy
+
+After working professionally designing and building networks and IT systems, I
+had learned a few rules. Networks in particular always consisted of several
+key elements:
+
+- **Three Tiers**. You needed Core, Distribution, and Access switches. This
+  helps to scale the network and keep things well balanced.
+- **VLANs**. Every packet needs to go through a VLAN. VLANs keep the network
+  segregated for security, and allow for smaller broadcast domains.
+- **Firewalls**. The network has to protect the vulnerable endpoints by
+  blocking packets that aren't permitted by policy.
+- **Virtualization**. Virtualize everything to decouple the systems from the
+  underlying infrastructure, keeping them portable.
+
+Naturally, I took these ideas home and built my personal networks accordingly.
+
+### Something's not right
+
+Eventually, I had built myself a network that looked something like the
+diagram below. I kept to the principles I was familiar with, and this was the
+result.
+
+![First Network Diagram](first_net.svg 'My last "orthodox" network architecture')
+
+I had VLANs for everything coming from the VM hosts to the physical switches.
+Traffic would loop through the tiers (in, out, and back in), routing
+everything like it's supposed to, but this redundancy introduced unecessary
+complexity. I had more VMs acting as routers than I had VMs doing productive
+activity.
+
+When I decided that I would like to start using IPv6 in my network, everything
+doubled. I kept the IPv4 and IPv6 traffic on separate VLANs, and had separate
+routers for everything, doubling what's in that diagram. It didn't take me
+long to notice that this wasn't working out, and started to think about a new
+approach.
+
+## New Design
+
+When I started to think about what I really needed in my home network, I
+came up with several principles:
+
+- **IPv6 First**. Using IPv6 makes so many things simpler. No NAT. Subnets
+  don't run out of addresses. Link-local addressing that works, and is useful.
+  No DHCP. *No NAT!*
+- **Zero Trust**. I'm a fan of zero-trust architectures. When you place the
+  responsibility for security on the network, it tends to get complicated. You
+  make physical design choices around isolating packets on the right segments
+  and getting them to the right firewalls, where the network should focus on
+  moving traffic through it as quickly as possible. This principle of simply
+  getting packets to where they need to be is how the Internet scales so well.
+  We can keep physical LANs simple and efficient like this, and leave the
+  security concerns to the endpoints. The endpoints need to be secure anyways,
+  and we now have more modern and effective tools to help us.
+- **Network Fabric**. Rather than redoing the same 3-tier model I was used to,
+  I wanted to do something more efficient. I was inspired by an article
+  written at Facebook about their
+  ["data center fabric"](https://engineering.fb.com/2014/11/14/production-engineering/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/)
+  architecture. This is obviously much larger than what anyone needs in a home
+  network, but I thought that these were good ideas that I could use.
+- **Routing Protocols**. I've traditionally used [OSPF](https://en.wikipedia.org/wiki/Open_Shortest_Path_First)
+  in the networks I've operated. When I decided to implement IPv6 in the
+  network, I was using [Quagga](https://www.nongnu.org/quagga/) for the
+  routing software. It doesn't support OSPF areas in OSPFv3 for IPv6, which
+  made me reconsider. I settled on [IS-IS](https://en.wikipedia.org/wiki/IS-IS),
+  as it supports both IPv4 and IPv6 at the same time, and could do everything
+  that I needed it to do.
+
+### Refresh, Version 1
+
+My first refreshed network design looked like this:
+
+![New Network Diagram, Version 1](first_new_net.svg 'My first updated network architecture')
+
+I did away with all of the traditional complexity, and established two network
+"fabrics" that all of the traffic would pass through. The fabrics do not
+connect directly at layer 2, each side is separate. Each fabric is a single
+flat network, there are no VLANs.
+
+These were the key design decisions:
+
+- **Routing over switching**. Every physical device connecting into the fabric
+  switches would conform to a basic set of requirements:
+  1. It will act as a router, running IS-IS to communicate with every other
+     device on the fabric switches.
+  2. Each endpoint uses a single loopback address as its network identity. It
+     advertises this address to the other nodes, as well as the subnets that
+     it can route to.
+  3. Routes are advertised over both fabrics, enabling [ECMP](https://en.wikipedia.org/wiki/Equal-cost_multi-path_routing)
+     for higher availability and bandwidth.
+- **IPv6 first**. The access routers and Wifi routers only had IPv6 subnets
+  available for client devices. This allowed me to do away with DHCP services
+  on the network, only using [SLAAC](https://en.wikipedia.org/wiki/IPv6_address#Stateless_address_autoconfiguration_(SLAAC)).
+  Access to IPv4-only resources was through the use of
+  [DNS64 and the NAT64 gateway](https://en.wikipedia.org/wiki/IPv6_transition_mechanism#NAT64).
+
+## Next Generation
+
+At this point, I was fairly happy with the result. The network was efficient
+and much easier to maintain. It was faster, thanks to ECMP and having fewer
+hops. As I was using it however, I started to think about the next set of
+improvements.
+
+- **Single point routers**. I had only single devices acting as my edge,
+  access, and Wifi routers. I wanted some redundancy in case one failed, and
+  to make maintenance more transparent with the ability to fail over.
+- **Virtual Machines**. Most of my workloads were set up as virtual machines.
+  I wanted to migrate to [Kubernetes](https://kubernetes.io/) as everything
+  I was running could be run there, along with many other benefits.
+- **NAT64**. Here I was running IPv6 to get away from needing NAT, but I still
+  needed NAT. This setup was mostly working fine, but there were a few small
+  irritations:
+  - There are not very many NAT64 implementations. I was using [JooL](https://jool.mx/);
+    it's a non-standard Linux kernel module, and it's not really actively
+    developed anymore.
+  - The path from the NAT64 gateway out the Edge router is still IPv4, and I
+    still need to do NAT for IPv4 at the edge.
+  - Applications connecting directly to an IPv4 address weren't able to do so.
+    I could use [464XLAT](https://en.wikipedia.org/wiki/IPv6_transition_mechanism#464XLAT)
+    on endpoints that supported it, but it's yet another thing to set up.
+  - There's the occasional device that still doesn't support IPv6, or doesn't
+    support it properly.
+- **BGP**. I was purely using IS-IS throughout the network, but Kubernetes
+  CNIs that work on bare metal systems like mine rely on BGP to advertise
+  routes into the network. I'd have to work out how to incorporate this.
+- **Easier WiFi**. I was using a WiFi router running [OpenWRT](https://openwrt.org/),
+  connecting to both fabrics and running IS-IS just like everything else.
+  OpenWRT is great, but it is challenging to keep devices up-to-date.
+- **Load Balancing**. I didn't have any solution for establishing network
+  load balancing for scale and availability.
+
+### Refresh, Version 2
+
+Incorporating the improvements I wanted to make, here is the resulting network
+architecture:
+
+![New Network Diagram, Version 2](second_new_net.svg 'My current network architecture')
+
+The key changes are:
+
+- **Redundant Routers**. I doubled up the edge and access routers. They can
+  effectively divide the load and fail over when needed.
+- **Anycast for Load Balancing**. I've standardized on making use of
+  [Anycast](https://en.wikipedia.org/wiki/Anycast) addressing for creating
+  load balanced and redundant network services. I'm using this a few ways:
+  - The API server for my Kubernetes cluster is on an anycast address. This
+    address is advertised from the three control plane nodes.
+  - Kubernetes `LoadBalancer` type services allocate an address from a pool
+    and advertise it out from any node that can accept traffic for the
+    service.
+  - My recursive DNS servers providing DNS lookups for the network are on two
+    anycast addresses. Each edge router runs an instance and advertises one of
+    the addresses; this is so I can "bootstrap" the network from the edge
+    routers. I also run the DNS service under Kubernetes, which advertises the
+    same anycast addresses using ordinary `LoadBalancer` services.
+- **IS-IS and BGP**. I took a few passes at getting this right. I first tried
+  to move fully from IS-IS to BGP only. This meant setting up peering
+  using IPv6 link local addresses, which worked, but it was a bit flaky under
+  [FRR](https://frrouting.org/). I settled on using IS-IS on the fabric
+  interfaces only to exchange the IPv6 loopback addresses of each node. I use
+  the globally routable loopback addresses for the BGP peering, which is much
+  easier in practice. All of the other routes (access subnets, Kubernetes
+  networks, anycast addresses, defaults from the edge routers) are exchanged
+  using BGP.
+- **No NAT64**. I decided to do away with NAT64 and provide dual-stack
+  connectivity to the access networks. I set up [Kea](https://www.isc.org/kea/)
+  as a cluster on the two access routers, which is thankfully rather low
+  maintenance.
+- **BGP Extended-Nexthop**. An added bonus to using BGP the way that I am is
+  that I could make use of the [BGP extended-nexthop](https://datatracker.ietf.org/doc/html/rfc8950)
+  capability. The old network with only IS-IS still required me to define IPv4
+  subnets on the switching fabrics, nodes used IPv4 addresses as the next hop
+  gateway addresses for IPv4 routes. With the extended-nexthop capability in
+  BGP, it uses the IPv6 link-local addresses for the next hop under both IPv4
+  and IPv6.
+
+### High Availability
+
+To migrate from single routers to redundant pairs, I needed to figure out a
+few things.
+
+#### Default Routes
+
+With a single edge router, this was easy. With two, it's a bit of a puzzle. My
+ISP doesn't actually provide fully routed IPv6 connectivity with my class of
+service. I do get static IPv4 addresses, however. I've been using Hurricane
+Electric's [tunnel broker](https://tunnelbroker.net/) service to get a routed
+`/48` IPv6 subnet.
+
+With a pair of edge routers, I've set them up with four static IPv4 addresses
+on their Internet-facing interfaces. Each router gets one address. I then have
+two [VRRP](https://en.wikipedia.org/wiki/Virtual_Router_Redundancy_Protocol)
+interfaces, one that I use to terminate the IPv6 tunnel, and the other I use
+for all IPv4 traffic. When both routers are up and running, one will have the
+IPv6 tunnel and the other will have the IPv4 interface. Each one advertises a
+default route for the address family it's taking care of. If one goes down,
+the interface will fail over and everything reconverges rather quickly. IPv6
+connections are unaffected, as the routing is stateless and traffic continues
+to flow normally. IPv4 connections may get interrupted as the NAT state is
+lost.
+
+#### Access Routers
+
+The interfaces facing the client machines provide connectivity for both IPv4
+and IPv6.
+
+The IPv6 configuration is much simpler. FRR can be configured to send router
+advertisements to the subnets. Both routers are configured to advertise their
+presence, as well as the subnet prefixes and DNS information. Client machines
+will pick these up, and then have both routers as their default gateways.
+
+While IPv6 configuration is seamless, IPv4 relies on VRRP to share a `".1"`
+default gateway address, which, though functional, lacks the elegance of
+IPv6's stateless design.
+
+## It Works
+
+After I got this all in place, it was finally possible to build myself a
+working Kubernetes cluster and migrate all of my old services over to it. The
+transition to Kubernetes not only streamlined service management but also laid
+the foundation for future scalability and automation. I'll get into that
+adventure in the next series of articles.