NixOS Raspberry Pi 4 Google Fiber Router

:: computers, networking, nixos, raspberry-pi

My home network is weird, and it’s taken me, ... uh... way too much work to get it set up how I want it. I’ve decided to post the bulk of my current router configuration as a help to other people who are similarly weird.

Currently my ISP is Google Fiber. I have some reservations about the world’s largest corporate surveillance company managing my home internet connection and therefore handling literally all of my traffic. They claim they don’t keep any of that data or correlate it with accounts. I bet the don’t now, but I don’t think we should feel comfortable putting our infrastructure eggs in a basket controlled by a tracking company that could change their minds on this in the future, or just accidentally store and correlate data.

That said, I love having fiber internet, and I was excited to get Google Fiber instead of cable crap from Comcast. Also, who could have predicted it, but a new ISP coming to town made Comcast suddenly, inexplicably, drastically improve their offerings as well. That’s a funny and unexpected turn of events. Maybe Google’s plan to spur ISP competition actually had some sense to it! Good job, Google! If only creating ISP competition weren’t full of nonsense legal red tape lobbied in by the current monopolies, it might actually have worked at a large scale! Some day I hope we will gather the political will to make sensible, locally controlled municipal fiber networks throughout the country. I can dream. At any rate, I hate Comcast and it felt good to give them the boot, at least temporarily©.

©: Watch me move to a region where Comcast is a practical monopoly again once I graduate.

Initially Google Fiber offered two tiers of service: 100 Mbps (symmetric up and down) for $50/month or 1 Gbps (symmetric) for $70/month. As much as I love gigabit, the difference is not huge for me right now. My household only contains 2 internet users, I don’t think my wife does anything where she would ever notice the difference, and I rarely do. (Though I would frequently notice if my home LAN were limited to 100 Mbps.) Part of me wanted the 10x speed, which I likely would have bought if I were single, but my wise and thrifty wife helped bring me down to earth to get the sensible plan. At any rate, Google Fiber recently eliminated their 100 Mbps tier, but grandfathered current customers to a $55/month 500 Mbps tier. This is actually the best of both worlds for me – it’s only an extra $60/year instead of an extra $240/year, and I get half-way to gigabit glory.𝐆

𝐆: Frankly, full gigabit requires better hardware than I use anyway – many things that are labelled as “1000 Mbps” in theory only handle a fraction of it in practice, maybe 200-700 Mpbs. Maybe I just need better cables, better than short CAT-6 cables? But I digress.

The router provided by Google was unsatisfactory. It constantly dropped wireless connections for some reason. Additionally, it can only be configured by logging into a Google account so they can send it configuration commands, and there is no way to turn off or otherwise configure its IPv6 firewall. Bogus!

At any rate, I prefer using a Linux box as a router anyway, since it’s easier to administer and, critically, I can save the configuration in a git repository and easily restore it. Recently I decided to try out a Raspberry Pi 4 to see how well it does, and it can manage the full half gigabit in either direction. One of its four cores gets pegged around 100% while sustaining a half-gigabit load, so it may not be suitable for full gigabit or a constantly loaded network (due to eventual overheating). However, it’s much lower power than the desktop computer I was using previouslyπ, and I’m not worried about overheating with a combination of heat sinks and generally only short spurts of peak network load. So I’m going to continue using it going forward.

π: The Raspberry Pi 4 uses around 3 watts idle and 7 watts at full load with all 4 cores. An old desktop uses something like 20 times that. Granted, the desktop was running a bunch of other things as well, but I plan to separate the various services to different, low-powered computers for both power savings and security improvements.

Setting up a Linux box to route IPv6 was poorly documented when I first did it, and probably still is. Additionally, Google Fiber has some nonsense requirements for its routers that are hard to find all in one place (Google certainly doesn’t tell their customers directly). I sank way more time into setting this up (and later porting it to NixOS) than I should have. At some point it hit that “this shouldn’t be that hard, and probably isn’t worth it, but gosh darn it I’m not going to let this beat me” stage of irrationality, though, so here we are. Here, for the world to see, is the current incaranation of this madness, with lots of comments for explanation.

△: Note that I’ve slimmed this down a bit from my actual configuration – there are some extra bits and conveniences that mine includes that are not really relevant to the router aspect of the configuration, and I’ve redacted some things specific to my network and hardware. But this configuration should work, or at worst have a couple typos from editing down after copy-pasting it from my actual configuration.

#/etc/nixos/configuration.nix
{ lib, config, pkgs, ... }:

# Here we define variables containing the names of our network interfaces.
# Actually, using ‘services.udev.extraRules‘ you can make udev give
# your interfaces consistent, memorable names!
# However, there is some bug I can’t get things to work if I use
# udev to rename the external interface.
# Your interface names may vary.
let
INTERNALINTERFACE = "eth0";
EXTERNALINTERFACE = "enp1s0u1";
in
{

  #### Booting

  # NixOS wants to enable GRUB by default, but we are using a Raspberry Pi.
  # Actually, this boot configuration should eventually be outdated, and
  # a more generic loader should work.  But for now, this boot mode works.
  boot.loader.grub.enable = false;
  boot.loader.raspberryPi = {
    enable = true;
    version = 4;
  };

  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.initrd.availableKernelModules = [ "usbhid" ];

  fileSystems."/" = {
    device = "/dev/disk/by-label/NIXOS_SD";
    fsType = "ext4";
  };
  fileSystems."/boot" = {
    device = "/dev/disk/by-label/FIRMWARE";
    fsType = "vfat";
  };


  #### Networking

  networking.hostName = "raspberry-pi-google-fiber-router";

  services.udev.extraRules =
    ”
    # By using udev extra rules, you can change interface names!
    # It’s really convenient... except that I can’t seem to get routing
    # to work if I change the name of the external interface.
    # Changing the name of the internal interface seems to work, though.
    #SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="12:34:56:78:9a:bc", NAME="better-than-enp0s2u9crap0"
    ”;


  # You’d better forward packets if you actually want a router.
  boot.kernel.sysctl = {
    "net.ipv4.ip_forward" = 1;
    "net.ipv6.conf.all.forwarding" = 1;
    "net.ipv6.conf.default.forwarding" = 1;
  };


  # Google fiber requires your router talk to them over vlan 2.
  # Genius network design, or jerk lockout move to keep
  # people using the equipment they want you to use because
  # vlans are generally poorly supported, complicated to
  # set up, and virtually unknown except to networking
  # professionals?  You decide.
  networking.vlans."wan" = {
    interface = EXTERNALINTERFACE;
    id = 2;
  };

  # Oh, also did I mention they will throttle you down to ~10 Mbps
  # if you don’t apply this egress map on the vlan?
  # Genius networking, or jerk lockout move that is even less supported,
  # more complicated, and less well known than vlans?  You decide.
  systemd.services.wan-egress-map = {
    enable = true;
    after = ["wan-netdev.service"];
    wantedBy = ["multi-user.target"];
    description = "set egress map required for full upload speed on Google Fiber";
    serviceConfig = {
      ExecStart = "${pkgs.iproute}/bin/ip link set wan type vlan egress 0:3";
      Type = "oneshot";
    };
  };


  # You probably want to configure a firewall.
  # I think firewalls are mostly stupid.
  #
  # Actually, I would love a protocol to tell router what firewall
  # settings you want for each client.  Then phones could say “Firewall
  # anything I’m not expecting so I can preserve battery life!”, while
  # desktops could say “Don’t bother firewalling anything because I’ll
  # just drop any packets I’m not expecting by myself.”  Generally I
  # would really love for my globally unique IPv6 address to mean I can
  # just host any services on the internet from any device I want.  Alas,
  # security cargo culting gives us firewalls everywhere.  Unless you
  # explicitly turn off firewalls on the router in your path you can’t
  # host anything without falling back to the crazy NAT hole punching
  # nonsense required with IPv4.  Oh, and you probably only control the
  # firewall on your home router.  So much for mobile, roaming internet
  # services that you might actually, you know, make an intelligent,
  # informed choice to host.
  networking.firewall.enable = false;


  networking.interfaces.wan.useDHCP = true;
  # You don’t actually want this on a Google Fiber router, but if you were
  # to use a Raspberry Pi 4 router on some more sensible network, you would
  # want this line on.
  #networking.interfaces.${EXTERNALINTERFACE}.useDHCP = true;

  networking.dhcpcd = {
    enable = true;
    allowInterfaces = [
      "wan"
      INTERNALINTERFACE
    ];
    extraConfig =
      ”
      # The man page says that ipv6rs should be disabled globally when
      # using a prefix delegation.
      noipv6rs

      interface wan
      # On the wan interface, we want to ask for a prefix delegation.
      ipv6rs
      ia_pd 2/::/60 ${INTERNALINTERFACE}/0/64

      # We don’t want dhcpcd to give us an address on the internal interface.
      interface ${INTERNALINTERFACE}
      noipv4
      ”;
  };

  # Now we set up our NAT for ipv4.
  networking.interfaces.${INTERNALINTERFACE}.ipv4 = {
    addresses = [{address = "192.168.1.1"; prefixLength = 24;}];
    routes = [{address = "192.168.1.0"; prefixLength = 24;}];
  };

  # Port forwarding is a finnicky thing.
  # The built-in port forwarding options only get us half way there.
  # We need to add additional rules to get hairpin NAT working.
  # (Hairpin NAT means you see the same translations inside the
  # network as outside it.)  Unfortunately, I still haven’t figured
  # out how to get the same NAT translations to work for the router
  # itself.  Clearly there is room for improvement in the built-in
  # port forwarding configuration.  But I’m not about to figure out
  # the hairpin-for-localhost issue and try to get things merged.

  networking.nat = {
    enable = true;
    #externalInterface = EXTERNALINTERFACE;
    externalInterface = "wan";
    internalInterfaces = [INTERNALINTERFACE];
    internalIPs = ["192.168.1.0/24" "127.0.0.1/32"];
    forwardPorts = [
      # For destination port range use "x.x.x.x:start-end".
      # For source port range use "start-end" in a string.
      {sourcePort = 19; destination = "192.168.1.19:19"; proto = "tcp";}
    ];
    extraCommands =
      ”
      # This is the first extra rule needed for hairpin NAT.
      # It can be done statically.
      iptables -t nat -A nixos-nat-post -o ${INTERNALINTERFACE} -s 192.168.1.0/24 -d 192.168.1.19/32 -p tcp -m tcp –dport 19 -j SNAT –to-source 192.168.1.1
      ”;
  };

  # The final NAT rule requires the IP address of your external interface,
  # and therefore must be done dynamically if you don’t have a static IP!
  # Let’s run it using the dhcpcd run hook.
  networking.dhcpcd.runHook = "/etc/nat-hairpin-rules";
  environment.etc.nat-hairpin-rules = {
    text =
      ”
      #!${pkgs.bash}/bin/bash
      # I perhaps ought to figure out for sure which packages each of these
      # belongs to and clutter the script with NixOS package paths.
      # But instead, let’s just source the profile.
      source /etc/profile
      ext_ip="$(ip -4 addr show dev wan | grep inet | awk ’{print $2}’ | cut -d / -f 1)"
      iptables -t nat -A nixos-nat-pre -i ${INTERNALINTERFACE} -s 192.168.1.0/24 -d "$ext_ip" -p tcp -m tcp –dport 19 -j DNAT –to-destination 192.168.1.19
      ”;
    mode = "0555";
  };


  # We need to run a DHCP server.  Let’s also reserve some IP addresses.
  # Additionally, we can set which DNS servers to use, a domain to search in,
  # etc.
  services.dhcpd4 = {
    enable = true;
    interfaces = [INTERNALINTERFACE];
    machines = [
      # Keep this in sync with home.williamhatch.net.zone file
      {hostName = "some-host"; ipAddress = "192.168.1.19"; ethernetAddress = "11:22:33:44:55:66";}
    ];
    extraConfig =
      ”
      #option domain-name "my.domain.tld";
      #option domain-name-servers 192.168.1.1;
      option domain-name-servers 1.1.1.1, 8.8.8.8;
      option subnet-mask 255.255.255.0;
      default-lease-time 604800;
      max-lease-time 60480000;
      subnet 192.168.1.0 netmask 255.255.255.0 {
        range 192.168.1.100 192.168.1.200;
        option subnet-mask 255.255.255.0;
        option broadcast-address 192.168.1.255;
        option routers 192.168.1.1;
      }
      ”;
  };

  # We also want to supply our network with IPv6 addresses, of course,
  # so they can be first-class citizens of the global network.
  services.radvd = {
    enable = true;
    config =
      ”
      interface ${INTERNALINTERFACE}
      {
        AdvSendAdvert on;
        # Advertise at least every 30 seconds
        MaxRtrAdvInterval 30;
        # This special prefix detects what the actual prefix
        # available is dynamically.  This should be the default.
        prefix ::/64
        {
        };
      };
      ”;
  };


  # You could set your router up to be a DNS server as well!
  #services.bind = {
  #  enable = true;
  #  # cacheNetworks controls who can use this DNS resolver RECURSIVELY.
  #  # It should be disallowed except for local requests.
  #  # Otherwise your DNS server could be used in bogus amplification attacks!
  #  cacheNetworks = ["127.0.0.0/24" "192.168.1.0/24"];
  #  forwarders = [
  #    # IE where does this DNS server look up things it doesn’t know.
  #    # Cloudflare
  #    "1.1.1.1"
  #    "1.0.0.1"
  #    # Google
  #    "8.8.8.8"
  #    "8.8.4.4"
  #  ];
  #};


  services.xserver.enable = false;
  services.openssh.enable = true;

  users.mutableUsers = false;
  users.users.yourusername = {
    isNormalUser = true;
    extraGroups = ["wheel"];
    hashedPassword = "output the result of ‘mkpasswd -m sha512‘ here";
    openssh.authorizedKeys.keys = ["your-ssh-key-here"];
  };

  # I could be missing some things, since I stripped some things out of my
  # actual configuration to arrive at something minimal for a blog post.
  # I think these are the packages needed for the hairpin nat script.
  # Of course, for your system administration convenience you likely want
  # some other things.  Say, an editor?
  environment.systemPackages = [
    pkgs.iproute
    pkgs.gawk
    pkgs.gnugrep
    pkgs.iptables
  ];

}