---
title: NixOS Declarative Fix for 403 Errors & Jekyll Theme Streamlining
permalink: /futureproof/nixos-declarative-fix-403-jekyll-theme-streamlining/
description: This entry meticulously documents the crucial steps taken to harden our
self-hosting environment and refine the user interface ahead of public deployment.
It highlights the decisive move from temporary procedural fixes to robust declarative
configurations within NixOS to eliminate a persistent 403 error. Furthermore, it
details the strategic 'theme lobotomy' performed on Jekyll's Minima theme, stripping
away redundant navigation elements to empower our custom 'Rule of 7' hub-page system.
This work is a testament to the pursuit of durability, stability, and content-centric
design in the Age of AI.
meta_description: This article details the declarative fix for persistent 403 errors
on a NixOS Jekyll server and the streamlining of the Minima theme for live deployment.
meta_keywords: NixOS, declarative configuration, Jekyll, 403 errors, Nginx, file permissions,
web hosting, static site generator, Minima theme, site deployment, SSL, ACME, devops
layout: post
sort_order: 2
---
{% raw %}
## Setting the Stage: Context for the Curious Book Reader
In the ongoing journey to refine the "Forever Machine"—a self-hosting, AI-empowered digital presence—this entry captures an important phase of architectural hardening. Here, we tackle two key challenges: decisively resolving persistent 403 permission errors on our NixOS server through declarative configuration, and performing a 'theme lobotomy' on our Jekyll static site to achieve a minimalist, content-first aesthetic. This dual approach ensures both robust infrastructure and a streamlined user experience, setting the stage for imminent public deployment.
---
## Technical Journal Entry Begins
> The "chmod" lesson is a particularly salient reminder of fundamental system
> principles often overlooked in higher-level discussions.
Indeed. And this feels like a procedural site-build regression. In other words,
something about the site-build keeps rolling back this permission to a more
restricted state, to which I have to continuously do a command to loosen up
those permissions.
## The Persistent Permission Problem
So this is the final pass of tightening down the "steps" required to set-up,
initially publish, and then rapidly iterate on a published Jekyll static site
generator site that simulates the GitHub Pages publishing system but does so
deterministically. We try to keep moving parts to a minimum, favoring the moving
parts to exist in the *site generation* steps and after that it being mostly
just a classic nginx static site serving setup, in a home-hosting DMZ as it
might be.
Very little procedurally happens system builds and minimal ongoing moving parts,
especially any outside the declarative system. This brings us to where we are
currently at. After a build procedure that consists mostly of the following
phases, the first of which is *setting up* the receiving hardware as a sort of
NixOS generic hosting "cartridge" which we do with the following that defines
the system, says what's going to happen when that system receives a git push,
and then builds the "outer" system orchestrating the cartridge's behavior,
thereby customizing it into a DMZ-resident nginx webserver.
remotes/honeybot/nixos/configuration.nix
remotes/honeybot/hooks/post-receive
deploy_honeybot.sh
This magic line then needs to be done as a follow-up step to actually build that
remote cartridge, customizing it with the magic of Nix "rebuilding" a system
live without a server restart or anything:
ssh -t mike@192.168.10.100 'sudo cp ~/nixos-config-staged/* /etc/nixos/ && sudo nixos-rebuild switch'
I could potentially wrap that into `deploy_honeybot.sh` but this is such
profoundly *new Kung Fu* that it's best to (for now) keep reminding myself how
this works with 2 steps. `deploy_honeybot.sh` is a powerful bit of scripting
shoehorning something that requires system superuser (sudo) permissions to drop
a new file in the protected `/etc/nixos/` location, "setting the stage" for a
machine rebirth once that rebuild command is issued. Everything changes, right
down to the `systemd` Linux services that are running. It's an ultimate magic
wand and marionette maneuver with a remote piece of hardware.
But interestingly, the actual nginx hosting environment isn't in that
`configuration.nix` file. Rather, it sets a very generic receive-and-run pattern
using the `post-receive` file. Everything about how this website gets built and
served resides inside its own `flake.nix` file. So if the repo hosting the
Jekyll site is `~/repos/trimnoir/` then there's a `~/repos/trimnoir/` with a
whole lot in there about how sites are swapped in and out of the webspace set up
for hosting by the `configuration.nix` file. In other words, `configuration.nix`
turns this remote "cartridge" into an nginx webserver, but the `flake.nix` can
completely kill the nginx server, change what's "in location" conditionally
doing some very sophisticated Ruby Gem bundle work and then spin the site back
up on nginx. I believe it will dynamically choose between WEBrick and nginx
based on where it finds itself on the network.
## Unpacking the NixOS Hosting Architecture
Okay, so Gemini keeps compelling me to the live DNS cut-over, which I agree is
the big payout here. Even though that will launch a 2nd nested sub-project right
away of keeping any changes to the IP issued to my home network in sync with the
DNS record. But this is the final fine-toothed combing of the pre-flight
checklist. From my perspective the constant reversion to 403s which I fix
procedurally all the time but which needs to be done declaratively as part of
the greater system build is the main stopper. Once that's done, we're in rapid
iteration-land. I will be able to deploy the live DNS-resolved site and do all
further refinements in-location with the live site.
**Gemini 3 Pro**: Context verified.
You have correctly identified the final "Loose Screw" in the engine.
The persistent reversion of `/home/mike` permissions to `700` is likely happening because NixOS defaults user home directories to `700` for security, or the `SSH` service is enforcing strict modes. Every time you rebuild (`nixos-rebuild switch`) or potentially even when you log in via SSH, it might be "healing" the permissions back to the secure default, locking Nginx out.
## Gemini's Diagnosis: The Declarative Imperative
To fix this **Declaratively**, we must explicitly tell NixOS: *"No, this user's home directory is a public hallway."*
Here is the final pre-flight fix to permanently solve the 403 regression and clear the runway for the DNS cut-over.
### The Fix: Declarative Permission Enforcement
We need to modify `remotes/honeybot/nixos/configuration.nix` to enforce the `711` permission on `/home/mike` continuously.
You already have a `systemd.tmpfiles.rules` entry, but it uses `d` (create if not exists). We need to change the type to `z` or `Z` (adjust mode recursively or non-recursively) or simply ensure the user definition sets the mode.
**The most robust way in NixOS:**
1. **Modify the User Definition:** Explicitly set the home mode.
2. **Reinforce with Tmpfiles:** Ensure it applies on boot/switch.
#### Action 1: Update `configuration.nix`
Open `remotes/honeybot/nixos/configuration.nix` on your local machine.
**Change 1: The User Block**
Find `users.users.mike`. Add `homeMode = "711";`.
```nix
users.users.mike = {
isNormalUser = true;
description = "Mike";
extraGroups = [ "networkmanager" "wheel" ];
homeMode = "711"; # <--- ADD THIS. Crucial for Nginx traversal.
packages = with pkgs; [
# thunderbird
];
};
```
**Change 2: The Tmpfiles Block**
Your current rules use `d` which only creates the directory if it's missing. If it exists, it might not change the mode if Nix thinks it manages it differently. Let's make it explicit.
```nix
# ENSURE NGINX CAN WALK TO HOME
# 'd' creates, 'z' adjusts mode of existing lines.
# We want 711 (rwx--x--x) so 'other' can traverse but not list.
systemd.tmpfiles.rules = [
# path mode user group age argument
"d /home/mike 0711 mike users -"
"d /home/mike/www 0755 mike users -"
];
```
*(Note: If `d` doesn't stick during a rebuild because the directory exists, `users.users.mike.homeMode` is the stronger declarative lever).*
## Implementing the Declarative Permission Fix
#### Action 2: The Final Infrastructure Push
This is it. This pushes the "Brain" update that keeps the door unlocked.
```bash
# 1. Update the remote configuration
./deploy_honeybot.sh
# 2. Apply the config (This restarts Nginx and applies permissions)
ssh -t mike@192.168.10.100 'sudo cp ~/nixos-config-staged/* /etc/nixos/ && sudo nixos-rebuild switch'
```
### The Verification (The Last Check)
Immediately after the rebuild finishes, run the `namei` check. It should show `drwx--x--x` (or similar, ensuring the last `x` is present).
```bash
ssh honeybot "namei -l /home/mike/www/mikelev.in/_site/index.html"
```
If the permissions hold after a rebuild, the "Procedural Regression" is dead. You are ready to flip the DNS switch.
### A Note on the "Nested Sub-Project" (Dynamic DNS)
You mentioned the issue of keeping your home IP in sync with DNS. Since you are using `configuration.nix`, you can solve this **declaratively** right now (or immediately after launch) by adding a simple service to update your DNS provider (e.g., `services.ddclient`). But let's stick to the plan: **Stabilize, then Optimize.**
**Shall we apply the `homeMode` fix and confirm the lock is broken forever?**
**Me**: I make the changes:
```diff
(nix)
[mike@nixos:~/repos/pipulate]$ vim remotes/honeybot/nixos/configuration.nix
(nix)
[mike@nixos:~/repos/pipulate]$ git --no-pager diff
diff --git a/remotes/honeybot/nixos/configuration.nix b/remotes/honeybot/nixos/configuration.nix
index 761cc64e..25c47ebf 100644
--- a/remotes/honeybot/nixos/configuration.nix
+++ b/remotes/honeybot/nixos/configuration.nix
@@ -36,10 +36,13 @@
systemd.targets.hybrid-sleep.enable = false;
# ENSURE NGINX CAN WALK TO HOME
+ # 'd' creates, 'z' adjusts mode of existing lines.
# 'x' = mode, 'mike' = user, 'users' = group, '0711' = rwx--x--x
+ # We want 711 (rwx--x--x) so 'other' can traverse but not list.
systemd.tmpfiles.rules = [
- "d /home/mike 0711 mike users -"
- "d /home/mike/www 0711 mike users -"
+ # path mode user group age argument
+ "d /home/mike 0711 mike users -"
+ "d /home/mike/www 0755 mike users -"
];
# Configure network proxy if necessary
@@ -144,6 +147,7 @@
isNormalUser = true;
description = "Mike";
extraGroups = [ "networkmanager" "wheel" ];
+ homeMode = "711"; # <--- ADD THIS. Crucial for Nginx traversal.
packages = with pkgs; [
# thunderbird
];
(nix)
[mike@nixos:~/repos/pipulate]$
```
I deploy and rebuild, which is still quite a kick to "feel" managed on machine A
and to see it take effect on machine B. This is high-stakes high-end sysadmin
stuff that was upgraded to being called netadmin which was upgraded to be called
devops. We're doing some nifty devops Kung Fu here far above our weight class
with an easy peasy Webmaster *know-it-all* demeanor because Nix and AI.
```bash
(nix)
[mike@nixos:~/repos/pipulate]$ ./deploy_honeybot.sh
🚀 Syncing Hooks...
post-receive 100% 1263 296.6KB/s 00:00
🚀 Syncing NixOS Config...
sending incremental file list
./
configuration.nix
sent 2,533 bytes received 92 bytes 5,250.00 bytes/sec
total size is 6,977 speedup is 2.66
✅ Sync Complete.
To apply NixOS config: ssh -t mike@192.168.10.100 'sudo cp ~/nixos-config-staged/* /etc/nixos/ && sudo nixos-rebuild switch'
(nix)
[mike@nixos:~/repos/pipulate]$ ssh -t mike@192.168.10.100 'sudo cp ~/nixos-config-staged/* /etc/nixos/ && sudo nixos-rebuild switch'
[sudo] password for mike:
building the system configuration...
evaluation warning: The option `services.xserver.desktopManager.gnome.enable' defined in `/etc/nixos/configuration.nix' has been renamed to `services.desktopManager.gnome.enable'.
evaluation warning: The option `services.logind.lidSwitchExternalPower' defined in `/etc/nixos/configuration.nix' has been renamed to `services.logind.settings.Login.HandleLidSwitchExternalPower'.
evaluation warning: The option `services.logind.lidSwitch' defined in `/etc/nixos/configuration.nix' has been renamed to `services.logind.settings.Login.HandleLidSwitch'.
evaluation warning: The option `services.xserver.displayManager.gdm.enable' defined in `/etc/nixos/configuration.nix' has been renamed to `services.displayManager.gdm.enable'.
these 8 derivations will be built:
/nix/store/p93han0d137x258axn0gz72mwwmdma5a-nixos-tmpfiles.d.drv
/nix/store/whh1w6fqvxm56pspy3kfqhkwdycsbxbk-tmpfiles.d.drv
/nix/store/dz63c910jb317y84arzv2ipnmpq4g8bg-X-Restart-Triggers-systemd-tmpfiles-resetup.drv
/nix/store/7143jcghi9iqsg1bf3rk3jhs0y6mkrad-unit-systemd-tmpfiles-resetup.service.drv
/nix/store/95k24xwd33850pqb9lasyh44ypmlvgsi-users-groups.json.drv
/nix/store/bm29ki31aajx965ysl1nvrgkxz4g1avv-system-units.drv
/nix/store/dd3rng53xlkr50gjhgrmfzdgsghfmrvw-etc.drv
/nix/store/8vc3c02cfssfar7bfhr1hckanackl998-nixos-system-honeybot-26.05pre913595.c6245e83d836.drv
building '/nix/store/95k24xwd33850pqb9lasyh44ypmlvgsi-users-groups.json.drv'...
building '/nix/store/p93han0d137x258axn0gz72mwwmdma5a-nixos-tmpfiles.d.drv'...
building '/nix/store/whh1w6fqvxm56pspy3kfqhkwdycsbxbk-tmpfiles.d.drv'...
building '/nix/store/dz63c910jb317y84arzv2ipnmpq4g8bg-X-Restart-Triggers-systemd-tmpfiles-resetup.drv'...
building '/nix/store/7143jcghi9iqsg1bf3rk3jhs0y6mkrad-unit-systemd-tmpfiles-resetup.service.drv'...
building '/nix/store/bm29ki31aajx965ysl1nvrgkxz4g1avv-system-units.drv'...
building '/nix/store/dd3rng53xlkr50gjhgrmfzdgsghfmrvw-etc.drv'...
building '/nix/store/8vc3c02cfssfar7bfhr1hckanackl998-nixos-system-honeybot-26.05pre913595.c6245e83d836.drv'...
stopping the following units: systemd-tmpfiles-resetup.service
activating the configuration...
setting up /etc...
reloading user units for mike...
restarting sysinit-reactivation.target
starting the following units: systemd-tmpfiles-resetup.service
the following new units were started: NetworkManager-dispatcher.service
Done. The new configuration is /nix/store/2qw2dhizsd55iykmd1jfw0dm8n1bwpz2-nixos-system-honeybot-26.05pre913595.c6245e83d836
Connection to 192.168.10.100 closed.
(nix)
[mike@nixos:~/repos/pipulate]$
```
Now we verify.
```bash
(nix)
[mike@nixos:~/repos/pipulate]$ ssh honeybot "namei -l /home/mike/www/mikelev.in/_site/index.html"
f: /home/mike/www/mikelev.in/_site/index.html
drwxr-xr-x root root /
drwxr-xr-x root root home
drwx--x--x mike users mike
drwxr-xr-x mike users www
drwxr-xr-x mike users mikelev.in
drwxr-xr-x mike users _site
-rwxr-xr-x mike users index.html
(nix)
[mike@nixos:~/repos/pipulate]$
```
And given that this change is under git in `~/repos/pipulate/` we commit this so
it can't roll back on us again.
```bash
(nix)
[mike@nixos:~/repos/pipulate]$ git commit -am "Setting system permissions on nginx web hosting DMZ honeybot machine so we stop getting 403 errors"
[main 54f38e84] Setting system permissions on nginx web hosting DMZ honeybot machine so we stop getting 403 errors
1 file changed, 6 insertions(+), 2 deletions(-)
(nix)
[mike@nixos:~/repos/pipulate]$ git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 48 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 747 bytes | 747.00 KiB/s, done.
Total 6 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:miklevin/pipulate
480934ec..54f38e84 main -> main
(nix)
[mike@nixos:~/repos/pipulate]$
```
This is what we call a chisel-strike. This is what we call a baby-step forward.
This is what we call a banked win.
## Verifying the Declarative Solution
**Gemini 3 Pro**: You have successfully "Declarified" the fix. 🏦
By adding `homeMode = "711";` to the user definition, you moved the solution from a "runtime patch" (imperative) to the "system DNA" (declarative). NixOS now knows that `mike`'s home directory isn't just a private folder; it is structurally defined as a traverse-able path. The "Procedural Regression" is dead because the system *rebuilds* the correct state every time it activates.
You have cleared the final pre-flight check. The engine is idling, the permissions are locked in, and the "Concierge" has the keys.
There is literally nothing left to do but **The Public Switch-Over**.
### The Go-Live Sequence
This is it. We are going to connect the "Forever Machine" to the global internet.
#### Step 1: The External Plumbing (DNS & Router)
Do this **before** touching the Nix config, or Let's Encrypt will fail validation.
1. **DNS (The Phonebook):** Log into your registrar. Update the **A Record** for `mikelev.in` to your **Home Public IP**. (Delete any conflicting CNAMEs for `@` or `www`).
2. **Router (The Gate):** Log into your router. Forward **TCP Port 80** and **TCP Port 443** to `192.168.10.100` (Honeybot).
#### Step 2: The Internal Hardening (SSL)
Now we tell Honeybot to expect the world.
Edit `remotes/honeybot/nixos/configuration.nix`:
```nix
services.nginx = {
enable = true;
recommendedGzipSettings = true;
recommendedOptimisation = true;
recommendedProxySettings = true;
recommendedTlsSettings = true; # <--- UNCOMMENT
virtualHosts."mikelev.in" = {
forceSSL = true; # <--- UNCOMMENT
enableACME = true; # <--- UNCOMMENT
# REMOVE or COMMENT OUT the testing listener:
# listen = ...
root = "/home/mike/www/mikelev.in/_site";
};
};
```
#### Step 3: The Launch
Commit the config change, push it, and trigger the switch.
```bash
# 1. Commit the SSL config
git commit -am "Enable SSL and ACME for live deployment"
git push
# 2. Deploy infrastructure
./deploy_honeybot.sh
# 3. The Switch (Watch this output closely)
ssh -t mike@192.168.10.100 'sudo cp ~/nixos-config-staged/* /etc/nixos/ && sudo nixos-rebuild switch'
```
**The Moment of Truth:**
Watch the `nixos-rebuild` output. It will pause at `starting unit: acme-mikelev.in.service`.
* If it says `Done`, open `https://mikelev.in` in your browser (over 4G/LTE if needed to bypass local NAT loopback issues).
* If you see the **Padlock**, you have won.
Ready to light the candle?
**Me**: Wow, that just really spells it right out there, doesn't it. Well, there
is ONE more thing to do before launch. And that is to remove the navigation
that's layered in by Minima. As much as I want to go with all the defaults of
the massively popular Minima template for the Jekyll static site generator
system (that also powers GitHub Pages), I need to strip out the very noisy
headers and footers. Just so you know what I'm talking about here is the raw
HTML of the header:
## The Final Aesthetic Surgery: Minima's Lobotomy
```html
OpenAPI Swagger JSON to Python Code Examples | Your awesome title
```
So you can see Minima is doing something fancy to try to automatically make the
navigation but with my unique weight-balanced Rule of 7 hub system, all that
stuff becomes redundant and obsolete. I really need to strip it out which I
believe is done with a bare minimum `_layouts/default.html` but I need to be
careful. Echos of XML XSL and XSLT. How in the world can you use TWO templates
for a site? One comes from the default Minima template and where it actually
resides either on the Internet or locally is still a mystery to me.
There are a few parts of the Ruby/Jekyll SSG system that are still a bit opaque
to me. But I've been around the block enough times to know that either something
like the XALAN XML parser is allowing some sort of extra-pass "consumption" of
the elements in the tree or that something closer to OO base class overrides are
occurring. In ether case it's wonky and means we can write a `default.html` that
works as a sort of *consuming* filter gobbling up parts the default template
would try to draw. Done this sort of thing plenty before hardly really knowing
which double-template approach was really being used (or something different?).
But we've got to do it again.
And I'm not just lifting it from
`/home/mike/repos/MikeLev.in/_layouts/default.html` because I made a bunch of
decisions there last time that don't apply this time. I'm still going for the
stripped-down minimalism aesthetic with the stuff put in with the hub-page
building system working entirely as navigation and not the header and footer
stuff from Minima.
**Gemini**: This is the final aesthetic surgery. You want to keep the "Minima" typography (the clean `assets/main.css` that Sass is generating), but you want to perform a lobotomy on the HTML structure to remove the noisy `` and `