kokada's blog

My coding experience with LLM

Mon, 02 Sep 2024 00:00:00 +0000

I don't want to talk about AI. AI fatigue is real, with (garbage) AI content everywhere you go. However, I like to talk about tools, and AIs, or more specifically, Large Language Models (LLM), can be really useful tools. Maybe someday we will have our AI overlords dominating everything (including my job), but right now I don't feel particularly threatened given my experience with AIs.

Given that Google Search is getting worse, I am slowly relying in LLMs more and more. This is the focus of this post, basically the good, the bad and the ugly (by the way, never saw this film) of LLMs, especially focused for solving coding problems. I will take a look at the my chat history in ChatGPT and see where it was a good tool, where it shined, and where it just completely wasted my time.

A disclaimer, I never bothered to buy a ChatGPT subscription because I don't feel that the tool is useful enough (at least yet), so most of the examples here are either with GPT-3.5 or GPT-4o mini, since those where the models offered by free in ChatGPT.

An addendum, I have some experience with GitHub Copilot (another LLM, but this one runs inside your IDE or favorite text editor) in $PREVIOUS_JOB. But my experience with it is even more underwhelming than what I am going to describe in this post: it was basically really good for generating fixture for tests and between ok-ish to plain bad for everything else. Yes, that's would fit in a tweet (or xeet?).

The good

Chat history.

Is there any way that I can pass a context for a net.conn to cancel its Read()?

This is one of the good cases where ChatGPT felt to me that I was talking with an expert in Go. To start, I didn't even mention in my initial prompt that I was talking about Go, but it correctly inferred so. Also, the solution overview was great:

Use a separate goroutine to handle the context cancellation.

Set a deadline on the connection using SetReadDeadline() to unblock the Read() when the context is canceled.

Before asking ChatGPT about this issue I had already coded my own naive solution that was something similar to this:

func foo(ctx context.Context) {
    // ...
    done := make(chan struct{})
    var n int
    var err error

    go func() {
        n, err = conn.Read(buf)
        done <-struct{}{}
    }()

    select {
    case <-ctx.Done():
        conn.Close()
        return 0, ctx.Err()
    case <-done:
        return n, err
    }
    // ...
}

Compared this solution with the one from ChatGPT:

func readWithContext(ctx context.Context, conn net.Conn, buf []byte) (int, error) {
    done := make(chan struct{})
    var n int
    var err error

    // Start a goroutine to perform the read
    go func() {
        n, err = conn.Read(buf)
        close(done)
    }()

    select {
    case <-ctx.Done(): // Context was canceled or timed out
        // Set a short deadline to unblock the Read()
        conn.SetReadDeadline(time.Now())
        <-done // Wait for the read to finish
        return 0, ctx.Err()
    case <-done: // Read finished successfully
        return n, err
    }
}

The ChatGPT version is much better because:

It doesn't close the connection socket allowing it to be reused
It waits until the goroutine is finished to avoid leaks
It closes the channel after it finishes

Now, I don't claim to be a Go expert (far from it), but this was one of those cases where I felt that I was being guided by someone that is coding in Go for years.

But it didn't end there, I asked:

If I want to reuse the connection, do I need to reset the SetReadDeadline?

Since in this particular case I wanted the underlying connection to be reused, and ChatGPT correctly amended the code:

func readWithContext(ctx context.Context, conn net.Conn, buf []byte) (int, error) {
    // Same as before
    select {
    case <-ctx.Done(): // Context was canceled or timed out
        // Set a short deadline to unblock the Read()
        conn.SetReadDeadline(time.Now())
        <-done // Wait for the read to finish

        // Reset the deadline so the connection can be reused
        conn.SetReadDeadline(time.Time{})
        return 0, ctx.Err()
    case <-done: // Read finished successfully
        return n, err
    }
}

Also the explanation on why this happens was good:

When you set a read deadline with conn.SetReadDeadline(time.Now()) to unblock the Read() call, this deadline remains in effect for future Read() operations. This means that if you attempt another read operation after this, it could immediately fail with a timeout error if the deadline is still in effect. To reuse the connection, you'll need to remove the deadline or set a new one. You can do this by calling conn.SetReadDeadline(time.Time{}), which clears any existing deadline.

I had one last question that I kind already knew the answer:

Do we need to defer the clearing?

And ChatGPT correctly returned:

Whether you need to defer the clearing of the read deadline depends on how your code is structured. If you want to ensure that the read deadline is always cleared after a context cancellation, even if the code panics or exits early, then deferring the conn.SetReadDeadline(time.Time{}) call is a good idea.

This whole interaction was one of the best ones I ever had in ChatGPT, one that it gave me an actual functional code with good explanation on what was happening. But still, it didn't talked at all about the gotcha of setting the read deadline and reusing the underlying connection, needing another prompt to give me this information.

Also, there is one issue in ChatGPT solution though: we are discarding the values of the buffer once the context is done by returning n = 0 instead of the number of bytes read. In this particular case this is what I wanted, but this is a possible issue that ChatGPT didn't talk.

For comparison, this is the best that I got searching in both DuckDuckGo and Google, that is where I kind based my first solution on. I still not sure where ChatGPT could give me such a good answer for this question, because I just couldn't find anything remotely similar during my searches.

The amazing

Chat history.

I want to write an extension to https://github.com/yuin/goldmark/ that allows me to rewrite [link](../2024-07-30/post.md) to [link](https://kokada.capivaras.dev/slug)

Here is the generated code:

package linkrewriter

import (
	"fmt"
	"path"
	"strings"

	"github.com/yuin/goldmark"
	"github.com/yuin/goldmark/ast"
	"github.com/yuin/goldmark/parser"
	"github.com/yuin/goldmark/renderer"
	"github.com/yuin/goldmark/util"
)

// LinkRewriter is the main struct for your extension
type LinkRewriter struct{}

// NewLinkRewriter returns a new instance of LinkRewriter
func NewLinkRewriter() *LinkRewriter {
	return &LinkRewriter{}
}

// Extend will be called by Goldmark to add your extension
func (e *LinkRewriter) Extend(m goldmark.Markdown) {
	m.Parser().AddOptions(parser.WithASTTransformers(util.Prioritized(e, 0)))
}

// Transform is the method that modifies the AST
func (e *LinkRewriter) Transform(node *ast.Document, reader text.Reader, pc parser.Context) {
	ast.Walk(node, func(n ast.Node, entering bool) (ast.WalkStatus, error) {
		if !entering {
			return ast.WalkContinue, nil
		}
		if link, ok := n.(*ast.Link); ok {
			rewriteLink(link)
		}
		return ast.WalkContinue, nil
	})
}

// rewriteLink modifies the link URL
func rewriteLink(link *ast.Link) {
	url := string(link.Destination)
	if strings.HasPrefix(url, "../") {
		// Modify this part according to your slug generation logic
		slug := generateSlug(url)
		link.Destination = []byte(fmt.Sprintf("https://kokada.capivaras.dev/%s", slug))
	}
}

// generateSlug generates a slug from the given URL
func generateSlug(url string) string {
	// Assuming the slug is the last part of the URL without the extension
	return strings.TrimSuffix(path.Base(url), path.Ext(url))
}

package main

import (
	"github.com/yuin/goldmark"
	"github.com/yuin/goldmark/renderer/html"
	linkrewriter "path/to/your/linkrewriter"
)

func main() {
	md := goldmark.New(
		goldmark.WithExtensions(
			linkrewriter.NewLinkRewriter(),
		),
		goldmark.WithRendererOptions(
			html.WithHardWraps(),
		),
	)

	// Use `md.Convert` to convert your markdown to HTML as usual
}

This one will need some context. Goldmark, the Markdown renderer that powers this blog, does not have good examples or documentation (at least I didn't found neither).

So I was not really hopeful I would get any answers in ChatGPT, but not only it put me in the correct path, the code was almost correct. It did had some unused and missing imports, and there were some compilation errors that were easily fixed. And when it registered the extension, it also added a html.WithHardWraps() renderer for no reason at all.

But this is one of the experiences in LLM that most impressed me. I had no idea at all on how to solve this problem, and ChatGPT basically gave me the solution. A modified version of the code from this chat is still used to power this blog to this day, that also demonstrates that the code is not garbage (since now I understand what it does and I can even extend it).

I am still not sure how the license situation of this code is though. Does licenses even make sense in AI generated code?

The bad

Chat history.

I want to configure VM options inside IntelliJ by project

My project uses sbt, and I want to set this in a way that I can share with my colleagues

I am using a build.sbt file, but IntelliJ is not loading it's JVM options and I need to set this up manually. I want to automate this

This is an example of my most common interaction with ChatGPT: it generally gives some options that could work, but not in my specific case. So I start to ask for more information until it eventually starts to hallucinate.

In this particular case, the hallucination was in:

The sbt-idea-plugin can generate IntelliJ IDEA project files based on your SBT configuration, including JVM options. This allows you to ensure that IntelliJ loads the correct settings from the SBT configuration.

Steps:

Add the sbt-idea-plugin to your project/plugins.sbt file:
addSbtPlugin("com.github.mwz" % "sbt-idea" % "2.4.0")
Run the following command in your terminal to regenerate IntelliJ IDEA project files:
sbt gen-idea

Yes, there is a sbt-idea-plugin, but no, it doesn't generate project files based in my SBT configuration. It is not even from the repository that it is supposed to be pointing too.

I will just not say that this interaction was a waste of time because some of the other ideas can be applied, but this was the kind of case where I ended up just being disappointed because the thing that seemed to be a possible solution for my problem ended up not existing at all.

The ugly

There are lots of examples where ChatGPT just plain disappointed me. Here is a few:

What is the language that is a fork of Go that has extra features like map?

Chat history.

This is one of those cases where I saw something once and was trying to remember. ChatGPT just gave me lots of answers that were not a fork of Go (V, Gluon, Pony), hallucinated a programming language (Gomini), and said that Zig is focused in STEM.

By the way, I eventually remember what the language was: Go+. To be fair with ChatGPT, I misremembered that the Go+ had map (as the high-order function, not a hashmap that both Go and Go+ have). But still, Go+ explicit announces that it is a language for STEM everywhere in their website, and even them ChatGPT couldn't answer.

In IntelliJ, I need to pass some VM options to the run configuration for my application to run tests, e.g.:
--add-exports=java.base/sun.security.x509=ALL-UNNAMED
--add-opens=java.base/sun.security.ssl=ALL-UNNAMED
-Dconfig.resource=api.test.conf
However, the api.test.conf may be jobs.api.conf or something else depending on the subproject I am running inside Idea. Can I parametrize this in some way?

Chat history

So quick answer: you can't. IntelliJ macros are not powerful enough for this. ChatGPT itself said so when I asked about the IntelliJ macros directly.

But in this particular case, ChatGPT happily said "yes, you can parameterize VM options in IntelliJ IDEA to make them more flexible depending on the subproject or other context". Thanks for wasting my time, I guess.

Conclusion

I do think that LLMs, in particular ChatGPT since this is the one I have more experience with, are improving. But still, if there is anything that I concluded after putting all those experiences in a post is that you still need someone that knows at least as much about whatever you're asking ChatGPT to actually validate the answer.

Not only because LLMs hallucinates a lot, but also because even in the "good" example the LLM needed extra input to figure out about the possible drawback of the solution. The only case where I felt that ChatGPT outsmarted me was the "amazing" case, but this was literally the only example that I could find in all my chat history with ChatGPT that was like this.

So yes, at least for now I still don't think AI will be our overlords. Maybe someday.

Making a blog for the next 10 years

Sat, 24 Aug 2024 00:00:00 +0000

So one thing that I realise after starting writing this blog is that I care more about it than some of my other projects. For some reason or another I want to make sure that this blog will continue with me for a long time. This is one of the reasons why I use GitHub as mirror blog and why I created a bunch of automation to make sure I never forget how to maintain this blog.

Still, there are a bunch of dependencies that I need to make sure they're working so I can publish blog posts:

Markdown
A text editor
Hosting and blog platform
Shell script and Makefile
Go toolchain and modules

Let's start with the most important one: the texts themselves, they're all written in Markdown. The nice thing about Markdown is that it is just plain text files with some special notation that allow you to format text, but the text itself is still legible even if you have no support to render Markdowns. So it means as long as I can read a plain text file, I can still read the posts, no issue there. Closely related issue here is the text editor, but the nice thing about Markdown is that if tomorrow I decide to change from neovim to, say, Zed, I can still write it Markdown files without issue. I also use a standardised Markdown implementation called CommonMark, that fixes a bunch of ambiguities of the original Markdown spec.

The next point is the host (capivaras.dev) and the blog platform (Mataroa). One of the nice things about Mataroa is that it avoids platform lock-in by having multiple ways to export your posts. I could export everything to Hugo, for example, and serve the blog from GitHub Pages.

This is especially nice consider how small capivaras.dev is, so it is good to know that if I eventually have issues I could move to somewhere else. I also have an e-mail backup every month with all posts made (another Mataroa feature), and of course I have a Git repo that also acts as a mirror of this blog. So I would say the chance of losing access to the content is slim.

One other issue is the URL of the posts that are indexed in many different places, like Hacker News, Lobte.rs, etc. This is why I finally decide to bite the bullet and purchase a proper domain, and this website should now be available in kokada.dev. This means that in my eventual exit from capivaras.dev, I can just point my new blog location to my own domain if needed (it is not as easy since I also need to preserve the post URLs, but shouldn't be difficult to fix this if I ever need to do so).

Now for the tools that I use to publish from the original Markdown files to everything else. Let's start with shell script(s) and Makefile: I decided that they're less of an issue if they eventually stop working: they're only used to make my life easier, but I can still publish files manually if needed. Still, I tried to rewrite both the shell and Makefile to avoid GNUisms, so in the eventual case that I decide to e.g.: stop using a GNU/Linux system like NixOS and use a *BSD system instead, I am covered.

In other projects I would probably add a shell.nix file and add the GNU versions of the tools that I am using, but I am trying to avoid Nix while building this blog as much as I can, mainly because it is not available everywhere (e.g.: *BSD systems).

Go is the more important part: the tooling used to publish this blog is written in Go. Go is a good language when you want to ensure that things will work for a long time because of its backwards compatibility guarantee. Also I don't expect Google dropping Go development soon, but even if this happen ("killed by Google" is a thing after all), it is very likely some other group or company would adopt its development quickly, considering how popular the language is.

However, the Go modules that I depend are another story:

elliotchance/orderedmap: an ordered map implementation that I use until Go adds it in the standard library
gorilla/feeds: a RSS generator library
gosimple/slug: a slug generator library
yuin/goldmark: a CommonMark parser and renderer
teekennedy/goldmark-markdown: a renderer for Goldmark to render back to Markdown (since Goldmark itself doesn't have this capacity)

In common for all those modules are that they're clearly small projects maintained mostly by one developer. They're all very good, don't get me wrong, but they're still an reliability issue in the future. There is no guarantee those repositories will not be deleted tomorrow, for example.

Yes, Go Proxy exist, but from what I understood reading its page is that while it caches modules contents, this is not guarantee:

proxy.golang.org does not save all modules forever. There are a number of reasons for this, but one reason is if proxy.golang.org is not able to detect a suitable license. In this case, only a temporarily cached copy of the module will be made available, and may become unavailable if it is removed from the original source and becomes outdated. The checksums will still remain in the checksum database regardless of whether or not they have become unavailable in the mirror.

This is why this is the first project that made sense to me to use go mod vendor. Now I have a copy of the source code of all modules inside the vendor directory in the repository, avoiding the risks I commented above. This allows me to ensure that this blog will still be publishable in the future, as long as I have a working Go toolchain (and Go toolchain makes this easy).

There are a few other things that can bitrot this blog, for example links going nowhere. I always try to use permalinks when it makes sense, but the only actual way to ensure those links would work in the future would be to point them to archive.org (but even archive.org may not exist forever). Maybe something to fix in the future, hope not in the far future, before things start to break.

An unordered list of hidden gems inside NixOS

Wed, 21 Aug 2024 00:00:00 +0000

After using NixOS for the last 5+ years as my main OS, I end up with a configuration with many things that are interesting for one reason or another, but it is not listed anywhere (well, except if you are the kind of person that reads man 5 configuration.nix or the release notes in every release).

So kind in the same spirit as my list of things that I miss in Go, here is a list of modules that I find neat in NixOS and are not default already. Again, the list is unordered since this makes it easier to update in the future if I find something else, but also I don't want to think too hard about an order here.

With all above, let's start.

`networking.nftables`

nftables is, accordingly to Wikipedia:

nftables is a subsystem of the Linux kernel providing filtering and classification of network packets/datagrams/frames.

It is basically a replacement of the venerable iptables, that still exist and is the default program to configure the famous networking.firewall, the declarative Firewall that NixOS enable by default.

To enable, it is simple, just add to your configuration:

{
  networking.nftables.enable = true;
}

And thanks to the fact that NixOS's Firewall is declarative, everything should still work as expect: any service that you set openFirewall = true will still have its ports open, if you set networking.firewall.allowPing = false it will still disable pings like before, etc.

If you look at the documentation of the above option, you will find the following warning:

Note that if you have Docker enabled you will not be able to use nftables without intervention. Docker uses iptables internally to setup NAT for containers. This module disables the ip_tables kernel module, however Docker automatically loads the module. Please see https://github.com/NixOS/nixpkgs/issues/24318#issuecomment-289216273 for more information.

I don't use Docker (switched to Podman instead for quite a long time), so I don't know how bad the situation is. Also keep in mind that nftables does offer iptables-compat for compatibility with old iptables scripts, so it is most likely Docker doing something weird here.

Now, the actual advantage from the user here is not clear: the main advantage from my point of view (and the reason I used to use in other distros like Arch) is the improved syntax, however if you are using the declarative NixOS's Firewall you are not interacting with either iptables or nftables directly anyway. nftables is supposed to be more efficient, but not sure most users will care about this.

However if you are the kind of person that needs custom rules, switching to nftables does bring a few benefits, including networking.nftables.checkRuleset (enabled by default), that checks if your ruleset has syntax errors during build time. Really valuable to avoid issues only after switch.

Anyway, this is one of those options that I think it should be the default for a long time, since most of the new development in NixOS firewall seems to be focusing nftables for a while.

`system.switch.enableNg`

This one I just discovered today, but it has been available for a while (~2 months if you're using nixos-unstable). Finally someone is rewriting switch-to-configuration.pl, the Perl script that is called everytime you run nixos-rebuild switch.

Now, I am not one of those "rewrite in Rust" zealots, but in this case this is definitely worth it: switch-to-configuration.pl is one of those pieces of code in NixOS that most people avoid touching at the fear of breaking something. There is a reason why nixos-rebuild is as convoluted as it is, because even if it is a messy shell script, most people preferred to workaround issues from the switch-to-configuration.pl inside it than trying to understand the mess that switch-to-configuration.pl is.

Trying this one is easy:

{
  system.switch = {
    enable = false;
    enableNg = true;
  };
}

Yes, you need to explicit set system.switch.enable = false, since the default is true.

By the way, what is the reason you would want to set system.switch.enable = false before the enableNg appeared you ask? For systems that are immutable and updated by e.g.: image upgrades instead of modifying root.

Enabling switch-to-configuration-ng right now is mostly for testing purposes, but one of the advantages that I saw is that system switches are (slightly) faster:

$ hyperfine "sudo nixos-rebuild switch" # switch-to-configuration.pl
Benchmark 1: sudo nixos-rebuild switch
  Time (mean ± σ):      3.576 s ±  0.035 s    [User: 0.004 s, System: 0.014 s]
  Range (min … max):    3.522 s …  3.645 s    10 runs

$ hyperfine "sudo nixos-rebuild switch" # switch-to-configuration-ng
Benchmark 1: sudo nixos-rebuild switch
  Time (mean ± σ):      3.394 s ±  0.080 s    [User: 0.004 s, System: 0.013 s]
  Range (min … max):    3.325 s …  3.608 s    10 runs

But yes, the difference is not enough to make a significant impact, and it is not the objective anyway. The real reason for the rewrite is to make it easier to collaborate. I hope one day we also have someone brave enough to rewrite the nixos-rebuild script in something saner.

`boot.initrd.systemd`

A quick recap on how a modern Linux distro generally boots: the first thing that the bootloader (say GRUB or systemd-boot) loads is initrd (initial ramdisk), a small image that runs from RAM and includes the Linux kernel and some utilities that are responsible for setting up the main system. For example, one of the responsibilities of the initrd is to mount the disks and start init system (systemd).

It may surprising that this initrd image does not generally include systemd. Traditionally initrd is composed by a bunch of shell scripts and a minimal runtime (e.g.: busybox), however systemd can also do this job since a long time ago. It is just the role of the distros to integrate systemd inside the initrd.

This is what boot.initrd.systemd does: enable systemd inside the initrd. It make a few subtle changes:

If you are using Full Disk Encryption via LUKS, you will get a different password prompt at login
You will get initrd time information if using systemd-analyze to measure boot time
- You can get even more information (bootloader) if you also use systemd-boot
You will also get systemd style status about services during initrd (not only afterwards)

But I think the main reason is that since systemd is event-driven, it should make boot more reliable, especially in challenging situations (like booting from network). I can't say that I have any system like this to test if it is actually more reliable or not, but I don't remember having any issues since I set boot.initrd.systemd.enable = true, so there is that.

`services.pipewire`

If there is something in that list that has a good chance that you're using already, it is this one, especially if you're using Wayland. Still, I think it is interesting to include in this list since PipeWire is great.

The experience with PipeWire until now for me was seamless: I never had any issues with it, all my applications still work exactly as it always worked. I also didn't had any issues with PulseAudio for a while, but I still remember when I first tried PulseAudio during the 0.x in Fedora and having tons of issues. So bonus points for PipeWire developers for polishing the experience of enough that most people will feel no difference.

To enable PipeWire, I would recommend:

{
  services.pipewire = {
    enable = true;
    alsa.enable = true;
    pulse.enable = true;
    # jack.enable = true;
  };
  security.rtkit.enable = true;
}

This enables both ALSA and PulseAudio emulation support in PipeWire for maximum compatibility with desktop applications (you can also enable jack if you use professional audio applications). It also enables rtkit, allowing PipeWire to get (soft) realtime, helping avoiding cracks during high CPU load.

I also recommend taking a look at the Wiki article, that has multiple interesting configurations that can be added for low-latency setups or improved codecs for Bluetooth devices.

`networking.networkmanager.wifi.backend = "iwd"`

There is a good change that you're using NetworkManager to manage network, especially for Wi-Fi. And if that is the case, I can't recommend enough changing the backend from the default wpa_supplicant to iwd.

If you think that your Wi-Fi takes a long time to connect/re-connect, it may be because wpa_supplicant. iwd seems much more optimised in this regard, and since switching to it I never felt that my Wi-Fi was worse than other OSes (and generally slightly better than Windows, but keep in mind that this is a non-scientific comparison).

Not saying that I never had Wi-Fi issues since switching to iwd, however switching back to wpa_supplicant in those cases never fixed the issue (it was the same or worse), so I assume either bad hardware or drivers in those cases.

`boot.tmp.useTmpfs`

Added in 2024-08-22

Most Linux distro nowadays mount /tmp inside tmpfs, storing the files inside RAM (and making them actually temporary, e.g.: does not persist between reboots). There is a long discussion whether this makes sense or not, but it is a fact that using /tmp as an storage for small files generally makes sense.

However, NixOS still stores /tmp inside / by default. The main reason for this is because Nix, by default, still builds everything in TMPDIR (that defaults to /tmp if not set). To fix the situation you can do:

{
  boot.tmp.useTmpfs = true;
  systemd.services.nix-daemon = {
    environment.TMPDIR = "/var/tmp";
  };
}

This ensures that Nix will build in /var/tmp, that is the temporary directory for large files.

If you don't want to use /tmp inside tmpfs for some reason, I recommend at least setting boot.tmp.cleanOnBoot = true. This is to avoid issues with e.g.: mistakenly writing a program that depends in /tmp being persistent, because this is the behavior that most other distros expects.

`zramSwap`

Added in 2024-08-22

Have a system with small amounts of RAM? ~~You can download more RAM for free, just click this link.~~ Well, just kidding, but you can actually improve RAM usage by compressing it. This is what zram does: it will create a block device using part of your RAM that will be compressed.

While this means that whatever is in the compressed part can't be used as-is (it needs to be decompressed first), it means you can store a lot more information. And since RAM is much faster than disk, this generally is a good trade-off. It is still recommended that you have some swap space though, e.g.: to swap pages that are unused.

To enable it, you can do:

{
  zramSwap = {
    enable = true;
    algorithm = "zstd";
  };
}

You can play a little with the zramSwap.algorithm parameter, but I recommend either lz4 or zstd. It is important for the compression algorithm to be fast, since you probably want to minimise CPU usage. Well, unless you have a very good CPU and small amounts of RAM, them it may make sense to use an algorithm that compress better but uses more CPU.

`services.fstrim`

Added in 2024-08-22

Do you have a SSD? If so, you probably want to trim it regularly to keep a good performance. This should be as easy as:

{
  services.fstrim.enable = true;
}

One caveat though: if you are using Full Disk Encryption with LUKS, you also need to enable boot.initrd.luks.devices.<name>.allowDiscards, otherwise fstrim will not work. The reason that this is not enabled by default is because there are some security implications. It shouldn't be an issue for most people, but if you are paranoid please assess your risks first.

`boot.binfmt.emulatedSystems`

Added in 2024-08-22

This is one of my favorites. Do you want to have the power to run binaries from other architectures like ARM or RISC-V? In NixOS it is easy:

{
  boot.binfmt.emulatedSystems = [ "aarch64-linux" "riscv64-linux" ];
}

This will install a QEMU emulator for the target architectures, and configure binfmt so that the kernel will automatically detect when a binary from another architecture is run, running it with the QEMU emulator instead.

Now, keep in mind that you still need e.g.: libraries for the target architecture. However this is where the Nix magic comes in, you can easily compile something to another architecture as:

$ uname -a
Linux sankyuu-nixos 6.10.3 #1-NixOS SMP PREEMPT_DYNAMIC Sat Aug  3 07:01:09 UTC 2024 x86_64 GNU/Linux
$ cd nixpkgs
$ nix build .#hello --system aarch64-linux
$ file ./result/bin/hello
./result/bin/hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/kz7xglxzhad64v667wwpn8vrxhjwcbna-glibc-2.39-52/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped
$ ./result/bin/hello
Hello, world!

And since the binary will be linked with the libraries from the target architecture, everything works as expect.

You also need to temper your expectations: QEMU emulation is slow. It is sufficient fast to build small programs, but if you need to build something bigger, I would recommend you to do it via cross-compilation instead.

`services.dbus.implementation = "broker"`

Added in 2024-08-22

Someone in Lobste.rs remembered me of this gem: dbus-broker, a high performance implementation of D-Bus.

For those who doesn't know, D-Bus is the Inter-Process Communication protocol used mostly in Linux desktop. And distros like Arch and Fedora already use it to default, so there aren't lots of reason to not use it.

`services.irqbalance`

Added in 2024-08-23

Here is one that I have no experience but got curious after this comment in Lobte.rs:

One more option to add is maybe services.irqbalance.enable. For my old laptop it’s a life savior: no more freezes of the DE or video playback during heavy operations like e.g. compilation.

Never tried it before but will give a try!

Quick bits: using forks with Go modules

Tue, 20 Aug 2024 00:00:00 +0000

There are 2 types of blog posts: the ones you write for others, and the ones that you write for yourself. This blog post is the latter kind. What I am going to talk here is probably something most people know, but I didn't, and the documentation is all scattered in the internet. So I am writing mostly to remember myself, in future, if I have the same kind of issue what I need to do.

The context: Mataroa, the blog platform capivaras.dev is hosted on, relies in python-markdown, predates CommonMark specification and as such, has some incompatibilities. One of those incompatibilities with CommonMark is the way sub lists are handled. From the documentation:

The syntax rules clearly state that when a list item consists of multiple paragraphs, “each subsequent paragraph in a list item must be indented by either 4 spaces or one tab” (emphasis added). However, many implementations do not enforce this rule and allow less than 4 spaces of indentation. The implementers of Python-Markdown consider it a bug to not enforce this rule.

CommonMark relax those restrictions, allowing a sublist to be defined with just 2 spaces of indentation.

So I have automated all posts from this blog using Go and a CommonMark renderer called Goldmark. I them re-render the Markdown to Mataroa using a Markdown renderer before publising to capivaras.dev, because this allow me to do some transformations in the original Markdown. It mostly works fine except for sub lists, thanks to the fact that the Markdown renderer I am using renders sub lists with 2 spaces.

The only reason sub lists are working right now is because @ratsclub fixed this issue in the fork that capivaras.dev runs. But I want to be compatible with the official instance if I ever need to migrate.

The solution? Let's fix this in a PR. However now that I have code to fix the issue, how can I use it without waiting upstream to merge my code?

If you are using Go modules it is easy, you just need to use the replace directive:

module github.com/thiagokokada/blog

go 1.23

require (
	github.com/elliotchance/orderedmap/v2 v2.4.0
	github.com/gorilla/feeds v1.2.0
	github.com/gosimple/slug v1.14.0
	github.com/teekennedy/goldmark-markdown v0.3.0
	github.com/yuin/goldmark v1.7.4
	github.com/yuin/goldmark-highlighting v0.0.0-20220208100518-594be1970594
)

require (
	github.com/alecthomas/chroma v0.10.0 // indirect
	github.com/dlclark/regexp2 v1.11.4 // indirect
	github.com/gosimple/unidecode v1.0.1 // indirect
)

replace github.com/teekennedy/goldmark-markdown => github.com/thiagokokada/goldmark-markdown v0.0.0-20240820111219-f30775d8ed15

This will replace all usages of github.com/teekennedy/goldmark-markdown to my fork in github.com/thiagokokada/goldmark-markdown. You even get all the reproducibility of modules since Go automatically pins the commit.

Since the Go format for versions is quite unique, you can just set to the desired branch (e.g.: instead of v0.0.0-20240820111219-f30775d8ed15, you can use add-sublist-length-opt that is the branch name) and run go mod tidy to fix the format.

Quick bits: Go automatically downloads a newer toolchain if needed

Mon, 19 Aug 2024 00:00:00 +0000

I am using elliotchance/orderedmap as my choice of ordered maps (since Go doesn't have one in standard library yet). I recently did a PR to implement Go 1.23 iterators, because I find them neat, however I was avoiding to use it in the code that generates this blog since Go 1.23 was just released and is still not the default Go in nixpkgs.

I decided that I would create a branch and leave there for a few months, until I decided to try to run the code locally and got this:

$ go build
go: downloading go1.23.0 (darwin/arm64)

Nice. And before you ask, yes, the compiled binary works perfectly:

$ make
./blog > README.md
./blog -rss > rss.xml

So how does this work? Take a look at the documentation in the official Golang page:

Starting in Go 1.21, the Go distribution consists of a go command and a bundled Go toolchain, which is the standard library as well as the compiler, assembler, and other tools. The go command can use its bundled Go toolchain as well as other versions that it finds in the local PATH or downloads as needed.

There are a bunch of rules here that I am not going to enter in detail (I recommend you to read the official documentation), but a quick summary:

Go will download a toolchain when either go or toolchain lines go.mod is set to a Go version higher than your current go binary
- But only if your go binary is at least version 1.21, since this is the version that introduces this behavior
You can force a specific toolchain with GOTOOLCHAIN environment setting, e.g.: GOTOOLCHAIN=1.23
- The default value for GOTOOLCHAIN is auto, that basically has the behavior described in this post
- You can also set to local to always use the current go binary, or the previous behaviour pre-1.21 Go
- There is also <name>+auto and path options, that can be seen in the docs

The downloaded toolchains go to whatever your GOPATH is, inside golang.org/toolchain module, and version v0.0.1-goVERSION.GOOS-GOARCH, for example:

$ ls -lah $GOPATH/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.darwin-arm64
total 64
dr-xr-xr-x  14 user  staff   448B 19 Aug 12:01 .
drwxr-xr-x   4 user  staff   128B 19 Aug 12:01 ..
-r--r--r--   1 user  staff   1.3K 19 Aug 12:01 CONTRIBUTING.md
-r--r--r--   1 user  staff   1.4K 19 Aug 12:01 LICENSE
-r--r--r--   1 user  staff   1.3K 19 Aug 12:01 PATENTS
-r--r--r--   1 user  staff   1.4K 19 Aug 12:01 README.md
-r--r--r--   1 user  staff   426B 19 Aug 12:01 SECURITY.md
-r--r--r--   1 user  staff    35B 19 Aug 12:01 VERSION
dr-xr-xr-x   4 user  staff   128B 19 Aug 12:01 bin
-r--r--r--   1 user  staff    52B 19 Aug 12:01 codereview.cfg
-r--r--r--   1 user  staff   505B 19 Aug 12:01 go.env
dr-xr-xr-x   3 user  staff    96B 19 Aug 12:01 lib
dr-xr-xr-x   4 user  staff   128B 19 Aug 12:01 pkg
dr-xr-xr-x  77 user  staff   2.4K 19 Aug 12:02 src

By the way, this only works well because Go toolchain binaries are static.

While I don't like a program downloading random binaries from the internet, I like what Go is doing here. It makes the whole bootstrapping process for a Go project much easier: as long as you have a reasonable up-to-date go binary in your PATH, you should be ready to go (pun intended). And Go modules are already reasonable secure, ensuring that each module have a proper checksum. As long as nobody else can publish modules in golang.org/toolchain namespace I can't see much of a security issue here, but I am not a security expert.

But if you don't like this behavior, you can always disable it by setting GOTOOLCHAIN=local. And just do not forget to set this in your CI, unless you don't care about Go versions.

An unordered list of things I miss in Go

Sat, 17 Aug 2024 00:00:00 +0000

I like Go. I think it is a reasonable good language, and has some good qualities that makes up for its flaws. However, this doesn't mean I think the language couldn't be better, far from it.

This blog post is a list of things that I miss from Go from other languages. Some of the things here could probably be implemented soon, some other would probably need a major revision of the language. The list is unordered, because this makes it easier for me to update in the future if I found something else, but also because I don't want to think too hard about giving each point here a rank.

With all above, let's start.

Ordered maps in standard library

When I first learned about dictionaries in Python it quickly became one of my favorite data structures ever. They're extremely versatile, and most modern programming languages have something similar in its standard library. Go isn't different, it has map, that is Go implementation of a hash table. However map in Go are quirky, for example:

package main

func main() {
	m := map[string]bool{"foo": true, "bar": false, "baz": true, "qux": false, "quux": true}

	for k := range m {
		println(k)
	}
}

$ go run ./test.go
bar
baz
qux
quux
foo

$ go run ./test.go
foo
bar
baz
qux
quux

$ go run ./test.go
qux
quux
foo
bar
baz

Now, I don't expect any hash table implementation to keep the order of the elements, but Go actually randomise each map instance:

But here’s the deal, while the hash function used for maps in Go is consistent across all maps with the same key type, the seed used by that hash function is different for each map instance. So, when you create a new map, Go generates a random seed just for that map.

While I understand the reason for this (i.e.: to avoid developers relying in a specific iteration order), I still find it weird, and I think this is something unique for Go. This decision means that even if you don't care about a specific order, you will still need to sort the map before doing something else if you want reproducibility, something that I care a lot.

The fix for this? Go could offer an ordered map implementation inside the standard library. An ordered map ensure that the iteration order of the map is the same as the insertion order (that is, by the way, a powerful property that allow maps to be used in other contexts, not just my pet peeve above).

Python actually does this for any dictionaries since Python 3.6, but it offered an OrderedDict before it (and OrderedDict still has some methods that normal dict doesn't, that maybe useful in specific cases).

Before generics it would be impossible to have a type-safe API for such data structure without introducing a new data type in the language (like slices), but now Go has generics so it is not an issue anymore. The other issue is that you would be forced to iterate manually in this new data structure, but thanks to the new range-over-func in Go 1.23, it means we can iterate in an ordered map as a library almost exactly like we can do as a map:

import "orderedmap"

func main() {
    m := orderedmap.New[string, bool]()
    m.Set("foo", true)
    m.Set("bar", false)
    m.Set("baz", true)

    for k := range m.Iterator() {
        println(k) // Order always will be: foo, bar, baz
    }
}

Now, of course the lack of Ordered Map in the standard library can be filled with third party implementations, e.g.: I am using this one in one of my projects. But being in standard library reduces the friction: if there was some implementation in standard library, I would generally prefer it unless I have some specific needs. However when the standard library doesn't offer what I need, I need to find it myself a suitable library, and this ends up taking time since generally there are lots of alternatives.

Keyword and default arguments for functions

Something that comes straight from Python that I miss sometimes in Go is that you can do things like this when declaring a function:

def hello(name="World"):
    print(f"Hello, {name}")

hello("Foo") # "normal" function call
hello(name="Bar") # calling with keyword arguments
hello() # calling with default arguments

$ python hello.py
Hello, Foo
Hello, Bar
Hello, World

The lack of default arguments especially affects even some of the API decisions for Go standard library, for example, string.Replace:

func Replace(s, old, new string, n int) string

Replace returns a copy of the string s with the first n non-overlapping instances of old replaced by new. If old is empty, it matches at the beginning of the string and after each UTF-8 sequence, yielding up to k+1 replacements for a k-rune string. If n < 0, there is no limit on the number of replacements.

If Go had default arguments, Replace could have e.g.: func Replace(s, old, new string, n int = -1) signature, that would mean we don't need strings.ReplaceAll (since this is basically what this function does, call strings.Replace(s, old, new, -1)).

Nullability (or nillability)

I talked I little about this in my previous post about Go, but I want to expand here.

First, I don't think the language needs to support the generic solution for nullability, that would be either having proper Union or Sum types. Kotlin AFAIK doesn't support neither, but my 2 years experience with Kotlin showed that just having nullable types already helped a lot in ensuring type safety.

Second, I do feel that Go has less issues with nil values, than say, Java, because its decision of using zero values instead of nil in many cases. So for example, a string can never be nil, however a string pointer can be. This means that this is fine:

func(s string) {
    // do something with s
}

However:

func(s *string) {
    // s maybe nil here, better check first
}

Still, I get more panic for nil pointer deference than I get in other languages that offer nullables (heck, even Python with mypy is safer).

Sadly this is the change in this post that is more likely to need a completely new revision of the language. nillability was proposed before, but it is really unlikely it can be done without breaking backwards compatibility.

It could be done the Java way by adding a nullable type to the standard library (JSR305), but the fact that JSR305 is considerd dead by many shows how difficult it is to do something like this without a major change in the language. Dart is the only language that I know that did this successfully, but definitely it was not without its pains. And the fact that most people that program in Dart probably does because of Flutter (that eventually required newer versions with null-safety) is not a good sign.

Lambdas

Added in 2024-08-18

Go is a surprising good language for some functional code, thanks to having first class functions and closures. Sadly the syntax doesn't help, since the only way you can use anonymous functions in Go is using func. Especially if the types are complex, this can result in some convoluted code. Take the example from the range-over-func experiment:

package slices

func Backward[E any](s []E) func(func(int, E) bool) {
    return func(yield func(int, E) bool) {
        for i := len(s)-1; i >= 0; i-- {
            if !yield(i, s[i]) {
                return
            }
        }
    }
}

If Go had a syntax for lambdas, especially if we could elide the types, this could be simplified a lot:

package slices

func Backward[E any](s []E) func(func(int, E) bool) {
    return (yield) => {
        for i := len(s)-1; i >= 0; i-- {
            if !yield(i, s[i]) {
                return
            }
        }
    }
}

Or even something like this would already help, no special syntax but allowing the types to be elided in an unnamed function:

package slices

func Backward[E any](s []E) func(func(int, E) bool) {
    return func(yield) {
        for i := len(s)-1; i >= 0; i-- {
            if !yield(i, s[i]) {
                return
            }
        }
    }
}

This feature I am still somewhat hopeful that may become a reality in some future version of the language, since they didn't close the issue yet, and the discussion about the possibility of this feature is still ongoing.

Error on unused return values

Added in 2024-09-12

In a recent post about LLMs, I talked about this function that I asked ChatGPT to generate to create a socket that accepted a context:

func readWithContext(ctx context.Context, conn net.Conn, buf []byte) (int, error) {
    done := make(chan struct{})
    var n int
    var err error

    // Start a goroutine to perform the read
    go func() {
        n, err = conn.Read(buf)
        close(done)
    }()

    select {
    case <-ctx.Done(): // Context was canceled or timed out
        // Set a short deadline to unblock the Read()
        conn.SetReadDeadline(time.Now())
        <-done // Wait for the read to finish
        return 0, ctx.Err()
    case <-done: // Read finished successfully
        return n, err
    }
}

However while reading this blog post about "The Error Model", I realised that this function is lacking error check during conn.SetReadDeadline() calls:

func readWithContext(ctx context.Context, conn net.Conn, buf []byte) (n int, err error) {
	done := make(chan struct{})

	// Start a goroutine to perform the read
	go func() {
		n, err = conn.Read(buf)
		close(done)
	}()

	select {
	case <-done:
		return n, err
	case <-ctx.Done():
		// Set a short deadline to unblock the Read()
		err = conn.SetReadDeadline(time.Now())
		if err != nil {
			return 0, err
		}
		// Reset read deadline
		defer func() {
			if e := conn.SetReadDeadline(time.Time{}); e != nil {
				err = errors.Join(err, e)
			}
		}()
		// Make sure that the goroutine is done to avoid leaks
		<-done
		return 0, errors.Join(err, ctx.Err())
	}
}

I could blame the LLM on this, but this is the type of mistake that I could see happening even in a Pull Request done by an actual human.

Ignoring errors is bad, especially because once the error is ignored it is gone, forever. You may have strange issues in the code that are impossible to debug because you can't know about something that doesn't exist. This is one of the reasons I think exceptions are better, since this is the kind of error that would be impossible to ignore if the language had exceptions, and the exception would eventually propagate in your stack until it is captured or you get a crash (with a stack trace to debug the issue).

Now, I don't think this is a fault of using error as values instead of exceptions. Go has the tools to encapsulate the errors and propagate them properly, like it is shown in the fixed code. But the fact that someone can do this mistake without any warning or error from the compiler is bad.

If Go had a warning or error for unused return values, this would be different:

func foo(conn *net.Conn) {
  // ...
  _ = conn.SetReadDeadline(time.Now())
}

Now in this case it is clear: I am ignoring the error explicitly, probably for a good reason. I can ask during a Pull Request why the committer is ignoring if it is lacking appropriate context, and maybe even ask for some comments to be added why this would be safe. What can't happen is this being ignored by mistake.

I am not completely sure if I want this only for errors or for any unused value though. There are lots of cases where ignoring non-error values is ok, but I also don't see the issue of having something like:

func foo() {
  // ...
  _, _ = FuncThatReturnIntAndError()
}

If anything, it is clear for whoever is reading this code later that you're only calling this function only for its side-effects.

By the way, yes, errcheck linter exists, but if the language itself doesn't enforce this it means that there are lots of libraries that are probably mistakenly ignoring errors. And if the library is ignoring errors by mistake, you can't do much when you actually have them.

Quick bits: NixOS's boot.kernelPackages is a attrset

Thu, 15 Aug 2024 00:00:00 +0000

I don't know if this is just something that got me by surprise or not, but boot.kernelPackages option does not receive a derivation like most other packages, but instead receives a function and returns an attribute set with all packages. Here is the documentation:

This option allows you to override the Linux kernel used by NixOS. Since things like external kernel module packages are tied to the kernel you’re using, it also overrides those. This option is a function that takes Nixpkgs as an argument (as a convenience), and returns an attribute set containing at the very least an attribute kernel. Additional attributes may be needed depending on your configuration. For instance, if you use the NVIDIA X driver, then it also needs to contain an attribute nvidia_x11.

The kernel package itself is referenced by the kernel derivation inside this attribute set:

nix-repl> nixosConfigurations.sankyuu-nixos.config.boot.kernelPackages.kernel
«derivation /nix/store/5zyjvf3qgfk52qmgxh36l4dkr9lf100x-linux-6.10.3.drv»

The reason for this is because it ensure that things like modules are built with the same kernel version you are booting.

However one less obvious consequence about this is that if you want packages that come from linuxPackages, say for example cpupower or turbostat, it is better to do:

{ config, ... }:
{
  environment.systemPackages = [
    config.boot.kernelPackages.cpupower
    config.boot.kernelPackages.turbostat
  ];
}

Instead of:

{ pkgs, ... }:
{
  environment.systemPackages = with pkgs; [
    linuxPackages.cpupower
    linuxPackages.turbostat
  ];
}

Now, I used the later for a long time and never had issues. But technically those packages depends in a specific kernel version for a reason, so getting them from config.boot.kernelPackages reduces the chance of you having some compatibility issue in the future.

Abusing Makefiles for fun and profit

Tue, 13 Aug 2024 00:00:00 +0000

If you are following this blog for a while, it should be no surprise that most of the workflow in this blog is automated using Go. I basically write Markdown files with some special rules inside the repository, commit and push it. In seconds, the CI (currently GitHub Actions) will take the latest commit, generate some files (since I use the repository itself as a backup blog) and publish to the capivaras.dev website.

Now, considering how much about Nix I talk in this blog, it should be a surprise that the workflow above has zero Nix code inside it. I am not saying this blog will never have it, but I am only going to add if this is necessary, for example if I start using a tool to build this blog that I generally don't expect it to be installed by the machine I am currently using. Go is an exception of this rule since it is relatively straightfoward to install (just download the binary) and because its stability guarantee means (hopefully) no breakage. But most other things I consider moving targets, and I wouldn't be comfortable to use unless I have Nix to ensure reproducibility.

This is why the other tool that this blog (ab)uses during its workflow is Make, one of the oldest build automation tool that exist. It is basically available in any *nix (do not confuse with Nix) system, from most Linux distros to macOS, by default. So it is the tool I choose to automatise some tasks in this blog, even if I consider writing a Makefile (the domain-specific language that Make uses) kind of a lost, dark art.

To be clear, the idea of this post is not to be a Makefile tutorial. I will explain some basic concepts, but if you want an actual tutorial a good one can be found here. Also, while I am using Make thanks to the reasons above, you can use many other tools for a similar objective, like Justfiles, Taskfiles (sadly it uses YAML), or even a small script written in any language you want. The reason that I am writing this post is why you should do it, not how.

A quick recap on how this blog works: inside the repository, a post is basically a Markdown post following the directory structure below (permalink for the current version of this blog):

.
<...>
├── 2024-08-07
│   ├── 01-quick-bits-is-crostini-a-microvm.md
│   └── 02-meta-are-quick-bits-really-quick.md
├── 2024-08-11
│   └── 01-building-static-binaries-in-nix.md
├── 2024-08-12
│   ├── 01-things-i-dont-like-in-my-chromebook-duet-3.md
│   └── Screenshot_2024-08-12_20.50.42.png
├── 2024-08-13
│   ├── 01-abusing-makefiles-for-fun-and-profit.md <-- this file
├── .github
│   └── workflows
│       └── go.yml
├── .gitignore
├── go.mod
├── go.sum
├── LICENSE
├── link_rewriter.go
├── Makefile
├── mataroa.go
├── README.md
├── rss.xml
└── .scripts
    └── gen-post.sh

So I just create a new Markdown file following the YYYY-MM-DD/XX-title-slug.md format. It must start with a h1 header, that will be automatically extract to be used as the post title, but otherwise there is no other formatting rules. It is a highly optionated structure, but the nice thing about being optionated is that we can extract lots of information just from how the files are organised in the filesystem.

Most of the magic that converts those Markdown files to actual blog posts are in the Go files that you can see above: blog.go is the main logic that walks in the repository and extracts the necessary information, mataroa.go is responsible for the capivaras.dev integration (that uses Mataroa platform), while link_rewriter.go is responsible to do some transformations in the Markdown files before posting.

While I could manage everything by just using go CLI and a few other *nix commands, to make it easier to manager everything I have the following Makefile:

MARKDOWN := $(wildcard ./**/*.md)

.PHONY: all
all: README.md rss.xml

blog: *.go go.*
	go build

README.md: blog $(MARKDOWN)
	./blog > README.md

rss.xml: blog $(MARKDOWN)
	./blog -rss > rss.xml

.PHONY: publish
publish: blog
	./blog -publish

DAY := $(shell date)
_PARSED_DAY := $(shell date '+%Y-%m-%d' -d '$(DAY)')
.PHONY: day
day:
	mkdir -p '$(_PARSED_DAY)'

TITLE = $(error TITLE is not defined)
.PHONY: post
post: blog day
	./.scripts/gen-post.sh '$(_PARSED_DAY)' '$(TITLE)'

FILE = $(error FILE is not defined)
.PHONY: draft
draft:
	cd '$(dir $(FILE))' && mv '$(notdir $(FILE))' '.$(notdir $(FILE))'

.PHONY: undraft
undraft:
	cd '$(dir $(FILE))' && mv '$(notdir $(FILE))' '$(patsubst .%,%,$(notdir $(FILE)))'

.PHONY: clean
clean:
	rm -rf blog

For those unfamiliar with Makefile, a quick explanation on how it works from Wikipedia:

Each rule begins with a dependency line which consists of the rule's target name followed by a colon (:) and optionally a list of targets on which the rule's target depends, its prerequisites.

So if we look for example at the blog binary, the dependencies are all the .go files and Go module files like go.mod and go.sum. We can make the blog binary by running:

$ make blog
go build

One nice thing about Makefile is that they track if any of the source files has a newer timestamp than the target file, and only trigger the build again if there are changes, for example:

$ make blog
make: 'blog' is up to date.

$ touch blog.go

$ make blog
go build

But sometimes this property is undesirable. In those cases we can declare a target as .PHONY, that basically instructs Makefile to always make the target. One classic example is clean target, that removes build artifacts:

$ make clean
rm -rf blog

$ make clean
rm -rf blog

By the way, it is better to declare a target as .PHONY than declaring dependencies incorrectly, especially in languages that has fast build times like e.g.: Go. The worst thing that can happen is something not being rebuild when it needs to. So my recomendation if you are writing your first Makefile is to just declare everything as .PHONY. You can always improve it later.

One last basic concept that I want to explain about Makefile is the default target: it is the target that is run if you just run make without arguments inside the directory that contains a Makefile. The default target is generally the first target in the Makefile. It is common to have an all target (that is also marked as .PHONY) that has as dependencies all the targets that you want to build by default. In this particular case I declare the README.md and rss.xml files to be build by default, and they themselves depends in blog binary being build. So once I run make you get as result:

$ make
go build
./blog > README.md
./blog -rss > rss.xml

And this result above highlights the first reason I think you should have a Makefile or something similar in your projects: you don't need to remember the exactly steps that you need to get things working. If I see one project of mine having a Makefile, I can be reasonably confident that I can get it working by just running make.

But now let's focus in the other targets that I have in the Makefile that are not related to the build process but are there to help me manage my blog posts. Remember the rules I explained above? Maybe not, but it should be no problem, because:

$ make post TITLE="My new blog post"
mkdir -p "2024-08-13"
./.scripts/gen-post.sh "2024-08-13" "My new blog post"
Creating file: 2024-08-13/02-my-new-blog-post.md

$ cat 2024-08-13/02-my-new-blog-post.md
# My new blog post

This command, make post, is responsible for:

Create a new directory for today, if it doesn't exist
Run the gen-post.sh script, that:
1. Enumerates all posts from the day, so we can number the new post correctly
  - We already had this post planned for 2024-08-13, so the new post is 02
2. Slugify the title, so we can create each Markdown file with the correct filename
3. Creates a new Markdown file with the title as a h1 header

The steps above may or may not seen trivial, and for a while I was doing them manually. But not having to think what is the current date or if I already posted that day or what is the slug is for the title make (pun intended) my like much easier.

Yes, the code is ugly. The way variables works in Make is that you can declare then inside the Makefile, but they can be overwritten in the terminal if you pass them. I used this to allow make post to also work for future posts:

$ make post TITLE="Another new blog post" DAY=2024-12-12
mkdir -p "2024-12-12"
./.scripts/gen-post.sh "2024-12-12" "Another new blog post"
Creating file: 2024-12-12/01-another-new-blog-post.md

So in the above case, DAY is filled with the value passed in the terminal instead of default (that would be the current day), and _PARSED_DAY is the day we use to actually create the directory. We can actually pass any date format recognised by date, not just YYYY-MM-DD.

I have 2 other phony targets that I want to talk, draft and undraft. They expect a FILE to be passed, and I use them to either hide or unhide a file:

$ make draft FILE=2024-12-12/01-another-new-blog-post.md
mv "2024-12-12/01-another-new-blog-post.md" "2024-12-12/.01-another-new-blog-post.md"

$ make undraft FILE=2024-12-12/.01-another-new-blog-post.md
mv "2024-12-12/.01-another-new-blog-post.md" "2024-12-12/01-another-new-blog-post.md"

Why? Because hidden files are explicit ignored during my directory parser to mean they're a draft post and not ready to be published. And the reason I created those targets is because I was tired of trying to hide or unhide a file manually.

So that's it, for the same reason you should probably automate everything, you also need to have some way to automate your tasks. Makefile is one way to do it, maybe not the best way to do it, but it works and it is available anywhere.

Things I don't like in my Chromebook Duet 3

Mon, 12 Aug 2024 00:00:00 +0000

So this is kind of a continuation from my previous post talking why my favorite device is a Chromebook. In this post I want to talk about what makes me this device unhappy, and comment about things that if changed would make it a much better device.

But before talking about the negative aspects, let me talk about a positive aspect that I just briefly talked in the previous post: the screen. It is a HiDPI screen (2000x1200 resolution in 10.95''), that is unexpected bright (400 nits according to the specs). It is difficult to find laptops at the same price with a screen that good. At 10.95'' in its default resolution I find it too small (like 1250x750), but I find the font size acceptable at 115% scale (1087x652). Yes, it result in a small workspace, but this is not a big issue for what I do in this device. It is also only 60Hz, but I thought I would miss high refresh rate more than I actually miss in this device.

Update: I forgot to say one thing about the screen: it scratches really easy. I got my screen scratched after the first day of usage, and considering the price I don't think the screen has a hardened glass. I bought a cheap glass screen protector and this did the trick though, even hiding the previous scratch, and I have zero issues with the screen now.

Now the first aspect that I don't like: the speakers. They sound tiny and even at maximum volume it is not really loud. The speakers is the only reason why I still keep my Xiaomi Pad 5, because I like to watch animes/videos before sleep and having good speakers is a must.

The keyboard has that issue that I mentioned in the previous post: sometimes the key get stuck, and I get duplicated characters. But it also has some minor issues that I didn't talked about: the first one is the UK layout that has some extra keys that I have no use for, but this also makes the keys that I use smaller. Very much a "me" problem, since if I had got a US version I wouldn't have those issues, but an issue nonetheless that gets worse considering how small the keyboard is. I am actually suprised how fast I can type considering how many issues this keyboard has, so maybe this is a testament that this keyboard is not actually that bad.

The other keyboard issue is a problem that affects all Chromebooks: its custom layout. Google replaced a few keys like Fn keys with shortcuts and replaced the Caps Lock with a "Everything" key (that is similar to the Windows Key), while removing Windows Key from its place. I actually have less issue with this than I initially though: I don't care too much about Fn keys (except when using IntelliJ, but that is something that I only use at $CURRENT_JOB), and ChromeOS is surprisingly powerful in its customisation, allowing you to swap key functionality. I remap Everything key with Esc, and Esc for the Everything key, and I can get productive in my neovim setup.

And finally, let me talk more about the performance: yes, it is bad, but bearable once you get used to. The issue is both the CPU and IO. While the CPU, a Snapdragon 7c Gen 2 is octa-core, it has only 2 high performance CPU cores vs 6 low performance ones (2xARM Cortex A76 vs 6xARM Cortex A55). If it was something like 4x4, it would be much better. The fact that the cores are old doesn't help either.

But the worst part is the IO. Not only it uses a eMMC module, it is slow:

I don't know how much more expensive it would be to put a UFS instead of eMMC in this device, but this is probably the choice that would most increase performance in this device, especially considering how aggressive Chromebooks use (z)swap.

Update 2: I forgot to talk about the fact that the exterior of the device is covered in cloth. I thought I would hate this at first, but nowadays I kind like it. And it is also nice that it will never get scratched, I don't care too much about the exterior of this device and it is the only device that I have at home that doesn't have additional protection (except the screen protector mentioned above).

Building static binaries in Nix

Sun, 11 Aug 2024 00:00:00 +0000

I recently had laser eye surgery for my near-sightedness, and while evaluating if I could have the surgery I discovered that I was suffering from dry eyes. Thanks to this, my ophthalmologist recommended that every 20 minutes while using screens, I look somewhere 20 feet away for 20 seconds, a technique known as 20-20-20 rule.

I had issues following this rule because I never remembered to do the pauses. I initially tried to setup an alarm, but this became annoying, so I decided to try to find a program. I found this one for macOS that worked fine, but it bothered me that it was compiled for x86_64 while I was using a M1 MacBook Pro at the time, and also I needed something that worked in Linux.

Since I never found a good multi-platform alternative, I decided to write my own. This became twenty-twenty-twenty, the first Go program that I ever wrote. I wrote it in Go because I wanted to learn the language, but also because Go made it easy to build static binaries. And the first version I could build static binaries without issues because I was using beeep, that uses pure Go code in all supported platforms. However, it also meant that the notifications in macOS looked ugly, since it used osascript.

I wanted better integration with macOS, so this meant switching libraries. After searching for a while, the notify library from GioUI is the one that seemed to work better. It implements notification in macOS using its native framework, so it works much better, but sadly it meant losing static binaries because it depends in CGO.

Not a big loss initially, because I am only depending in Foundation inside macOS (that should always be available), and in Linux I could still statically compile. However I eventually added more features like sound (via beep) and tray icon (via systray), that meant I needed CGO in both macOS and Linux.

Losing static binaries in Linux is a much bigger deal, since Linux is a moving target. The general recommendation for building CGO binaries statically is using musl, but this also means building all dependencies that we need using musl (e.g.: ALSA for [beep/oto]). This generally means pain, but Nix makes it easy.

Let's start by creating a Nix file that builds our Go module (simplified below for brevity):

{ lib
, stdenv
, alsa-lib
, buildGoModule
, pkg-config
, withStatic ? false
}:

buildGoModule {
  pname = "twenty-twenty-twenty";
  version = "1.0.0";
  src = lib.cleanSource ./.;
  vendorHash = "sha256-NzDhpJRogIfL2IYoqAUHoPh/ZdNnvnhEQ+kn8A+ZyBw=";

  CGO_ENABLED = "1";

  nativeBuildInputs = lib.optionals (stdenv.hostPlatform.isLinux) [
    pkg-config
  ];

  buildInputs = lib.optionals (stdenv.hostPlatform.isLinux) [
    alsa-lib
  ];

  ldflags = [ "-X=main.version=${version}" "-s" "-w" ]
    ++ lib.optionals withStatic [ "-linkmode external" ''-extldflags "-static"'' ];

  meta = with lib; {
    description = "Alerts every 20 minutes to look something at 20 feet away for 20 seconds";
    homepage = "https://github.com/thiagokokada/twenty-twenty-twenty";
    license = licenses.mit;
    mainProgram = "twenty-twenty-twenty";
  };
}

And we can build it with the following flake.nix:

{
  description = "twenty-twenty-twenty";

  inputs = {
    nixpkgs.url = "nixpkgs/nixpkgs-unstable";
    flake-compat.url = "github:edolstra/flake-compat";
  };

  outputs = { self, nixpkgs, ... }:
    let
      supportedSystems = [ "x86_64-linux" "aarch64-linux" ];

      # Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'.
      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;

      # Nixpkgs instantiated for supported system types.
      nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; });
    in
    {
      packages = forAllSystems (system:
        let pkgs = nixpkgsFor.${system};
        in {
          default = self.packages.${system}.twenty-twenty-twenty;
          twenty-twenty-twenty = pkgs.callPackage ./twenty-twenty-twenty.nix { };
          twenty-twenty-twenty-static = pkgs.pkgsStatic.callPackage ./twenty-twenty-twenty.nix {
            withStatic = true;
          };
        });
    };
}

I think this shows how powerful Nix is: the only difference between the normal build and a static build the usage of pkgs.pkgsStatic instead of pkgs. This automatically builds all packages statically with musl. Also we need pass some extra flags to the Go compiler (i.e.: -linkmode external -extldflags "-static"), but this is a requirement from Go.

So, does it work? Let's test:

$ nix build .#twenty-twenty-twenty-static

$ file result/bin/twenty-twenty-twenty
result/bin/twenty-twenty-twenty: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

$ ./result/bin/twenty-twenty-twenty
ALSA lib pcm.c:2712:(snd_pcm_open_conf) Either /nix/store/dhn51w2km4fyf9ivi00rz03qs8q4mpng-pipewire-1.2.1/lib/alsa-lib/libasound_module_pcm_pipewire.so cannot be opened or _snd_pcm_pipewire_open was not defined inside
ALSA lib pcm.c:2712:(snd_pcm_open_conf) Either /nix/store/ly9d7llymzjyf6gi1455qzayqipk2kab-pipewire-1.2.1/lib/alsa-lib/libasound_module_pcm_pipewire.so cannot be opened or _snd_pcm_pipewire_open was not defined inside
ALSA lib pcm.c:2712:(snd_pcm_open_conf) Either /nix/store/dhn51w2km4fyf9ivi00rz03qs8q4mpng-pipewire-1.2.1/lib/alsa-lib/libasound_module_pcm_pipewire.so cannot be opened or _snd_pcm_pipewire_open was not defined inside
ALSA lib pcm.c:2712:(snd_pcm_open_conf) Either /nix/store/ly9d7llymzjyf6gi1455qzayqipk2kab-pipewire-1.2.1/lib/alsa-lib/libasound_module_pcm_pipewire.so cannot be opened or _snd_pcm_pipewire_open was not defined inside
ALSA lib pcm.c:2712:(snd_pcm_open_conf) Either /nix/store/dhn51w2km4fyf9ivi00rz03qs8q4mpng-pipewire-1.2.1/lib/alsa-lib/libasound_module_pcm_pipewire.so cannot be opened or _snd_pcm_pipewire_open was not defined inside
ALSA lib pcm.c:2712:(snd_pcm_open_conf) Either /nix/store/ly9d7llymzjyf6gi1455qzayqipk2kab-pipewire-1.2.1/lib/alsa-lib/libasound_module_pcm_pipewire.so cannot be opened or _snd_pcm_pipewire_open was not defined inside
2024-08-11T19:26:33+01:00 INFO Running twenty-twenty-twenty every 20.0 minute(s), with 20 second(s) duration and sound set to true

There are some warns and sadly the sound doesn't work. I think the issue is related because of my usage of PipeWire and the binary may work in a pure ALSA system, but I don't have access to one. Maybe adding pipewire to buildInputs would fix this issue, but I can't get pipewire to be compiled statically (because of its dependencies). I think this is a good show how easy it is to statically compilation is in Nix, but also how complex static binaries are to get correctly.

Bonus points for cross-compilation. We can easily cross-compile by using pkgsCross:

{
  # ...
  outputs = { self, nixpkgs, ... }:
    let
      # ...
    in
    {
      packages = forAllSystems (system:
        let pkgs = nixpkgsFor.${system};
        in {
          twenty-twenty-twenty-static-aarch64 = pkgs.pkgsCross.aarch64-multiplatform.pkgsStatic.callPackage ./twenty-twenty-twenty.nix {
            withStatic = true;
          };
        });
    };
}

The idea of pkgsCross is to select a target platform (e.g.: aarch64-multiplatform) and use it as any other pkgs. We can even chain pkgsStatic to statically cross compile binaries:

$ nix build .#twenty-twenty-twenty-static-aarch64

$ file result/bin/twenty-twenty-twenty
result/bin/twenty-twenty-twenty: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped

Meta: are quick bits really quick?

Wed, 07 Aug 2024 00:00:00 +0000

When I wrote my first quick bits post in this blog, I gave that title prefix without much thought: I knew it was supposed to be a quicker post than my previous one, but I never thought too deeply about it. But after:

$ ls -lah **/*quick-bits*.md | wc -l
4

Well, 4 blog posts starting with the same prefix, I was curious: are those quick bits really quick, or at least quicker? Let's see:

$ wc -w **/*.md
 1107 2024-07-26/01-writing-nixos-tests-for-fun-and-profit.md
 1220 2024-07-26/02-using-github-as-a-bad-blog-platform.md
  286 2024-07-27/01-quick-bits-nix-shell-is-cursed.md
  387 2024-07-29/01-quick-bits-why-you-should-automate-everything.md
 1060 2024-07-29/02-go-a-reasonable-good-language.md
 1380 2024-07-30/01-first-impressions-fpgbc.md
 1238 2024-07-31/01-generating-yaml-files-with-nix.md
 2308 2024-08-01/01-troubleshooting-zsh-lag-and-solutions-with-nix.md
  504 2024-08-01/02-quick-bits-realise-nix-symlinks.md
  834 2024-08-04/01-make-nixd-modules-completion-work-anywhere-with-flakes.md
 1147 2024-08-05/01-my-favorite-device-is-a-chromebook.md
  394 2024-08-07/01-quick-bits-is-crostini-a-microvm.md
  120 README.md
11985 total

While using wc is probably not the best way to measure word count (especially in this blog, since I tend to write lots of code snippets), I think this at least it gives me a good insight: yes, quick bits are quicker, and they're basically posts with a soft limit around 500 words. So expect in future this limit to be used.

By the way, at:

$ wc -w 2024-08-07/02-meta-are-quick-bits-really-quick.md
220 2024-08-07/02-meta-are-quick-bits-really-quick.md

This post is also technically a quick bits post, but "quick bits meta" would be too much. And yes, that last block of code is also meta ;).

Quick bits: is Crostini a micro VM?

Wed, 07 Aug 2024 00:00:00 +0000

Disclaimer: I am not a Virtual Machine specialist, this post is mainly the conclusion I got after some searching.

Someone asked me in my previous post about my Chromebook if Crostini could be considered a micro VM. This was a interesting question, so I decided to do another post.

To start, it is really difficult to get a good definition of what a "micro VM" is. Firecracker defines itself as a micro VM, and describes itself in its website as:

Firecracker is a virtual machine monitor (VMM) that uses the Linux Kernel-based Virtual Machine (KVM) to create and manage microVMs. Firecracker has a minimalist design. It excludes unnecessary devices and guest functionality to reduce the memory footprint and attack surface area of each microVM. This improves security, decreases the startup time, and increases hardware utilization.

Now looking at Crostini, its heart is a VMM called crosvm. It is described in its README as:

crosvm is a virtual machine monitor (VMM) based on Linux’s KVM hypervisor, with a focus on simplicity, security, and speed. crosvm is intended to run Linux guests, originally as a security boundary for running native applications on the ChromeOS platform. Compared to QEMU, crosvm doesn’t emulate architectures or real hardware, instead concentrating on paravirtualized devices, such as the virtio standard.

Similar descriptions right? Actually Firecracker website says it "started from Chromium OS's Virtual Machine Monitor, crosvm, an open source VMM written in Rust". So I would say it is safe to say crosvm itself is a micro VM.

But Crostini itself is a combination of virtualization AND containerization. Basically inside the VM it runs a Linux kernel and LXC, that can start arbitrary containers inside it. From the Crostini documentation this choice seems to be to keep startup times down, and also to increase security (e.g.: in case of a security issue inside the container).

This is definitely an interesting choice, since containers allow the overhead of each distro that you run inside Crostini to be low, and the main VM itself (called Termina) should have low overhead too thanks to crosvm.

By the way, if you want to learn more how "devices" works inside a micro VM like crosvm, I recommend this blog post talking about paravirtualized devices in crosvm.

My favorite device is a Chromebook

Mon, 05 Aug 2024 00:00:00 +0000

Most of the posts in this blog (including this one) and most of I would call "personal computing" that I do nowadays is mostly done in one of the most unremarkable devices that I own: a cheap Chromebook Duet 3, that I bought for around EUR300. I was thinking why, because it is woefully underpowered: a Snapdragon 7c Gen 2, a CPU that was already considered slow 3 years ago, coupled with an eMMC for storage, that is not much faster than a HDD. At least I have the 8GB RAM version instead of the 4GB one.

It is a hybrid device, one that can be used as either a tablet or laptop, but it is compromised experience in both cases: as a tablet, it lacks the better touch optimised interface from iOS or Android; as a laptop, you have to depend on the stand to adjust the screen, and the detachable keyboard is worse than any laptop I have ever owned: getting keys stucked and characters being duplicated as a result is a common occurence. It is not so bad that I can't get things done though. About the trackpad: its biggest quality is that I never feel the need to use the touchscreen in laptop mode, that is to say that it is acceptable. Just crank up the pointer speed in ChromeOS settings, otherwise you never get anywhere since the trackpad is so small. There is also an active stylus, that helped me sometimes when I needed to sign something but otherwise I can't comment too much.

But I really love this device. It is generally the only device that I bring in trips nowadays, because while it is compromised it works well enough: I can use to consume media in tablet mode (the fact that ChromeOS supports Android apps is a plus in those cases), browse the web and even do Linux stuff (more about this later). The fact that it is small (the size remembers me of a netbook), lightweight (~1KG, including the keyboard), has a good screen (that is bright and HiDPI) and good battery life (I don't have numbers but I almost never worry about it) is what makes this device the perfect companion to trips.

Also, it has 2 USB-C ports and supports DisplayPort alt-mode, so it means you can charge it, connect to a external display and peripherals, all at the same time. Sadly, the maximum output resolution I got was 1080p (2560x1080), although some people at Reddit seems to have success at 1440p, and the specs suggests it supports 4k. It may be my Dell S3423DWC monitor being wonky, the fact that it is Ultrawide or the cable, who knows? I even tried to change the monitor to "High Resolution" mode in settings, but to no avail.

Update: looking at the product brief for Snapdragon 7c Gen 2, it seems it supports up to QHD@60Hz (1440p) for the external display, and not 4k. This explains why it doesn't work at maximum resolution in my Dell S3423DWC, since while it is 1440p it is Ultrawide, so the resolution is bigger than QHD (3440x1440 vs 2560x1440).

ChromeOS is also really interesting nowadays. To start, it is designed from the ground up to be a secure computing environment, probably the most secure OS for consumers right now. Being a Chrome-first OS makes it a compromised experience, for example, it is the only device that I use Chrome as my main browser (since I personally prefer Firefox). But having a OS that boots fast is great: I never worry about OS updates because I know the device will be ready in seconds after a reboot. And the whole desktop experience inside the ChromeOS desktop is good, having shortcuts for many operations so you can get things done fast, and support for virtual desktops (ChromeOS call it "desks") means you can organise your windows as much as you want.

And what I think makes ChromeOS really powerful is Crostini, a full Linux VM that you can run inside ChromeOS. It runs Debian (it seems you can run other distros though) with a deep integration with ChromeOS, so you can run even graphical programs without issues (including OpenGL!):

This is all thanks to Sommelier, a nested Wayland compositor that runs inside Crostini and allow both Wayland and X11 applications to be forwarded to ChromeOS. The integration is so good that I can run Firefox inside Crostini and works well enough, but sadly Firefox is too slow in this device (I am not sure if the issue is ChromeOS or Firefox, but I suspect the later since Google does some optimisation per device).

One interesting tidbit about the OpenGL situation in this device: this seems to be the first Chromebook to ship with open source drivers, thanks to Freedreno. There is this very interesting presentation done by Rob Clark in XDC 2021, that I recommended anyone interested in free drivers to watch (the reference design of Duet 3 is Strongbad).

The Crostini integration is probably the best VM integration with Linux I ever saw in an OS: you can manage files inside the VM, share directories between the OS and VM, copy and paste works between the two, GUI applications installed inside the VM appear in the ChromeOS menu, memory allocation inside the VM is transparent, etc. Even the themes for Linux GUI applications are customised to match ChromeOS. It is unironically one of the best Linux desktop experiences I ever had.

Of course I am using Nix, but since the Crostini integration depends on some services being configured and installed, I decided to run Nix inside Debian instead of NixOS and run Home-Manager standalone. I recommend checking the official NixOS Wiki article about Crostini, that details how to register applications in ChromeOS (so desktop applications appear in menu) and use nixGL to make OpenGL applications work.

Like I said at the start of the article, the device is woefully slow thanks to its CPU and eMMC. It does mean that, for example, activating my Home-Manager configuration takes a while (around 1 minute, vs a few seconds in my laptop). But it is much faster than say, nix-on-droid, that the last time I tried in a much more powerful device (Xiaomi Pad 5), took 30 minutes until I just decided to cancel the operation. Having a proper VM instead of proot makes all the difference here.

I can even do some light programming here: using my ZSH and neovim configuration (including LSP for coding) is reasonable fast. For example, I did most of the code that powers this blog using this Chromebook. If I need more power, I can use the Tailscale app for Android to connect to any other of my hosts via SSH. Yes, the Tailscale app works in Crostini, sadly without MagicDNS, so you need to use the internal Tailscale IPs instead.

Until Google decides to give us a proper VM or user namespaces in Android or release a hybrid Chromebook device with better specs, this small Chromebook will probably stay as my travel companion, and is one of my favorite devices.

Make nixd module completion to work anywhere (with Flakes)

Sun, 04 Aug 2024 00:00:00 +0000

If you want the TL;DR, go to the bottom of the post (search for "final result").

I recently switched from nil to nixd as my LSP of choice for Nix. I was curious in nixd for a long time since the fact that it can eval Nix code means it can offer much more powerful completion than the nil's static analysis, however it used to be difficult to setup. Nowadays it is much easier, basically doing the right thing as long as you have NIX_PATH setup, and you get both package and NixOS modules completion.

Getting Home-Manager modules though needs some setup. The recommended way to setup accordingly to the official documentation is to use the following for Flake based configurations (using neovim configuration here, but it should be easy to adapt to other editors):

{
  nixpkgs = {
    expr = "import <nixpkgs> { }",
  },
  options = {
    nixos = {
      expr = '(builtins.getFlake ("git+file://" + toString ./.)).nixosConfigurations.miku-nixos.options',
    },
    home_manager = {
      expr = '(builtins.getFlake ("git+file://" + toString ./.)).homeConfigurations.home-linux.options',
    },
  },
  -- ...
}

This works, but it should be pretty clear the limitations of using ./.: this will only work if you open your editor in your Nix configuration repository. For any other repository, nixosConfigurations.miku-nixos or homeConfigurations.home-linux will not exist and the completion will not work.

It may look like this is easy to fix if you have specialArgs (or extraSpecialArgs in Home-Manager) set to have your Flakes inputs, but:

# By the way, ${self} does not exist in the Flake output by default, you need
# to explicit add `inherit self` to your outputs:
# https://discourse.nixos.org/t/who-is-self-in-flake-outputs/31859/4
nix-repl> (builtins.getFlake "git+file://${self}").nixosConfigurations.miku-linux.options
error:
       … while calling the 'getFlake' builtin
         at «string»:1:2:
            1| (builtins.getFlake "git+file://${self}")
             |  ^

       … while evaluating the argument passed to builtins.getFlake

       error: the string 'git+file:///nix/store/avr1lcmznj8ghynh5vj1kakgfdf0zrxx-source' is not allowed to refer to a store path (such as 'avr1lcmznj8ghynh5vj1kakgfdf0zrxx-source')

Well, it was worth a try. Another option would be to:

(builtins.getFlake "github:thiagokokada/nix-configs").nixosConfigurations.miku-linux.options
# Or even something like this
# However, using ${rev} means this wouldn't work in dirty Flake repos, since
# ${rev} is not set in those cases
(builtins.getFlake "github:thiagokokada/nix-configs/${rev}").nixosConfigurations.miku-linux.options

But while it works, it is slow, because it needs network to evaluate (and it is impure, since there is no flake.lock).

The default configuration for nixd makes NixOS completion work even outside of my configuration repo, and it is fast. How? I decided to take a look at the nixd source code and found this (formatted here for legibility):

(
  let
    pkgs = import <nixpkgs> { };
  in
  (pkgs.lib.evalModules {
    modules = (import <nixpkgs/nixos/modules/module-list.nix>) ++ [
      ({ ... }: { nixpkgs.hostPlatform = builtins.currentSystem; })
    ];
  })
).options

Interesting, so they're manually loading the modules using evalModules. As I said above, it depends in NIX_PATH being correctly set. Can we fix this to use our Flake inputs instead? After some tries in the Nix REPL, I got the following:

(
  let
    pkgs = import "${inputs.nixpkgs}" { };
  in
  (pkgs.lib.evalModules {
    modules = (import "${inputs.nixpkgs}/nixos/modules/module-list.nix") ++ [
      ({ ... }: { nixpkgs.hostPlatform = builtins.currentSystem; })
    ];
  })
).options

So we can adapt this to the neovim configuration:

{
  options = {
    nixos = {
      expr = '(let pkgs = import "${inputs.nixpkgs}" { }; in (pkgs.lib.evalModules { modules =  (import "${inputs.nixpkgs}/nixos/modules/module-list.nix") ++ [ ({...}: { nixpkgs.hostPlatform = builtins.currentSystem;} ) ] ; })).options',
    },
  },
}

This was easy. But the main issue is Home-Manager. How can we fix it? I needed to take a look at the Home-Manager source code to find the answer:

(
  let
    pkgs = import "${inputs.nixpkgs}" { };
    lib = import "${inputs.home-manager}/modules/lib/stdlib-extended.nix" pkgs.lib;
  in
  (lib.evalModules {
    modules = (import "${inputs.home-manager}/modules/modules.nix") {
      inherit lib pkgs;
      check = false;
    };
  })
).options

The interesting part is: Home-Manager has its own extension of the module system (including evalModules). This includes e.g.: extra types used in Home-Manager only. Also, we need to disable checks, otherwise we will hit some validations (e.g.: missing stateVersion). I am not sure if this causes any issue for module completion yet, I may set it in the future.

And for the final result:

{
  nixpkgs = {
    expr = 'import "${flake.inputs.nixpkgs}" { }',
  },
  options = {
    nixos = {
      expr = '(let pkgs = import "${inputs.nixpkgs}" { }; in (pkgs.lib.evalModules { modules =  (import "${inputs.nixpkgs}/nixos/modules/module-list.nix") ++ [ ({...}: { nixpkgs.hostPlatform = builtins.currentSystem;} ) ] ; })).options',
    },
    home_manager = {
      expr = '(let pkgs = import "${inputs.nixpkgs}" { }; lib = import "${inputs.home-manager}/modules/lib/stdlib-extended.nix" pkgs.lib; in (lib.evalModules { modules =  (import "${inputs.home-manager}/modules/modules.nix") { inherit lib pkgs; check = false; }; })).options',
    },
  },
}

Yes, it is quite a mouthful, but it makes module completion work in any repository, as long as you're using Flakes. And it is fast, since it doesn't need any network access. Since we are already here, let's define nixpkgs to not depend in the NIX_PATH being set too.

Quick bits: realise Nix symlinks

Thu, 01 Aug 2024 00:00:00 +0000

When you are using Nix, especially with Home-Manager, there are times when you want to test something or maybe debug some issue in your configuration. Those times it would be really convenient if you could avoid a rebuild of your Home-Manager configuration, since this takes time until evaluation and activation.

For those times I have this small script in my Nix configuration called realise-symlinks, that is defined as:

{ pkgs, ... }:
let
  realise-symlink = pkgs.writeShellApplication {
    name = "realise-symlink";
    runtimeInputs = with pkgs; [ coreutils ];
    text = ''
      for file in "$@"; do
        if [[ -L "$file" ]]; then
          if [[ -d "$file" ]]; then
            tmpdir="''${file}.tmp"
            mkdir -p "$tmpdir"
            cp --verbose --recursive "$file"/* "$tmpdir"
            unlink "$file"
            mv "$tmpdir" "$file"
            chmod --changes --recursive +w "$file"
          else
            cp --verbose --remove-destination "$(readlink "$file")" "$file"
            chmod --changes +w "$file"
          fi
        else
          >&2 echo "Not a symlink: $file"
          exit 1
        fi
      done
    '';
  };
in
{
  home.packages = [ realise-symlink ];
}

The idea of this script is that you can call it against a symlink against Nix store and it will realise, e.g.: convert to an "actual" file, e.g.:

$ ls -lah .zshrc
lrwxrwxrwx 1 thiagoko users 69 Aug  1 00:10 .zshrc -> /nix/store/glz018yyh0qfqc9lywx1yhr7c3l96lv7-home-manager-files/.zshrc

$ realise-symlink .zshrc
removed '.zshrc'
'/nix/store/glz018yyh0qfqc9lywx1yhr7c3l96lv7-home-manager-files/.zshrc' -> '.zshrc'
mode of '.zshrc' changed from 0444 (r--r--r--) to 0644 (rw-r--r--)

$ ls -lah .zshrc
-rw-r--r-- 1 thiagoko users 5.8K Aug  1 00:16 .zshrc

It also add write permissions to the resulting file, to make it easier to edit. By the way, it also works with directories:

$ ls -lah zim-completion
lrwxrwxrwx 1 thiagoko users 90 Aug  1 00:10 zim-completion -> /nix/store/glz018yyh0qfqc9lywx1yhr7c3l96lv7-home-manager-files/.zsh/plugins/zim-completion

$ realise-symlink zim-completion
'zim-completion/init.zsh' -> 'zim-completion.tmp/init.zsh'
'zim-completion/init.zsh.zwc' -> 'zim-completion.tmp/init.zsh.zwc'
'zim-completion/LICENSE' -> 'zim-completion.tmp/LICENSE'
'zim-completion/README.md' -> 'zim-completion.tmp/README.md'
mode of 'zim-completion/init.zsh' changed from 0444 (r--r--r--) to 0644 (rw-r--r--)
mode of 'zim-completion/init.zsh.zwc' changed from 0444 (r--r--r--) to 0644 (rw-r--r--)
mode of 'zim-completion/LICENSE' changed from 0444 (r--r--r--) to 0644 (rw-r--r--)
mode of 'zim-completion/README.md' changed from 0444 (r--r--r--) to 0644 (rw-r--r--)

$ ls -lah zim-completion
total 28K
drwxr-xr-x 1 thiagoko users   72 Aug  1 00:18 .
drwxr-xr-x 1 thiagoko users  130 Aug  1 00:18 ..
-rw-r--r-- 1 thiagoko users 5.3K Aug  1 00:18 init.zsh
-rw-r--r-- 1 thiagoko users  12K Aug  1 00:18 init.zsh.zwc
-rw-r--r-- 1 thiagoko users 1.3K Aug  1 00:18 LICENSE
-rw-r--r-- 1 thiagoko users 2.2K Aug  1 00:18 README.md

After you finish whatever you are testing, to return to your configuration you can just delete those files and re-run your Home-Manager activation:

$ rm -rf .zshrc

$ sudo systemctl restart home-manager-<user>.service # or `home-manager switch`

$ ls -lah .zshrc
lrwxrwxrwx 1 thiagoko users 69 Aug  1 00:20 .zshrc -> /nix/store/glz018yyh0qfqc9lywx1yhr7c3l96lv7-home-manager-files/.zshrc

It even works with system files (i.e.: NixOS):

$ sudo realise-symlink /etc/nix/nix.conf
[sudo] password for thiagoko:
removed 'nix.conf'
'/etc/static/nix/nix.conf' -> 'nix.conf'
mode of 'nix.conf' changed from 0444 (r--r--r--) to 0644 (rw-r--r--)

But I never needed for this case since it is more rare to me to experiment with OS level configuration.

Troubleshooting: ZSH lag and solutions with Nix

Thu, 01 Aug 2024 00:00:00 +0000

Inspired by this blog post from Tavis, I decided to document my own recent journey of reducing terminal (ZSH) lag startup. This post is way less interesting than the one from Tavis that uses a debugger to patch applications on the fly, but should still be interesting for some. And it also shows how powerful Nix can be for some things.

For context, I have basically 3 systems where I interact with terminal frequently:

Thinkpad P14s Gen 1 running NixOS, with a reasonable fast CPU (AMD Ryzen 7 PRO 4750U) and disk (NVMe)
MacBook Pro with a really fast CPU (M1 Pro) and disk, but of course running macOS, being phased-out since this is a job owned machine and I am changing jobs right now, but should be replaced with another one soon™
Chromebook Duet 3 running ChromeOS, with slow CPU (Snapdragon 7c Gen 2) and disk (eMMC, really?)

My experience is similar to Tavis, at around 300ms of startup time I don't care too much, but around 500ms+ is where I start to notice. I never had any issues with startup time in NixOS itself (I had issues with macOS before, but it was not actually the fault of macOS), but in the Chromebook it was awful: 600ms+ with hot start, while cold start it could take multiple seconds.

We can check how long ZSH takes to start by using:

$ time zsh -ci exit
zsh -ic exit  0.04s user 0.10s system 100% cpu 0.143 total

The -i flag here is important, because we are interested in the interactive use of ZSH. Without this flag ZSH will ignore your ~/.zshrc file, and the results will be meaningless.

To do a more interesting benchmark, we can use hyperfine:

$ hyperfine "zsh -ic exit"
Benchmark 1: zsh -ic exit
  Time (mean ± σ):     145.4 ms ±   4.2 ms    [User: 49.8 ms, System: 97.3 ms]
  Range (min … max):   138.6 ms … 155.3 ms    19 runs

Hyperfine will run the command multiple times and take care of things like shell startup time. A really great tool to have in your toolbox by the way, but I digress.

So let's do a little time travelling. Going back to commit b12757f from nix-configs. Running hyperfine like above from my NixOS laptop, we have:

$ hyperfine "zsh -ic exit"
Benchmark 1: zsh -ic exit
  Time (mean ± σ):     218.6 ms ±   5.1 ms    [User: 70.6 ms, System: 151.5 ms]
  Range (min … max):   210.3 ms … 227.0 ms    13 runs

This doesn't look that bad, but let's see the same commit in my Chromebook:

$ hyperfine "zsh -ic exit"
Benchmark 1: zsh -ic exit
  Time (mean ± σ):     679.7 ms ±  40.2 ms    [User: 230.8 ms, System: 448.5 ms]
  Range (min … max):   607.3 ms … 737.0 ms    10 runs

Yikes, this is much worse. And those are the results after I retried the benchmark (so it is a hot start). The cold start times were above 3s. So let's investigate what is happening here. We can profile what is taking time during the startup of ZSH using zprof. You can add the following in your ~/.zshrc:

# At the top of your ~/.zshrc file
zmodload zsh/zprof

# ...

# At the end of your ~/.zshrc file
zprof

Or if using Home-Manager, use the programs.zsh.zprof.enable option. Once we restart ZSH, we will have something like:

num  calls                time                       self            name
-----------------------------------------------------------------------------------
 1)    1          36.91    36.91   34.29%     30.47    30.47   28.31%  (anon) [/home/thiagoko/.zsh/plugins/zim-completion/init.zsh:13]
 2)    1          25.43    25.43   23.63%     25.43    25.43   23.63%  (anon) [/home/thiagoko/.zsh/plugins/zim-ssh/init.zsh:6]
 3)    1          22.00    22.00   20.45%     21.92    21.92   20.36%  _zsh_highlight_load_highlighters
 4)    1          12.32    12.32   11.45%     12.32    12.32   11.45%  autopair-init
 5)    1           6.44     6.44    5.98%      6.44     6.44    5.98%  compinit
 6)    1           3.56     3.56    3.31%      3.48     3.48    3.23%  prompt_pure_state_setup
 7)    2           3.79     1.89    3.52%      2.85     1.43    2.65%  async
 8)    1           0.93     0.93    0.87%      0.93     0.93    0.87%  async_init
 9)    6           0.93     0.15    0.86%      0.93     0.15    0.86%  is-at-least
10)    6           0.67     0.11    0.63%      0.67     0.11    0.63%  add-zle-hook-widget
11)    1           8.25     8.25    7.66%      0.61     0.61    0.57%  prompt_pure_setup
12)    1           0.40     0.40    0.37%      0.40     0.40    0.37%  (anon) [/nix/store/p1zqypy7600fvfyl1v571bljx2l8zhay-zsh-autosuggestions-0.7.0/share/zsh-autosuggestions/zsh-autosuggestions.zsh:458]
13)    5           0.31     0.06    0.29%      0.31     0.06    0.29%  add-zsh-hook
14)    1           0.60     0.60    0.56%      0.29     0.29    0.27%  (anon) [/home/thiagoko/.zsh/plugins/zim-input/init.zsh:5]
15)    1           0.21     0.21    0.20%      0.21     0.21    0.20%  compdef
16)    1           0.10     0.10    0.09%      0.10     0.10    0.09%  _zsh_highlight__function_is_autoload_stub_p
17)    1           0.26     0.26    0.24%      0.08     0.08    0.08%  _zsh_highlight__function_callable_p
18)    1           0.08     0.08    0.08%      0.08     0.08    0.08%  prompt_pure_is_inside_container
19)    1           0.07     0.07    0.07%      0.07     0.07    0.07%  _zsh_highlight__is_function_p
20)    1           0.01     0.01    0.01%      0.01     0.01    0.01%  __wezterm_install_bash_prexec
21)    1           0.00     0.00    0.00%      0.00     0.00    0.00%  _zsh_highlight_bind_widgets
# ...

I ommited some output for brevit. The first 2 things that shows are from the zimfw, the framework that I use to configure my ZSH (similar to Oh-My-Zsh). I actually don't use zimfw directly, instead I just load some modules that I find useful, like the zim-completion and zim-ssh that we can see above. By the way, Zim is generally really well optimised for startup time, but those 2 modules are kind slow.

For zim-completion, after taking a look at it, there isn't much I could do. It seems that the reason zim-completion takes so long during startup is because it is trying to decide if it needs to recompile the completions (and replacing it with just a naive autoload -U compinit && compinit is even worse for startup performance). I may eventually replace it for something else, but I really like what Zim brings here, so I decided to not touch it for now.

However zim-ssh is another history. The only reason I used it is to start a ssh-agent and keep it between multiple ZSH sessions. It shouldn't have that much influence in startup time. So I took a look the code (since it is small, I am reproducing it here):

#
# Set up ssh-agent
#

# Don't do anything unless we can actually use ssh-agent
(( ${+commands[ssh-agent]} )) && () {
  ssh-add -l &>/dev/null
  if (( ? == 2 )); then
    # Unable to contact the authentication agent

    # Load stored agent connection info
    local -r ssh_env=${HOME}/.ssh-agent
    if [[ -r ${ssh_env} ]] source ${ssh_env} >/dev/null

    ssh-add -l &>/dev/null
    if (( ? == 2 )); then
        # Start agent and store agent connection info
        (umask 066; ssh-agent >! ${ssh_env})
        source ${ssh_env} >/dev/null
    fi
  fi

  # Load identities
  ssh-add -l &>/dev/null
  if (( ? == 1 )); then
    local -a zssh_ids
    zstyle -a ':zim:ssh' ids 'zssh_ids'
    if (( ${#zssh_ids} )); then
      ssh-add ${HOME}/.ssh/${^zssh_ids} 2>/dev/null
    else
      ssh-add 2>/dev/null
    fi
  fi
}

Well, this is bad. Let's assume the common path, where the ssh-agent is already running but you open a new shell instance (that doesn't have the connection info yet so it will need to load). This will run ssh-add at 4 times. How long does ssh-add takes to run?

$ hyperfine -Ni "ssh-add -l"
Benchmark 1: ssh-add -l
  Time (mean ± σ):       4.6 ms ±   1.1 ms    [User: 2.0 ms, System: 2.0 ms]
  Range (min … max):     3.4 ms …   8.7 ms    619 runs

  Warning: Ignoring non-zero exit code.

For those curious, -N disables the Shell usage, that works better when the command being tested is too fast.

In average we have 4x4ms=16ms of startup time. But keep in mind the worst case can be much worse. The question is, how can we improve the situation here?

After taking a look, I decided to write my own code, based in some ideas stolen from Oh-My-Zsh ssh-agent plugin. Here is final version of my code:

zmodload zsh/net/socket

_check_agent(){
  if [[ -S "$SSH_AUTH_SOCK" ]] && zsocket "$SSH_AUTH_SOCK" 2>/dev/null; then
    return 0
  fi
  return 1
}

_start_agent() {
  # Test if $SSH_AUTH_SOCK is visible, in case we start e.g.: ssh-agent via
  # systemd service
  if _check_agent; then
    return 0
  fi

  # Get the filename to store/lookup the environment from
  local -r ssh_env_cache="$HOME/.ssh-agent"

  # Check if ssh-agent is already running
  if [[ -f "$ssh_env_cache" ]]; then
    source "$ssh_env_cache" > /dev/null

    # Test if $SSH_AUTH_SOCK is visible, e.g.: the ssh-agent is still alive
    if _check_agent; then
      return 0
    fi
  fi

  # start ssh-agent and setup environment
  (
    umask 066
    ssh-agent -s >! "$ssh_env_cache"
  )
  source "$ssh_env_cache" > /dev/null
}

_start_agent
unfunction _check_agent _start_agent

The idea here is simple: using zsocket module from ZSH itself to check if the ssh-agent is working instead of executing ssh-add -l. The only case we run any program now is to start the agent itself if needed. Let's run hyperfine again:

$ hyperfine "zsh -ic exit"
Benchmark 1: zsh -ic exit
  Time (mean ± σ):     188.3 ms ±   8.2 ms    [User: 61.1 ms, System: 130.0 ms]
  Range (min … max):   170.9 ms … 198.4 ms    16 runs

Got a good improvement here already. Let's see zprof again:

num  calls                time                       self            name
-----------------------------------------------------------------------------------
 1)    1          41.23    41.23   48.66%     33.52    33.52   39.56%  (anon) [/home/thiagoko/.zsh/plugins/zim-completion/init.zsh:13]
 2)    1          22.23    22.23   26.24%     22.12    22.12   26.10%  _zsh_highlight_load_highlighters
 3)    1           8.90     8.90   10.51%      8.90     8.90   10.51%  Gautopair-init
 4)    1           7.71     7.71    9.10%      7.71     7.71    9.10%  compinit
 5)    1           5.74     5.74    6.77%      5.60     5.60    6.60%  prompt_pure_state_setup
 6)    6           1.19     0.20    1.41%      1.19     0.20    1.41%  add-zle-hook-widget
 7)    2           1.97     0.99    2.33%      1.14     0.57    1.34%  async
 8)    6           0.87     0.15    1.03%      0.87     0.15    1.03%  is-at-least
 9)    1           0.84     0.84    0.99%      0.84     0.84    0.99%  async_init
10)    1           9.30     9.30   10.97%      0.72     0.72    0.84%  prompt_pure_setup
11)    5           0.63     0.13    0.75%      0.63     0.13    0.75%  add-zsh-hook
12)    1           0.41     0.41    0.48%      0.41     0.41    0.48%  _start_agent
13)    1           0.31     0.31    0.37%      0.31     0.31    0.37%  (anon) [/nix/store/p1zqypy7600fvfyl1v571bljx2l8zhay-zsh-autosuggestions-0.7.0/share/zsh-autosuggestions/zsh-autosuggestions.zsh:458]
14)    1           0.55     0.55    0.64%      0.24     0.24    0.28%  (anon) [/home/thiagoko/.zsh/plugins/zim-input/init.zsh:5]
15)    1           0.14     0.14    0.16%      0.14     0.14    0.16%  prompt_pure_is_inside_container
16)    1           0.14     0.14    0.16%      0.14     0.14    0.16%  compdef
17)    1           0.09     0.09    0.11%      0.09     0.09    0.11%  _zsh_highlight__function_is_autoload_stub_p
18)    1           0.25     0.25    0.29%      0.08     0.08    0.09%  _zsh_highlight__function_callable_p
19)    1           0.07     0.07    0.09%      0.07     0.07    0.09%  _zsh_highlight__is_function_p
20)    1           0.01     0.01    0.01%      0.01     0.01    0.01%  __wezterm_install_bash_prexec
21)    1           0.01     0.01    0.01%      0.01     0.01    0.01%  _zsh_highlight_bind_widgets
# ...

Well, there is nothing interesting here anymore. I mean, zim-completion is still the main culprit, but nothing to do for now. Instead of looking at zproof, let's take a look at my ~/.zshrc instead:

# ...
if [[ $options[zle] = on ]]; then
  eval "$(/nix/store/sk6wsgp4h477baxypksz9rl8ldwwh9yg-fzf-0.54.0/bin/fzf --zsh)"
fi

# ...
/nix/store/x3yblr73r5x76dmaanjk3333mvzxc49r-any-nix-shell-1.2.1/bin/any-nix-shell zsh | source /dev/stdin

# ...
eval "$(/nix/store/330d6k81flfs6w46b44afmncxk57qggv-zoxide-0.9.4/bin/zoxide init zsh )"

# ...
eval "$(/nix/store/8l9j9kdv9m0z0s30lp4yvrc9s5bcbgmx-direnv-2.34.0/bin/direnv hook zsh)"

So you see, starting all those programs during ZSH startup can hurt the shell startup considerable. Not necessary for commands fast like fzf (that is written in Go), but let's see any-nix-shell, that is written in shell script:

$ hyperfine "any-nix-shell zsh"
Benchmark 1: any-nix-shell zsh
  Time (mean ± σ):      16.0 ms ±   1.8 ms    [User: 5.6 ms, System: 10.5 ms]
  Range (min … max):    11.3 ms …  20.3 ms    143 runs

This is bad, consistently bad actually. Even for commands that are fast, keep in mind that there is a difference between the cold and hot start again. For example, fzf:

$ hyperfine -N "fzf --zsh"
Benchmark 1: fzf --zsh
  Time (mean ± σ):       2.9 ms ±   0.9 ms    [User: 0.6 ms, System: 2.3 ms]
  Range (min … max):     1.7 ms …   6.8 ms    1113 runs

See the range? While 1.7ms is something that is probably difficult to notice, 6.8ms can be noticiable, especially if this accumulates with other slow starting apps.

And the thing is, all those commands are just generating in the end a fixed output, at least for the current version of the program. Can we pre-generate them instead? If using Nix, of course we can:

# You need to disable the default integration
programs.direnv.enableZshIntegration = false;
programs.fzf.enableZshIntegration = false;
programs.zoxide.enableZshIntegration = false;

programs.zsh.initExtra =
  # bash
  ''
    # any-nix-shell
    source ${
      pkgs.runCommand "any-nix-shell-zsh" { } ''
        ${lib.getExe pkgs.any-nix-shell} zsh > $out
      ''
    }

    # fzf
    source ${config.programs.fzf.package}/share/fzf/completion.zsh
    source ${config.programs.fzf.package}/share/fzf/key-bindings.zsh

    # zoxide
    source ${
      pkgs.runCommand "zoxide-init-zsh" { } ''
        ${lib.getExe config.programs.zoxide.package} init zsh > $out
      ''
    }

    # direnv
    source ${
      pkgs.runCommand "direnv-hook-zsh" { } ''
        ${lib.getExe config.programs.direnv.package} hook zsh > $out
      ''
    }
  '';

So we can use pkgs.runCommand to run those commands during build time and source the result. fzf actually doesn't need this since we have the files already generated in the package. I think this is one of those things that really shows the power of Nix: I wouldn't do something similar if I didn't use Nix because the risk of breaking something later is big (e.g.: forgetting to update the generated files), but Nix makes those things trivial.

Let's run hyperfine again:

$ hyperfine "zsh -ic exit"
Benchmark 1: zsh -ic exit
  Time (mean ± σ):     162.3 ms ±   4.9 ms    [User: 52.7 ms, System: 111.1 ms]
  Range (min … max):   153.0 ms … 173.4 ms    19 runs

Another good improvement. The last change I did is switching between zsh-syntax-highlighting to zsh-fast-syntax-highlighting, that is supposed to be faster and have better highlighting too. I got that from _zsh_highlight_load_highlighters using 26% of the time from my zprof above. And for the final hyperfine in my laptop:

$ hyperfine "zsh -ic exit"
Benchmark 1: zsh -ic exit
  Time (mean ± σ):     138.3 ms ±   7.1 ms    [User: 47.5 ms, System: 91.9 ms]
  Range (min … max):   123.8 ms … 157.9 ms    21 runs

A ~36% improvement, not bad. Let's see how it fares in my Chromebook:

$ hyperfine "zsh -ic exit"
Benchmark 1: zsh -ic exit
  Time (mean ± σ):     278.2 ms ±  46.9 ms    [User: 88.0 ms, System: 184.8 ms]
  Range (min … max):   204.7 ms … 368.5 ms    11 runs

An even more impressive ~59% improvement. And yes, the shell startup now feels much better.

Generating YAML files with Nix

Wed, 31 Jul 2024 00:00:00 +0000

I hate YAML. Instead of writing an essay on why I hate YAML, I can just link to noyaml.com. In my personal projects I will never use it, preferring either JSON, TOML or even plain old INI files depending on the use case. However the ship has sailed already, there are tons of projects everywhere that uses YAML: from most CI systems (GitHub Actions, CircleCI, Travis, et tu builds.sr.ht), to Kubernetes, or in almost every Rails application.

One way to avoid at least some issues with the language is to write YAML in another language. I will show my solution in one of my personal repositories, writing Nix to generate GitHub Actions configuration files. Bonus points for validating the result against the schema of GitHub Actions, so the famous "this is supposed to be string instead of a list of strings" is gone.

Let's start with the basics: YAML is supposed to be a superset of JSON. What that means is that a JSON file can be parsed by a YAML parser. And Nix itself generates JSON natively, after all, Nix can be imagined as "JSON with functions".

To make things easier, I will assume that you have the nix-commands and flakes enabled as experimental-features in your Nix configuration. If not, go here.

Using the nix eval command, we can generate a JSON expression from Nix by:

$ nix eval --expr '{ foo = "bar"; }' --json
{"foo":"bar"}

However, typing long excerpts of Nix code inside the console would be impractical. We can write the following code inside a foo.nix file instead:

{
  foo = "bar";
}

And:

$ nix eval --file foo.nix --json
{"foo":"bar"}

While you can use a JSON output as an input for YAML parsers, it is probably not the best idea. Sadly (or maybe not), Nix has no native functionality to export data to YAML. However, since we are using Nix, it is trivial to use nixpkgs to use some program to convert from JSON to YAML.

To start, let's create a new directory, move our foo.nix file to it, create a new flake.nix file and put the following contents:

{
  description = "Generate YAML files with Nix";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = { nixpkgs, ... }:
    {
      packages.x86_64-linux =
        let
          inherit (nixpkgs) lib;
          pkgs = import nixpkgs { system = "x86_64-linux"; };
        in
        {
          toYAML = pkgs.runCommand "toYAML" {
            buildInputs = with pkgs; [ yj ];
            json = builtins.toJSON (import ./go.nix);
            passAsFile = [ "json" ]; # will be available as `$jsonPath`
          } ''
            mkdir -p $out
            yj -jy < "$jsonPath" > $out/go.yaml
          '';
        };
    };
}

We are loading the ./foo.nix as a Nix file, converting it to JSON with builtins.toJSON function, and finally, using pkgs.runCommand and its passAsFile option to load the contents of the JSON file into yj, that converts between serialisation formats (-jy flag means "JSON to YAML"). The reason I choose yj is mostly because it is a single binary Go program, but you can use whatever you prefer.

By the way, there is a lib.generators.toYAML inside nixpkgs.lib, but as of the day of this post it only calls lib.strings.toJSON (that in turn, calls builtins.toJSON). So it doesn't really help here. Another option would be pkgs.formats.yaml.generate, that converts between formats, but it calls remarshal (in Python), so not my favorite choice.

If we run the following commands, we can see the result:

$ nix build .#packages.x86_64-linux.toYAML
$ cat result/foo.yaml
foo: bar

That is the basic idea. To have a more realistic example, let's convert the go.yml, that builds this blog, to Nix:

{
  name = "Go";
  on.push.branches = [ "main" ];

  jobs = {
    build = {
      runs-on = "ubuntu-latest";
      permissions.contents = "write";
      steps = [
        { uses = "actions/checkout@v4"; }
        {
          name = "Set up Go";
          uses = "actions/checkout@v4";
          "with".go-version = "1.21";
        }
        {
          name = "Update";
          run = "make";
        }
        {
          name = "Publish";
          run = "make publish";
          env.MATAROA_TOKEN = ''''${{ secrets.MATAROA_TOKEN }}'';
        }
        {
          name = "Commit";
          uses = "stefanzweifel/git-auto-commit-action@v5";
          "with".commit_message = "README/rss:update";
        }
      ];
    };
  };
}

Some interesting things to highlight: with is a reserved word in Nix, so we need to quote it. Not a problem, but something to be aware. And the template string in GitHub Actions uses the same ${} that Nix uses, so we need to escape.

And after running the following commands:

$ nix build .#packages.x86_64-linux.toYAML
$ cat result/go.yaml
jobs:
  build:
    permissions:
      contents: write
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Go
        uses: actions/checkout@v4
        with:
          go-version: "1.21"
      - name: Update
        run: make
      - env:
          MATAROA_TOKEN: ${{ secrets.MATAROA_TOKEN }}
        name: Publish
        run: make publish
      - name: Commit
        uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: README/rss:update
name: Go
"on":
  push:
    branches:
      - main

Yes, the keys are not in the same order as we defined, since Nix, like most programming languages (with the exception of Python), do not guarantee the insertion order in maps/dicts/attrsets/whatever. But I really hope whatever is consuming your YAML is not relying in the order the keys are defined (this would be more cursed than YAML already is).

So that is basically it. For the bonus points that I talked at the start of the post, we can modify pkgs.runCommand to run some kind of validator. I use action-validator, one that I particularly packaged in nixpkgs to use in those cases. But you could use e.g.: a validator of Kubernetes YAML. Or a generic YAML lint like this one. The possibilities are endless.

Let's modify our flake.nix to add the validation:

{
  # ...
  outputs = { nixpkgs, ... }:
    {
      packages.x86_64-linux =
        let
          inherit (nixpkgs) lib;
          pkgs = import nixpkgs { system = "x86_64-linux"; };
        in
        {
          toYAML = pkgs.runCommand "toYAML" {
            buildInputs = with pkgs; [ action-validator yj ];
            json = builtins.toJSON (import ./go.nix);
            passAsFile = [ "json" ];
          } ''
            mkdir -p $out
            yj -jy < "$jsonPath" > $out/go.yaml
            action-validator -v $out/go.yaml
          '';
        };
    };
}

And let's add an error in our go.nix file:

diff --git a/go.nix b/go.nix
index 25e0596..8c00033 100644
--- a/go.nix
+++ b/go.nix
@@ -5,7 +5,7 @@
   jobs = {
     build = {
       runs-on = "ubuntu-latest";
-      permissions.contents = "write";
+      permissions.contents = [ "write" ];
       steps = [
         { uses = "actions/checkout@v4"; }
         {

Finally, let's try to build our YAML file again:

$ nix build .#packages.x86_64-linux.toYAML
error: builder for '/nix/store/j8wr6j1pvyf986sf74hqw8k31lvlzac5-toYAML.drv' failed with exit code 1;
       last 25 log lines:
       >                                 "Additional property 'runs-on' is not allowed",
       >                             ),
       >                             path: "/jobs/build",
       >                             title: "Property conditions are not met",
       >                         },
       >                         Properties {
       >                             code: "properties",
       >                             detail: Some(
       >                                 "Additional property 'steps' is not allowed",
       >                             ),
       >                             path: "/jobs/build",
       >                             title: "Property conditions are not met",
       >                         },
       >                         Required {
       >                             code: "required",
       >                             detail: None,
       >                             path: "/jobs/build/uses",
       >                             title: "This property is required",
       >                         },
       >                     ],
       >                 },
       >             ],
       >         },
       >     ],
       > }
       For full logs, run 'nix log /nix/store/j8wr6j1pvyf986sf74hqw8k31lvlzac5-toYAML.drv'.

Yes, the output of action-validator is awfully verbose, but it is still better than making "8 commits/push in one hour".

If you are interested in how a more advantage usage of this technique is, including usage of functions and constants to share common steps between different actions, please take a look at the actions (permalink) in my nix-config repository.

First impressions: FPGBC

Tue, 30 Jul 2024 00:00:00 +0000

Here is something for nostalgia: I just put together a Game Boy Color made of completely new parts for a friend: here is the FPGBC.

The FP part of the name comes from FPGA, because instead of software emulation this device use FPGA to reproduce the device. While I am not convinced that FPGA is necessary more accurate than a good software emulator, one advantage of FPGA is the (possible) lower input latency thanks to the avoidance of complexity to handle the user input (e.g.: the Operational System). A quick playthrough against Motocross Maniacs seems to be fine, but I can't see much difference from my Miyoo Mini+ (I will do more comparisons between the two devices later), that is a software emulation device.

But I think focusing in accuracy is wrong, the main reason of getting a device like this one is for nostalgia, and this definitely hit the mark. The quality of the case is as good as I remember the original, and most of the details are replicate perfectly, including reproduction stickers in the back of the device. The only differences that I can find is the usage of USB-C port for charging in place of the barrel jack power adapter (thanks!), and the fact that the screen bezels are smaller compared to the original (because the screen is bigger) and doesn't include the Game Boy Color logo (that is fine in my opinion, since it would look weird in the fine bezels). It even has a supposedly working Link Cable (I don't have another Game Boy to test). Sadly it is missing the infrared sensor, but the usage of that was pretty limited anyway.

So how well does it work? I can't say for sure. I don't have any original games with me, so I am relying in backups and a flashcard for now. Many games that I tested works fine, a few of them have graphical issues that can be fixed in the menu (more about it later), and some of them doesn't boot. But I don't know if the issue with the games not booting are because of the roms, the flashcard or the device itself.

By the way, the flashcard I am using is a cheap knockoff of an Everdrive GB. This FPGBC came with firmware v1.09, while there is an update available for v1.10 in the website. I had an weird issue in the new firmware where no games would boot with this knockoff Everdrive so I had to go back to v1.09, but again, I am not sure if the issue was fact that I am using a knockoff device or this would happen with an original Everdrive GB. If you are going to buy a proper Everdrive, you probably wouldn't get a Everdrive GB anyway since it is discontinued, and it seems the newer versions have better compatibility with FPGBC.

Sadly that the update didn't work, since there is this repository that patches the firmware to boot the original logo instead of the ugly FPGBC one. And yes, for some reason the v1.09 firmware from this repository still doesn't work with my knockoff Everdrive.

By the way, it seems the device is not easy to brick: I borked the firmware update process once while trying to downgrade back to v1.09, resulting in a black screen when I turned on the console. But just connecting the device to the computer and powering on, I could flash the firmware again and the device came back to life.

About the features of the device: if you press the volume button (yes, you can press it now), it opens the following menu:

The first 2 options are the LCD backlight (BKLT) and volume (VOL). I didn't talk about those, but the LCD screen seems to be IPS, and the quality is really good, and also looks bright enough to play even under bad lightining conditions. And the speaker has good quality, the sound is better than I remember, but sadly the maximum volume is kind low. Still should be enough for playing in a quiet room.

DISPMOD is probably the most controversial option: it allow you to set which scale you want. Anything with EMU at the end means emulating the original colors, and as far I remember it gets really close. You can also chose betwen X4, X4P and FUL, the last one is the one shown in the photos where the image fills the whole screen at the cost of non-integer scaling. X4 is integer scaling, however the image doesn't fill the whole screen. The X4P also includes a pixel effect that makes the image closer than the original screen. It actually looks good, but the fact that I chose a white border for this FPGBC makes the border really distracting. Maybe the black one is a better choice if you want integer scale.

CORE is simple: you can choose between GB (Game Boy) or GBC (Game Boy Color). For those who don't know, you can run Game Boy games in Game Boy Color and they will be automatically colorised. Some people don't like this and prefer the colors of GB, so you have this option. The GB_PALETTE allows you to chose the color in GB mode, for example, the green-ish colors from the original Game Boy or the blue-ish colors from Game Boy Light. And yes, you can choose the color palette for Game Boy games running in GBC mode by pressing a button combination at the boot screen, but it seems not working in my unit and again, not sure if the fault is my knockoff Everdrive.

FRAME_MIX basically is an option that makes some effects, like transparency in Wave Race, to work at the cost of introducing blurriness. The reason for this is that those effects depends in the fact that the Game Boy screen was slow refresh, so you could rely on it by rapidly changing pixels to create some interesting effects, but sadly those effects doesn't work well in modern displays.

GB_CLRFIX is the option I mentioned before, where some Game Boy games just get completely wrong colors for some reason, e.g.: The Addams Family. Turning on fixes those games, but I am not sure if this option breaks other games.

Finally, SPD allows you to increase or decrease the CPU clock, slowing or speeding up the games (including the sound). The result can be hilarious, so I think this is a nice addition to the features. Sadly you can't know what the default speed is, so you need to rely on sound to adjust back to the default.

So in the end, can I recommend a FPGBC? I am not sure. If you want a device to play games, I still think something like a Miyoo Mini+ is a better choice. Not only you will have access to more games from different platforms, you also don't need to rely on flashcards or cartridges. Also it has way more features than FPGBC, like wireless multiplayer, RetroArchivements and save states.

But the actual reason to get a FPGBC is nostalgia, and for that I think the FPGBC is difficult to beat. The price of the kit to assemble ($69.90) is cheaper than most Game Boy's in good condition you can find in eBay, and you get for that price a rechargable battery, an amazing quality screen, the PCB and the speaker. You need to buy separately the case and the buttons, but in total you will still end up paying less, and allows you to fully customise your build. And the result device is not only in mint condition, it is really convenient too: recharging batteries (via USB-C even) is much more convenient than buying AA batteries, and the screen not only is better but it even has backlight. You can also buy a fully built console for $99.00, but you have less options of customisation.

This is the classic case of do what I say, don't do what I do. This FPGBC is a gift, and I will buy another one soon. Can't wait to play Pokémon Gold in (almost) original hardware again.

Go, a reasonable good language

Mon, 29 Jul 2024 00:00:00 +0000

Go was one of the languages that I always was interested to learn, but never got the hang of it. I first got interested in the language when I was in my first job, between 2016-2018. At the time the language was a completely different beast: no modules, no generics, no easy way to error wrap yet, etc.

Go forward 2023 (no pun indented), I wrote my first project in Go, wrote some scripts at $CURRENT_JOB in the language, and now wrote my first library (with an interesting way to run CI). I am also writing more scripts in the language, where I would prefer to use Bash or Python before. Heck, even this blog is automatically published with a Go script, that used to be a Python one before. I can say that nowadays it is another language in my toolbox, and while it is still a love and hate relationship, recently it is more about love and less about hate.

The points that I love about Go is probably obvious for some, but still interesting to talk about anyway. The fact that the language generates static binaries by default and have fast compilation times is something that I apreciate since I first heard about the language, and now that I am using the language frequently, are points thatI appreciate even more. Something about getting almost instant feedback after changing a line of code and running go run (even with its quirks) are great for the developer experience. This is the main reason why I am using the language more frequently for scripts.

Then we have the fast startup times. I am kind of sensitive to latency, especially of command line utilities that need to answer fast when I expect them to be fast (e.g.: foo --help). This is one part where I could have issues in Python, especially for more complex programs, but in Go it is rarely an issue.

Modules are also fantastic. It is not without its weirdness (like everything in Go ecossystem), but the fact that it is so easy to add and manage dependencies in a project using only the go CLI is great. I also like that it generates a hash of every dependency, making it reproducible (well, probably not at Nix level, but still reproducible).

Since I started to talk about go CLI, what a great tool! The fact that you can manage dependencies, generate documentation, format code, lint, run tests/benchmarks/fuzzing, check code for races etc., all with just the "compiler" for the language is excelent. Still probably one of the best developer experiences I know in any programming language.

I will not even talk about the things that everyone talks about Go, like goroutines, because I just don't think I can add anything interesting to the topic.

Now for the parts that I like less, the test part still quirks me that it is not based in assertions, but thankfully it is easy to write assertions with generics nowadays:

func Equal[T comparable](t *testing.T, got, want T) {
	t.Helper()
	if got != want {
		t.Errorf("got: %#v, want: %#v", got, want)
	}
}

func GreaterOrEqual[T cmp.Ordered](t *testing.T, actual, expected T) {
	t.Helper()
	if actual < expected {
		t.Errorf("got: %v; want: >=%v", actual, expected)
	}
}

// etc...

Just one of those things that I end up re-writing in every project. Yes, I know about testify and other assertion libraries, but quoting Rob Pike here, "a little copying is better than a little dependency". As long the code you write is trivial, it is better to duplicate the code than try to import a dependency.

About another piece of code that generics allows me to write and I always end up re-writing in every project is the must* family of functions:

func must(err error) {
	if err != nil {
		panic(err)
	}
}

func must1[T any](v T, err error) T {
	must(err)
	return v
}

func must2[T1 any, T2 any](v1 T1, v2 T2, err error) T {
	must(err)
	return v1, v2
}

// must3, must4, etc...

Those functions are so useful, especially for scripts where I generally don't want to handle each error: if I have an error, I want the program to halt and print a stack trace (exactly as I would have with a language with exceptions). It basically allow me to convert code from:

contents, err := os.ReadFile("file")
if err != nil {
    panic(err)
}

To:

contents := must1(os.ReadFile("file"))

This brings Go closer to Python to me, and I think for scripts this is something great.

Finally, for the things that I hate, well the biggest one currently is the lack of nullability (or in Go terms, nillability). After using languages that has it, like Kotlin, or even something like mypy, this is one of those things that completely changes the developer experience. I also still don't like the error handling (but must* goes far by improving the situation, when it is possible to use it), especially because it is easy to lose context on it:

// bad
func readFileContents(file) ([]byte, error) {
    contents, err := os.ReadFile(file)
    if err != nil {
        return nil, err
    }
    return contents, nil
}

// good
func readFileContents(file) ([]byte, error) {
    contents, err := os.ReadFile(file)
    if err != nil {
        return nil, fmt.Errorf("readFileContents: error while reading a file: %w", err)
    }
    return contents, nil
}

I also have some grips about the mutate everything approach of the language. I prefer immutability by default, but I find that in general as long as you split your functions at a reasonable size it is generally fine.

And for a language that is supposed to be straightforward, it is strange on how much magic the language relies on, in the form of things like internal and main packages, name capitalisation to indicate visibility (private vs Public), conditional compiling by filenames (e.g.: foo_amd64.go, bar_linux.go), magic comments (e.g.: //go:build), etc.

I expect to write more Go code going forward. Not because it is the perfect language or whatever, but just because it a is language that has some really good qualities that makes the language attractive even with the issues that I have. That makes it a reasonable good language, and at least for me this is good enough.

Quick bits: why you should automate everything

Mon, 29 Jul 2024 00:00:00 +0000

If everything works as expected this blog post should appear in in my blog without I ever touching the capivaras.dev website. I rewrote my previous Python script to Go (permalink) since my attempt to generate proper description to the RSS feed resulted in slow startup times (not because of Python, but because of my usage of nix-shell since I didn't want to deal with venv or anything to manage my Python dependencies).

My previous workflow of this blog already involved me writing the texts in neovim, copying and pasting the result in the capivaras.dev website and publishing. This was not that bad, except that it seems I have a heavy tendency of editing my posts multiple times. Copying and pasting data between neovim and the website became tedious, so I decided to give up and automate the whole process.

Mataroa (the blog platform capivaras.dev run) has a reasonable good API, and it only took a few hours to get a version of publishing working (it would take less if Django, the framework Mataroa is written, did not have a weird behavior with URLs missing a trailing /). An additional few lines of YAML to make GitHub Actions trigger a pipeline and now I should never have to manually update my blog again.

I could have not done this. I mean, I probably wasted more time writing an automation than I actually wasted publishing manually. But the manual process is easy to miss, and I already did multiple mistakes publishing in the manual method. For example, when writing the Markdown files, each post is written in a particular format, where the first header is considered the title, so I need to remove it from the contents during publication. But of course, this is easy to miss, and I had to fix this multiple times already.

So yes, I think this is a good lesson on why you should automate everything. It is more than just about time savings, it is about reducing mistakes and even documenting (even if poorly) a process. I mean, the code I wrote is not that great, but I can definitely rely on it in the future to remember what I need to do. It will be much faster than trying to learn from scratch again.

Quick bits: nix-shell is cursed

Sat, 27 Jul 2024 00:00:00 +0000

The other day I had to run a PHP project in my machine. I have no idea how PHP ecosystem work, I just wanted to get it to run.

The easiest way to get a script to run if you use Nix is to use nix-shell. As many of you probably know, you can add nix-shell as a shebang in your scripts to run them as ./script. This was a PHP script so I wanted to do the same. Easy right?

#!/use/bin/env nix-shell
#!nix-shell -i php -p php83
<?php
declare(strict_types=1);

And:

$ ./index.php
Fatal error: strict_types declaration must be the very first statement in the script in index.php on line 4

So it seems that declare(strict_types=1) needs to be the first line in a PHP script if used. I removed declare(strict_types=1) and while the script works, I don't have enough expertise in PHP to know if this would be safe or not.

I decided to try something that initially looked really dumb:

#!/use/bin/env nix-shell
<?php
declare(strict_types=1);
#!nix-shell -i php -p php83

And:

$ ./index.php
Works

Wat? I mean, it is not dumb if it works, but this at least looks cursed.

Eventually I found this comment in a Nix issue talking about cases where nix-shell shebang doesn't work. It looks like the classic case of a bug that becomes a feature.

Update: after posting this in Lobte.rs, it seems someone decided to open a Pull Request to document this behavior. Also the equivalent for the new nix CLI explicitly documents this behavior:

Note that the #! nix lines don't need to follow after the first line, to accomodate other interpreters.

Using GitHub as a (bad) blog platform

Fri, 26 Jul 2024 00:00:00 +0000

I finally started a new blog, thanks to the offer of @ratsclub to give me free access to capivaras.dev. But considering how small this blog platform is supposed to be, I want to have at least somewhere to have a backup of my posts. I know Mataroa, the blog platform that capivaras.dev runs, has automatic e-mail backups, but I want something more reliable.

I am writing all my posts in Markdown (the format that Mataroa supports) files inside neovim anyway, so why not store all my Markdown files in Git? So this is what I did, I now have an unofficial mirror in GitHub.

While I am here, why not overcomplicate? Can I make an usable blog platform from GitHub? And by that I don't mean GitHub pages, the repository itself. I mean, it already renders Markdown files by default, so no need to do anything in that space. To reach feature parity with capivaras.dev, I only need to have an index and RSS (since comments are not supported anyway). No need for newsletter since GitHub has a watch feature already.

After a couple of hours hacking a Python script, you can see the result of this monstrosity here. The script, called gen_blog.py, is available at the same repository (here is a permalink). It automatically generates an index at README.md with each blog post and a rss.xml file at the root of the repository.

Instead of trying to explain the code, I am going to explain the general idea, because I think that if you want to replicate this idea it is better to rewrite it in a way that you understand. It shouldn't take more than 2 hours in any decent programming language. But if you really want, the script itself is licensed in WTFPL license. The code only uses Python 3's standard library and should work in any relatively recent version (anything newer than 3.9 should work).

So the idea is basically to organise the repository and the Markdown files in a easy way that makes it trivial to parse in a deterministic way. For example, my repository is organised in the following way:

.
├── 2024-07-26
│   ├── 01-writing-nixos-tests-for-fun-and-profit.md
│   └── 02-using-github-as-a-bad-blog-platform.md <- this file
├── gen_blog.py
├── README.md
└── rss.xml

Each day that you write a new blog post will be on its own directory. This is nice because Markdown files may include extra files than the posts themselves, e.g.: images, and this organisation make it trivial to organise everything.

Each post has its own Markdown file. I put a two digit number before each post, to ensure that when publishing multiple posts at the same day I keep them in the same order of publishing. But if you don't care about it, you can just name the files whatever you want.

Also, I am assuming that each Markdown file has a header starting with # , and that is the title of the blog post.

Using the above organisation, I have this function that scrapes the repository and collect the necessary information to generate the index and RSS files:

def grab_posts(pwd: Path):
    posts = defaultdict(list)

    for dir in sorted(pwd.iterdir(), reverse=True):
        # Ignore non-directories or hidden files
        if not dir.is_dir() or dir.name[0] == ".":
            continue

        # Try to parse date from directory name
        try:
            date = datetime.strptime(dir.name, "%Y-%m-%d")
        except ValueError:
            print(f"WARN: ignoring non-date directory: {dir}", file=sys.stderr)
            continue

        # Iterate between the files in the date directory
        for post in sorted(dir.iterdir(), reverse=True):
            # Ignore non-markdown files or hidden files (draft)
            if not post.suffix == ".md" or post.name[0] == ".":
                continue

            # Grab the first H1 section to parse as title
            text = post.read_text()
            mTitle = re.match(r"# (?P<title>.*)\r?\n", text)
            if mTitle and (title := mTitle.groupdict().get("title")):
                posts[date].append({"title": title, "file": str(post)})
            else:
                print(f"WARN: did not find title for file: {post}", file=sys.stderr)

    return posts

Some interesting tidbits: if a Markdown file has a . at the start I assume it is a draft post, and ignore it from my scraper. I added a bunch of WARN prints to make sure that the me in the future doesn't do anything dumb. Also, sorting in reverse since reverse chronological order is the one most people expect in blogs (i.e.: more recent blog posts at top).

After running the function above, I have a resulting dictionary that I can use to generate either a README.md file or Markdown:

def gen_readme(posts):
    titles = []

    for date, dayPosts in posts.items():
        for post in dayPosts:
            # This creates a relative link to the Markdown file, .e.g.:
            # ./02-using-github-as-a-bad-blog-platform.md
            link = os.path.join(".", post["file"])
            # This formats the title, e.g.:
            # - [Using GitHub as a (bad) blog platform](./2024-07-26/02-using-github-as-a-bad-blog-platform.md) - 2024-07-26
            title = date.strftime(f"- [{post['title']}]({link}) - %Y-%m-%d")
            # This appends to the list to generate the content later
            titles.append(title)

    # README_TEMPLATE is a string with the static part of the README
    print(README_TEMPLATE.format(posts="\n".join(titles)))


def gen_rss(posts):
    # Got most of the specification from here:
    # https://www.w3schools.com/XML/xml_rss.asp
    rss = ET.Element("rss", version="2.0")

    # Here are the RSS metadata for the blog itself
    channel = ET.SubElement(rss, "channel")
    ET.SubElement(channel, "title").text = "kokada's blog"
    ET.SubElement(channel, "link").text = "https://github.com/thiagokokada/blog"
    ET.SubElement(channel, "description").text = "dd if=/dev/urandom of=/dev/brain0"

    # You create one item for each blog post
    for date, dayPost in posts.items():
        for post in dayPost:
            item = ET.SubElement(channel, "item")
            link = urljoin(RSS_POST_LINK_PREFIX, post["file"])
            ET.SubElement(item, "title").text = post["title"]
            ET.SubElement(item, "guid").text = link
            ET.SubElement(item, "link").text = link
            ET.SubElement(item, "pubDate").text = date.strftime('%a, %d %b %Y %H:%M:%S GMT')

    # Generate the XML and indent
    tree = ET.ElementTree(rss)
    ET.indent(tree, space="\t", level=0)
    tree.write("rss.xml", xml_declaration=True, encoding="UTF-8")

To publish a new Post, a basically write a Markdown file, run `./gen_readme.py

README.md` at the root of the repository, and see the magic happen.

It works much better than I initially anticipated. The README.md is properly populated with the titles and links. The RSS is kind of empty since it has no description, but it seems to work fine (at least in Inoreader, my RSS reader of choice). I can probably fill the post description with more information if I really want, but it is enough for now (update: it is working now, you just need to render the Markdown as HTML and escape the tags; permalink for the updated script). Not sure who is that interested in my writing that will want to use this RSS feed instead the one available in capivaras.dev anyway.

Also, while I am using GitHub here, the same idea would work in GitLab, Gitea, sr.ht or whatever. As long as your source hub supports Markdown files it should work.

So that is it. I am not saying this is a good idea for your primary blog platform or whatever, and I still prefer to publish to a platform that doesn't track users or have tons of JavaScript or whatever. But if you want a backup of your posts and you are already writing Markdown anyway, well, there are worse ways to do it I think.

Update: I rewrote the script again using Go (permalink). The reason for it is because when I started rendering Markdown (for descriptions) the Python version got quite slow (not the fault of Python itself, mostly because of the usage of nix-shell to manage dependencies; something that Go doesn't need). Took about half an hour, showing how easy it is to do the same.

Writing NixOS tests for fun and profit

Fri, 26 Jul 2024 00:00:00 +0000

I recently started a new side project writing an IPC library in Go for Hyprland, a Window Manager for Wayland.

Once I got past the Work-in-Progress phase, I realise I had an issue: I wrote some tests, but I was running then inside my system running Hyprland. And the tests themselves were annoying: since they send commands to the current running Hyprland instance, I was having programs being opened and settings being changed, because this was the only way to have a reasonable good confidence that what I was doing was correct. So I need to do like any good developer and implement a CI, but how?

One approach would be to create something like a mock client and test against my mock. Since this mock wouldn't need a running Hyprland instance the tests could run everywhere (even in non-Linux systems!), but they wouldn't be much useful. Mocks are great for testing business logic, but not really for making sure everything is working correctly.

I need something more akin to an integration test, but this is tricky. It is not like I am doing integration with e.g.: PostgreSQL that has thousands of libraries available to make integration tests easier, I am doing integration with a Window Manager that is a moving target with multiple breaking changes in each release. And this is where NixOS tests enter, a way to run tests inside Virtual Machines configured in Nix.

I am a long time NixOS user and commiter, but I never wrote a NixOS test outside of nixpkgs itself. However I knew it was possible, and after doing a quick reading of the Wiki entry about it, I was ready to start.

The first part is to call pkgs.testers.runNixOSTest and configure the machine as any other NixOS system, e.g.:

{ pkgs, ... }:
pkgs.testers.runNixOSTest {
  name = "hyprland-go";

  nodes.machine =
    { config, pkgs, lib, ... }:
    {
      # bootloader related configuration
      boot.loader.systemd-boot.enable = true;
      boot.loader.efi.canTouchEfiVariables = true;

      # enable hyprland
      programs.hyprland.enable = true;

      # create a user called alice
      users.users.alice = {
        isNormalUser = true;
      };

      # add some extra packages that we need during tests
      environment.systemPackages = with pkgs; [ go kitty ];

      # auto login as alice
      services.getty.autologinUser = "alice";

      # configure VM, increase memory and CPU and enable OpenGL via LLVMPipe
      virtualisation.qemu = {
        options = [
          "-smp 2"
          "-m 4G"
          "-vga none"
          "-device virtio-gpu-pci"
        ];
      };

      # Start hyprland at login
      programs.bash.loginShellInit = "Hyprland";
    };

  testScript = "start_all()";
}

A few details that I want to bring to attention. The first one is how easy it is to setup things like a normal user account, add some extra packages we need for testing, add Hyprland itself and configure auto-login. I have no idea how painful it would be to automatise all those steps in e.g.: Ansible, but here we are in a few lines of Nix code. This is, of course, thanks to all the contributors to nixpkgs that implement something that help their own use case, but once combined make it greater than the sum of the parts.

Second is something that I took a while to figure out: how to enable GPU acceleration inside the VM. You see, Hyprland, different from other Window Managers, requires OpenGL support. This is basically why the flag -device virtio-gpu-pci is in virtualisation.qemu.options, this enables OpenGL rendering via LLVMPipe, that while being slow since it is rendered in CPU, is sufficient for this case.

Putting the above code inside a flake.nix for reproducibility, I had something similar to:

{
  description = "Hyprland's IPC bindings for Go";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = { nixpkgs, ... }:
    {
      checks.x86_64-linux =
        let
          pkgs = import nixpkgs { system = "x86_64-linux"; };
        in
        {
          testVm = pkgs.testers.runNixOSTest {
            # the code above
          };
        }
    };
}

I can now run nix build .#checks.x86_64-linux.testVm -L to build and run the VM. However it is not really useful right now, since we didn't add any useful code in testScript, the core of the NixOS test framework. We can also run nix build .#checks.x86_64-linux.testVm.driverInteractive and ./result/bin/nixos-test-driver: this will start a Python console where we can manually play with the VM (try typing start_all() for example).

The testScript is a sequence of Python statements that perform various actions, such as starting VMs, executing commands in the VMs, and so on. More about it in the official documentation. For our case we can start with something like this:

{
    testScript = /* python */ ''
      start_all()

      machine.wait_for_unit("multi-user.target")
      machine.wait_for_file("/home/alice/test-finished")
    '';
}

The first statement, start_all(), starts all VMs, in this case we have only one, called machine. We send two further commands to machine: wait_for_unit("multi-user.target") and wait_for_file("/home/alice/test-finished").

The first command waits until systemd's multi-user.target is ready, a good way to ensure that the system is ready for further commands. The second one we wait for a file called test-finished to appear in Alice's $HOME (basically, a canary), but how can we generate this file?

Remember that we added programs.bash.loginShellInit = "Hyprland", that automatically starts Hyprland when Alice logs in. We need to modify that command to run the Go tests from our library. The good thing is that Hyprland configuration file supports a exec-once command that runs a command during Hyprland launch. We can abuse this to launch a terminal emulator and run our tests:

{
  programs.bash.loginShellInit =
    let
      testScript = pkgs.writeShellScript "hyprland-go-test" ''
        set -euo pipefail

        trap 'echo $? > $HOME/test-finished' EXIT # creates the canary when the script finishes

        cd ${./.} # go to the library directory
        go test -v ./... > $HOME/test.log 2>&1 # run Go tests
      '';
      hyprlandConf = pkgs.writeText "hyprland.conf" ''
        exec-once = kitty sh -c ${testScript}
      '';
    in ''
      Hyprland --config ${hyprlandConf}
    '';
}

So we are basically creating a custom Hyprland config that starts a Kitty terminal emulator, that then launches a shell script that runs the test. Since we have no way to get the results of the test, we pipe the output to a file that we can collect later (e.g.: machine.succeded("cat /home/alice/test.log")). And once the script exit, we create the canary file $HOME/test-finished, that allows the testScript knows that the test finished and it can destroy the VM safely.

If you want to take a look at the final result, it is here. This tests run in any Linux machine that supports KVM, and also works in GitHub Actions thanks to the the nix-installer-action.

And now I have a proper CI pipeline in a way that I never imagined would be possible, especially considering how simple it was.