James' Digital Garden

RTX ON - it's execution time

Fri, 21 Jul 2023 00:00:00 +0000

I’ve been using previously used sdkman for a few years to manage my JDK & Scala installations. It supports a good amount of tooling, but it’s very JVM focused. You may know nvm or n for managing and switching Node versions, or Volta/fnm for more general Javascript tooling management.

Recently I’ve been getting into Elixir & Phoenix LiveView, and I came across a similar tool called asdf. Actually though, thanks to its “plugins” system and almost 700 plugins, you can install so many different tools. Sounds good to me - I have projects using Python, Elixir, Java/Scala, Node, Terraform, AWS CLI, etc.. With one application, I can have tooling defined locally (per-project) so it’s all independent and easy to get the tooling right.

This was working great for Elixir & Erlang, but the ergonomics felt a little off. In order to list versions, you have to first download the plugin. And due to its “shim” mechanism, it adds about 100ms delay to each command that passes through the asdf executable (my ELI5 understanding).

I then came across rtx, a Rust tool inspired by asdf that takes a different approach. Here’s some features I’m really liking:

Speed - rtx points to tooling versions via the PATH, and updates the PATH when necessary - this keeps interactions fast (it doesn’t go through a “shim” unless it has to, unlike asdf)
- Also, apparently python called via rtx is much more response than python with pyenv
Installs - If you have a bunch of microservices on different Node/Java versions, rtx reloads the relevant version via the PATH when you switch project in your terminal. You don’t need to run commands like nvm use node 16 - it’s automatic. Global installs are supported too.
Plugins - asdf’s amazing plugins are here still, but you don’t have to explicltly install them first!
- rtx does have it’s own plugins, but <10 at the time of writing. Re-using asdf’s plugins is smart
Documentation - the CLI & interactions are friendly, and setup is (almost) frictionless
Configuration - .rtx.toml and the CLI interactions with it are easy to use, and really powerful - see below!

Show me the config

For our documentation website, I suggested we move from install nvm/node/yarn/sbt to just this configuration file (.rtx.toml):

[tools]
node = "16"
yarn = "1.22.19"
sbt = "1.9.2"

and this one-click script (in IntelliJ) to go from 0 to READY:

curl https://rtx.pub/install.sh | sh # install rtx 
echo 'eval "$(~/bin/rtx activate zsh)"' >> ~/.zshrc # hook rtx into shell
rtx install # install tooling
yarn # install deps 
yarn run # launch dev website

That’s a nice developer experience. I’m happy to know someone can clone a repo, click a button, grab a drink, and come back to a website!

CI - does it add value here?

So, rtx is pretty cool for local development - but what about CI?

For our main project, we use a JDK, Scala, and Mill. There’s a few Actions for setup (setup-java, coursier-setup, mill-setup, etc.) - but they usually want a version typing out. This could lead to drift between development and CI, and introduce a bit of toil when somebody finally notices or remembers.

setup-java

steps:
- uses: actions/checkout@v3
- uses: actions/setup-java@v3
 with:
 distribution: 'openjdk' # See 'Supported distributions' for available options
 java-version: '17'

But it’d be nice if we could set up more, with less lines right? See Coursier’s setup-action:

steps:
- uses: actions/checkout@v3
- uses: coursier/setup-action@v1
 with:
 jvm: adopt:17
 apps: sbtn bloop ammonite

Ah, so it seems the versions can’t be specified (other than for the jvm).

With the rtx Action, our .rtx.toml files can be used - which is accurate, and brief:

steps:
 - uses: actions/checkout@v3
 - uses: jdxcode/rtx-action@v1

I ran this as a workflow dispatch. The first run took 3m36s (it takes a while locally to install Elixir & Erlang too), but the second run (started soon after) took only 20 seconds! GitHub Actions seems to have nicely cached the worker for my master branch (270MB total). Apparently there’s a 10GB total limit - though I can’t see how long it lasts. That’s cool though - our action just works in CI, is super clean, and in Public GitHub they help us keep things fast with zero-configuration caches!

… but my versions for different tools are scattered around my source!

Let’s say you use three tools, which are specified in different files:

openjdk-17 - in a Dockerfile
Scala 2.13.xy - in a Dependencies.sc file
mill - in .mill-version

Fortunately, rtx uses the tera templating engine so we can grab these dynamically. These commands are kinda grim (I couldn’t use " or ‘; I found cut to be a good command, thanks ChatGPT), but are probably “good enough” to not need updating. The sources they read won’t be changing spacing much:

[tools]
mill = { version = "{{exec(command='echo $(cat .mill-version)')}}" }
java = { version = "{{exec(command='grep -m 1 openjdk docker/Dockerfile | cut -c 12- | tr : -')}}" }
scala = { version = "{{exec(command='grep -m 1 2.13 dependencies/Dependencies.sc | cut -c 33-39')}}" }

The secret sauce here is:

grep -m 1 <phrase> returns the first line that matches
cut 12- gives from the 12th char onwards, cut 33-39 does what you’d think

Yep, it does look dumb, but:

For Java we just pin to a major version; if we stick to the same vendor, there’ll be no issue
For Mill, it’s just a plain cat. Not too bad :)
For Scala, until there’s a migration to Scala 3 then we’ll just see 2.13.11 -> 2.13.xy

Additions & Alternatives

For asdf, there is lazyasdf - it’s a TUI for asdf (like how k9s is a TUI for k8s)
An alternative to asdf/rtx is aqua, written in Go. The local configuration (like .rtx.toml) is aqua.yaml, and it supports global installs too
- The Aqua registry has gives 1200+ results - but I see nothing for Elixir, Java/JDK/JVM, and the only node result is “kubectl-node-shell”
- The fzf-esque interactive search for packages with aqua g is nice, even if I can’t find what I want

What’s bad about rtx (security)?

There’s a good security write-up on the rtx repo.

As you can see, with Tera templating you can run some arbitrary commands (firstly in PRs/GitHub Actions, then locally if a change is merged). There is a command rtx trust, meaning “rtx will parse the file with potentially dangerous features enabled” - I guess that’d be useful if you clone some OSS repo and don’t want the tooling. There’s also configuration via environment variables, e.g. RTX_TRUSTED_CONFIG_PATHS that may be useful.

Even tools like gradlew have risks though, and that’s massively popular.

To answer “can/should I use rtx?”, at this point you need to do your own homework ;)

Conclusion

I encourage you to give rtx a try. It’ll be my tooling manager of choice for personal projects now, and I’m encourgaging its use at work. So far I’m using it in a backend JVM repo, a yarn/node documentation repo, and an Elixir/Erlang repo.

ChatOps - just say the word

Thu, 18 May 2023 00:00:00 +0000

ChatOps: you say some trigger, and you get some response. The processing behind the scenes can be as complex or niche as you like.

In this blog, we’ll talk about three different ChatOps tools (Slack, GitHub Actions, Hubot) and how they can:

Set up a basic reminder (Slack)
Trigger image builds, performance tests, etc. on PRs (GitHub Actions)
Send a list of open PRs, and their review counts, to Slack (Hubot)

Slack

Slack has a few baked in commands (“Shortcuts”). The most useful I’ve seen is reminders - whether for yourself or for your team. Here’s a few examples, with the format []/remind [yourself or #channel] [what] [when].](https://slack.com/intl/en-gb/help/articles/208423427-Set-a-reminder):

/remind #my-team to join Google Meet on Wednesday at 4:30pm
/remind me to file TPS reports in 20 minutes
/remind me to have a great weekend every Friday at 5pm

GitHub Actions

Note: If you have a project on Public GitHub, you can use their action “runners” for free. If you’re in the GitHub Enterprise Suite, you’ll need to deploy your own runs-on: [self-hosted] “runners”. There are some projects that can help you kick-start self-hosting, such as the scalable spot instance setup “terraform-aws-github-runner”.

perf tests awkward process, evolving over time as we change our Jenkins pipelines to multibranch etc.

very valuable - perf test before merge

low barrier to entry: chat ops. Ask a PR to be perf tested, and an hour later you have

docker images build
applications performance tested

Hubot

Ways of Working 'checklist'

Fri, 05 May 2023 00:00:00 +0000

Every team will figure out their own unique ways of working through “Forming, Storming, Norming and Performing” - but here are some techniques that I’ve seen provide lots of value - usually with little effort!

✏ Why not take a note of each one you aren’t using yet as you read?

GitHub

PR Templates

Open source projects often have multiple PR templates, to help capture context on bug reports, feature requests, and security vulnerabilities.

In your team’s day-to-day repositories, it’s likely you aren’t using templates. Maybe they “get in the way” and “just get deleted”, but these two features might make it more interesting!

Markdown comments

GitHub uses MarkDown (their own special flavour of MarkDown, really) - and it supports comments:

<!-- Please enter the ticket number below (GitHub will autolink to Jira), e.g. JIRA-1234 -->
JIRA-420

<!--
For this feature/fix, please link any tests from other repositories too:
- [ ] e2e: E2E-PR-1337
- [ ] perf-tests: PERF-404
-->

These comments are only visible when editing - you can’t see them on the posted description. You can use comment to give friendly reminders on:

Providing context
Linking to the ticket (rather than paraphrasing all that context!)
Ensuring tests (unit, integration, performance) are covered
- You could give a commented-out checklist if you want

AutoLinks

In each GitHub repository, you can set up “AutoLinks”. They’re basically an autogenerated, tidy hyperlink.

The GitHub Docs give a good example - but I’ll extract a snippet and save you getting distracted:

Reference prefix: JIRA-
Target URL: https://jira.example.com/issue?query=<num>
Preview: JIRA-123 is converted to https://jira.example.com/issue?query=123

Combined with templates, you can ensure that every PR has a short link to the relevant tickets. This is much better than just having the ticket number (and no link) in the title/description/branch/commits:

It saves the PR author time in making these links
It saves the reviewers time fishing around in Jira and getting distracted
It ensures everyone has easily accessible context, so the PR description can focus on the actual changes

Until I knew this I was using an Espanso text expansion macro :JIRA to do similar, but this setup gives your whole team an awesome shared capability

Conventional comments

Stating the importance & intent of your message up front can make communication clearer, and decisions faster.

Read more in detail at https://conventionalcomments.org/, but basically comments on PRs can look more valuable like this:

issue: this mock never gets called!

praise: this method is really easy to read, and handles the logic very well

nitpick: these two tests could be combined

Compare the latter example to how it might be expressed without “conventional comments”:

These two tests could be combined, but it's not a blocker for this PR and I'll approve

Conventional commits

Commits can look like:

docs: add javadocs for user-facing swagger api
bug: fix a flaky test
chore: bump dependency version x->y
feat: AI face detection when user blinks

It could help you have a more atomic git history, which may make PRs easier to comb through. You can also use the prefixes to group changes, and make prettier changelogs.

Tickets / work capture

Make templates

Context, task, ACs, key contacts,
User stories

Templates are all about adding context in an organised way. Having this context gives your team more autonomy and interest in the problems, and can lead to better outcomes. Think “Context over control”

Knowledge management

How are you recording your knowledge? Probably in a few places! It might look like this:

Slack (short term - threads can be linked, good for captured asynchronous discussion)
Google Docs (also great for captured async discussion, but leaning towards )
Confluence (longer-term storage for internal decisions - awkward to collaborate on, not good for regular updates)
Websites (static sites like Hugo’s Doks & Docusaurus can make information presentable, searchable, and written in simple markdown & managed by e.g. Git)
Some loose markdown files in various repos (maybe some readme.md)

To get someone up to speed on your project, you’d probably start with the higher-level, more organised/presentable formats. Hopefully there is a natural flow through the information, otherwise someone who knows the scenery may have to plan a route for you.

But what if there were tools that combined the strengths of these platforms. Are there any services that are:

simple: are written in simple markdown-like language
collaborative: many users can write in real-time
efficient: easy to convert discussions into clean documentation
searchable: text search, or even graph search (how do ideas and documentation naturally relate?)

Notion

I use Notion, which covers these points well. It’s becoming more and more popular, and I’ve seen a few companies using it. Despite appearing simple, there’s plenty of power features under the hood:

You can create tables of data - and create views over them, filter, sort, label and organise in helpful ways
You can make timelines, calendars
You can use it like a task management system (GTD, four quadrants - whatever you want)
You can use it as a sprint/kanban board

It’s powerful - even for free users. For businesses, it’s at least $15 per user, per month. That sounds like quite a lot, but it looks like Jira costs the same.

I’ve never worked in a company using it as a central tool in all the ways above - so I can’t actually vouch for it. Maybe it only works up to a certain scale of organisation - but maybe that could be your organisation.

Obsidian

I didn’t get far into Obsidian - the theory can get pretty deep, and there’s many methodologies. You might have heard of:

“Second brain”, and
“Zettelkasten”

The key concept is that ideas are related, and naturally link up (like in our brain). You can view them as a graph (looks like synapses in our brain), converge, and diverge thoughts whilst keeping them linked. There’s also a cool plugin system. I had a play and set up cloud sync to GitHub and OneDrive. I’ve already got years of notes in Notion, and didn’t quite get sold on Obsidian.

Arguably, the notes (and their relations!) might not be personal enough to be maximally useful for everyone. However, across a small team it could work really well. $50 per user, per year - plus about $100 a year for sync.

If you think you’re having problems in some of those areas, maybe try a different knowledge management system. This is totally not an advert ;) I’d just love to see first-hand how these tools could work for some teams instead of the usual Jira/Confluence fare! If you have some 💲 and some ⌛, your team could spike using these tools.

Discussions

Problem	Solution
Rabbit holes	Be mindful of topics dominating meetings - consider a separate meeting/thread to go into specifics
Circling	Be mindful of discussions looping. Raise the concern, or capture notes everyone can see and help to align on a plan
Uncaptured discussion	Take notes and share them. Ask for corrections, as you may have misunderstood. Whether it’s minutes for regular meetings, or details in ticket refinement - capturing 5 minutes of context now can save a few minutes in the future
Discussion contains too many moving pieces	Make a quick sketch, in TLDRAW or even Mermaid. Humans find it hard to remember 5-9 “bits” of info - compress ideas into a visual)
Out of office/ill teammates missed important discussion meetings	Record the meetings, and update the calendar invite with the recording link. Transcribe the audio, provide a searchable interface to save time
Solutionising without being aligned	Define the problem statement. “What questions are we trying to answer?”, “What data lets us answer these questions?”, “How can we get that data?”

Collaborative working

Visible, welcoming huddles

Is your team pairing and mobbing, but in private calls? It’s not transparent or welcoming - so consider this:

Take the number of devs you have, and divide by two. Make this many “pairing” Slack channels. It gives a space for everyone to pair, or form mobs. If someone needs help, it’s easy to hop in and out without the complexity of setting up more calls.

Remote pairing

Tool	Review
Zoom	Solid screenshare quality, awkward drawing tools, and confusing controls. Preferred screen-sharing platform for now
Slack	Worst screenshare & audio quality. Most convenient to drop in/out with pairing channels
VS Code	Shared editor & terminal are good - but the file explorer seems too strict to let the guest explore the project and be productive
Intellij	Code With Me is decent now - my main issue is the Shared Terminal is completely broken for the host (a big problem if you run your tests there!)
Tuple	Great screenshare quality (configurable), good interactivity. Configuration maybe a bit too permissive, but necessary. Great features like “pebble drop” to show where you’re looking

Quality checks

git hooks
github actions has a broad marketplace

Rota Driven Development ⁉

Fri, 28 Apr 2023 00:00:00 +0000

How can you take a bunch of T-shaped developers and upskill everyone to be a 🟩-shaped developer?

You might know a few ways you can do this already:

Knowledge sharing sessions
Pairing/Mobbing
Giving regular, honest feedback

You might know why it could be a good idea:

Employees want to be empowered
Learning, teaching, and broadening horizons can be rewarding & fun
Reduced “bus factor”

But what would something really extreme look like? Enter: “Rota Driven Development” Note: You might even want to experiment with this setup if you already have pairing/mobbing as your main way of working. Otherwise, this might sound quite terrible! But let’s see how it might be valuable. This post isn’t about why pairing is good or bad - but what an extreme variant of it could look like

Illustrative example

Let’s say we have three developers:

BackEnd expert (B), with a bit of Cyber (c) [Bc_]
Cyber expert (C), no other experience [C]
FrontEnd expert (F), with a bit of BackEnd (b) [b_F]

When you pair these developers, they’ll level up by working on tasks together:

[Bc_] + [b_F] -> [Bcf] + [B_F] (let’s say the BackEnd expert didn’t share much on Cyber this time)
[B_F] + [C] -> [BcF] + [bCf] (they worked across all three topics)
Already, the team is becoming much more well-rounded. Everyone has picked up at least the basics of every field.

Of course, it’s an extreme example. More realistically, there could be many domains (framework, syntax, literally domain knowledge, etc.) within any of these three fields - so it can still make sense for e.g. BackEnd developers only.

In a real team, there would probably be a few more people as well - so everyone can Always Be Transferring Knowledge

Your team’s “skill matrix”

To find out what some quality pairings would be, you can make a shared table of people and how they feel their skills are out of 5. It should highlight gaps, and if you update it few weeks/months you use it to track progress.

Planning work for maximal learning

In “second language acquisition”, there is a theory called “i+1”: To have a smooth, low-stress learning environment, you feed someone content that is slightly more complex than their current level. In other words, don’t throw people in at the deep end. If you can estimate the complexity (via story points or some other metric), you could combine that with the skill matrix to optimise growth in your employees’ skill set.

But tickets and rotations don’t line up nicely

There’s a few approaches you could try:

Set pairings for a whole sprint
Set pairings for only the first ticket in a sprint, and then let people self-organise

How can my team see if this works for us?

Check if your team is even interested in such an idea
Make the up-front investment (skills matrix), and continued effort investment (changes to planning/ticket preparation)
Run a trial for a few weeks. Maybe run a retro on the rotation process, and iterate if you see value there. Scrap it and move on if not.

Conclusion

For teams that have already bought into pairing and want to try a more focused approach to maximise their learning: “Rota Driven Development” could be an interesting experiment to try.

Symlinks, syncs, and app configuration

Wed, 18 Jan 2023 00:00:00 +0000

When setting up a new Windows device, there are some tools that can help you to install your apps quickly and easily from the command line (WinGet, chocolatey). Whilst these are great (and can actually install most of your apps), one thing they can’t do is transfer your configuration between devices.

In this short blog, we’ll be using:

A cloud sync of choice (Google Drive / OneDrive / DropBox etc.)
- So I have my configuration files on any device
Symbolic links, in a tiny script file
- So apps can easily access the backed-up configuration, and sync changes
An application we love, which uses simple configuration files

For this worked example, we’ll use Espanso, and a folder “Junction” symlink.

The folder linking must be done before the source folder (Espanso’s config) is created. Either do this before installing the app, or do some file/folder shuffling. It doesn’t matter if the target exists or not.

In your cloud-sync’d folder (%HomePath%\OneDrive\Apps\espanso), create a file symlink.bat and add:

mklink /J %APPDATA%\espanso %HomePath%\OneDrive\Apps\espanso

In Windows, the %APPDATA% folder will look like this if the command succeed; note the little arrow on our symlinked folder:

In ......./OneDrive/apps/espanso/match/base.yml, add this to the main matches body:

- trigger: ":synctest"
replace: "your espanso config was sync'd sucessfully!"

Install espanso with default settings

Espanso startup logs show our symlink folder being loaded in. It does this anyway (nothing special is happening to the file loading… other than it actually loading from our cloud sync’d folder!):

... reading configs from: "C:\\Users\\james\\AppData\\Roaming\\espanso"
... reading packages from: "C:\\Users\\james\\AppData\\Roaming\\espanso\\match\\packages"

So if we enter our special phrase, :synctest, we should see it expanded to:

your espanso config was sync’d sucessfully!

On another computer (or a fresh installation of Windows), all you have to do is double-click the .bat file. In just a few minutes, we’ve made a system that will always take care of our favourite Espanso packages & personalised keywords. Also note that any changes we make will be backed up to the cloud too!

What we’ve done here can be applied to more applications which have simple configuration. The scripts to create junctions/symbolic links could be per-app, or you could put them all in one script (to be run before a big WinGet/Brew install).

Here’s a few configurations ones I may set up this synchronisation for:

Headphone EQ - Peace/EqualizerAPO - .txt files represent frequencies/decibels etc.
GPU Overclock/Undervolt - MSI Afterburner - Custom curve is a bit tricky to make (I have to search for my notes on instructions every time I set up)
Calibre, for digital book management
… I can’t think of more, but I’m happy even with just Espanso having this :-)

The same principles apply to Ubuntu and MacOS - but implementation of your symlink script files might look more like this StackExchange answer.