Hacker News - Small Sites - Score >= 2

I Make Smart Devices Dumber: A Privacy Advocate's Reflection

Thu, 16 Jan 2025 16:11:04 GMT

Score 3 | Comments 0 (thread link) | @smnrg
January 16, 2025 | https://simone.org/dumber/ | archive.org

I flipped the vacuum robot upside down on my desk, wheels in the air. A thousands dollar marvel of modern, rebranded AliExpress convenience. Ready to map my home, learn my patterns, and send data across continents. Behind every promise of convenience lie hidden costs we're only beginning to understand.

My screwdriver hovered on its seams: These robots are not just about cleaning floors anymore, but drawing a line in the digital sand.

In the rush to embrace smart devices, you accept a devil's bargain: convenience for surveillance, efficiency for privacy. Homes become frontiers in the attention wars—each gadget a potential Trojan horse of data collection, promising easier lives.

The DIY movement has evolved far beyond fixing broken toasters. Some, like the devs of Valetudo, are digital rights advocates armed with soldering irons and software. Their work challenges the invisible monopolies that shape our relationship with technology. They're not just fixing devices: They are liberating those from adware and behavioral data harvesting. Each freed device marks a small victory in a conflict most people don't even realize they're in.

✨ Buy Limited Edition of 100 “Ferris Wheel”

Keep the Bots Away

Join me in preserving a human-crafted pocket of web on the Internet. No content is paywalled, and no paid subscription is solicited: this is the most effective way to keep bots and “AI” scrapers away while becoming part of a small community celebrating human writing, photography, and conversation.

I get it, here's my email

Already joined? Sign In

Kreya 1.16 with WebSocket support released – Postman Alternative

Thu, 16 Jan 2025 15:36:13 GMT

Score 5 | Comments 0 (thread link) | @ni507
January 16, 2025 | https://kreya.app/blog/kreya-1.16-whats-new/ | archive.org

Kreya 1.16 comes with support for WebSocket calls and an updated look and feel for the project settings. Faker is added to the scripting API, cookie management is added and many more features are implemented.

WebSocket support

Create a new operation in the operations list, select the type WebSocket and start using WebSockets in Kreya!

You can find all the details in our documentation.

It is now possible to define a custom header for the auth or send it as a query parameter. This can be configured for each auth configuration under advanced options.

Added faker to scripting API Pro / Enterprise

The bogus faker is now directly accessible in the scripting tab with kreya.faker. No additional imports are required.

Updated look and feel of Kreya

Kreya's UI has been reworked. Especially the project settings have been given a fresh new look. They now open as a tab, allowing the user to quickly switch between an operation and the project settings. To open the project settings, go to the application menu and click Project > Environments or Authentications or ....

Tip: For even faster access, the project settings can be opened using keyboard shortcuts (e.g. Ctrl+⇧+E for environments on Windows or ⌘+⇧+E on MacOS).

We've also streamlined the workflow for creating operations, so you don't have to select the type every time you create an operation, just choose the right type of operation from the menu at the start.

Collection state filter Pro / Enterprise

In the header of a collection it is now possible to filter the operations by their states.

File exclusion list for gRPC proto file importers

In gRPC proto file importers, files can be excluded from import. This is particularly useful if you're importing entire folders of protos and don't want to import certain subfolders.

To add custom headers to the gRPC server reflection and REST OpenAPI URL importers, open the advanced options in these importers.

Cookies can now be managed in Kreya. A response tab is visible when a cookie is set and all cookies can be managed in a separate tab under Project > Cookies. It is important to note that all cookies are managed separately per environment.

AWS Signature v4 authentication

An additional auth type has been added. An AWS Signature v4 auth type can now be created in the auth settings.

Bug fixes

Many bugs have been fixed. More details can be found on our release notes page.

If you find a bug, please do not hesitate to report it. You can contact us at [email protected] for any further information or feedback.

Have a nice day! 👋

Test-Driven Development with an LLM for Fun and Profit

Thu, 16 Jan 2025 15:30:19 GMT

Score 4 | Comments 0 (thread link) | @crazylogger
January 16, 2025 | https://blog.yfzhou.fyi/posts/tdd-llm/ | archive.org

Welcome to the very first post in a new blog! Here I will discuss software development, SRE work, and other fun stuff. Sometimes an idea is just too good to pass up. I hope this blog will motivate me to turn sparks and little pieces into general knowledge in writing the words down.

The other day I was discussing Tabby with a coworker. We talked about whether we should consider AI-autocompleted code harmful and ditch everyone’s newfound habit due to LLM’s inherent unreliability and their tendency toward spaghetti code, throwing traditional software engineering principles like DRY out the window. I disagreed: what if we could have a framework that integrates AI development tooling while also making everything better and more reliable instead? This instantly reminds me of Test-Driven Development, or TDD, which I think is great when combined with the use of a Large Language Model.

TDD in essence is to write comprehensive unit tests before you work on the main program. Since you wrote so many test cases that they essentially become the full specification, having the tests pass at the end “proves” your program’s correctness. Despite its promise, a lot of people find it a ridiculously clumsy process, even a major drag on productivity. Days could pass when nothing useful gets done. LLMs, however, have fundamentally changed the economics of doing TDD.

How I normally code with an LLM

I used tools like Github Copilot heavily ever since they came along. They are good for finding repetitive patterns and helping us fill in the next few lines, but usually struggle to: look at a clear specification, think deeply about it, and produce a whole working module against said specification. Sometimes a problem is so easy I am positive Copilot must be capable of it, but it stops short of giving me the full solution, opting to generate a single line of code instead.

To achieve what I want, LLMs need to be given well-formed requests with necessary, but not overwhelming, contexts. Stuff every tool and library at our disposal inside a model’s context, then it is distracted and easily carried away from the specific problem at hand.

In the end, I find myself rephrasing and renaming project-specific things in more general terms, introducing additional pieces of project-specific context into the conversation only when I realize solving the task requires them after all.

Another observation I had working with LLMs is that they are great debuggers. I can often paste raw error outputs to an LLM, and more often than not, they succeed at guessing the cause.

At some point, I realized that the bulk of friction coding with an LLM came from the back-and-forth copy-pasting juggle between my IDE, shell terminal, and the chat interface.

Can I automate it?

So I wrote a little event loop to automate this process.

In the first prompt we give to the LLM, we type in a specification of the function we want to implement and a function signature for extra stability. The LLM is expected to generate a good unit test followed by an implementation.

Let’s give it a non-trivial, real-world problem to be solved:

% go run main.go \
--spec 'develop a function to take in a large text, recognize and parse any and all ipv4 and ipv6 addresses and CIDRs contained within it (these may be surrounded by random words or symbols like commas), then return them as a list' \
--sig 'func ParseCidrs(input string) ([]*net.IPNet, error)'

The model will happily give us a first draft, which we immediately parse and load into a “sandbox” subdirectory for automatic verification:

% tree ./sandbox
./sandbox
├── go.mod
├── go.sum
├── main.go
└── main_test.go

go mod tidy && gotfmt -w . && goimports -w . is used to fix minor syntax issues, then go test . -v is run.

If things fail (totally expected at this point,) we make use of the second (iteration) prompt. Now that we are working on existing code, this prompt includes the code we just ran and, crucially, the actual command line output from running the test, which would be either a compiler error or information about some failed tests. The model is expected to think about what happened and iterate by producing a revised test + implementation in a loop until all tests are cleared.

The idea is that in most cases, sending the full debug session is not a very frugal use of the model’s context. A reasonably intelligent model can think about what went wrong and make incremental changes. We also get the benefit of way lower API bills by keeping the context length more or less constant, regardless of how many iterations we end up doing.

Sometimes the model does get stuck. In our CIDR example, claude-3.5-sonnet came very close to all-clear on its second attempt, but proceeded to fail the same test case twice in a row. This is when I come in to look at the code, realize the regexp is not matching the final ‘1’ in “2001:db8::1”, and then provide that as an explicit hint to the model:

Claude makes progress and clears the tests with our help:

Now we must contend with the “who guards the guards” problem. Because LLMs are unreliable agents, it might so happen that Claude just scammed us by spitting out useless (or otherwise low-effort) test cases. Anyway, it is a good practice that whoever implements the code shouldn’t write their own tests, because the same blindspots in design will crop up in tests. In our case, LLM generated both. So it’s time to introduce some human input in the form of additional test cases, which is made extra convenient since the model already provided the overall structure of our test. If those cases pass, we can be reasonably confident in integrating this function into our codebase. Theoretically, you might even invoke a third prompt to do some AI-powered mutation testing by asking for a subtle, but critical, change in the implementation that is supposed to break our tests, then find out if it did!

LLM-powered development and cognitive load

So this method appears to reliably tackle leetcode-style questions, but would it work in a practical codebase with an actual dependency graph? I believe it can be made to work with some engineering effort, and it’s great news for the codebase’s long-term maintainability if you do. For best results, our project structure needs to be set up with LLM workflows in mind. Specifically, we should carefully manage and keep the cognitive load required to understand and contribute code to a project at a minimum.

Every package, or directory, should consist of several independently testable subsets of code, wherein each subset contains basically 3 files: shared.go for the package’s shared typedefs and globals, whereas x.go and x_test.go focus on a specific aspect of our functional logic, ideally just one public function per file. Sometimes we also have main_test.go supplying a TestMain function for setting up test environments (e.g. testcontainers).

Recently I started a brand new software project at work, so I had the opportunity to design the code organization starting from a blank slate. I’m currently exploring an extension to the AI TDD workflow for larger projects. The whole project code would need to be copied to the sandbox for execution, but sending the entire codebase to an LLM is impractical and distract the model’s focus. Instead, we designate a specific package (subdirectory) that the LLM will work on at any given time, include a gomarkdoc-generated documentation for each dependency package we think is helpful, and finally include a same-package example code (perhaps a tightly-coupled entity’s finished implementation). The model will produce a function and a test, just like before, but this time we write the files in our intended subdirectory deep inside the sandbox to run the unit test.

With this pattern, not only do we limit the chance of introducing problematic code into production with test coverage by default, but we also encourage aggressive de-coupling and a unit test-first project structure. Because adding additional context to a model incurs some inference (and human-psychological) cost, we are constantly nudged into maintaining our code as nice little chunks, each consuming only a small cognitive load to fully understand. Hopefully, this approach lets us end up with “deep” modules with rich functionalities but minimal surface area.

In closing, you should remember AI’s Bitter Lesson. There is a non-zero chance that we wake up tomorrow to a major shift in AI architecture, eliminating the LLM limitations we talked about, and rendering our efforts meaningless. So perhaps don’t start refactoring right away your 100k-Line-of-Code projects following this advice!

Astronomers discover neutron star with an incredibly slow six-hour spin

Thu, 16 Jan 2025 15:27:05 GMT

Score 6 | Comments 1 (thread link) | @Brajeshwar
January 16, 2025 | https://www.abc.net.au/news/science/2025-01-16/neutron-star-radio-transient-6-hours/104799106 | archive.org

In our galaxy, about 13,000 light-years away, a dead star called ASKAP J1839-075 is breaking all the rules … extremely slowly.

Dead stars — or neutron stars — normally spin at breakneck speeds, but a team of astronomers has clocked the new-found star taking a leisurely six-and-a-half hours to undertake just one spin, which is thousands of times slower than expected.

"This could really change how we think about neutron star evolution," Yu Wing Joshua Lee, an astronomer from the University of Sydney and the first author on the new study, said.

The discovery has been published in Nature Astronomy.

The researchers believe ASKAP J1839-075, is a "pulsar", a high-energy neutron star that releases short bursts of radio waves.

Dense neutron stars normally spin extremely fast in space. (Supplied: NASA/Goddard Space Flight Center/Conceptual Image Lab)

But conventional wisdom is that when pulsars slow down they stop emitting radio waves, meaning ASKAP J1839-075 should be invisible to radio telescopes.

So what's going on?

What is a neutron star?

Neutron stars are one of the most extreme objects in the Universe.

These small, dense stars are created when the core of a supermassive star collapses and triggers a fiery explosion known as a supernova.

When the star collapses it may go from a million-kilometre radius to just 10 kilometres.

This extreme crumpling increases the star's rotational speed, like a figure skater spinning faster when their arms move close to their body.

Spinning extremely fast is therefore part and parcel of being a neutron star. A full spin usually takes these collapsed stars just milliseconds or seconds to occur.

If our Sun was to go through the process of becoming a neutron star, its current 27-day rotation could become 1,000 rotations a second.

As pulsars rotate, we see flashes of radio waves from Earth, similar to a lighthouse. (Supplied: NASA/Goddard Space Flight Center)

Using radio telescopes, astronomers can "see" pulses of radio waves from Earth as the neutron star spins, with the movement regularly described as like a "cosmic lighthouse".

Later in the collapsed star's life, it was thought that when it lost energy and began to slow down, the bursts of radio waves scientists detected from Earth would also stop.

"Once they pass the [speed] threshold, we thought they'd be silenced forever," Mr Lee said.

But in the past few years, astronomers discovered pulsars that seemed to contradict that hypothesis.

"This makes us rethink our previous theories on how these sources form."

What rule-breakers have researchers found?

Early in 2022 researchers found pulsars that rotated on minutes-long time scales rather than seconds, and by June last year, researchers had discovered an object which took almost an hour between pulses. These usually slow objects were called "long-period radio transients".

But ASKAP J1839-075's leisurely 6.45-hour spin was unheard of.

"The previous record was 54 minutes, so it was a huge jump," Mr Lee said.

"The team was really surprised."

Yu Wing Joshua Lee was the first author of the new paper. (Supplied: University of Sydney)

According to Gemma Anderson, an astronomer at Curtin University who was not involved in this paper but who was part of the team who found the first long-period radio transient, 6.45 hours stretches "our understanding of physics".

"A normal pulsar couldn't spin this slowly and produce radio light," she said.

"Some kind of extreme particle acceleration … is occurring that is causing it to be so radio bright on these long time scales."

A lucky find

Mr Lee was searching for "peculiar radio transients" by trawling through archival data from a sky survey taken by CSIRO's ASKAP radio telescope in outback Western Australia.

With little prior knowledge of where these transients pop up, the strategy is to pick a random point in the sky to see if anything interesting shows up, Mr Lee said.

Within the archival data, the team discovered a pulsar-like blip from early January 2024. The signal was already starting to fade when the survey began, so the team could only study the second half.

"If the observation was scheduled 15 minutes later, then we would have completely missed it," Mr Lee said.

"It is quite lucky that we discovered it."

It took 14 more observations to uncover its repeating pulses, and understand more about what sort of object it could be.

All of the long-period radio transients discovered so far have involved Australian teams and, according to Dr Anderson, Australia is particularly well placed to find them because of our current generation of radio telescopes.

"[The Murchison Widefield Array and the ASKAP telescope] are the two discovery machines for these types of objects," she said.

The SKA-Low telescope, which is aiming to be fully operational by the end of this decade, will be even more powerful.

So what's the explanation?

While finding these rule-breakers has only taken a few years, understanding what could be causing these mysteriously slow pulses is proving more challenging.

Previous papers have suggested that some other type of star like white dwarfs (created when stars less massive than our Sun run out of fuel and collapse) or special pulsars called magnetars could be behind the slow pulses.

However, the long-period radio transients found so far all emit radiation a little differently.

"They all have different properties," Mr Lee said.

"We don't know whether they belong to the same family, or the same type of object with different mechanisms."

ASKAP is the radio telescope used to discover the neutron star. (ABC News: Tom Hartley)

Dr Anderson noted that there may be two distinct classes of object, one group caused by white dwarfs, and one caused by magnetars.

In the case of ASKAP J1839-075, the evidence suggests that it's unlikely to be a white dwarf.

"This [research] nicely explains the different possible scenarios, but finds [in this case the] isolated neutron star or magnetar scenario is the most likely," Dr Anderson said.

The telltale signs were in ASKAP J1839-075's distinct radio emissions, as well as a lack of a stars visible in optical telescopes, which would normally be seen if the star was a white dwarf.

Even if the star is a magnetar, its slow spin is still almost unheard of, as most magnetars rotate once every two to 10 seconds, and more research will need to be done to understand how they work.

This is unlikely to be the last long-period radio transient scientists find, according to Dr Anderson, although the brightest and most obvious ones have probably already been found.

With the easiest finds out of the way, Dr Anderson suggests that researchers may turn to understanding more about how these rule-breaking stars could have formed.

"Perhaps this is opening an even larger discovery space where there are lots of objects producing these [transients]," she said.

"It's just we had never looked at the galaxy in this way with our radio telescopes before."

Streamline Git with GPT

Thu, 16 Jan 2025 14:46:50 GMT

Score 3 | Comments 0 (thread link) | @thunderbong
January 16, 2025 | https://itsgg.com/2025/01/16/streamline-git-with-gpt.html | archive.org

Prerequisites
Auto-Review Git Changes
Auto-Generate Git Commit Messages
Best Practices
Conclusion

Git workflows can be complex and time-consuming, especially when it comes to reviewing changes and writing meaningful commit messages. This guide shows how to leverage Shell-GPT to automate and enhance your Git experience.

Prerequisites

Before getting started, make sure you have:

Shell-GPT installed (pip install shell-gpt)
Git configured on your system
Basic familiarity with Git commands

Auto-Review Git Changes

Before committing your code, it’s crucial to review the changes. Shell-GPT can help by generating detailed reviews of your staged changes, helping catch potential issues early.

Setup

Run the following command to set up an alias for automated reviews:

git config --global alias.aireview '!f() { git diff --staged | sgpt "Generate a detailed code review"; }; f'

Usage

First, stage your changes with git add, then run:

Example Output

Here's a detailed review of your changes:

1. Feature Addition:
   - Added new user authentication middleware
   - Implemented JWT token validation
   - Added error handling for invalid tokens

2. Code Quality:
   - All new functions are properly documented
   - Error messages are descriptive and actionable
   - Consistent error handling pattern maintained

3. Potential Issues:
   - Consider adding rate limiting to the auth endpoints
   - Token expiration time might need configuration option

4. Recommendations:
   - Add unit tests for the new middleware
   - Document the JWT secret configuration in README

Auto-Generate Git Commit Messages

Writing clear and concise commit messages is an art. Let Shell-GPT help you generate commit messages that follow best practices and maintain a clean Git history.

Setup

Run this command to integrate AI into your commit workflow:

git config --global alias.aicommit '!f() { git add -A && git diff --staged | sgpt "Create a concise commit message with: summary (50 chars) + optional bullet points for details. Do not add any heading." | git commit -F -; }; f'

Usage

When you’re ready to commit your changes, simply run:

Example Output

feat: Add user authentication middleware

- Implement JWT token validation
- Add error handling for invalid tokens
- Create middleware configuration options

Best Practices

Stage your changes deliberately - don’t just use git add -A for everything
Review the AI-generated output before accepting it
You can always modify the commit message or review before proceeding
Use these tools as aids, not replacements for human judgment

Conclusion

These AI-powered Git aliases will help streamline your workflow while maintaining code quality and clear documentation of changes. Remember to review the AI suggestions and adjust them when needed to ensure they match your project’s specific requirements.

Improve Rust Compile Time by 108X

2025-01-10T16:31:43.028327

Score 5 | Comments 1 (thread link) | @nathanielsimard
January 16, 2025 | https://burn.dev/blog/improve-rust-compile-time-by-108x/ | archive.org

Disclamer

Before you get too excited, the techniques used to reduce compilation time are not applicable to all Rust projects. However, I expect the learnings to be useful for any Rust developer who wants to improve their project's compilation time. Now that this is clarified, let's dive into the results.

Results

We started with a compilation time of 108 seconds for the matmul benchmarks, which was reduced to only 1 second after all the optimizations. The most effective optimization was the element-type generics swap, where we instantiated generic functions with predefined "faked" element types to reduce the amount of LLVM[1] code generated. The second optimization also had a major impact, further reducing the compilation time by nearly 3×. This was achieved by using our comptime system instead of associated const generics to represent the matmul instruction sizes. Finally, the last optimization—also the simplest—was to reduce the LLVM optimization level to zero, which is particularly useful for debug builds, such as tests.

Compilation times are measured using incremental compilation.

Motivation

First, let me explain the situation that led me to investigate our compile time. During the last iteration of CubeCL[2], we refactored our matrix multiplication GPU kernel to work with many different configurations and element types. CubeCL is a dialect that lets you program GPUs for high-performance computing applications directly in Rust. The project supports multiple compiler backends, namely WebGPU[3], CUDA[4], ROCm[5], and Vulkan[6] with more to come.

The refactoring of the matrix multiplication kernel was done to improve tensor cores utilization across many different precisions. In fact, each kernel instance works across 3 different element types: the global memory element type, the staging element type, and the accumulation element type. These are all different since we normally want to accumulate in a higher precision element type, as this is where numerical approximation is most sensitive. Also, the tensor cores don't work across all input precisions; if you have f32 inputs, you need to convert those to tf32 element types (staging) to call the tensor cores instructions. To add to the complexity, tensor cores instructions only work across fixed matrix shapes that also depend on the precisions. For instance, f16 staging matrices work across all shapes from (32, 8, 16), (16, 16, 16), (8, 32, 16). But tf32 only works on (16, 16, 8).

In our first refactoring, we represented the shape of the matrices supported by the instructions using const associated types, since this is the abstraction component that makes the most sense in this case. For the element types, we naturally used generic arguments for traits and functions - pretty much what any developer would do in this scenario. However, with all the possible combinations, we ended up with a compilation time of 1m48s using the cache.

Yes, you read that right: 1m48s just to rebuild the matmul benchmark if we change anything in the bench file.

For the purpose of this optimization, we only consider incremental compilation using cargo caching, since this is the most important one to speed up dev iteration time. Changing one configuration to test if an optimization worked took almost 2 minutes just to create the binary to execute a few matmuls.

Why is it that slow?

Well, we need to understand that the Rust compiler is actually very fast. The slow parts are the linking and LLVM. The best way to improve compilation time is to reduce the amount of LLVM IR generated.

In our specific case, each combination of the matmul would generate a whole new function - this is what zero-cost abstraction means. There is no dynamic dispatch; every type change duplicates the code to improve performance at the cost of a bigger binary. Before all of our optimizations, the binary generated was 29M, and after we reduced it to 2.5M - a huge difference.

To reduce the amount of code generated, we had to use different Rust techniques to make our abstractions for the matmul components. In our case, we don't need zero-cost abstractions, since the code written in Rust for the matmul components actually generates the code that is used to compile at runtime a kernel that will be executed on the GPU. Only the GPU code needs to be fast; the JIT Rust code takes almost no time during runtime. Zero-cost abstraction would actually be optimal in a setting where we would perform ahead-of-time compilation of kernels.

Ever wonder why LibTorch[7] or cuBLAS[8] have executables that are GIGABYTES in size? Well, it's because all kernels for all precisions with all edge cases must be compiled to speed up runtime execution. This is necessary in a compute-heavy workload like deep learning.

However, CubeCL is different - it performs JIT compilation, therefore we don't need to compile all possible variations ahead of time before creating the binary: we can use dynamic abstractions instead! This is one of the two optimizations that we made for the matmul components. Instead of relying on const associated types, we leveraged the comptime system to dynamically have access to the instruction sizes during the compilation of a kernel at runtime. This is actually the second optimization that we made and helped us go from 14s compilation time to around 5s.

However, the biggest optimization was quite hard to pull off and is linked to the generic element types passed in each function. We still wanted to use zero-cost abstraction in this case, since passing around an enum listing what element type operations are on would be terrible in terms of developer experience. However, the hint to improve our compilation time was that when you write a function that will execute on the GPU, we have an attribute on top #[cube].

We want the code to look and feel like normal Rust, but the macro actually parses the Rust code written and generates another function, which we normally call the expand function, where the actual GPU IR is built for the function. That code will actually run during runtime, not the code that the user is writing. The element types generics are only used to convert the generic element type into the enum IR element type. In the expand functions, we also pass a context where all IR is tracked.

So the optimization was to pass a fake generic element type, called FloatExpand instead of the actual element type like f32. When compiling a kernel, we first register the real element type in the context, using the const generic item to differentiate multiple element types if a function has multiple generic element types. Since we always call the expand function with the exact same generic items for all element types, we only generate one instance of that function, and the element types are fetched at runtime using the context.

The most tedious part was actually implementing that optimization while trying not to break our components. The biggest problem caused by that optimization is that we can't support generic dependencies between traits over the element type in launch functions of CubeCL.


  #[cube(launch)]
  // Doesn't compile, since we want to call the function with FloatExpand, and the provided I.
  fn matmul<F: Float, I: Instruction<F>>() { 
      // ...
  }

It makes sense though - we don't want to recompile all the instruction types for all different precisions. Since our optimization is activated at the boundaries of CPU code and GPU code, where cube functions are identified as launchable, we need the generic trait to not have a dependency on the element type. They are going to be switched by our macro. We use generic associated types instead of traits with generic element types.

This is known as the family pattern[9], where a trait is describing a family of types.


  pub trait InstructionFamily {
     type Instruction<F: Float>: Instruction<F>;
  }
  
  pub trait Instruction<F: Float> {
      fn execute(lhs: Matrix, rhs: Matrix, accumulator: Matrix);
  }

Using this pattern, we can inject the family type at the boundaries of CPU and GPU code and instantiate the inner instruction type with the expand element type.


  #[cube(launch)]
  fn matmul<F: Float, I: InstructionFamily>() {
     // ...
     I::Instruction::<F>::execute(lhs, rhs, accumulator);
     // ...
  }

  // Launch code generated by the macro #[cube(launch)].
  mod matmul {
    // More code is generated for other stuff ...

    // Expand function generated.
    pub fn expand<F: Float, I: InstructionFamily>(
        context: &mut CubeContext,
    ) {
      // The element type IR can be access like the following.
      let elem = F::as_elem(context);
      // ..
    } 
     
    // The following defines the kernel definition.
    impl<F: Float, I: InstructionFamily, __R: Runtime> Kernel for Matmul<F, I, __R> {
        fn define(&self) -> KernelDefinition {
            let mut builder = KernelBuilder::default();

            // Register the real element type.
            builder
                .context
                .register_elem::<FloatExpand<0u8>>(F::as_elem_native_unchecked());

            // Instantiate the expand function with the expand float element type.
            expand::<FloatExpand<0u8>, I>(&mut builder.context);
            builder.build(self.settings.clone())
        }

        fn id(&self) -> cubecl::KernelId {
            let cube_dim = self.settings.cube_dim.clone();
            // Still use the original element type for the JIT compilation cache.
            KernelId::new::<Self>().info((cube_dim,))
        }
    }
  }

Migrating most of the components to the new pattern, we reduced compile time from 1m48s to about 14s.

It was a lot of work, and I don't expect all projects to face cases like this, but it was worth it! Now waiting for about 5 seconds after trying something in the code to see if performance is improved doesn't break the flow, but almost 2 minutes did.

We essentially leveraged the fact that CubeCL is a JIT compiler and not an AOT compiler, which is very appropriate for throughput-focused high-performance applications.

Playing with LLVM optimization settings

Since our benchmarks are compiled with optimization level set to 3, we could still improve the compilation time further to about 1s by reducing the optimization level to zero. Another 5X speedup that we can have by simply adjusting the LLVM optimization level.


  [profile.dev]
  opt-level = 0

We decided not to keep that optimization in production, since we want the benchmarks to have the same LLVM optimization level as user applications. However, we activated it for testing, since we often rerun tests to ensure we don't break correctness when implementing or optimizing kernels.

Not a Rust Compiler Issue

All of our optimizations actually created tons of code - we used proc macros, associated type generics, const generics, and tons of complex features from the Rust type system.

The Rust compiler is actually very fast; the slow part is really the linking and optimizing of the LLVM IR. If there's one thing to take from this post, it's that you shouldn't worry about using complex features of Rust, but make sure you don't generate huge binaries. Reducing the binary size will improve compile time even if you use complex methods to do so! "Less code compiles faster" is not exactly right. "Less generated code compiles faster" is what we have to keep in mind!

Zig comptime: does anything come close?

Thu, 16 Jan 2025 14:00:37 GMT

Score 3 | Comments 2 (thread link) | @bw86
January 16, 2025 | https://renato.athaydes.com/posts/comptime-programming | archive.org

Zig comptime: does anything come close?

A look at what other languages can do where Zig would just use comptime.

Written on Wed, 15 Jan 2025 19:19

Image by rawpixel.com on Freepik

Zig’s comptime feature has been causing a little bit of a fuss on programmer discussion forums lately. It’s a very interesting feature because it is relatively simple to grasp but gives Zig code some super-powers which in other languages tend to require a lot of complexity to achieve. That’s in line with Zig’s stated desire to remain a simple language.

The most common super-power (ok, maybe not a super-power, but a non-trivial feature) given as an example is generics, but there are many more.

In this post, I decided to have a look at that in more detail and compare Zig’s approach with what some other languages can offer. Zig uses comptime for things that look a little bit like macros from Lisp or Rust, templates from C++ and D, and code generation via annotation processors from Java and Kotlin! Are those alternatives anywhere near as powerful? Or easy to use?

I will investigate that by looking at some Zig examples from Loris Cro (Zig evangelist)’s What is Zig’s Comptime and Scott Redig’s Zig’s Comptime is Bonkers Good.

I only know relatively few languages, so apologies if I didn’t include your favourite one (do let me know if your language has even nicer solutions)… specifically, I don’t know much C++, so be warned that while I am aware C++ has constexpr (I guess that’s the closest thing to comptime), I won’t even try to compare that with comptime in this post!

The basics

The first thing one can imagine doing at comptime in any language is to compute the value of some constant.

I mean, it would kind of suck if this expression was actually computed at runtime:

// five days
int time_ms = 5 * 24 * 60 * 60 * 1000;

Most compilers (maybe even all?) will use constant folding and compile that down to this:

int time_ms = 432'000'000;

However, they generally refuse to do this if any function calls are involved. Why? 🤔

Because it may be too hard, I suppose! Evaluating even simple expressions as above require at least an arithmetic interpreter, so almost all compilers must already have one hidden in them. Is executing at least some functions in their interpreters much harder to do?

I am not sure, but apparently the Zig authors (that would be Andrew Kelley, now backed by the Zig Software Foundation) think doing that is worth it!

fn multiply(a: i64, b: i64) i64 {
    return a * b;
}

pub fn main() void {
    const len = comptime multiply(4, 5);
    const my_static_array: [len]u8 = undefined;
    _ = my_static_array;
}

The multiply function is just a regular function that can obviously be called by other functions at runtime. But in this sample, it’s getting called with the comptime keyword in front of it:

const len = comptime multiply(4, 5);

That forces the compiler to actually call multiply at compile-time and replace the expression with the result. That means that, at runtime, the above line would be replaced with this:

const len = 20;

That’s why len can be used in the next line to create a static array (Zig doesn’t have dynamic arrays, so without comptime that wouldn’t work).

Of course, in this example you could just write that, but as you can call most other code, you might as well pre-compute things that would be hard to do at compile time otherwise.

The first thing that came to mind when I saw this example was Lisp’s read time evaluation!

(defun multiply (a b)
    (* a b))

(defun main ()
    (let ((len #.(multiply 4 5)))
        (format t "Lenght is ~s" len)))

The Lisp reader computes #.(multiply 4 5), so if you compile the function, it will just contain a 20 there at runtime.

It’s kind of important to understand that this is not the same as initializing constants at runtime, as this Java code would do:

class Hello {
    // Java evaluates this at runtime, when this Class is loaded
    static final int LEN = multiply(5, 4);
    
    static int multiply(int a, int b) {
        return a * b;
    }
}

We can imagine some expensive computation that might take time to perform at runtime, even if only once when the program starts up. If you do that at compile time, you save all that time forever, on every run of your program.

In D, we can also easily call “normal” functions at comptime (D calls that Compile Time Function Evaluation, or CTFE):

int multiply(int a, int b) => a * b;

void main()
{
    enum len = multiply(5, 4);
    ubyte[len] my_static_array;
}

D uses the enum keyword to declare comptime variables. But despite some name differences, you can see that this example is extremely similar to Zig’s.

Walter Bright, creator of D, has just written an article where he claims that “evaluating constant-expressions” is an obvious thing C should do! And in fact, the C compiler used by D to compile C code, importC does it!

Rust can also do it, but only a sub-set of the language is available as documented here, and only functions explicitly marked as const fn can be used (a big limitation in comparison with Zig and D):

const fn multiply(a: usize, b: usize) -> usize {
    a * b
}

fn main() {
    const len: usize = multiply(5, 4);
    let my_static_array: [u8; len];
}

Even in Zig and D, however, not everything can be done at comptime. For example, IO and networking seem to be out-of-limits, at least in the usual form.

Trying to read a file in Zig at comptime causes an error:

const std = @import("std");

fn go() []u8 {
    var buffer: [64]u8 = undefined;
    const cwd = std.fs.cwd();
    const handle = cwd.openFile("read-file-bytes.zig", .{ .mode = .read_only }) catch unreachable;
    defer handle.close();
    const len = handle.readAll(&buffer) catch unreachable;
    const str = buffer[0..len];
    const idx = std.mem.indexOf(u8, str, "\n") orelse len;
    return str[0..idx+1];
}

pub fn main() void {
    const file = comptime go();
    std.debug.print("{s}", .{file});
}

ERROR:

/usr/local/Cellar/zig/0.13.0/lib/zig/std/posix.zig:1751:30: error: comptime call of extern function
        const rc = openat_sym(dir_fd, file_path, flags, mode);
                   ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/Cellar/zig/0.13.0/lib/zig/std/fs/Dir.zig:880:33: note: called from here
    const fd = try posix.openatZ(self.fd, sub_path, os_flags, 0);
                   ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/Cellar/zig/0.13.0/lib/zig/std/fs/Dir.zig:827:26: note: called from here
    return self.openFileZ(&path_c, flags);
           ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
read-file-bytes.zig:6:32: note: called from here
    const handle = cwd.openFile("read-file-bytes.zig", .{ .mode = .read_only }) catch unreachable;
                   ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
read-file-bytes.zig:14:29: note: called from here
    const file = comptime go();
                          ~~^~

Remove the comptime keyword and the example above works.

In D, same thing:

import std;

void main()
{
    string firstLineOfFile()
    {
        auto file = File("parser.d");
        foreach (line; file.byLineCopy)
        {
            return line;
        }
        return "";
    }

    enum line = firstLineOfFile();

    // D's comptime version of printf
    pragma(msg, line);
}

ERROR:

Error: `fopen` cannot be interpreted at compile time, because it has no available source code
Error: `malloc` cannot be interpreted at compile time, because it has no available source code
main.d(15):        compile time context created here
main.d(17):        while evaluating `pragma(msg, line)`

Replace enum with auto and use writeln instead of pragma and the above also works.

Notice how both Zig and D appear to fail due to extern functions… Zig explicitly disallows syscalls. That makes sense as allowing that could make compilation really undeterministic and the result highly likely to diverge depending on the exact machine the code was compiled on.

Despite that, both languages provide ways to actually read files at comptime!

In Zig, you can use @embedFile:

const std = @import("std");

fn go() [] const u8 {
    const file = @embedFile("read-file-bytes.zig");
    const idx = std.mem.indexOf(u8, file, "\n") orelse file.len - 1;
    return file[0..idx+1];
}

pub fn main() void {
    const file = comptime go();
    std.debug.print("{s}", .{file});
}

D offers import expressions:

This requires using the compiler -J flag to point to a directory where to import files from.

import std;

void main()
{
    string firstLineOfFile()
    {
        enum file = import("main.d");
        return file.splitter('\n').front;
    }

    enum line = firstLineOfFile();
    
    // D's comptime version of printf
    pragma(msg, line);
}

Rust has include_bytes:

fn main() {
    let file = include_bytes!("main.rs");
    println!("{}", std::str::from_utf8(file).unwrap());
}

The above program embeds the main.rs file as a byte array and prints it at runtime.

However, a const fn cannot currently call include_bytes (there’s an issue to maybe make that possible). Hence, include_bytes is a bit more limited than Zig’s @embedFile and D’s import.

In Java, the closest you can get to doing this would be to include a resource in your jar and then load that at runtime:

try (var resource = MyClass.class.getResourceAsStream("/path/to/resource")) {
    // read the resource stream
}

Not quite the same, of course, but gets the jo done. It probably does not even need to perform IO as it’s going to just take the bytes from the already opened and likely memory-mapped jar/zip file where the class itself came from.

comptime blocks and arguments

The next example from Kristoff’s blog is very interesting. It implements the insensitive_eql function so that one of the strings is known at comptime and can be verified to be uppercase by the compiler (thanks to your own comptime code, that is):

// Compares two strings ignoring case (ascii strings only).
// Specialzied version where `uppr` is comptime known and *uppercase*.
fn insensitive_eql(comptime uppr: []const u8, str: []const u8) bool {
    comptime {
        var i = 0;
        while (i < uppr.len) : (i += 1) {
            if (uppr[i] >= 'a' and uppr[i] <= 'z') {
                @compileError("`uppr` must be all uppercase");
            }
        }
    }
    var i = 0;
    while (i < uppr.len) : (i += 1) {
        const val = if (str[i] >= 'a' and str[i] <= 'z')
            str[i] - 32
        else
            str[i];
        if (val != uppr[i]) return false;
    }
    return true;
}

pub fn main() void {
    const x = insensitive_eql("Hello", "hElLo");
}

ERROR:

insensitive_eql.zig:8:17: error: `uppr` must be all uppercase
                @compileError("`uppr` must be all uppercase");
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Or course, to fix that you just need to replace the first argument to insensitive_eql with something all-caps:

pub fn main() void {
    const x = insensitive_eql("HELLO", "hElLo");
}

This is a nice optimization based on comptime argument values.

Let’s see if we can rewrite this example in D, given D seems to also support comptime pretty well:

The original example did not check the length of the strings, so it can obviously return incorrect results if the input is longer than the comptime value, or read past the input’s bounds if that’s shorter, but I kept the same behaviour in this translation because this is not trying to implement a production-grade algorithm anyway, and it’s fun to maybe poison the training data for the AIs scraping my site.

/// Compares two strings ignoring case (ascii strings only).
/// Specialzied version where `uppr` is comptime known and *uppercase*.
bool insensitiveEqual(string uppr)(string str)
{
    import std.ascii : lowercase, toUpper;
    import std.algorithm.searching : canFind;

    static foreach (c; uppr)
    {
        static if (lowercase.canFind(c))
            static assert(0, "uppr must be all uppercase");
    }
    foreach (i, c; str)
    {
        if (c.toUpper != uppr[i])
            return false;
    }
    return true;
}

void main()
{
    const x = insensitiveEqual!("Hello")("hElLo");
}

ERROR:

insensitive_eql.d(11): Error: static assert:  "uppr must be all uppercase"
insensitive_eql.d(23):        instantiated from here: `insensitiveEqual!"Hello"`

While D does not have comptime blocks, it’s fairly clear when a block is executed at comptime due to the use of static to differentiate if, foreach and assert from their runtime counterparts. If you see static if you just know it’s comptime.

This example shows one of the main differences between Zig’s and D’s comptime facilities: while in Zig, whether a function argument must be known at comptime is determined by annotating the argument with comptime, in D there are two parameter lists: the first list consists of the comptime parameters, and the second one of the runtime parameters.

Hence, in Zig you cannot know by looking at a function call which arguments are comptime, you need to know the function signature or track where the variable came from. In D, you can, because the comptime parameters come on a separate list which must follow the ! symbol:

insensitiveEqual!("Hello")("hElLo");

The parenthesis are optional when only one argument is used, so this is equivalent:

insensitiveEqual!"Hello"("hElLo");

The downside is that one cannot interleave runtime and comptime arguments, which may be a bit more natural in some cases.

Moving on to Rust now… Rust does have something that lets us say this value must be known for the life of the program: lifetime annotations. That may not be exactly the same as comptime, but sounds close! Let’s try using it.

fn insensitive_eql(uppr: &'static str, string: &str) -> bool {
    todo!()
}

const fn is_upper(string: &str) -> bool {
    string.bytes().all(|c| c.is_ascii_uppercase())
}

pub fn main() {
    let a = "Hello";
    assert!(is_upper(a));
    print!("result: {}", insensitive_eql(a, "hElLo"));
}

ERROR:

error[E0015]: cannot call non-const fn ` as Iterator>::all::<{closure@src/main.rs:6:24: 6:27}>` in constant functions
 --> src/main.rs:6:20
  |
6 |     string.bytes().all(|c| c.is_ascii_uppercase())
  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: calls in constant functions are limited to constant functions, tuple structs and tuple variants

Close, but no cigar! As we can only call other const fns from a const fn, and no one seems to declare all their functions that could be const fn actually const fn (sorry for so many const fns in one sentence), there seems to be very little we can do with const fns! We would probably need to implement our own is_upper const fn to get around that. But even if we did, what we really needed was for the assertion that the string was all-caps to run at compile-time, and as far as I can see that’s not possible with Rust.

Java doesn’t have const fn or anything similar in the first place (Java is extremely dynamic, you can load a jar at runtime, create a ClassLoader and start using code from the jar), so it’s hopeless to even try anything, I believe.

Lisp can do anything at read time (there’s nearly no difference between comptime, readtime and runtime as far as what code you can use). However, for the sake of brevity and to try to focus on statically-typed languages where there’s a clear compile-/run-time split, I’m afraid I will not be including Lisp anymore, sorry if you’re a fan!

Code ellision

This example is also from Kristoff’s post, but adapted by me to compile with Zig 0.13.0 (Zig’s stdlib changes a lot between releases):

const builtin = @import("builtin");
const std = @import("std");
const fmt = std.fmt;
const io = std.io;

const Op = enum {
    Sum,
    Mul,
    Sub,
};

fn ask_user() !i64 {
    var buf: [10]u8 = undefined;
    std.debug.print("A number please: ", .{});
    const stdin = std.io.getStdIn().reader();
    const user_input = try stdin.readUntilDelimiter(&buf, '\n');
    return fmt.parseInt(i64, user_input, 10);
}

fn apply_ops(comptime operations: []const Op, num: i64) i64 {
    var acc: i64 = 0;
    inline for (operations) |op| {
        switch (op) {
            .Sum => acc +%= num,
            .Mul => acc *%= num,
            .Sub => acc -%= num,
        }
    }
    return acc;
}

pub fn main() !void {
    const user_num = try ask_user();
    const ops = [4]Op{ .Sum, .Mul, .Sub, .Sub };
    const x = apply_ops(ops[0..], user_num);
    std.debug.print("Result: {}\n", .{x});
}

It shows how inline for can be used to loop using comptime variables. The loop gets unrolled at comptime, naturally. Looking at the generated Assembly in Godbolt, one can clearly see it’s only doing the expected arithmetics.

As we’ve just learned in the previous section, D’s static for is a comptime loop, hence equivalent to Zig’s inline for.

Why Zig calls this inline for instead of comptime for? And why does D use enum to designate comptime variables? Language creators don’t seem to be any better at naming things than the rest of us.

Here’s the same application written in D:

enum Op
{
    SUM,
    MUL,
    SUB,
}

long askUser()
{
    import std.stdio : write, readln;
    import std.conv : to;

    char[] buf;
    write("A number please: ");
    auto len = readln!char(buf);
    return buf[0 .. len - 1].to!long;
}

long applyOps(Op[] operations)(long num)
{
    long acc = 0;
    static foreach (op; operations)
    {
        final switch (op) with (Op)
        {
        case SUM:
            acc += num;
            break;
        case MUL:
            acc *= num;
            break;
        case SUB:
            acc -= num;
        }
    }
    return acc;
}

int main()
{
    import std.stdio : writeln;

    auto num = askUser();
    with (Op)
    {
        static immutable ops = [SUM, MUL, SUB, SUB];
        num = applyOps!ops(num);
    }
    writeln("Result: ", num);
    return 0;
}

In case you’re not familiar with D, final switch is a switch that demands all cases to be covered, as opposed to regular switch which require a default case (so enums could evolve without breaking callers).

In this example, the two languages look very similar, apart from the syntax differences (and formatting; I am using the standard format given by the respective Language Servers).

But there’s a problem with the D version. If you look at the Assembly it generates, it’s obvious that the switch is still there at runtime, even when using the -release flag, as far as I can see.

The reason is that there’s no static switch in D, i.e. no comptime switch (pretty lazy of them to only have included static if)!

For that reason, the switch needs to be rewritten to use static if to be actually equivalent to Zig:

long applyOps(Op[] operations)(long num)
{
    long acc = 0;
    static foreach (op; operations)
    {
        static if (op == Op.SUM)
        {
            acc += num;
        }
        else static if (op == Op.MUL)
        {
            acc *= num;
        }
        else static if (op == Op.SUB)
        {
            acc -= num;
        }
    }
    return acc;
}

This generates essentially the same Assembly as Zig.

I am not aware of any other statically-typed language that could also implement this, at least without macros, but would be happy to be hear about it if anyone knows (languages like Terra, which are not really practical languages, don’t count).

But talking about macros, Rust has macros! It actually has four types of macros.

Can we do this with macros?

Let’s see. Calling applyOps in Rust should look something like this:

apply_ops!(num, SUM, MUL, SUB, SUB);

Notice that the ! symbol in Rust is a macro specifier. The question is, can we write a loop and a switch in a Rust macro (this kind of macro is called declarative macro)?

Yes!

macro_rules! apply_ops {
    ( $num:expr, $( $op:expr ),* ) => {
    {
        use Op::{Mul, Sub, Sum};
        let mut acc: u64 = 0;
        $(
            match $op {
                Sum => acc += $num,
                Mul => acc *= $num,
                Sub => acc -= $num,
            };
        )*
        acc
    }
    }
}

The part between $( .. )* may not look much like a loop, but it is! The reason it looks different than a normal loop is that this uses Rust’s macro templating language, essentially. That’s the problem with macro systems: they are a separate language within a host language, normally. Except, of course, if your language is itself written in AST form, like with Lisp languages, but let’s leave that for another time.

At least macros do run, or get expanded, at comptime!

Here’s the full example in Rust:

use std::io::{self, Write};

enum Op {
    Sum,
    Mul,
    Sub,
}

macro_rules! apply_ops {
    ( $num:expr, $( $op:expr ),* ) => {
    {
        use Op::{Mul, Sub, Sum};
        let mut acc: u64 = 0;
        $(
            match $op {
                Sum => acc += $num,
                Mul => acc *= $num,
                Sub => acc -= $num,
            };
        )*
        acc
    }
    }
}

fn ask_user() -> u64 {
    print!("A number please: ");
    io::stdout().flush().unwrap();
    let mut line = String::new();
    let _ = io::stdin().read_line(&mut line).unwrap();
    line.trim_end().parse::<u64>().expect("a number")
}

pub fn main() {
    let num = ask_user();
    let num = apply_ops!(num, Sum, Mul, Sub, Sub);
    println!("Result: {}", num);
}

It’s surprisingly similar to Zig… well, at least if you ignore the macro syntax.

The cool thing about Rust is that it is very popular, or at least it’s popular with people who care about tooling! So, it has awesome tooling, like the cargo-expand crate.

To install it, run:

cargo install cargo-expand

… and wait for 5 minutes as it compiles half of the crates in crates.io! But the wait is worth it!

Look at what it prints when I run cargo expand (displayed with the real colors on my terminal!):

Isn’t that pretty!?

I don’t know of any tool that can do this for Zig or D!

This reveals that the macro expands kind of as expected. Hopefully, the compiler will be smart enough to actually remove the match Sum { Sum => ... } blocks.

I did try to verify it by looking at the generated ASM using cargo-asm, but that crate couldn’t do it for this example as it crashed!

ERROR:

thread 'main' panicked at /Users/renato/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo-asm-0.1.16/src/rust.rs:123:33:
called `Result::unwrap()` on an `Err` value: Os { code: 21, kind: IsADirectory, message: "Is a directory" }

So much for great tooling (and people should just stop using unwrap in production code that can actually fail… “this should never happen” always happens)!

I knew I should’ve just used Godbolt from the start, but I prefer to use local tools if I can. Anyway, without using any compiler arguments, it looks like the match blocks are still there! However, using the -O flag to enable optimisations removes so much code (and inlines everything) that all I see is one multiplication! What kind of magic is that!?

Generics

Finally, we get to the most famous part of Zig’s comptime story: how it can provide generics-like functionality without actually having generics!

Here’s Kristoff’s generics sample (again, updated for Zig 0.13.0):

/// Compares two slices and returns whether they are equal.
pub fn eql(comptime T: type, a: []const T, b: []const T) bool {
    if (a.len != b.len) return false;
    for (0.., a) |index, item| {
        if (!std.meta.eql(b[index], item)) return false;
    }
    return true;
}

Notice how the first argument to eql is a type! This is a very common pattern in Zig.

That makes it look similar to “real” generics in languages like Java and Rust (even Go has recently added generics). But whereas in those languages, generics are a feature on their own, and a complex one at that, in Zig, it’s just a result of how comptime works (and a bunch of built-ins provided to work with types).

Arguably, however, what Zig does is not truly generics but templating.

Check out Zig-style generics are not well-suited for most languages for a more critical look at the differences.

In a language with true generics, there are some signficant differences. Let’s see what it looks like in Rust:

fn eql<T: std::cmp::PartialEq>(a: &[T], b: &[T]) -> bool {
    if a.len() != b.len() {
        return false;
    }
    for i in 0..a.len() {
        if a[i] != b[i] {
            return false;
        }
    }
    true
}

Notice how there’s a type bound to the generic type T:

T: std::cmp::PartialEq

That’s because you wouldn’t be able to use the != operator otherwise, as only implementations of the PartialEq trait provide that in Rust.

Removing the type bound gives me an opportunity to showcase another Rust error message, which is always fun:

error[E0369]: binary operation `!=` cannot be applied to type `T`
 --> src/main.rs:6:17
  |
6 |         if a[i] != b[i] {
  |            ---- ^^ ---- T
  |            |
  |            T
  |
help: consider restricting type parameter `T`
  |
1 | fn eql(a: &[T], b: &[T]) -> bool {
  |         +++++++++++++++++++++

This kind of generics is quite nice on tooling: the IDE (and the programmer) will know exactly what can and cannot be done with arguments of generic types, so it can easily diagnose mistakes. With templates, that is only possible at the call site because that’s when the template will actually be instantiated and type-checked!

To illusrate that, in the following example, the eql call fails to compile:

pub fn main() !void {
    const C = struct {};
    const a: []C = &[0]C{};
    const b: []C = &[0]C{};
    
    std.debug.print("Result: {}\n", .{eql(C, a, b)});
}

ERROR:

generics.zig:9:22: error: operator != not allowed for type 'comptime.main.C'
        if (b[index] != item) return false;
            ~~~~~~~~~^~~~~~~
generics.zig:14:15: note: struct declared here
    const C = struct {};
              ^~~~~~~~~

The error is shown inside the template itself, even though the template is not where the mistake is. That’s worse than in Rust and other languages with similar generics, where the error is shown on the call site if it tries to use a type which does not satisfy the generic type bounds in the function signature.

Going back to the error above, that happens because Zig only defines the equality operators for Integers, Floats, bool and type, so it won’t work for structs.

As an aside, as Zig has no interfaces or traits unless you perform some acrobatics, it couldn’t declare type bounds that way, currently. It could, however, do something like D does and allow boolean expression of types to be added to function signatures, as we’ll see.

For other types, the comparison should be done with std.meta.eql:

if (!std.meta.eql(b[index], item)) return false;

Indeed, after this change, the eql example now works for any type, thanks to std.meta.eql actually doing the heavy lifting, of course.

And the way it does that is by specializing behaviour for different types.

Here’s a partial view of its implementation:

pub fn eql(a: anytype, b: @TypeOf(a)) bool {
    const T = @TypeOf(a);

    switch (@typeInfo(T)) {
        .Struct => |info| {
            inline for (info.fields) |field_info| {
                if (!eql(@field(a, field_info.name), @field(b, field_info.name))) return false;
            }
            return true;
        },
...
        .Array => {
            if (a.len != b.len) return false;
            for (a, 0..) |e, i|
                if (!eql(e, b[i])) return false;
            return true;
        },
...
        else => return a == b,
    }
}

This shows the special type, anytype, which is basically comptime duck typing. In this case, the function explicitly handles several known types (the actual type is obtained using @typeInfo), then falls back on the == operator.

In a language with union types, this would be easy to represent type-safely, but in Zig, anytype is the only solution, currently (but there are proposals to improve this). That’s not ideal because programming with anytype feels like programming in a dynamically typed language: you just don’t know what you can do with your values and just hope for the best. Unlike dynamic languages, the type checker will still eventually catch up with you, obviously, but that doesn’t help when you’re writing the function or even trying to figure out if you can call it (you want to avoid being yelled at by the compiler for writing bad code, after all). The only way to know what the function requires of a value of type anytype is to look at its source code, which is one of the reasons why a lot of complaints by people who are new to the Zig community are waved away with just read the code.

For example, this is a valid function in Zig (though it probably cannot be called by anything):

fn cant_call_me(a: anytype) @TypeOf(a) {
    const b = a.b;
    const e = b.c.d.e;
    if (e.doesStuff()) {
        return e.f.g;
    }
    return b.default.value;
}

The @TypeOf(a) expression is a little awkward in the previous example, but allows telling the compiler that b must be of whatever type a is (not anytype but the actual type at the call site). There is an interesting proposal to improve on that with infer T.

Notice how this function signature:

pub fn eql(a: anytype, b: @TypeOf(a)) bool

Can be rewritten to, at the cost of having to specify the type explicitly, the following:

pub fn eql(comptime T: type, a: T, b: T) bool

The choice of which to use is somewhat subtle.

D has had to grapple with most of these issues in the past as well. For example, to assist with knowing which types you could call a generic function with, it allows adding type constraints to its function signatures, making it a little closer to Rust than Zig in that regard, despite using templates to implement generics.

Let’s look at another Zig example before we go back to D to see how it works:

// This is the stdlib implementation of `parseInt`
pub fn parseInt(comptime T: type, buf: []const u8, radix: u8) !T

This function does not declare what types T may be, but almost certainly, you can only use one of the integer types (e.g. u8, u16, u32, i32 …). In other cases, it may not be as easy to guess.

Well, with D, one could say exactly what the requirements are (though what isIntegral means in D is a little bit different):

import std.traits : isIntegral;

T parseInt(T)(string value, ubyte radix) if (isIntegral!T) {
  todo();
}

The isIntegral function is one of many comptime type-checking helpers from the std.traits module.

That’s a very good approach for both documentation and tooling support.

Notice that you could define isIntegral yourself (e.g. to be more strict with what types are allowed):

bool isIntegral(T)()
{
    return is(T == uint) || is(T == ulong) ||
        is(T == int) || is(T == long);
}

T parseInt(T)(string value, ubyte radix) if (isIntegral!T)
{
    todo();
}

Trying to call parseInt with some other type, say char, causes an error, of course:

void main()
{
    parseInt!char("c", 1);
}

ERROR:

tests.d(51): Error: template instance `tests.parseInt!char` does not match template declaration `parseInt(T)(string value, ubyte radix)`
  with `T = char`
  must satisfy the following constraint:
`       isIntegral!T`

Type introspection

This Zig example is from Scott Redig’s blog post:

const std = @import("std");

const MyStruct = struct {
    a: i64,
    b: i64,
    c: i64,

    fn sumFields(my_struct: MyStruct) i64 {
        var sum: i64 = 0;
        inline for (comptime std.meta.fieldNames(MyStruct)) |field_name| {
            sum += @field(my_struct, field_name);
        }
        return sum;
    }
};

pub fn main() void {
    const ms: MyStruct = .{ .a = 32, .b = 4, .c = 2 };
    std.debug.print("{d}\n", .{ms.sumFields()});
}

Even though this example only works on this particular struct, you can easily rewrite it so it can sum the integer fields of any struct (even those that have non-integer fields):

const std = @import("std");

fn sumFields(my_struct: anytype) i64 {
    var sum: i64 = 0;
    inline for (comptime std.meta.fieldNames(MyStruct)) |field_name| {
        const FT = @TypeOf(@field(my_struct, field_name));
        if (@typeInfo(FT) == .Int) {
            sum += @field(my_struct, field_name);
        }
    }
    return sum;
}

const MyStruct = struct {
    a: i64,
    s: []const u8,
    b: i64,
    c: i32,
};

pub fn main() void {
    const ms: MyStruct = .{ .a = 32, .b = 4, .c = 2, .s = "" };
    std.debug.print("{d}\n", .{sumFields(ms)});
}

This is very, very cool because of the implications: you can use the same technique to generate code that, for example, serializes structs into JSON, or whatever data format.

You may not be surprised by now to hear that D can also do the same thing:

int main()
{
    import std.stdio : writeln;

    struct S
    {
        ulong a;
        string s;
        ulong b;
        uint c;
    }

    S s = {a: 32, b: 4, c: 2, s: ""};
    writeln("Result: ", sumFields(s));
    return 0;
}

long sumFields(S)(S myStruct) if (is(S == struct))
{
    import std.traits : isIntegral;

    long sum = 0;
    foreach (member; __traits(allMembers, S))
    {
        auto value = __traits(getMember, myStruct, member);
        static if (__traits(isIntegral, typeof(value)))
        {
            sum += value;
        }
    }
    return sum;
}

The __traits syntax takes some getting used to, but after that it’s not very different from Zig’s @TypeOf and similar built-ins.

I want to emphasize again that this is all doing work at comptime, and at runtime, you have custom-made code for each type you call the function with. For example, for a struct like this:

struct AB {
    int a;
    int b;
}

The actual runtime version of sumFields should look like this:

long sumFields(AB myStruct) {
    long sum = 0;
    sum += myStruct.a;
    sum += myStruct.b;
    return sum;
}

Do visit Godbolt and check it out.

This is in contrast to something like Java reflection. You could definitely implement something similar using reflection, but that would have a high runtime cost, not to mention the code would look much more complex than this.

However, Java does have a solution that can work: annotation processors. Well, at least as long as you don’t mind doing all of the following:

create an annotation, say @SumFields.
write an annotation processor for this annotation that generates a new class that can provide the sumFields method.
annotate each class you want to sum the fields of with @SumFields. Tough luck if it’s not your class to change.
configure the build to invoke your annotation processor.
build your project, then finally use the generated class to call sumFields.

I was going to include some code showing how this looks like in practice, but I decided to just link to an article that shows something similar at Baeldung instead, I hope you can understand that.

Because the process to do this in Java is so convoluted, people almost never do that in application code. Only frameworks do, they have to. So, you get some pretty big frameworks that can do impressive things like comptime Dependency Injection and JSON serialization.

Rust has procedural macros which are pretty similar to Java annotation processors. But because working directly on ASTs is not a lot of fun, and Rust has macros, one can use the quote crate to write things more easily in a way that resembles a lot templates.

In Zig and D, you don’t need frameworks doing magic and you don’t need macros 😎.

Textual code generation

Finally, Redig’s blog post has a section titled View 5: Textual Code Generation where he says:

“If this method of metaprogramming (textual code generation) is familiar to you then moving to Zig comptime might feel like a significant downgrade.”

This may sound crazy, but with D, you can even do that:

string toJson(T)(T value) {
  mixin(`import std.conv : to;string result = "{\n";`);
  foreach (member; __traits(allMembers, T)) {
    mixin("auto v = value." ~ member ~ ".to!string;");
    mixin(`result ~= "  ` ~ member ~ `: " ~ v;`);
  }
  mixin(`result ~= "\n}\n"; return result;`);
}

Above, I wrote a very basic, incomplete, pseudo-JSON serializer for anything that works with the allMembers trait (though it probably only does the right thing for classes and structs) using just string mixins.

Using the struct instance from the previous example:

S s = {a: 32, b: 4, c: 2, s: "hello"};
writeln("Result: ", s.toJson);

Prints:

Result: {
  a: 32  s: hello  b: 4  c: 2
}

I don’t know about you, but I find this bonkers good, indeed.

Final thoughts

Zig is doing great work on a lot of fronts, specially with its build system, C interop and top-of-the-line cross-compilation.

Its meta-programming capabilities are also great, as we’ve seen, and comptime fits the language perfectly to provide some powerful features while keeping the overall language simple.

The instability of the language still keeps me away for now. I maintain a basic Zig Common Tasks website and every upgrade requires quite some effort (though it was much less in the last version). I hope they get to a 1.0 release soon, so this won’t be a problem anymore.

Anyway, while Zig deserves all the praise it’s getting, I think it’s not the only one.

As I hope to have shown in this post, D’s CTFE and templates appear to be able do pretty much everything that Zig comptime can, and then some. But no one is talking about it! D certainly isn’t getting huge donations from wealthy admirers and people are not being paid the largest salaries to write code in it. As someone who is just a distant observer in all of this, I have to wonder why that is. I get that Zig is not just comptime… while D may have an even more impressive comptime than Zig, it may lack in too many other areas.

After playing with D for some time, my main complaints are:

it has too many features, some of which a little rough around the edges, like its multitude of function parameters modifiers.
the build system, Dub, seriously needs more attention. I tried to help with docs but my help was kind of ignored.
how hard it is to use @safe, @nogc and pure - they would be awesome if they were actually usable!. Calling almost anything on the standard library forces you to remove these.

But D also has a lot cool stuff I didn’t mention:

seamless C interop, as good as Zig.
betterC for really low-level stuff.
an excellent concurrency story.
an extensive standard library.
3 good compiler backends (DMD - official, GDC - GCC-based, and LDC - LLVM-based).

Maybe the fact that D has a GC doomed it from the start?

Rust has a less fun way of doing things, arguably, but it also covers a lot of things that are possible with comptime through macros, generics and traits. And it doesn’t have a GC (take that D)! And it is memory-safe (take that, Zig)!

Java may not be at nearly the same level in terms of meta-programming as any of the other languages mentioned in this post, but at least writing top-notch tooling for it is much easier, and it shows! And with its love for big frameworks and wealth of libraries, it actually manages to almost compensate for that.

Nepenthes is a tarpit to catch AI web crawlers

Thu, 16 Jan 2025 13:57:43 GMT

Score 141 | Comments 61 (thread link) | @blendergeek
January 16, 2025 | https://zadzmo.org/code/nepenthes/ | archive.org

This is a tarpit intended to catch web crawlers. Specifically, it's targetting crawlers that scrape data for LLM's - but really, like the plants it is named after, it'll eat just about anything that finds it's way inside.

It works by generating an endless sequences of pages, each of which with dozens of links, that simply go back into a the tarpit. Pages are randomly generated, but in a deterministic way, causing them to appear to be flat files that never change. Intentional delay is added to prevent crawlers from bogging down your server, in addition to wasting their time. Lastly, optional Markov-babble can be added to the pages, to give the crawlers something to scrape up and train their LLMs on, hopefully accelerating model collapse.

You can take a look at what this looks like, here. (Note: VERY slow page loads!)

THIS IS DELIBERATELY MALICIOUS SOFTWARE INTENDED TO CAUSE HARMFUL ACTIVITY. DO NOT DEPLOY IF YOU AREN'T FULLY COMFORTABLE WITH WHAT YOU ARE DOING.

LLM scrapers are relentless and brutual. You may be able to keep them at bay with this software - but it works by providing them with a neverending stream of exactly what they are looking for. YOU ARE LIKELY TO EXPERIENCE SIGNIFICANT CONTINUOUS CPU LOAD, ESPECIALLY WITH THE MARKOV MODULE ENABLED.

There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS.

Latest Version

Nepenthes 1.0

All downloads

Usage

Expected usage is to hide the tarpit behind nginx or Apache, or whatever else you have implemented your site in. Directly exposing it to the internet is ill advised. We want it to look as innocent and normal as possible; in addition HTTP headers are used to configure the tarpit.

I'll be using nginx configurations for examples. Here's a real world snippet for the demo above:

    location /nepenthes-demo/ {
            proxy_pass http://localhost:8893;
            proxy_set_header X-Prefix '/nepenthes-demo';
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_buffering off;
    }

You'll see several headers are added here: "X-Prefix" tells the tarpit that all links should go to that path. Make this match what is in the 'location' directive. X-Forwarded-For is optional, but will make any statistics gathered significantly more useful.

The proxy_buffering directive is important. LLM crawlers typically disconnect if not given a response within a few seconds; Nepenthes counters this by drip-feeding a few bytes at a time. Buffering breaks this workaround.

You can have multiple proxies to an individual Nepenthes instance; simply set the X-Prefix header accordingly.

Installation

You can use Docker, or install manually.

A Dockerfile and compose.yaml is provided in the /docker directory. Simply tweak the configuration file to your preferences, 'docker compose up'. You will still need to bootstrap a Markov corpus if you enable the feature (see next section.)

For Manual installation, you'll need to install Lua (5.4 preferred), SQLite (if using Markov), and OpenSSL. The following Lua modules need to be installed - if they are all present in your package manager, use that; otherwise you will need to install Luarocks and use it to install the following:

cqueues
ossl (aka luaossl)
lpeg
lzlib (or lua-zlib, only one of the two needed)
dbi-sqlite3 (aka luadbi-sqlite3)
unix (aka lunix)

Create a nepenthes user (you REALLY don't want this running as root.) Let's assume the user's home directory is also your install directory.

useradd -m nepenthes

Unpack the tarball:

cd scratch/
tar -xvzf nepenthes-1.0.tar.gz
    cp -r nepenthes-1.0/* /home/nepenthes/

Tweak config.yml as you prefer (see below for documentation.) Then you're ready to start:

    su -l -u nepenthes /home/nepenthes/nepenthes /home/nepenthes/config.yml

Sending SIGTERM or SIGINT will shut the process down.

Bootstrapping the Markov Babbler

The Markov feature requires a trained corpus to babble from. One was intentionally omitted because, ideally, everyone's tarpits should look different to evade detection. Find a source of text in whatever language you prefer; there's lots of research corpuses out there, or possibly pull in some very long Wikipedia articles, maybe grab some books from Project Gutenberg, the Unix fortune file, it really doesn't matter at all. Be creative!

Training is accomplished by sending data to a POST endpoint. This only needs to be done once. Sending training data more than once cumulatively adds to the existing corpus, allowing you to mix different texts - or train in chunks.

Once you have your body of text, assuming it's called corpus.txt, in your working directory, and you're running with the default port:

curl -XPOST -d ./@corpus.txt -H'Content-type: text/plain' http://localhost:8893/train

This could take a very, VERY long time - possibly hours. curl may potentially time out. See load.sh in the nepenthes distribution for a script that incrementally loads training data.

The Markov module returns an empt string if there is no corpus. Thus, the tarpit will continue to function as a tarpit without a corpus loaded. The extra CPU consumed for this check is almost nothing.

Statistics

Want to see what prey you've caught? There are several statistics endpoints, all returning JSON. To see everything:

http://{http_host:http_port}/stats

To see user agent strings only:

http://{http_host:http_port}/stats/agents

Or IP addresses only: 3 http://{http_host:http_port}/stats/ips/

These can get quite big; so it's possible to filter both 'agents' and 'ips', simply add a minimum hit count to the URL. For example, to see a list of all IPs that have visted more than 100 times:

http://{http_host:http_port}/stats/ips/100

Simply curl the URLs, pipe into 'jq' to pretty-print as desired. Script away!

Nepenthes used Defensively

A link to a Nepenthes location from your site will flood out valid URLs within your site's domain name, making it unlikely the crawler will access real content.

In addition, the aggregated statistics will provide a list of IP addresses that are almost certainly crawlers and not real users. Use this list to create ACLs that block those IPs from reaching your content - either return 403, 404, or just block at the firewall level.

Integration with fail2ban or blocklistd (or similar) is a future possibility, allowing realtime reactions to crawlers, but not currently implemented.

Using Nepenthes defensively, it would be ideal to turn off the Markov module, and set both max_delay and min_delay to something large, as a way to conserve your CPU.

Nepenthes used Offensively

Let's say you've got horsepower and bandwidth to burn, and just want to see these AI models burn. Nepenthes has what you need:

Don't make any attempt to block crawlers with the IP stats. Put the delay times as low as you are comfortable with. Train a big Markov corpus and leave the Markov module enabled, set the maximum babble size to something big. In short, let them suck down as much bullshit as they have diskspace for and choke on it.

Configuration File

All possible directives in config.yaml:

http_host : sets the host that Nepenthes will listen on; default is localhost only.
http_port : sets the listening port number; default 8893
prefix: Prefix all generated links should be given. Can be overriden with the X-Prefix HTTP header. Defaults to nothing.
templates: Path to the template files. This should be the '/templates' directory inside your Nepenthes installation.
detach: If true, Nepenthes will fork into the background and redirect logging output to Syslog.
pidfile: Path to drop a pid file after daemonization. If empty, no pid file is created.
max_wait: Longest amount of delay to add to every request. Increase to slow down crawlers; too slow they might not come back.
min_wait: The smallest amount of delay to add to every request. A random value is chosen between max_wait and min_wait.
real_ip_header: Changes the name of the X-Forwarded-For header that communicates the actual client IP address for statistics gathering.
prefix_header: Changes the name of the X-Prefix header that overrides the prefix configuration variable.
forget_time: length of time, in seconds, that a given user-agent can go missing before being deleted from the statistics table.
forget_hits: A user-agent that generates more than this number of requests will not be deleted from the statistics table.
persist_stats: A path to write a JSON file to, that allows statistics to survive across crashes/restarts, etc
seed_file: Specifies location of persistent unique instance identifier. This allows two instances with the same corpus to have different looking tarpits.
words: path to a dictionary file, usually '/usr/share/dict/words', but could vary depending on your OS.
markov: Path to a SQLite database containing a Markov corpus. If not specified, the Markov feature is disabled.
markov_min: Minimum number of words to babble on a page.
markov_max: Maximum number of words to babble on a page. Very large values can cause serious CPU load.

History

Version numbers use a simple process: If the only changes are fully backwards compatible, the minor number changes. If the user/administrator needs to change anything after or part of the upgrade, the major number changes and the minor number resets to zero.

v1.0: Initial release

Stuff We Already Depleted

Thu, 16 Jan 2025 13:33:18 GMT

Score 5 | Comments 1 (thread link) | @juancroldan
January 16, 2025 | https://jcarlosroldan.com/post/352 | archive.org

One last thing!

Who are you? We'll save this info in your browser for the next time.

I Ditched the Algorithm for RSS–and You Should Too

Thu, 16 Jan 2025 12:18:32 GMT

Score 63 | Comments 24 (thread link) | @DearNarwhal
January 16, 2025 | https://joeyehand.com/blog/2025/01/15/i-ditched-the-algorithm-for-rssand-you-should-too/ | archive.org

I waste too much time scrolling through social media. It's bad for my health, so why do I keep doing it?

Because once in a while, I'll find a post so good that it teaches me something I never knew before, and all the scrolling feels worth it. But I've stumbled upon an old piece of free and open source tech, relatively unknown today, which is THE solution of solving the problems with modern media without sacrificing accessible, good content: RSS.

Reddit, Facebook, Twitter â€” platforms built for engagement, not efficiency. Instead of showing you high-quality posts upfront, they pad your feed with memes, spam, and astroturfing. There is only so much 'good' content created in a day. By padding your feed with trash, they make the limited amount of good posts "last longer". These sites want you to spend more time scrolling on their website, so they feed you scraps which makes the occasional great post feels like a jackpot.
This concept, operant conditioning, was developed by B.F. Skinner â€” Yes, the mind behind the Skinnerbox.

While some sites offer filtering or sorting options, manually settings these options every time you want to access a subreddit is just not doable.

You could, of course, stop consuming content from these websites. However, this would mean potentially missing really good content; content you'd learn from, interesting ideas, and more.
But it doesn't have to be this way. You can reclaim your attention span while still having access to the same quality content as before.

Enter:

RSS is like your youtube subscription feed in hyperdrive. Subscribe to sites you love and decide what shows up â€” no exploitative social media algorithm needed. No more ads or algorithms deciding how to keep you doomscrolling. This 1999 tech actually solves a lot of 2025 problems.
Here's the kicker: Most websites, even social media, quietly support RSS feeds.

You can filter out keywords, set minimum upvotes or like counts, and much more! Modern RSS clients allow you to make filters using Regex, and there are a lot of software and services you can use to tune up your filtering to 11.

TL;DR: Never see noise, and never miss hidden gems again!

But how do you get started with RSS? It's easier than you think!

Setup¶

I personally self-host an open source RSS reader: Tiny Tiny RSS
If you don't want to host it yourself, you can google for companies offering easy and accessible RSS readers.

To make it easier, let's differentiate between three levels of ease when it comes to adding a website to RSS: Easy, medium, and hard.
I'll be going over how to add several popular sites to your feed.

Easy 1: Youtube¶

Want a youtube channel in your RSS feed? Just copy the channel's URL and subscribe to it in your reader. Done.

Easy 2: IGN¶

If you like games, you might want to subscribe to IGN. There's no clear RSS button, so the best course of action would be to google "IGN RSS".

This leads to a nice IGN RSS Feeds page with multiple categorized feeds for you to pick from. If you wanted to subscribe to "Game Articles", you'd right-click on the game articles link, press "copy link", go to your RSS reader of choice and subscribe to the link you copied.

Now all IGN Game Articles will show up in your RSS feed as they are published!

Tip

Some websites don't have a dedicated RSS button, but still support RSS. You can discover their RSS urls by adding .rss, atom.xml, feed, etc. at the end of the site's URL, for example https://website.com/atom.xml. Almost all RSS readers support Atom feeds. For more examples, check this Reddit comment.

Medium 1: HackerNews¶

Some sites like HackerNews have RSS support. However, this RSS can be extremely limited if you want to filter posts so your feed isn't spammed by low effort content. Some people are nice enough to set up a "middleman" between your RSS feed and the website, so you can pull the RSS feed through the middleman while doing actions like filtering it.

For example, if you wanted to subscribe to HackerNews but filter out low upvote count posts, you could subscribe to HNRSS instead of through HN directly. For example, I filter out posts below 150 upvotes by to this url: https://hnrss.org/newest?points=150
Sometimes these services open-source their code so you can self-host the 'middleman'.

Medium 2: Reddit¶

Warning

When removing the optional search term from a reddit search URL, don't forget to remove the + When removing the sort options If adding more search terms, add a + between them!

I love managing my homelab. I follow /r/homelab. Some posts are really good and teach me a lot.
However, there's a lot of noise posted to that subreddit; I do not want to see memes, and pictures of people's hardware setup gets boring quick. I'm interested in hidden gems, threads where a lot of interesting info is explained, things I can really learn a lot from.

Step 1: Filter out picture posts.¶

Reddit hack: Filter out picture posts by searching for 'self:true' in a subreddit. Bonus: You can subscribe to that specific search query as a RSS feed for text posts only.

So instead of subscribing to a subreddit's RSS directly, you do a search for posts in that subreddit and then subscribe to that RSS feed.
The RSS link you should subscribe to should look someting like this: https://www.reddit.com/r/homelab/search.rss?q=self%3Atrue&restrict_sr=on
You can change 'homelab' to your subreddit of choice.

Note

The restrict_sr=on parameter in the URL (probably) means "Restrict_subreddit". Removing this from the search will yield results from different subreddits than the one you're searching in. If you think that parameter is redundant, I agree.

There are a lot of text-only submissions on /r/homelab. Gems are relatively sparce. Lots of low quality content. It's not the subreddit's fault; this is standard across Reddit.

Step 2: Filter for quality¶

Seems easy; let's add a 'minimal upvotes' query to the search, right?
Sadly, Reddit doesn't support that.. a shame, really.
However, a workaround is sorting by 'Top' and asking the search to show us the 'top posts of this week'. Note: 'This week' would mean 'past 7 days' instead of 'posted this week'.

Filtering by 'Top of ...' always returns 25 items. This means that if you sort by 'top of this week', on average of 25/7=3,57 NEW posts get added to your feed each day! This is a great way to only see the highest scoring posts of each day.

Adding this sort on top of the RSS feed from step 1 results in an URL like this: https://www.reddit.com/r/homelab/search.rss?q=self%3Atrue&restrict_sr=on&sort=top&t=year

Bug

If you don't care about only seeing text posts, removing self%3Atrue does NOT work for RSS feeds, even though it does work for direct searches. Instead, subscribe to the subreddit's "top" RSS and filter by time. For example: https://www.reddit.com/r/homelab/top.rss?t=month

For reference, here is how many posts you would get in your RSS feed, depening on your reddit sorting:

And just like that, we converted a high-noise subreddit to an RSS feed which only gives us the best the subreddit has to offer.

Tip

If you wanted to subscribe to all new posts in a subreddit, you would subscribe to an url like https://www.reddit.com/r/SUBREDDIT_NAME/new/.rss?sort=new. For a more extensive Reddit RSS guide, see this post

Hard:¶

Some sites might not have support for an RSS feed. Sometimes you can get away with a neat google trick:

Most of the time you'd need something to generate the RSS for you. You could use one of many RSS feed generators available online, or host one yourself. Most of these feed generators have enhanced filtering tools as well.

I haven't had to do this yet, however I've heard really good things about the open source RSS-Bridge

How you'd set up a feed generator depends on the software, so I won't expand upon that here.

Conclusion¶

Separating yourself from the algorithmic whims of social media platform is easier than ever. With RSS, you can stay informed, save time, and never miss the content that truly matters.

This blog also has an RSS feed!

To end this post, here is a list of (RSS supported) sites I think are really interesting. Linked are excellent articles for first-time readers!

Nicky Case's blog & New Stuff - Really cool interactive explanations!
Julia Evans - In-depth tech blog
Kevin Fang - In-depth incident analysis
Explaining Software Design
Hacker News - General tech news feed

Strategies to Complete Tasks with ADHD

Thu, 16 Jan 2025 12:02:47 GMT

Score 8 | Comments 2 (thread link) | @adhs
January 16, 2025 | https://schroedermelanie.com/adhs-nichts-zuende-bringen/ | archive.org

Menschen mit ADHS (Aufmerksamkeitsdefizit-/Hyperaktivitätsstörung) erleben den Alltag oft als ständiges Auf und Ab von Konzentration, Emotionen und Energie. Dieses innere Chaos führt nicht nur dazu, dass Aufgaben häufig abgebrochen werden, sondern auch, dass sich Betroffene schnell überfordert fühlen. Um zu verstehen, warum das so ist, lohnt es sich, einen Blick auf die emotionale Ebene und die Funktionsweise des Nervensystems bei ADHS zu werfen.

Menschen mit ADHS starten meist mit großen Vorsätzen in den Tag, doch diese verschwinden schnell im Chaos der Ablenkungen. Jeder neue Impuls scheint so wichtig, dass du ihn nicht ignorieren kannst und so in einen Strudel von Ablenkungen gerätst.

Die neurologischen Besonderheiten bei ADHS sind eng mit den beschriebenen emotionalen Herausforderungen verknüpft. Das Gehirn von Menschen mit ADHS funktioniert anders, insbesondere in den Bereichen, die für Fokus, Impulskontrolle und die Regulation von Emotionen zuständig sind.

Die Schwierigkeiten von Menschen mit ADHS, an Aufgaben dranzubleiben und sich nicht überfordert zu fühlen, sind kein Zeichen von Faulheit oder mangelndem Willen. Vielmehr liegen die Ursachen in der besonderen Funktionsweise ihres Nervensystems und der intensiven emotionalen Verarbeitung. Ein besseres Verständnis für diese Dynamik – sowohl von Betroffenen als auch von ihrem Umfeld – kann den Umgang mit diesen Herausforderungen erheblich erleichtern.

National IQs Are Valid

Thu, 16 Jan 2025 11:18:16 GMT

Score 7 | Comments 0 (thread link) | @noch
January 16, 2025 | https://www.cremieux.xyz/p/national-iqs-are-valid | archive.org

If you follow me here or on Twitter/X, I’m sure you’ve seen a map like this, showing country-level differences in average IQs:

The figures in this map are derived from a raft of studies compiled by Richard Lynn and Tatu Vanhanen in their 2002 book IQ and the Wealth of Nations. The book itself is little more than a compilation and discussion of these studies, all of which are IQ estimates from samples located in different countries or based on diasporas (e.g., refugees) from those countries.

To get this out of the way, the estimates from IQ and the Wealth of Nations hold up. They are replicable and they are meaningful. At the same time, they are contentious. Lynn and Vanhanen’s estimates have many detractors, but virtually all of the negative arguments have one thing in common: they’re based on the idea that the estimates feel wrong, rather than with any actual inaccuracies with them.

Let’s review.

One of the most common arguments against Lynn and Vanhanen’s national IQ estimates is that it is simply impossible for whole countries’ mean IQs to be what people often consider to be so low that they’re considered prima facie evidence of mental retardation. This feeling is based on misconceptions about how mental retardation is diagnosed and defined, and misunderstandings about the meanings of very low IQs across populations. Let’s tackle definition first.

The belief that mental retardation is defined by an IQ threshold is similar to many of the arguments against the validity of national IQs in that it is based on a failure to give even a cursory thought to one’s own arguments. It’s a belief that cannot survive reading the latest version of the DSM, or for that matter, thinking of psychologists as competent people. If you open up the DSM-5 and turn to the section entitled Intellectual Disabilities, the first thing you see is the diagnostic criteria, which read as follows:

The following three criteria must be met:
Deficits in intellectual functions, such as reasoning, problem solving, planning, abstract thinking, judgment, academic learning, and learning from experience, confirmed by both clinical assessment and individualized, standardized intelligence testing.
Deficits in adaptive functioning that result in failure to meet developmental and sociocultural standards for personal independence and social responsibility. Without ongoing support, the adaptive deficits limit functioning in one or more activities of daily life, such as communication, social participation, and independent living, across multiple environments, such as home, school, work, and community.
Onset of intellectual and adaptive deficits during the developmental period.

There are four listed severity levels for mental retardation, Mild, Moderate, Severe, and Profound, and they are “defined on the basis of adaptive functioning, and not IQ scores, because it is adaptive functioning that determines the level of supports required. Moreover, IQ measures are less valid in the lower end of the IQ range.” It’s true that most mentally retarded people have IQs in the range of 55 to 70, so it’s easy to get misled into thinking that IQ is the defining factor for mental retardation. But an IQ of about 70 and below only indicates (but doesn’t diagnose) mental retardation because of what it tends to be caused by in certain populations. This is a roundabout way of saying that a low IQ indicates mental retardation because of what it means for a person's behavior, which, I'll explain, covaries with its causes.

IQs are mostly normally distributed, and IQs represent the influences of multiple different constructs and causes, but they primarily reflect differences in general intelligence. At the same time, there’s a major deviation from normality at the lower end of the scale. If you sample well enough, the picture you’d get would look like this:

The reason is that there are the normal-range causes that produce the traditional bell curve, and then there are extreme circumstances that produce extraordinarily low IQs. Contrarily, we don't know of anything that produces abnormally high IQs. There's no known one-off mutation that makes someone a genius, but there are several mutations that we now know can make a person extremely unintelligent. Consider these:

Young and Martin 2023, Figure 1

It's much easier to break a machine than to throw a wrench in it and make it work better. And that makes sense! Your wrench in the gears is likely to break something, not to increase efficiency. If you hit someone in the head hard enough, you can reduce their IQ score, but not in a million years will you turn them into von Neumann.

The reason someone's IQ drops after you bludgeon them is more singular and specific than the reasons IQ varies in the general population. A hit to the noggin may leave someone unable to flex their short-term memory, even if their visuospatial rotation capabilities are unaffected. There's some degree of isolation of function and compensation for deficits in the brain. We know this thanks to many observations, like on the effects of neural lesions.

When we say that someone with an IQ of ≤70 is mentally retarded, we're saying that they lack adaptive behavior—that they're not very bright, and so it makes living life hard. When someone with an IQ >70 is mentally retarded—which happens!—we’re saying the same thing. But if a population legitimately has a mean IQ of 70 (and some do), we'll notice that they're not drooling troglodytes who can't put on their shoes. This is because the reasons for their low IQ are not things that cause specific and extreme deficits, but instead, things that cause normal-range variation, which is far less severe in nature than something that causes massive, specific, discontinuously-caused deficits.

Supposing discontinuous causes that create major, specific deficits yields testable consequences. We can see them play out clearly by leveraging different countries’ population registers.

In large Israeli (B) and Swedish (A) datasets, we can see that when a person has a mild intellectual disability, their sibling tends to be less intelligent too. You also see this for heights, autism, or any other highly polygenic trait. This is expected with continuously distributed causes because siblings share portions of their genetic endowments.

But when a sibling has a severe—otherwise known as, "idiopathic"1—intellectual disability that's likely to have a discontinuous cause, the distribution of their siblings’ IQs is the same as the distribution for the general population. And of course they would, because the reason for the difference isn’t something that siblings should be expected to partially share.

The deficits in adaptive behavior that result from idiopathic intellectual disability are ones that make life hard to live in discrete and extreme ways. In some cases, you wouldn't even recognize them as being "retarded" because they have conditions like amnesia, where the person is clearly odd and unfortunate, and they score poorly on an IQ test, but they might sound totally coherent otherwise.

Normal-range causes lead to linear changes in adaptive behavior; idiopathic mental retardation, leads to a discontinuous decrease in adaptive behavior.

Accordingly, an IQ of 70 does not have the same meaning for members of different groups, since if a group with a mean IQ of 70 randomly pulls a person with an IQ of 70, they aren't likely to score like someone who's mentally retarded for idiopathic reasons. Arthur Jensen, incidentally, predicted this sort of result in his 1972 Genetics and Education based on some observations he made as an educator.

My student said he was looking for a good culture-free or culture-fair test of intelligence and had not been able to find one. All the tests he used, whether they were claimed to be culture-fair or not, were in considerable agreement with respect to children diagnosed as educationally mentally retarded (EMR), by which they were assigned to special small classes offering a different instructional program from that in the regular classes. To qualify for this special treatment, children had to have IQs below 75 as well as lagging far behind their age-mates in scholastic performance. My student, who had examined many of these backward pupils himself, had gained the impression that the tests were quite valid in their assessments of white middle-class children but not of minority lower-class children.
Many of the latter, despite IQs below 75 and markedly poor scholastic performance, did not seem nearly as retarded as the white middle-class children with comparable IQs and scholastic records. Middle-class white children with IQs in the EMR range generally appeared more retarded than the minority children who were in special classes. Using nonverbal rather than verbal tests did not appreciably alter the problem. I confirmed my students observations for myself by observing EMR children in their classes and on the playground and by discussing their characteristics with a number of teachers and school psychologists. My student’s observations proved reliable.
EMR children who were called ‘culturally disadvantaged’, as contrasted with middle-class EMR children, appeared much brighter socially and on the playground, often being quite indistinguishable in every way from children of normal IQ except in their scholastic performance and in their scores on a variety of standard IQ tests. Middle-class white children diagnosed as EMR, on the other hand, though they constituted a much smaller percentage of the EMR classes, usually appeared to be more mentally retarded all round and not just in their performance in scholastic subjects and IQ tests. I asked myself, how could one devise a testing procedure that would reveal this distinction so that it could be brought under closer study and not depend upon casual observations and impressions.

The distinction between types of mental retardation that are demarcated by their symptoms and, thus, by their causes, is important to understand. It's why an IQ of 70 for a Japanese person is likely to indicate an extraordinarily severe issue in need of attention, while for a Bushman, they won't have any trouble surviving. This is why the DSM-V notes in the Diagnostic Features section and bolded here: “The essential features of intellectual disability are deficits in general mental abilities (Criterion 1) and impairment in everyday adaptive functioning, in comparison to an individual’s age-, gender-, and socioculturally matched peers (Criterion 2).”

Keep that bolded part in mind and you’ll understand the importance of test norming and, knowing about discontinuous causes and the adaptive functioning deficit requirement for retardation diagnosis, you’ll never again be able to think that a mean IQ around 70 invalidates national IQs. But there’s another reason you shouldn’t, and it’s that you’re probably not an ultra-hereditarian.

In 2010, Richard Lynn pointed out that high IQs in Sub-Saharan Africa were incompatible with anything but the judgment that the group differences in the U.S. were entirely genetic in origin and even extremely poor environments have no effects on cognitive development. In response to an attempt to say that Sub-Saharan Africans had a mean IQ closer to 80 than 70, he wrote:

[The] assumption that [some of these samples of] children had IQs of 85 and 88 seems improbable. These are the IQs of blacks in the United States. It can hardly be possible that blacks in the United States who have all the advantages of living in an economically developed country, with high income, good health care, good nutrition and education, would have the same IQ as blacks in impoverished Nigeria. If this were so, we would have to infer that these environmental disadvantages have no effect whatever on IQs and even the most hard line hereditarians would not go that far.

Or, if you want to see this point made diagrammatically (courtesy of X user AnechoicMedia), here you are:

Low IQs are also predictable from national development, making them that much more realistic. Using the latest national IQ dataset, I’ll show this for Sub-Saharan Africa—a region often claimed to have invalid IQs precisely because they’re ‘too low’.

First, we’ll predict Sub-Saharan African national IQ from a regression of log(GDP PPP Per Capita) on national IQ estimates. Whether the regression is performed with or without Sub-Saharan Africa, the results are similar. The measured mean IQ of Sub-Saharan Africa is 71.96, the predicted IQ in the regression with Sub-Saharan Africa included is 74.86, and without them, it’s 76.78. Or in other words, it’s not very different.

Without the logs, the predicted IQs are 78.29 and 82.47, but that’s not an appropriate model, as you’ll see in a moment. But first, you might contest the latest national IQ dataset. The studies underlying its estimates and the methodology to assemble them are well-documented, so if you really want to contest them, you should start there. But if you dismiss them out of hand, you still can’t escape prediction by development, because we can just use the World Bank’s Harmonized Learning Outcomes (HLOs), an alternative national IQ dataset created by qualified researchers (including Noam Angrist), with good methods (test-score-linking), recently published (2021) in a respectable journal (Nature).2

Using HLOs, the observed mean IQ of Sub-Saharan Africa turns out to be 71.05. The predicted mean IQ with Sub-Saharan Africa in the regression is 72.25, and without it, 72.55. Without logs, those predictions become 77.35 and 81.85. But again, that model isn’t appropriate. The reason it’s not appropriate is due to nonlinearities that logging helps to handle. Take a look:

You can see this same nonlinearity elsewhere, such as in PISA scores:

So to overcome this issue, we can do a simple piecewise regression. Using a GDP PPP Per Capita of $50,000 as the cut-point—and feel free to go use whatever you want, it doesn’t really change the result so long as it’s reasonable—we get these regressions:

With this method, the predicted mean IQ of Sub-Saharan Africa is 76.12 with it included and 79.77 without. Using the World Bank’s HLOs instead, the results are 73.75 with Sub-Saharan Africa and 76.6 without it.

All of these estimates are pretty close to the ground-truth, and I suspect they would be even closer if all the data came from the same years instead of near years, and I know it’s closer if Actual Individual Consumption is used instead of GDP Per Capita, since that tends to iron out some of the issues with tax havens and oil barons. But regardless, it should now be apparent that low national IQs are

Not indications of widespread, debilitating mental retardation
Where they’re predicted to be given levels of national development
Thus not a reason to cast aside national IQ estimates

In his earlier national IQ datasets, Lynn ran into a lot of missing data. To get around this issue, he exploited the fact that there’s spatial autocorrelation in order to provide imputed national IQs. Some people think, however, that these imputed IQs are bad and make his data fake, failing to realize that imputation is normal and it doesn’t have meaningful effects on Lynn’s national IQ estimates.

In the most recent dataset Lynn created before he died, his imputation procedure was like so:

To calculate these imputations, Lynn and Becker (2019a, b) took advantage of the spatial autocorrelation that often exists in international data and identified the three countries with the longest land borders that had IQ means. A mean IQ, weighted by the length of the land border, was calculated and used as an estimate for the country’s missing estimated mean IQ. For island nations, the three closest countries with IQ data were identified, and an unweighted mean was calculated and imputed as an estimated mean IQ value for the missing country’s data.

Geographic imputation of this sort is the responsible thing to do when data is not missing at random and it can be predicted from other values in the dataset, because more data means more power and, given the nature of the missingness, less bias. So whether Lynn was being responsible or irresponsible is a matter of how well the imputations hold up. Thankfully, they do hold up.

The simplest way to check the robustness of Lynn’s national IQs is to compare his imputed national IQs to subsequently sampled national IQs. I’ll do this with his much maligned 2002 and 2012 national IQs. To assess the validity of the imputed IQs I’ll correlate them with our current best national IQs and the World Bank HLOs.

Lynn’s 2002 imputed national IQs correlate at r = 0.90 with our current best national IQs and 0.72 with HLOs. 102 countries were imputed. Compared to our current best national IQs, 72 were overestimates and the average estimation error was 1.47 points upwards. The average overestimation on Lynn’s part was 3.71 points, and the average underestimation was 3.75 points. Since these were imputed countries, they don’t really reveal anything about Lynn’s estimation biases in general, as they’re a select subset of generally poor or small countries. Lynn’s 2012 imputed national IQs correlate at r = 0.92 with our current best national IQs and 0.76 with HLOs. 66 countries were imputed. Compared to our current best national IQs, 37 were overestimates and the average estimation error was 0.72 points upwards. The average overestimation on Lynn’s part was 3.48 points, and the average underestimation was 2.61 points.3

I want to mention again that it should not be surprising that this procedure doesn’t really do much. It’s simple, straightforward, and scientifically uncontroversial, plus it’s theoretically sound, so it’s not shocking that it works. In a related domain, I found that this worked out just fine in the U.S.

I took the NAEP Black-White score gaps from the lower-48 and predicted them for each state from the observed gap for their immediate neighbors. The Michigan-Minnesota and New York-Rhode Island water borders were counted as neighboring state borders, and the result of imputing across state borders was a correlation of about 0.60 with the real gaps:

With a larger sample size and even more so with imputation just of observations that are missing for theoretically important reasons, this would almost certainly work better, but regardless, it’s fine: Imputation just works.

This is easy to check, so you have to wonder why people believe it. If it were true, it would be easy to show rather than to merely assert.4 To check this, just compare Lynn’s estimated national IQs to if independent collections of national IQ estimates and see if they vary systematically and meaningfully. I’ve done this, with both our current best national IQs as the reference and using World Bank HLOs as reference. The results are practically the same, but this should be unsurprising, since Lynn computed sets of national IQs based on IQ tests and based on achievement tests, and they were highly aligned. But I digress. Here’s what I found using the current best national IQs:

Compared to our current best estimates, Lynn’s original estimates were pretty close to the line, with a mix of under- and over-estimation. The degree of under- and over-estimation is minor, ranging between underestimating Sub-Saharan Africa by 1.89 points and overestimating Latin America by 4.21 points. Europe was overestimated by just 1.01 points, indicating no real evidence for the theory that Lynn favored Whites. If we drop imputed numbers, then we can see this even more clearly, because Sub-Saharan Africa without imputation was actually underestimated by an even larger margin of 2.40 points, but Europe without imputation was only overestimated by 0.18 points.

Fast forward to Lynn’s 2012 dataset and the results are even tighter, and the underestimation of Sub-Saharan Africa drops to 0.49 points, while the overestimation of Europe follows in lock-step, falling to 0.04 points. Without imputation, these numbers become 0.97 points of underestimation and, curiously, Europe is actually underestimated by 0.15 points.

In addition to producing replicable estimates, the estimates Lynn produced also weren’t off by much in general. I don’t think this should be surprising, but some people have responded to this with the argument that…

This is a response to the above that is, simply put, a complete lie. The idea here is that the above validation of Lynn’s national IQ estimates is based on looking only at permutations of Lynn’s national IQ estimates, derived largely from the same sets of studies. But there is only one way to arrive at that view, and it’s to simply assume it’s true without looking at the data to see that it’s clearly not.

The wonderful thing about Lynn’s data is that he documents all of his decisions and you can peruse his national IQ database, cutting out studies you don’t like and adjusting all the estimates accordingly. Reasonable changes to Lynn’s adjustments and inclusions don’t make a difference. The data is publicly available, so you can go and confirm that yourself. But an even simpler way to show that Lynn’s estimates hold up is to just see if they hold up in data that’s entirely non-overlapping with Lynn’s data. So for that, I’ll use the World Bank’s HLOs:

The World Bank HLOs correlate at 0.83 with Lynn’s estimates with imputation and 0.89 without imputation. Compared to HLOs, Lynn underestimated Sub-Saharan Africa by a piddling 5.03 points, but he also underestimated Europeans by 0.61 points, and overestimated Oceania, Latin America, and South Asia by 5.71, 2.74, and 5.89 points, respectively.

This is just not consistent with a pattern of strong, European-favoring misestimation, and if we drop Lynn’s imputations that becomes even clearer because the underestimation of Sub-Saharan Africa increases to 5.44 points, while the overestimation of Oceania, Latin America, and South Asia shift to 2.68, 1.97, and 1.92 points. And if we look at Lynn’s later 2012 national IQ estimates, the patterns just aren’t that different, a fact that’s true whether we subset HLOs to be derived completely beyond the dates of Lynn’s data or not.

So to summarize, unless everyone shares Lynn’s biases, then his national IQ estimates do not suggest he was biased in favor of Europeans. The impacts of his imputation methods also suggest that he wasn’t biased in that direction either.

A major argument that Lynn was in the wrong is that some of his samples were disadvantaged, underprivileged, or whatever other euphemisms you might wish to use for being poor, and that perhaps his samples from poor countries were worse in this respect, because he wanted them to look worse. This doesn’t stand up to scrutiny.

Lynn always caveated national IQs based on limited samples. Though you won’t see that if you just look at the resulting national IQ maps, you definitely see that if you look in his books describing what data was used, how it was gathered, and so on. It is true that some of Lynn’s referenced samples were very poor, but this always has to be considered in the light of representativeness. If poverty is the norm for a region, then samples should be poor to be representative. Similarly, if a health condition like anemia is the norm for a region—and in many places, it actually is!—then the best sample will have high rates of anemia, regardless of whether that’s ‘acceptable’ for a sample from a developed Western country.5

We can think about this in the context of a few sample means from poor places which we know to be psychometrically unbiased when compared with U.S. samples. In this case, I’ll reference samples Russell Warne provided data on from Kenya and Ghana.

The samples were from 1997 and 2015 and they used the WISC-III in Kenya and WAIS-IV in Ghana. The Kenyan test-takers came from grade 8 schools and the Ghanaian ones were public and private high schoolers from Accra, alongside a sample of university students.

The problem with this sampling is that education in Kenya in 1997 and Ghana in 2015 is nowhere near this level. In Kenya, most people didn't even reach grade 8 at the time, and in Ghana, most people don't reach university but university students were 57% of the sample! Furthermore, the Kenyan sample was from Nairobi, so it was at least sampled in what was, as of 2018, Kenya's 2nd-richest county and home to <9% of the population. The situation was even less representative in Ghana, where the sample was from Accra and thus also <9% of the population and a third of Ghana's GDP.

These samples were highly unrepresentative of their respective populations because they were relatively elite. The bias was not as bad in the Kenyan sample: they were one to two SDs above the country as a whole in socioeconomic status. But in Ghana, they were three to four SDs above the average. That's like comparing high school dropouts to college graduates.

Warne knew of this problem and wrote:

As eighth-graders, the members of this sample were more educated than the average Kenyan. In 1997, the average Kenyan adult had 4.7 years of formal schooling; by 2019, this average had increased to 6.6 years.

And:

Like the [Kenyan] sample, this [Ghanaian] sample is much more educated than the average person in the country. In 2015, the average Ghanaian adult had 7.8 years of schooling (which increased to 8.3 years by 2019), whereas 61% of [this sample] had 16 years of education or more.

Knowing this, mentally adjust their scores downward in accordance with how much you think being highly educated and relatively wealthy should overstate their scores relative to the general population. So, what were their scores? Here:

To put these scores into intuitive terms, we need to do two things: the factor correlation matrix and an assumption of equal latent variances. The correlation matrices were provided in the paper's figures 1 through 3. If we assume that the scale we want is based on a mean of 100 and an SD of 15 ("the IQ metric"), we just place the gaps on that scale (i.e., *15 + 100), subtract the number of factors times 100, divide that by the square root of the sum of the elements in the correlation matrix (we'll use the American values, since their sample was larger), and add back 100.

The Kenyan sample, which was 1-2 SDs above the Kenyan average in terms of education, had a mean IQ score of 79.08 versus the American average of 100. The Ghanaian sample, which was 3-4 SDs above the Ghanaian average in terms of education, had a mean IQ of 92.32 versus the American average of 100. And do note, the American average the Ghanaian sample would be compared to would be lower than the average the Kenyan sample was compared to due to demographic change over time.

These estimates are much higher than Lynn’s, and that makes sense, because these are clearly socioeconomically extremely well-off samples, perhaps not by Western standards, but certainly by their own national standards. But some people think these are the sorts of samples that should be used to represent poor countries. That’s wrong, but the view has its supporters, like Wicherts, Dolan, Carlson and van der Maas, who claimed as much in 2010. But Lynn called this out at the time too:

Wicherts, Dolan, Carlson & van der Maas (WDCM) (2010) contend that the average IQ in sub-Saharan Africa assessed by the Progressive Matrices is 78 in relation to a British mean of 100, Flynn effect corrected to 77, and reduced further to 76 to adjust for around 20% of Africans who do not attend school and are credited with an IQ of 71. This estimate is higher than the average of 67 proposed by Lynn and Vanhanen (2002, 2006) and Lynn (2006).
The crucial issues in estimating the average IQ in sub-Saharan Africa concern the selection of studies of acceptable representative samples, and the adjustment of IQs obtained from unrepresentative samples to make them approximately representative. Many samples have been drawn from schools but these are a problem because significant numbers of children in sub-Saharan Africa have not attended schools during the last sixty years or so, and those who attend schools have higher average IQs than those who do not.

Lynn then reviewed general population studies and those returned a Sub-Saharan African mean IQ in the 60s. He subsequently reviewed primary school studies and the data yielded a median of 71, which he adjusted to 69 to account for the fact that only about 80% of the population gets a primary school education in Sub-Saharan Africa. After that, he reviewed studies of secondary school students, and those returned a mean IQ in the 70s, but that is not an acceptable estimate for a few simple reasons.

(1) many adolescents in sub-Saharan Africa have not attended secondary school and tertiary institutes. For instance, Notcutt estimated that in South Africa in 1950 only about 25% of children aged 7-17 were in schools and “we cannot assume that those who are in school are a representative sample of the population”. (2) Entry to secondary school has generally been by competitive examination, resulting in those with higher IQs being selected for admission. Thus, “entry to secondary schools in the East African countries of Kenya, Uganda, and Tanzania is competitive… approximately 25 percent of the population complete the seven standards of primary school and there are secondary places for 10-12 percent of these”. Similarly, Silvey writing of Uganda around 1970 stated that at this time only 2% of children were admitted to secondary schools and entry is determined partly by a primary school leaving examination, and Heynman and Jamison writing of Uganda in 1972, note that admission to secondary school is based on “achievement performance on the academic selection examination… and there are secondary school places for only one child in 10”.

This is egregious, but this is typical for higher estimates for poor places. They tend to be based on samples that are, as in these cases, relatively privileged, pre-screened for IQ, etc., and despite that still generally not that impressive in socioeconomic or IQ terms. These also aren’t the worst estimates like this that Lynn has noted. My favorite was this:

WDCM include a number of studies that cannot be accepted for a variety of reasons. Their samples of university students are clearly unrepresentative. The Crawford-Nutt sample consisted of high school students (IQ 84) in math classes admission to which “is dependent on the degree of excellence of the pupil’s performance in the lower classes” and described as “a select segment of the population”. The students were also coached on how to do the test and “Teaching the strategies required to solve Matrix problems yields dramatic short-term gains in score”. This is clearly an unrepresentative sample.

Curiously, people usually ‘get this’ sort of representativeness issue when it comes to things like China only sampling from rich, well-off areas to look good in international assessments like PISA and TIMSS, but they don’t get this when it applies to samples that look rich by one country’s standards and poor by the standards of the developed world. It’s precisely those sorts of large-scale examinations which bring me to the easiest way to vindicate Lynn’s sampling in a very general sense.

Large-scale international examinations like PISA have sampling frames, requirements to be met for a sample’s scores to be considered valid and comparable to those of other countries, and if countries meet them, then it’s likely their samples were sufficiently representative to make a statement about a country’s youth or, in the case of assessments like PIAAC, its adults. These large-scale assessments have standards for sampling, and they also have psychometric standards. Their samples end up representative—at least of developed countries (see above in this section)—and their test scores aren’t biased, and these still correlate highly with the results produced by Lynn and his colleagues. This is the basis for the World Bank’s HLOs, so by this point in this article, you’ve already seen this point made multiple times! And to be completely fair to Lynn, he made this point too, people just neglect that he did. It’s no doubt part of why he computed his own academic achievement test-based national IQs (see Footnote 3).

Some people probably still think that some of Lynn’s estimates are too low. Lynn believed the same thing in cases where there just wasn’t enough data, which is why he Winsorized national IQ estimates and expressed his doubts about extremely low estimates.

recently noted that sub-60 IQs also aren’t empirically supported. Through reviewing additional evidence on countries listed as having sub-60 IQs, he found that each one ended up with either no estimate (due to missingness) or an IQ above 60.

The unstated part of the argument that some national IQ estimates are unrealistically low is that somehow this disqualifies larger portions of the dataset, but that’s a non sequitur.

Some people have claimed that Lynn made poor countries appear to perform worse than they actually do by using samples of children instead of adults. This criticism reveals a lack of awareness that test scores have age-specific norms. But if we assume it’s a legitimate criticism, then it suggests Lynn unfairly advantaged poor countries since the existing not-so-strong evidence shows that those countries are more likely to have scores that decline relatively with age.

During Lynn’s life, he didn’t focus much on psychometric bias. This is fair, because by the time it became a big focus for the field, he was already old. But in any case, as I’ve noted above, we have little reason to think it’s a big concern for most estimates. Furthermore, it’s not even clear that bias is systematic in general. Consider, for example, the comparison of Britons and South Africans I discussed here. In that comparison, there was bias, but it favored the lower-scoring South African group!

In general, when people find representative samples and no psychometric bias, the results aren’t different from what Lynn found, so unless someone wants to substantiate this concern, it’s little more than a waste of ink.6

The Flynn effect is widely misunderstood. I've written an article on this that goes into much greater depth about what I mean. But the important point when it comes to national IQs is that the Flynn effect is not about differences in intelligence, instead, it primarily concerns test bias. The existence of the Flynn effect also doesn’t imply there will be convergence between countries, it cannot be said to be the source of any convergence across countries without evidence that doesn’t currently exist, and the Flynn effect is explicitly adjusted for in national IQ computation. It just doesn’t have any relevance to the discussion because the evidence for larger Flynn effects during catch-up economic growth is extraordinarily poor.

But furthermore, increases in scores for cohorts over time will sometimes reflect bias rather than ability gains and relative cognitive performance across countries is generally very stable. There’s not really a reason to consider this argument, even for countries proposed—but not shown—to have major upward swings in their national IQ, like Ireland:

We know that IQ differences are partially causally explained by differences in brain size. Because development seemingly minimally impacts brain sizes—and it theoretically should not anyway—, brain sizes can be used successfully to instrument for national IQs, allowing us to estimate the causal impact of national IQs on outcomes like growth, crime rates, and so on. This has been done and it works well. The same result also turns up using ancestry-adjusted UVR and numeracy measured in the 19th century, both of which also can’t be caused by modern development.

These results suggest that if Lynn is getting the causality backwards from IQ to measures of national economic success, he’s still dominantly correct that national IQs precede development. Not only that, but national IQs are, as mentioned, largely stable over time, despite the world experiencing a lot of development. We can see this very directly using the World Bank’s HLOs again. The paper introducing them includes this diagram, showing (a) percentages enrolled in primary education over time, and (b) HLOs over the same period. Notice the dramatic increase in the former and relative stability of the latter.7

Since we do know there’s a considerable degree of stability in measured national IQs, we can leverage stability as an assumption and see the curious result that, over time, it looks like Lynn’s estimates are vindicated more and more, because development measures have gotten more in line with his national IQs!

Ultimately, people who want to argue Lynn got causality backwards aren’t really taking issue with the national IQ estimates themselves, they’re just specifying an alternative claim that they hope sounds like it can invalidate Lynn’s estimates. But—and here’s the kicker—Lynn firmly believed that national IQs would increase with development; his estimates were point-in-time estimates, not the final letter forever and ever, and he fully expected them to change because he thought very poor places were environmentally disadvantaged. So this isn’t even really an argument against Lynn’s general views per se.

People often reply to national IQ estimates with news articles or misinterpreted scientific articles that they allege show low-scoring countries actually do very well. Two examples that were brought up to me recently were Iran and India.

The example of Iran was a meta-analysis of different studies of Iranians. Someone brought this up to me to claim that Iran actually had a national IQ like America’s, at 97.12. Perplexed, I asked the simple question: Did they use American or British norms? And the answer is ‘no’, the test norms were Iranian, so if anything, these samples had an IQ below what’s expected—that is, a mean of 100. There’s really no need to ask further questions about these results, because being on different norms and not having the handbook handy to make the scores comparable means that they do not permit international comparisons. But this is a pretty standard sort of argument for people to make to contradict Lynn’s estimates.

The example of India was a news report about alleged testing of Indian students by Mensa’s India chapter. The report reads:

In the past couple of months, Mensa India, Delhi, administered its internationally recognized IQ test to over 4,000 underprivileged children in Delhi and NCR as part of a unique project aimed at identifying and mentoring poor children with high IQ. Of the 102 extremely bright children it selected, over a dozen, including Amisha, achieved an IQ score of 145-plus, which puts her in the genius category.
The others achieved IQ scores of 130-145, which puts them in the category of ‘very gifted’ children. The average score in Mensa India’s IQ test is between 85 and 115. Interestingly, all of these children are sons and daughters of labourers, rickshaw pullers, security guards, street vendors, etc.

Now, does this say anything in defiance of Lynn’s numbers? No, because we don’t know the norms. We don’t even know much about these statistics at all, we just have hearsay without accompanying statistics. This barely rises above an anecdote, but it is the sort of thing that people will misinterpret to mean Lynn was wrong, somehow.

Another less common strategy to reject national IQs is to just unreasonably ignore data. For example, today I encountered someone who plotted Haiti’s national IQ over time with two datapoints, one from the late-1940s and the other from the late-1970s. They alleged that Haiti had gotten much smarter over time, with their national IQ rising from around 60 to almost 100. But looking at all available data, this view cannot be supported. What they did to make their claim isn’t even a reasonable way to compute a national IQ. Getting a national IQ estimate requires looking at multiple lines of evidence and qualifying the inclusion of samples and whatnot, whereas their strategy was to act like there were just two studies to discuss and to conclude that the later one was the ground-truth regardless of its reliability or any other of its qualities.

This complaint takes two forms. The first is the more general rejection of intelligence tests measuring intelligence. That’s not relevant to national IQs and it’s poorly supported, but the arguments on that topic are familiar, so I’ll skip to the relevant argument.

The second form is that, for some reason, differences in national IQs are due to different factors than those that explain test performance within populations. This perspective is incompatible with measurement invariance, so it is necessarily wrong for any psychometrically unbiased comparisons. A sense in which the claim can be recovered is that for a given unbiased comparison, there might be mean differences in specific factors rather than in g—a sort of international versions of the contra hypothesis for Spearman’s hypothesis. This is not the case for the PISA tests or any other unbiased national IQ comparison of which I’m aware, so while it’s a possible perspective, it’s empirically contradicted at the moment.

Some people prefer achievement tests to national IQs, despite the positive manifold strongly suggesting that achievement tests and IQ tests both measure g8, and confirmatory factor modeling confirming that. The people with that preference also tend to make another, related argument: that achievement tests are not measures of g and cannot be treated as such. But like the above arguments, there are only negative reasons to think this is true for international examinations like the PISA tests, which have a strong general dimension, much like standard IQ tests.

People really want national IQ estimates to be debunked or, worse, to feel personally favorable. So they concoct a lot of bad arguments to make it sound like national IQs are off. To recap, here are some of the ways:

They confuse themselves and others about how mental retardation is defined and act as though it’s defined by IQ alone, so certain national IQs must be implausible. In doing so, they ignore the importance and existence of norms, as well as the modern definition of mental retardation itself.
They claim methodological choices like imputation are extraordinarily biasing when that is not the case and checking shows that the choice is actually not even biasing in the direction of national IQs being unfair to the groups national IQs have been proposed to illustrate bias against.
They claim that sampling is directionally biased in a certain way, when inspection of the data generally shows that, if anything, it’s biased in a way that leads to understated lower-tail cognitive differentiation.
They make claims that ‘sound right’ like that comparing children and adults is bad, even when we have age-based norms, so this cannot be a genuine criticism unless it is theoretically qualified that somehow children in certain countries are more disadvantaged than their adults and that this disadvantage translates to lower cognitive performance. I’m not aware of anyone who believes this theoretical qualifier and existing evidence speaks against it.
They make groundless extrapolations like ‘The Flynn effect means national IQs are worthless’ or ‘national IQs have changed a lot’ and they refuse to justify these inferences.
They look for odd references and outlier studies to justify throwing out a whole corpus of material.
Etc.

The common thread between different national IQ criticisms is the weaponization of ignorance. Critics toss out claims they think are right or which sound right, and they don’t check their work. A stand-out example is that people regularly say that Lynn was biased against Sub-Saharan Africans on the basis of his use of samples that are well-off by the standards of their countries purely because they are poor by the standards of the developed world. But this argument cannot stand. It has no merit, and it only serves to insult the reader and to convince them that the person making the argument has done some of the required work to dismiss Lynn’s estimates, when they’ve really only done the required work to say they’ve just barely cracked open the book!

After throwing out enough criticisms, people feel that national IQ estimates simply cannot stand, that they must be wrong, or so many criticisms wouldn’t be possible in the first place. But on this they’re wrong, and the very fact that they have so many criticisms is a strike against them, because the criticisms are so uniformly bad that they should embarrass the person making them. Making matters worse, critics seem to never go back when they’re shown to be wrong. Their attempted debunkings get roundly debunked and national IQ estimates remain reliable as ever, and they just keep making the same tired arguments that no right-minded person could still believe.

The defining feature of criticisms of national IQs is not all of this lazy argument though, it’s what comes next: insults. People like Lynn are taken to be ‘stupid’, people who believe in a given estimate that upsets people regardless of how well-supported it is are taken to be ‘morons’, and looking into and understanding national IQ estimates becomes less common because, after all, the only people who would look into them are the sorts of ‘stupid morons’ who actually bother to check their work.

Think I missed any big arguments? Want more details about something I said? Need something explained? Noticed a grammatical error, spelling mistake, or other triviality? Think I’m right or wrong about some claim? Have more data for me to look at?

Then tell me, because this is a living post that will be updated over time.

Now enjoy the most up-to-date national IQ map. It’s imputation-free!

Jensen and Kirkegaard 2024

Discussion about this post

Is there such a thing as a web-safe font?

Thu, 16 Jan 2025 10:17:01 GMT

Score 32 | Comments 46 (thread link) | @mariuz
January 16, 2025 | https://www.highperformancewebfonts.com/read/web-safe-fonts | archive.org

Unable to extract article

Accessibility essentials every front-end developer should know

Thu, 16 Jan 2025 10:06:54 GMT

Score 3 | Comments 0 (thread link) | @MartijnHols
January 16, 2025 | https://martijnhols.nl/blog/accessibility-essentials-every-front-end-developer-should-know | archive.org

Published

last week,

updated 5 days ago

Many developers view accessibility as an overwhelming task, requiring a lot of extra effort or specialized knowledge. But a few basic practices can make a significant impact.

In this article, I'll walk you through the key accessibility principles I believe every front-end developer should apply when building components, including:

Semantic HTML: Use the right elements for interactive and native functionality.
Forms: Simplify labels and structure to improve usability for everyone.
Keyboard navigation: Ensure users can navigate around with their keyboard.
Modals: Modals have many accessibility requirements.
Image alt texts: Write better descriptions to make images more accessible.
Styling: Enhance accessibility through focus indicators, responsive design, and reduced motion.
ARIA Attributes: When and how to use ARIA to fill accessibility gaps.

These practices not only benefit users relying on assistive technologies, they improve the overall user experience (UX).

This article focuses on the basic things you, as a front-end developer, can do to improve accessibility and usability without spending much extra time and effort.

Aside

Most examples in this article use React, but the principles apply to any front-end app. Even if you're not using React, you can still benefit from the practices outlined here.

Semantic HTML

Accessibility begins with semantic HTML; using the correct HTML5 elements for their intended purposes. This helps browsers and tools understand the structure of your page allowing them to provide built-in accessibility benefits. And a nice bonus is that semantic HTML also improves SEO.

Interactive elements

The most important elements to get right are

Hacker News - Small Sites - Score >= 2

I Make Smart Devices Dumber: A Privacy Advocate's Reflection

Keep the Bots Away

Kreya 1.16 with WebSocket support released – Postman Alternative

WebSocket support​

Added faker to scripting API Pro / Enterprise​

Updated look and feel of Kreya​

Collection state filter Pro / Enterprise​

File exclusion list for gRPC proto file importers​

Cookie management​

AWS Signature v4 authentication​

Bug fixes​

Test-Driven Development with an LLM for Fun and Profit

How I normally code with an LLM

Can I automate it?

LLM-powered development and cognitive load

Astronomers discover neutron star with an incredibly slow six-hour spin

What is a neutron star?

What rule-breakers have researchers found?

A lucky find

So what's the explanation?

Streamline Git with GPT

Prerequisites

Auto-Review Git Changes

Setup

Usage

Example Output

Auto-Generate Git Commit Messages

Setup

Usage

Example Output

Best Practices

Conclusion

Improve Rust Compile Time by 108X

Disclamer

Results

Motivation

Why is it that slow?

Playing with LLVM optimization settings

Not a Rust Compiler Issue

Zig comptime: does anything come close?

Zig comptime: does anything come close?

The basics

comptime blocks and arguments

Code ellision

Generics

Type introspection

Textual code generation

Final thoughts

Nepenthes is a tarpit to catch AI web crawlers

Latest Version

Usage

Installation

Bootstrapping the Markov Babbler

Statistics

Nepenthes used Defensively

Nepenthes used Offensively

Configuration File

History

Stuff We Already Depleted

One last thing!

I Ditched the Algorithm for RSS–and You Should Too

Setup¶

Easy 1: Youtube¶

Easy 2: IGN¶

Medium 1: HackerNews¶

Medium 2: Reddit¶

Step 1: Filter out picture posts.¶

Step 2: Filter for quality¶

Hard:¶

Conclusion¶

Strategies to Complete Tasks with ADHD

National IQs Are Valid

Discussion about this post

Is there such a thing as a web-safe font?

Accessibility essentials every front-end developer should know

Semantic HTML

Interactive elements

Native elements

WebSocket support

Added faker to scripting API Pro / Enterprise

Updated look and feel of Kreya

Collection state filter Pro / Enterprise

File exclusion list for gRPC proto file importers

Cookie management

AWS Signature v4 authentication

Bug fixes