# zsh-completion-sync
A zsh plugin that automatically loads completions added dynamically to FPATH or XDG_DATA_DIRS

## What does it do?

Anytime after the start of your shell if new entries have been added to your FPATH, this plugin will reload your completion system to make the new completions functions immediately available. It also loads extra paths from `$XDG_DATA_DIRS` to comply with the XDG spec.
This plugin is especially useful/important for users of dynamic development shells via tools like `direnv`, `nix-direnv`, `nix develop`, which dynamically put specific versions of executables on the front of `$PATH` and typically surface matching completion scripts via `$XDG_DATA_DIRS`

## Motivation
Normally zsh only builds the pairing of executable name and completion functions once at shell startup, when `compinit` is called, typically while sourcing `~/.zshrc`. For this, only directories on `$FPATH` are considered. Any additions to `$FPATH` after this point, will make the functions themselves available for autoloading, but the completion system will not consider them anymore. This means that any tools that are installed non-permanently, will never have their completions picked up, since they will generally not be on the startup `$FPATH` (or `$PATH` for that matter).
Users of `nix` and `direnv` are hit especially hard by this, since they rely on non-permanent changes to the running shell or shells with dynamically generated `$PATH`. This plugin bridges that [still unresolved need](https://github.com/direnv/direnv/issues/443) for zsh users. (Note, for fish users, there is a similar plugin [here](https://github.com/pfgray/fish-completion-sync))


## Installation

When sourced, this plugin only defines functions and registers hooks. The hooks are only executed the first time the prompt is rendered. Therefore it is not very sensitive to load order, however it should be loaded late to ensure that it only runs after all other hooks that could affect the relevant env vars have run.

## Quickstart

### Nix `mkShell` (via `nix-direnv` or `nix develop`)

Should work out of the box.

Most packages have shell completion scripts for bash,zsh, and fish available in their share directory. `mkShell` automatically adds their share directories to `$XDG_DATA_DIRS`, so if they ship a `zsh/site-functions` dir in them (i.e. they used `installShellCompletion --zsh`, which is most), this plugin will add them to the `$FPATH` and make the functions available for completion.


### `nix-shell -p`

Currently, `nix-shell` does not currently set nativeBuildInputs (https://github.com/NixOS/nix/issues/4254) and therefore does not set/expand `$XDG_DATA_DIR`.
This plugin offers a slightly hacky workaround which tries to [discover fpaths from entries on `$PATH`](#path-sync).
You can enable it by setting `zstyle ':completion-sync:path' enabled true`. Since this behaviour does not conform to the meaning of the `$PATH` variable (as opposed to `$FPATH` and `$XDG_DATA_DIR`) it is not enabled by default. Putting relevant packages into `nativeBuildInputs` (once supported) is the preferred way.

### `direnv` (without nix)

Ensure that `.envrc` exports an extended `$XDG_DATA_DIRS`. each path should contain a folder `zsh/site-functions` or `zsh/$ZSH_VERSION/functions`, (often this path also contains completion scripts for other shells and is therefore shell agnostic and preferrable over setting fpath directly). Alternatively, expand `$FPATH` with the folders containing the completion scripts (Remember that direnv uses a bash subshell, so you need to use the scalar `$FPATH` and not the array `$fpath`)

## How does it work?

This plugin registers a hook for both `precmd` (run each time before the prompt is displayed) and `chpwd` (run when changing directories). It registers itself at the end of the hook chain when it is loaded, in the hope of going last after all hooks that might change the env vars (especially after the dir env hook).

### XDG_DATA_DIRS sync

The first time this feature loads, it will enumerate all subpaths of XDG_DATA_DIRS that contain zsh functions (`$dir/zsh/site-functions`, `$dir/zsh/$ZSH_VERSION/functions`, or `$dir/zsh/vendor-completions` which is a non-standard path used by debian packages) and then prepend all of these paths *that are not on the FPATH yet* to the FPATH. This avoids overriding custom overrides for internal functions again by putting the zsh install directory on the front of the path, because it is also reachable via `$XDG_DATA_DIRS`.

Then everytime `$XDG_DATA_DIRS` changes, the plugin then enumerates the zsh function subpaths again and then diffs that against the last state of the function dirs. It then adds/removes these from fpaths as indicated by the diff. Note, that unlike in the initialization, this will always prepend or remove the first occurence of a path from the fpath. If a directory is dynamically added during runtime, we assume that the user wants it to take priority.

### PATH sync
The PATH sync feature is a workaround for [`nix-shell -p`](#nix-shell--p), which puts the `bin/` folder of nix a set of nix derivations on the PATH, but doesn't export their `share/` directory on `$XDG_DATA_DIRS`. (This is in tune with how nix works, since only `nativeBuildInputs` and not `packages` are meant to export their data dirs). In nix, these `share/` dirs are siblings to the `bin/` path on `$PATH` and if they have a corresponding zsh completions, those are installed in `share/zsh/site-functions`. So, to discover these fpaths, the plugin tests for the existence of a directory at `$p/$relPath` for every `$p` from `$PATH` and `$rp` from the [configured array of relative paths](#path-to-fpath-sync--nix-shell-compat). The default works for discovering any zsh FPATHs in nix derivations on `$PATH`. The resultant array of paths is then added to the front of `$FPATH` (preserving priority order from `$PATH`) *if the relevant path if not on `$FPATH` yet*. When there is a change in `$PATH`, the plugin builds the array from the current `$PATH` again and diffs it against the last state, adding or/removing to/from the front of `$FPATH` as needed.

### `zsh-autocomplete` Compatibility

Since [`zsh-autocomplete`](https://github.com/marlonrichert/zsh-autocomplete) includes very heavy customization of the zsh compsys, we cannot simply use the existing mechanism of calling `compinit` again. Instead we need to source (i.e. initialize) all of `zsh-autocomplete` again. To accomplish this, the plugin tries to detect if a named directory `~autocomplete` exists. (ZAC defines this directory and points it at the directory containing it's plugin file). We then attempt to source a file named `zsh-autocomplete.plugin.zsh` in this directory. If such a file is found, sourcing it again replaces our calls to compinit with. This detection is pretty robust. Should it ever fail, it can either be turned off by setting `zstyle ':completion-sync:compinit:compat:zsh-autocomplete' enabled false`. Alternatively, you can directly set the path to the zsh-autocomplete plugin file [by using a custom hook](#the-hard-way)

In the future we hope that the integration can be made more lightweight with specific functions to just reload the matching of commands and completion functions, or even allow for adding/removing specific completion entries.

## Options

You can use zstyle to control the behaviour of the plugin. This includes enabling/disabling certain feature-sets and fine-grained control over debug logging.

### XDG_DATA_DIRS sync

To turn the XDG_DATA_DIRS feature on or off, set `zstyle 'completion-sync:xdg' enabled` (default enabled).
Note that disabling this feature during runtime will not remove the dirs added to the fpath at startup, it will only pause the syncing. To avoid adding paths from `$XDG_DATA_DIRS` to `$FPATH`, set the zstyle in `.zshrc` (i.e. before the first prompt is rendered)

### PATH to FPATH sync / nix-shell compat

To enable detecting zsh function paths from the binary search path, set `zstyle :completion-sync:path enabled true` (default false).

The plugin searches the relative paths indicated in the array set in `zstyle :completion-sync:path rel_dirs` (default: `("../share/zsh/$ZSH_VERSION/functions" '../share/zsh/site-functions')`). The default is setup based on the typical nix derivation structure, which has its `share/`/`XDG_DATA_DIR` as a sibling to the `bin/` directory on the path.

Note that if you want nix-shell to put you into zsh by default and only need support for `nix-shell` compatibility, you should install [this plugin for compatibility](https://github.com/chisui/zsh-nix-shell) which does put nix fpaths onto path already and does not require this plugin's workaround.


### Custom Pre- and Post- Hooks

This plugin allows running custom hooks just before and/or after reloading compinit. This is especially helpful to reintroduce changes which where overridden by compinit reinvocation. (For example zsh-autocomplete binds tab to its completion function, but the user might want to use tab for something else or use something like fzf as entrypoint to the completion functions)

```
zstyle ':completion-sync:compinit:custom:pre-hook' enabled true
zstyle ':completion-sync:compinit:custom:pre-hook' command 'this string will be eval'ed'

zstyle ':completion-sync:compinit:custom:post-hook' enabled true
zstyle ':completion-sync:compinit:custom:post-hook' command 'this string will be eval'ed'
```

### Priority of custom compinit options

The following options all control how the zsh compsys is reloaded when a change in fpath is detected. These options take priority in the following order
1. [Custom](#custom-command) (default: disabled)
2. [zsh-autocomplete](#zsh-autocomplete-compatibility-1) (if enabled and zsh-autocomplete found, default: enabled)
3. [Builtin compinit](#builtin-compinit) (default: disabled)
4. `compinit` (using the currently loaded compinit)

### Custom Command
If the existing options are not sufficient or you want to fine tune how compsys is reloaded, you can set a custom command

```zsh
zstyle ':completion-sync:compinit:custom' enabled true
zstyle ':completion-sync:compinit:custom' command 'this string will be eval'ed'
```

### Compinit arguments

When using zsh-autocomplete, default or builtin `compinit`, you can use `zstyle ':completion-sync:compinit' arguments` to set an argument of arguments to use when invoking compinit. Note that `-d /path/to/compdump` is managed outside of this zstyle and should not be set inside arguments

Example: This will disable compdumping, and fpath testing for existing compdump files, both of which are expensive processes
```zsh
zstyle ':completion-sync:compinit' arguments -D -C
```

### `zsh-autocomplete` Compatibility

The plugin `zsh-autocomplete` (ZAC) customizes the behaviour of zsh's compsys to an extreme degree. it does this so much, that any calls to the builtin `compinit` will break the plugin in subtle to overt ways. As a protection, the plugin disables `compinit` completely after it has loaded (see [Builtin Compinit](#builtin-compinit) for details). This leaves us without a good option on how to reload our completions system. While [forcing use of the builtin compinit function](#builtin-compinit) allows reloading, this will lead to aforementioned breakage. Instead, we can make use of the fact, that [zsh-autocomplete initialization is idempotent](https://github.com/marlonrichert/zsh-autocomplete/issues/283#issuecomment-902850335) and source the complete plugin again to ensure that zsh-autocomplete re-initializes on the current FPATH

#### The automagic way
By default, this plugin will try to detect the installation path of zsh-autocomplete for you.
```zsh
zstyle ':completion-sync:compinit:compat:zsh-autocomplete' enabled true # default: true
```

#### The hard way
In order to re-initialize ZAC , this plugin needs to know where ZAC is installed
```zsh
zstyle ':completion-sync:compinit:custom' enabled true
zstyle ':completion-sync:compinit:custom' command 'source <path to zsh-autocomplete.plugin.zsh>'
```

### Builtin Compinit

Caution: This is hacky and will probably break something.

```zsh
zstyle ':completion-sync:compinit:builtin-compinit' enabled true #default: false
```

Some plugins patch or otherwise modify the compinit function. this can leave us unable to access the functionality necessary for reloading the completion system. As a workaround, this option, if enabled will first copy the existing compinit function out of the way and autoload `compinit` from the fpath again. The result of this is almost always the builtin `compinit` or a function meant to act as a drop-in replacement and therefore safe to call.
Note that while this will bruteforce re-enable the ability to reload completions, there is often a very good reason why plugins patch compinit. Sometimes, it is only the goal of improving performance ensuring that multiple calls to `compinit`  in `.zshrc` via different plugins do not slow down shell start. In this case, where `compinit` was no-op'ed for performance, it is safe to enable this option.
On the other hand, plugins like [zsh-autocomplete](https://github.com/marlonrichert/zsh-autocomplete/) (which made me originally write this workaround) no-op the function, because it's default behaviour will break `zsh-autocomplete` in subtle ways. Using builtin-compinit with it gets completions for new commands, but at degraded autocomplete functionality across the board.

### Debug Logging
Examples:
```zsh
  # Enable debug logging by default
  zstyle ':completion-sync:*' debug true
  # Turn off the very verbose line diffs
  zstyle ':completion-sync:*:diff' debug false
  # Turn off debug logging about candidate paths
  zstyle ':completion-sync:**:candidate' debug false
  zstyle ':completion-sync:**:candidate:*' debug false
```
You can control the debug logging very precisely if you need to, but for most use cases the options above are sufficient. If you need more precision, search for `:completion-sync:` in the source.


### Optimizations

Zsh's completion system is necessarily opiniated because it was designed and built in a very different time, including where installed packages/binaries didn't change quickly and cpu time was sparse compared to today. Design decisions (especially around caching) made sense then, but are standing in the way of performant dynamic updates to the completion system. To regain some performance this plugin uses a few optimizations. These are:

#### fast-add

When adding entries to the (front of) the FPATH, we can optimize the performance of the plugin by skipping a full reload of the compinit system and only manually adding in any `compdef` and `autoload` calls that a related to the added entries on the FPATH. This should still lead to the exact same state, since adding completion definitions to the front of the fpath will mean that they become the preferred function definition for said completion. When removing entries from the FPATH, a full reload is still required.

Overall, the optimization is crude because it can lead to a lot of independent `compdef` calls, therefore it is still considered experimental for now.

This optimization has graduated from expiremntal and is enabled by default, to *disable* this optimization, use

```zsh
zstyle ':completion-sync:compinit:optimizations:fast-add' enabled false
```

#### no-caching

zsh's compinit caching is very limited, it implicitly assumes that the set of completions changes very infrequently and not during the lifetime of a shell (basically whenever a user installs packages or changes their `.zshrc`). Therefore caching only works if the compdump file is an exact match for the rarely changing fpath. Inside of the circumstances where this plugin is typically used, this will mostly lead to unnecessary churn due to almost constant cache-misses. This option sets the compdump location (`$_comp_dumpfile` and similar variables) to `/dev/null`, and adds `-D -C` to the [compinit arguments](#compinit-arguments), forcing instant cache misses without extraneous checks, and disables constructing a useless compdump file

This optimization has graduated from expiremntal and is enabled by default, to *disable* this optimization, use

```zsh
zstyle ':completion-sync:compinit:optimizations:no-caching' enabled false
```


#### copy-compdump (experimental)

By default, this plugin uses a separate, initially empty zcompdump file per shell to avoid temporary shell environments polluting the user-global compdump file. With the experimental "copy-compdump" option, each shell will start out with a copy of the user-global compdump file and the plugin will attempt to maintain an up-to-date user-global compdump file by copying the cache dump back to the user-global spot when run inside a login shell (since a login shell should never start out in a temporary shell environment).

Given the limitations of zsh's caching system it is unclear if there are many usecases where this optimization will help much. Most use-cases will be sped up more by using [no-caching](#no-caching).

Disabled if [no-caching](#no-caching) is enabled

To enable this optimization, use

```zsh
zstyle ':completion-sync:compinit:experimental:copy-compdump' enabled true
```
#### frozen-first-cache (experimental)

(only relevant if [no-caching](#no-caching) is also enabled).

Modifies the logic of [no-caching](#no-caching), so that the compdump file path is not set to `/dev/null` until just before this plugin's `precmd` hook runs for the first time. This allows for an optimization in which the first compinit call (which should always lead to the same state of the compsys unless `.zshrc` changed) is loaded from cache for a faster startup. Afterwards, behaves exactly like `no-caching` (i.e. forces full reloads to traverse the `$FPATH` without caching).

Note that when using this option, you need to handle cache-invalidation yourself, otherwise your completions might be incomplete or broken on shell start.

Most use cases will not need this option and can simply ensure that `zsh-completion-sync` is initialized after the synchronous call to `compinit [-C -D] -d /path/to/compdump`, however when using [zsh-autocomplete](https://github.com/marlonrichert/zsh-autocomplete/), you should not call compinit directly, but let ZAC call it in its initialization hook. With [no-caching](#no-caching), the compdump path will then already have been set to `/dev/null`, negating the benefit of having a properly maintained cache file for startup.

TL;DR: If you use a plugin which runs `compinit` late in a `precmd` hook (like [zsh-autocomplete](https://github.com/marlonrichert/zsh-autocomplete/)), use this option, and make sure this plugin is initialized after the `compinit` calling plugin

To enable this optimization, use

```zsh
zstyle ':completion-sync:compinit:experimental:frozen-first-cache' enabled true
```