- [Start the daemon](#start-the-daemon)
    * [Systemd](#systemd)
- [How to use the client](#how-to-use-the-client)
    * [Adding commands](#adding-commands)
    * [See what's going on](#see-whats-going-on)
- [Controlling Tasks](#controlling-tasks)
    * [Parallel tasks](#parallel-tasks)
    * [Pause, resume and start tasks](#pause-resume-and-start-tasks)
    * [Manipulate multiple tasks at once](#manipulate-multiple-tasks-at-once)
    * [Dependencies](#dependencies)
    * [Delays and immediate](#delays-and-immediate)
- [Good to know](#good-to-know)
    * [Shell Escaping](#shell-escaping)
    * [Environment-Variables](#environment-variables)
    * [Load Balancing](#load-balancing)


## Start the Daemon

<a name="headers"/>

Before you can use the `pueue` client, you have to start the daemon.

**Local:**
The daemon can be run in the current shell.
Just run `pueued` anywhere on your command line. It'll exit if you close the terminal, though.

**Background:**
To fork and run `pueued` into the background, add the `-d` or `--daemonize` flag. E.g. `pueued -d`. \
The daemon can always be shut down using the client command `pueue shutdown`.

### Systemd

Systemd user services allow every user to start/enable their own session on Linux operating system distributions.

If you didn't install Pueue with a package manager, follow these instructions first:

1. Download `pueued.service` from the GitHub Releases page.
1. Place `pueued.service` in `/etc/systemd/user/` or `~/.config/systemd/user/`.
1. Run `chmod +x pueue` and `chmod +x pueued` to ensure these binaries can be executed.
1. Make sure the `pueued` binary is placed at `/usr/bin/`, which is where `pueued.service` expects it to be.

Then, regardless of how you installed Pueue, run:

1. `systemctl --user start pueued`, to start the `pueued` service.
1. `systemctl --user enable pueued`, to run the `pueued` service at user login.
1. `systemctl --user status pueued`, to ensure it is **active (running)**.

### Windows

In 4.0+, `pueued` has built in support for Windows services. 

The following commands exist on `pueued`. All of them must be run with administrator privileges:

* `pueued [-c config] [-p profile] service install` - Install the service. The service is set to autostart, but it is not yet running after the initial install. To start it after installation, see the below `start` command. The path to `pueued` is hardcoded. If you wish to move the binary, please first uninstall, move the binary, then reinstall the service. `-c`/`-p`, if provided, will be used in daemon startup.
* `pueued service uninstall` - Uninstall the Windows service.
* `pueued service start` - Start the service. This may also be done manually in `services.msc`.
* `pueued service stop` - Stop the service. This may also be done manually in `services.msc`.

Changing the startup type (automatic, manual, disabled) can be done from `services.msc`.

For the Windows service, please also note that it [currently only supports 1 user per computer](https://github.com/Nukesor/pueue/issues/574). It does not support multiple users being logged in at the same time.

If you do not wish to use the service functionality, in 4.0+ `-d`/`--daemonize` mode has been fixed. This means you can also use any other Windows autostart functionality with daemon mode, such as the task scheduler or autostart shortcuts.

## How to use the client

### Adding commands

To add a simple command, just write: `pueue add sleep 60`.

If you want to add flags to the command, pass them behind a double dash `--`:\
`pueue add -- ls -al /tmp/no_spaces_allowed`

If your command contains characters that need escaping, surround the whole command with a string: \
`pueue add 'ls -al /tmp/dir\ with\ spaces/'`

For more information, please read the [shell escaping](#shell-escaping) section.

If you're having trouble with something else, please take a look at the [common pitfalls](https://github.com/Nukesor/pueue/wiki/Common-Pitfalls-and-Debugging) page.

### See what's going on

To get the status of currently running commands, just type `pueue status` or `pueue` as a shortcut.\
To look at the current output of a command, use `pueue log` or `pueue log $task_id`.\
If you want to follow the output of a running command, use `pueue follow $task_id`.
To follow stderr, use the `-e` flag.

## Controlling tasks

### Pause, resume and start tasks

Without any parameters, the `pause` subcommand pauses all running tasks and the daemon itself.
A paused daemon won't start any new tasks, until it's started again.

To resume normal operation, just write `pueue start`.
This will continue all paused tasks and the daemon will continue starting tasks.

However, you can also pause specific tasks, without affecting the daemon's state or any other tasks.
Just add the ID of the task as a parameter, e.g., `pueue pause 1`.
It can be resumed the same way with the `start` command.

`start` can also force tasks to be started, which ignores any constraints on parallel tasks.

### Manipulate multiple tasks at once

Most commands can be executed on multiple tasks at once.
For instance, you can look at specific logs like this:\
`pueue log 0 1 2 3 15 19`.

This also works with your shell's range parameter, e.g., `pueue log {0..3} 15 19`.

### Parallel Tasks

By default, pueue only executes a single task at a time.
This can be changed in the configuration file, but also on-demand during runtime.
Just use the `parallel` subcommand, e.g., `pueue parallel 3`.
Now there'll always be up to three tasks running in parallel.

### Stashed Tasks

Tasks may be stashed when adding tasks, while they're queued via the `stash` subcommand or when restarting tasks.

A `Stashed` task is **not** `Queued` and will thereby **not** be automatically started, which might be useful when you know that you have to execute a command, but don't want it to run just yet.
`Stashed` can be normally enqueued via the `enqueue` subcommand or they can be immediately started via `start $task_id`.

### Dependencies

Pueue allows one to specify dependencies for tasks.
A task will only be executed if all dependencies were successful.

A dependency can be specified by using the `--after/-a` flag on the `add` command.

It is advised to use this in combination with the `pause_on_failure` setting.
This will prevent all dependent tasks of a failed task to fail as well. \
Instead, one can now go ahead, debug/fix the failed task and `restart` it with the `--in-place` flag.

Any dependency handling will then continue as expected without breaking the whole dependency chain.

### Delays and immediate

There are multiple other ways to specify when a command should be executed.
Check the help text of the `add` subcommand to see all options.

As an example, you can:

- Set a delay. The task will be scheduled after, e.g., 5 hours.
- force a start. The task will be started immediately.

## Good to know

### Shell Escaping

Let's take this example: `pueue add -- echo "&& rm /tmp"` \
It might seem a little unintuitive at first, but this task will try to delete `/tmp`.

Pueue sees these parameters as two distinct strings `['echo', '&& rm /tmp']`. \
Pay attention to the fact that the quotes around the `&& rm /tmp` part didn't get passed through.
This is due to the way shells parse parameters and the way parameters are passed through to a called process.

Let's take a look at another example: `pueue add ls /tmp/Hello\ World.txt`. \
Pueue will see those parameters like this: `['ls', '/tmp/Hello World.txt']`. \
As you can see, the shell escape character `\` has been removed!

The shell removes all shell specific characters before the command line arguments are passed to the called program!
Under normal circumstances, this is exactly what you would want, but not in the case of Pueue. \
Pueue takes the given command and places it in a `sh -c '$command'` call.
However, since some critical information may have been lost, such as the `\` and `"` characters, commands may no longer work or misbehave.

`echo "&& rm /tmp"` becomes `sh -c 'echo && rm /tmp'`. \
`ls /tmp/Hello\ World.txt` becomes `sh -c 'ls /tmp/Hello World.txt'`, which then looks for `/tmp/Hello` and the file `World.md`.

The safest way to prevent any of this, is to surround the command with quotes!
That way, no shell specific syntax is removed and everything will be perfectly preserved. \
In our example, this would look like this: \
`pueue add 'echo "&& rm /tmp"'` \
`pueue add 'ls /tmp/Hello\ World.md'`


There's also the special `--escape` flag, but this is mainly for scripting.
A side effect is that it disables any shell specific syntax, such as `&&` or `&>`. \
It is not advised to use it unless you know what you're doing.


### Environment variables

Pueue preserves the current environment variables when calling `pueue add -- some_task`.
This ensures, that the command behaves as close as possible to executing it in the shell right now and then.

### Load Balancing

Pueue doesn't do any real load-balancing itself, but Pueue has an internal representation of worker pools. \
This representation is leaked to the called commands via two environment variables:

- `$PUEUE_WORKER_ID` - The current "worker" this task is assigned to.
    There'll never be two concurrent tasks with the same `$PUEUE_WORKER_ID` in the same group.
- `$PUEUE_GROUP` - The name of the group, the task is currently in.

Pueue doesn't have any real worker pools per se, but introduced this concept to allows users to map Pueue's internal group and "pool" representation to external resources:

```
pueue group add cluster_1
pueue group add cluster_2

pueue parallel 2 -g cluster_1
pueue parallel 1 -g cluster_2

pueue add -g cluster1 -- './run_on_gpu_pool experiment1 --gpu $PUEUE_WORKER_ID --cluster $PUEUE_GROUP'
pueue add -g cluster1 -- './run_on_gpu_pool experiment2 --gpu $PUEUE_WORKER_ID --cluster $PUEUE_GROUP'
pueue add -g cluster2 -- './run_on_gpu_pool experiment3 --gpu $PUEUE_WORKER_ID --cluster $PUEUE_GROUP'
pueue add -g cluster2 -- './run_on_gpu_pool experiment4 --gpu $PUEUE_WORKER_ID --cluster $PUEUE_GROUP'
```

This will create two groups with two worker pools.

- `experiment1` will be called with `$PUEUE_WORKER_ID=0` and `$PUEUE_GROUP=cluster_1`
- `experiment2` will be called with `$PUEUE_WORKER_ID=1` and `$PUEUE_GROUP=cluster_1`
- `experiment3` will be called with `$PUEUE_WORKER_ID=0` and `$PUEUE_GROUP=cluster_2`
- `experiment4` will be called with `$PUEUE_WORKER_ID=0` and `$PUEUE_GROUP=cluster_2` after `experiment3` finished.