# Operations
What a deploy has to get right: configuration, key prefixes, every TTL, and the
two ceilings that decide whether a fleet stays up. Nothing here is about game
logic, and all of it has bitten somebody.
## Configuration is injected, never read
```mermaid
flowchart LR
ENV["process.env"] --> H["gamebaseOptionsFromEnv
slackLogWriterOptionsFromEnv
s3cbClientOptionsFromEnv
s3cbLogFlushOptionsFromEnv"]
H --> O["an options object"]
O --> F["createGamebaseContext, createSlackLogger, ..."]
ENV -.->|"never"| F
```
**Library code never reads `process.env` and never reaches for a global clock**,
and nothing prints because of an environment variable. (One helper,
`createLambdaS3Logger`, does print to the console by default — see
[Logging](logging.md#composition).) A package may export exactly one
`OptionsFromEnv()` helper for callers who want env wiring, and only that
helper touches the environment. Calling it is your choice; the library's own
path is the options object.
That is not purity for its own sake. Those seams are the test seams, and they
are why [`examples/actor-game`](../examples/actor-game/README.md) can run a real
game loop with no AWS and no Redis.
`gamebaseOptionsFromEnv()` reads `REDIS_HOST`, `REDIS_PORT`, `REDIS_USER`,
`REDIS_PASSWORD`, `REDIS_TLS` (any non-empty value enables TLS), `WS_ENDPOINT`,
`GAME_ACTOR_LAMBDA_NAME` and `IS_OFFLINE`.
## Key prefixes and the Redis ACL
```mermaid
flowchart TD
ACL["your channel's ACL user
scoped to one key pattern and one channel pattern"]
ACL --> K1["eventKeyPrefix + gameId"]
ACL --> K2["queueKeyPrefix + gameId"]
ACL --> K3["channelPrefix + gameId"]
ACL -.->|"NOPERM"| X["anything outside the scope"]
X --> Y["including a prefix retyped by hand"]
```
The console issues a credential scoped to your channel's key and pub/sub
patterns. A wrong prefix therefore fails **loudly** with `NOPERM` rather than
quietly reading another game's queue — which is the point of the scoping.
**Copy the prefixes as one block; never retype them.** A prefix typed by hand
usually stays _inside_ the ACL's pattern, so it does not fail at all: it writes
to a key nobody drains, and both sides look healthy. The console presents them
as a single copyable block for exactly this reason.
The same credential denies key enumeration — `KEYS`, and explicitly `SCAN`,
`RANDOMKEY` and `DBSIZE`, since Redis does not filter `SCAN` by pattern. So you
cannot go looking for the key you actually wrote, which is the other half of why
the TTL rule below is absolute.
The prefixes are configuration in three places — the yyt service, tslib, and
your own code — and **a mismatch is a silent no-op, not an error**.
## Every TTL
| Setting | Where | Default | Why it exists |
| ----------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------- |
| `queueTtlSeconds` | `handleConnect`, `handleDisconnect`, `handleMessages`, `createActorSubsystem` | **required** | only a producer pushes, so only a producer can re-apply it; a queue abandoned by a dead actor must expire |
| `queueTtlSeconds` on `handleActor` | the actor's own subsystem | `lifetimeSeconds + 10` | inert unless `gameMain` pushes, since the actor only drains |
| `ttlSeconds` on `createRedisQueue` | the queue itself | **required**, throws on non-positive | `EXPIRE` takes whole seconds; a fraction would floor to 0 and delete the key after every push |
| `lockTimeout` on `createRedisLock` | the lease | **required** | a lock that never expires deadlocks its actor forever when the holder crashes |
| `lockTimeoutSeconds` | `handleActor` | 30, heartbeated at a third | short so a crash frees the `gameId` in seconds, not for the rest of the game |
| `defaultConnectionMappingTtlMillis` | the `connectionId` to `gameId` map | 900000 | refreshed on every inbound message, so it bounds idle time, not session length |
| `expiresInMillis` | `repository-redis`, the document factories | **required in effect** | `set()` throws without it |
**"No TTL" is not an option anywhere.** The instance is shared and runs
`allkeys-lru`, so a key that never expires evicts someone else's first. Where
tslib can enforce it, it does — by making the option required rather than by
documenting a default, because a default that is unsafe should not exist.
## The concurrency ceiling
```mermaid
flowchart LR
L["your Lambdas"] --> C["one pinned connection each
a frozen container holds it until wait_timeout"]
C --> DB[("MariaDB max_connections 60")]
RC["reservedConcurrency"] -->|"the only clamp"| L
```
**Keep `reservedConcurrency` set on every function** — platform stacks and any
example you copy. It is free; it is _provisioned_ concurrency that bills. It is
the only thing holding a fleet under the database's connection ceiling, because
each container pins one connection and a frozen container keeps it.
It throttles silently past its number, so raise it with your party size and
player count rather than removing it.
## Sizing a run
```mermaid
flowchart LR
A["Lambda hard ceiling
900 s"] --> B["your run budget
the reference design uses 12 min"]
B --> C["lifetimeSeconds"]
C --> D["queueTtlSeconds
lifetimeSeconds + 10"]
C --> E["the function timeout
waiting + running + margin"]
```
The Lambda timeout must exceed `gameWaitingSeconds + gameRunningSeconds` plus a
margin. The start-event TTL is a _different_ number — `lifetimeSeconds + 10`,
from `handleActor`'s own option — so keep `lifetimeSeconds` at or above the sum
of the two stage budgets, or the start event can expire under a game that is
still being played. Set `maximumRetryAttempts` to
0 on the actor: a retried invocation would replay the game from the start.
**The lease has to outlive a stalled Redis round trip, several times over.**
`handleActor` beats the heartbeat every `max(1000, lease / 3)` ms and skips a
beat while one is still in flight, so a renew that hangs swallows every beat it
overlaps. A hung command can occupy up to twice the connection's `timeoutMillis`
([Redis and sockets](redis-and-sockets.md#what-timeoutmillis-measures)), which
at the 5000 default is 10 s — so `lockTimeoutSeconds` at the default 30 leaves
three attempts inside the lease, while a lease of 3 s would leave exactly one
and expire under the first stall. Keep `lockTimeoutSeconds * 1000` at or above
six times `timeoutMillis`.
## Nothing after the game may fail it
Once the outcome is decided, everything left is delivery: `onGameEnd`, the
end-stage announcement, the disconnects, the start-event delete, and the actor
lock release. All of them log and continue instead of throwing, for two reasons.
A failure there is not the game's result. A run that was played to the end and
then could not delete a key is a successful run, and reporting it as a failed
invocation hides real failures behind noise — with `maximumRetryAttempts` at 0
there is nothing to retry either way.
And a step that throws takes the steps after it with it. The repeats exist for a
delivery that did not land the first time (`endRepeatCount` with a pub/sub
transport), so an announcement that threw used to cancel exactly the retry meant
to cover it — and so did a refused disconnect, which took the whole remaining
schedule with it. Within one round every socket was always attempted, because
`Promise.all` starts them all before it observes a rejection; guarding per
connection is what names the one that refused and lets the next round run.
"Log and continue" is not "skip". The lock release is the one step with no
fallback when the lock was configured without an expiry, so `eventLoop` retries
it once and reports the second failure at `error`; the rest are logged and
forgotten because a TTL or the next run cleans up after them.
None of this reaches the invocation, and neither does the game itself: a throw
out of the game loop becomes `reason: "error"` inside `runGameAllTogether`, and
`startActorLoop` logs anything `gameMain` throws as `"unexpected error from
game"` without rethrowing. A failed invocation therefore points at something
outside the game — loading the start event, taking the lock, the readiness
handshake — not at a game that went wrong.
## Never log
The short version, because it is an operational property and not only a coding
one: no credential, no request event, no connection-id list, and no payload.
Two shipped packages do log a payload at `debug`, so a production logger's
severity is a real decision — [Logging](logging.md#what-must-never-be-logged)
has the full list, the exceptions, and what to log instead.
## Read next
[Logging](logging.md), or [Troubleshooting](troubleshooting.md) when something
is already wrong.