---
name: openstack-keystone
description: "OpenStack Keystone identity service skill for deploying, configuring, operating, and troubleshooting the authentication and authorization backbone of an OpenStack cloud. Covers identity management, token lifecycle (Fernet provider with rotation), service catalog registration, RBAC policy customization, domain/project/user hierarchy, federation basics (SAML/OIDC), credential encryption, and endpoint management. Use when deploying Keystone via Kolla-Ansible, managing users and projects, debugging 401 errors, rotating Fernet keys, configuring RBAC policies, or integrating services through the service catalog."
user-invocable: true
allowed-tools: Read Grep Glob
metadata:
  extensions:
    gsd-skill-creator:
      version: 1
      createdAt: "2026-02-22"
      triggers:
        intents:
          - "keystone"
          - "identity"
          - "authentication"
          - "service catalog"
          - "RBAC"
          - "token"
          - "federation"
          - "policy.json"
          - "endpoint"
        contexts:
          - "deploying openstack"
          - "configuring identity"
          - "troubleshooting authentication"
          - "managing users and projects"
---

# OpenStack Keystone Identity Service

Keystone is the identity service for OpenStack. Every API call to every OpenStack service passes through Keystone for authentication and authorization. It is the first service deployed and the last decommissioned. If Keystone is down, the entire cloud is down.

Keystone provides five core functions: **identity** (users, groups), **resources** (projects, domains), **assignment** (roles mapped to users on projects), **token** (authentication proof with configurable lifetime), and **catalog** (service endpoint registry). Understanding Keystone means understanding how every OpenStack service discovers and trusts every other service.

## Deploy

### Kolla-Ansible Configuration

**globals.yml settings:**

```yaml
# Required -- set strong passwords
keystone_admin_password: "{{ vault_keystone_admin_password }}"
keystone_database_password: "{{ vault_keystone_database_password }}"

# TLS (recommended for production)
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_copy_ca_into_containers: "yes"

# Token provider (Fernet is default and recommended)
keystone_token_provider: "fernet"

# Optional tuning
keystone_token_expiration: 3600  # seconds, default 1 hour
```

**Deployment sequence:**

```bash
# 1. Bootstrap -- creates databases, service users, initial endpoints
kolla-ansible -i inventory bootstrap-servers

# 2. Deploy Keystone (runs as part of full deploy or targeted)
kolla-ansible -i inventory deploy --tags keystone

# 3. Post-deploy -- creates admin credentials file
kolla-ansible -i inventory post-deploy
```

**Container verification:**

```bash
# Verify Keystone containers are running
docker ps --filter "name=keystone" --format "table {{.Names}}\t{{.Status}}"
# Expected: keystone_api, keystone_fernet (both Up)

# Verify service responds
source /etc/kolla/admin-openrc.sh
openstack token issue  # Must return a valid token table
```

**Service catalog registration:**

Kolla-Ansible auto-registers Keystone in the service catalog. Verify:

```bash
openstack service list  # Should show "identity" service
openstack endpoint list --service keystone
# Expected: 3 endpoints (public, internal, admin) per region
```

**Initial admin setup:**

```bash
# admin-openrc.sh is generated by post-deploy
source /etc/kolla/admin-openrc.sh

# Verify admin project and user exist
openstack project show admin
openstack user show admin
openstack role assignment list --user admin --project admin
# admin should have "admin" role on "admin" project
```

## Configure

### Fernet Token Provider

Fernet tokens are cryptographic tokens validated without database lookup. They require synchronized key repositories across Keystone nodes.

**Key repository:** `/etc/kolla/keystone/fernet-keys/` (inside container)

**Key types:**
- **Staging key (index 0):** Used for decryption only; becomes primary on rotation
- **Primary key (highest index):** Used for encryption; tokens it created remain valid
- **Secondary keys (middle indices):** Decrypt only; pruned after max_active_keys exceeded

**Rotation procedure:**

```bash
# Rotate Fernet keys (Kolla-Ansible managed)
kolla-ansible -i inventory keystone_fernet_rotate

# Manual rotation (if needed)
docker exec keystone_api keystone-manage fernet_rotate \
  --keystone-user keystone --keystone-group keystone
```

**Configuration:** `keystone.conf [fernet_tokens]`
- `max_active_keys`: Default 3. Set to (token_expiration / rotation_interval) + 2

### Domain, Project, and User Hierarchy

```
Domain (organizational boundary)
  +-- Project (resource container, formerly "tenant")
       +-- User (identity with credentials)
       +-- Group (collection of users)
       +-- Role Assignment (user/group + role on this project)
```

**Best practices:**
- Create a domain per organization or department
- Use the "default" domain for service accounts
- Never modify the "admin" project; create separate operator projects
- Assign roles to groups, not individual users, for scalability

### RBAC Policies

**Policy files:** `/etc/kolla/keystone/policy.yaml` (override defaults)

```yaml
# Example: restrict user creation to domain-scoped admins
"identity:create_user": "rule:admin_required and domain_id:%(target.user.domain_id)s"

# Example: allow project members to list users in their project
"identity:list_users": "role:member and project_id:%(scope.project.id)s"
```

**Policy evaluation order:**
1. Check explicit rule match in policy.yaml
2. Fall back to code default (oslo.policy)
3. Deny if no rule matches

**Verify policy changes:**

```bash
# Test a specific policy rule
openstack --os-auth-url http://keystone:5000/v3 \
  --os-username testuser --os-password testpass \
  --os-project-name testproject --os-user-domain-name default \
  --os-project-domain-name default user list
```

### Service Catalog Management

```bash
# List all registered services and endpoints
openstack service list
openstack endpoint list

# Register a new service
openstack service create --name nova --description "Compute" compute
openstack endpoint create --region RegionOne \
  compute public http://controller:8774/v2.1
openstack endpoint create --region RegionOne \
  compute internal http://controller:8774/v2.1
openstack endpoint create --region RegionOne \
  compute admin http://controller:8774/v2.1
```

### Federation (SAML/OIDC)

Federation allows external identity providers to authenticate OpenStack users.

**Key concepts:**
- **Identity Provider (IdP):** External auth source (AD, Okta, Keycloak)
- **Mapping:** Rules that translate IdP assertions to Keystone users/groups/projects
- **Protocol:** SAML 2.0 or OpenID Connect
- **Service Provider (SP):** Keystone acting as relying party

**Configuration reference:** `keystone.conf [federation]`, `[saml]`, `[openid]`

### Credential Encryption

```bash
# Initialize credential encryption (first time)
docker exec keystone_api keystone-manage credential_setup \
  --keystone-user keystone --keystone-group keystone

# Rotate credential encryption keys
docker exec keystone_api keystone-manage credential_rotate \
  --keystone-user keystone --keystone-group keystone

# Migrate credentials to new encryption
docker exec keystone_api keystone-manage credential_migrate \
  --keystone-user keystone --keystone-group keystone
```

## Operate

### User and Project Management

```bash
# Create domain
openstack domain create --description "Engineering" engineering

# Create project in domain
openstack project create --domain engineering --description "Dev team" dev-team

# Create user in domain
openstack user create --domain engineering --password-prompt \
  --email dev@example.com devuser

# Assign role
openstack role add --project dev-team --user devuser --user-domain engineering member

# List role assignments
openstack role assignment list --project dev-team --names
```

### Token Debugging

```bash
# Issue a token and inspect it
openstack token issue -f json

# Check token expiration
openstack token issue -c expires -f value
# Compare with system clock -- clock skew causes premature expiration

# Revoke a specific token
openstack token revoke <token-id>
```

### Fernet Key Rotation Procedure

**Schedule:** Rotate at half the token expiration interval. Default token lifetime is 1 hour, so rotate every 30 minutes in production.

```bash
# 1. Check current key count
docker exec keystone_api ls /etc/kolla/keystone/fernet-keys/ | wc -l

# 2. Rotate keys
kolla-ansible -i inventory keystone_fernet_rotate

# 3. Verify rotation
docker exec keystone_api ls -la /etc/kolla/keystone/fernet-keys/
# New highest-numbered key should be primary

# 4. Verify tokens still work
openstack token issue
```

### Backup and Restore

```bash
# Backup Keystone database
docker exec mariadb mysqldump -u root -p keystone > keystone_backup.sql

# Backup Fernet keys
docker cp keystone_api:/etc/kolla/keystone/fernet-keys/ ./fernet-keys-backup/

# Restore database
docker exec -i mariadb mysql -u root -p keystone < keystone_backup.sql

# Restore Fernet keys
docker cp ./fernet-keys-backup/ keystone_api:/etc/kolla/keystone/fernet-keys/
docker restart keystone_api
```

### Quota Management

```bash
# Keystone itself has no quotas, but controls project-level quotas for other services
# Set Nova quotas for a project
openstack quota set --instances 20 --cores 40 --ram 81920 dev-team

# Set Cinder quotas
openstack quota set --volumes 50 --gigabytes 1000 dev-team
```

## Troubleshoot

### 1. "401 Unauthorized" on All Services

**Symptoms:** Every `openstack` CLI command returns 401. All service-to-service calls fail.

**Root causes and resolution:**

| Cause | Diagnosis | Fix |
|-------|-----------|-----|
| Expired Fernet keys | `docker exec keystone_api ls -la /etc/kolla/keystone/fernet-keys/` -- check modification timestamps | `kolla-ansible -i inventory keystone_fernet_rotate` |
| Clock skew | `date` on controller vs compute nodes -- difference > 30s causes token validation failure | Sync NTP: `chronyc sources` and `chronyc tracking` |
| Wrong endpoint in admin-openrc.sh | `cat /etc/kolla/admin-openrc.sh` -- verify OS_AUTH_URL matches actual Keystone endpoint | Regenerate: `kolla-ansible -i inventory post-deploy` |
| Keystone container down | `docker ps --filter name=keystone` -- not running | `docker restart keystone_api keystone_fernet` |
| MariaDB connection failure | `docker logs keystone_api 2>&1 \| grep -i "database\|maria\|mysql"` | Check MariaDB: `docker exec mariadb mysql -u root -p -e "SELECT 1"` |

### 2. Service Catalog Endpoint Mismatch

**Symptoms:** Services fail to find each other. Errors like "Could not find service endpoint" or wrong URLs in API responses.

**Diagnosis:**

```bash
openstack endpoint list --long
# Check: public/internal/admin URLs match actual service locations
# Check: region names are consistent across all services
```

**Fix:** Correct the mismatched endpoint:

```bash
openstack endpoint set --url http://correct-host:port/v3 <endpoint-id>
```

### 3. Token Validation Failures

**Symptoms:** Tokens issue successfully but fail validation on other services. Intermittent auth failures.

**Root causes:**

| Cause | Diagnosis | Fix |
|-------|-----------|-----|
| Fernet key mismatch (multi-node) | Compare key files across Keystone nodes | `kolla-ansible -i inventory keystone_fernet_rotate` to sync |
| Memcached connectivity | `docker logs keystone_api 2>&1 \| grep memcache` | Verify memcached: `docker exec memcached memcstat --servers=localhost` |
| keystonemiddleware misconfigured | Check `[keystone_authtoken]` in service configs | Verify `auth_url`, `memcached_servers`, `project_name` |

### 4. Database Connection Failures

**Symptoms:** Keystone API returns 503. Container logs show database errors.

**Diagnosis:**

```bash
docker logs keystone_api 2>&1 | tail -50
# Look for: "OperationalError", "Can't connect to MySQL", "Access denied"

# Test MariaDB directly
docker exec mariadb mysql -u keystone -p -e "USE keystone; SELECT COUNT(*) FROM project;"
```

**Fix:**
- Connection refused: `docker restart mariadb`, verify MariaDB is listening on expected port
- Access denied: Check credentials in `/etc/kolla/keystone/keystone.conf` `[database]` section
- Database missing: `kolla-ansible -i inventory deploy --tags mariadb,keystone`

### 5. Federation/SSO Issues

**Symptoms:** External IdP users cannot authenticate. SAML assertions rejected.

**Diagnosis:**

```bash
docker logs keystone_api 2>&1 | grep -i "federation\|saml\|oidc\|mapping"
# Check: certificate trust chain, metadata freshness, mapping rules
```

**Common fixes:**
- Update IdP metadata: re-download and reimport
- Fix certificate chain: ensure CA cert is in Keystone trust store
- Debug mapping rules: use `keystone-manage mapping_engine` to test rule evaluation

### 6. RBAC Policy Denials

**Symptoms:** Users get 403 Forbidden despite having seemingly correct roles.

**Diagnosis:**

```bash
# Enable debug logging temporarily
docker exec keystone_api sed -i 's/^#debug = .*/debug = true/' /etc/keystone/keystone.conf
docker restart keystone_api

# Check policy evaluation
docker logs keystone_api 2>&1 | grep "Policy check"
# Look for the specific rule that denied the request
```

**Common fixes:**
- Policy syntax error: validate YAML with `python3 -c "import yaml; yaml.safe_load(open('policy.yaml'))"`
- Rule evaluation order: explicit rules override code defaults; check for conflicting rules
- Scope mismatch: request is project-scoped but rule requires domain-scope, or vice versa

## Integration Points

Every OpenStack service authenticates through Keystone. The integration pattern is consistent:

**keystonemiddleware configuration** (in each service's config file):

```ini
[keystone_authtoken]
www_authenticate_uri = http://keystone:5000
auth_url = http://keystone:5000
memcached_servers = memcached:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova          # service-specific
password = nova_pass     # service-specific
```

**Service user pattern:** Each service has a dedicated user in the `service` project with the `admin` role. Kolla-Ansible creates these automatically.

**Endpoint discovery:** Services find each other through the Keystone service catalog. When Nova needs to fetch an image from Glance, it queries the catalog for the `image` service endpoint.

**Token flow:**
1. User authenticates to Keystone, receives scoped token
2. User sends request to Nova with token in `X-Auth-Token` header
3. Nova's keystonemiddleware validates token against Keystone (or memcached cache)
4. Nova uses its own service user token to call Glance, Neutron, Cinder as needed

## NASA SE Cross-References

| SE Phase | Keystone Activity | Reference |
|----------|-------------------|-----------|
| Phase C (Final Design) | Configure Keystone in globals.yml: admin passwords, TLS settings, token provider, federation | SP-6105 SS 5.1 |
| Phase D (Integration & Test) | Verify: `openstack token issue`, `openstack endpoint list`, `openstack user list` -- prove identity service is operational | SP-6105 SS 5.2-5.3 |
| Phase E (Operations) | Fernet key rotation, user/project management, credential rotation, RBAC policy updates, service catalog maintenance | SP-6105 SS 5.4-5.5, NPR 7123.1 SS 5.4 |
| Phase F (Closeout) | Export user/project data, decommission service catalog entries, archive Keystone database | SP-6105 SS 6.1 |