# Content Output Review and Generation Interface (CORGI) ![CORGI](docs/_static/images/corgi.jpg) _FKA: "CORGI", Content Output Review and Generation Interface_ --- - **What does CORGI do? 🤔** [Check out the high level docs](https://openstax.atlassian.net/wiki/spaces/CE/pages/2017918977/CORGI+Service) - **I'd like to read the detailed docs 🤓** [Read The Docs](https://corgi.readthedocs.org/) - **I'm ready to install and run CORGI 🚀** [Installation](#development-internals) ## Summary The CORGI system consists of 2 parts: 1. CORGI Job Dashboard 2. [Enki](https://github.com/openstax/enki) Pipeline ### CORGI Job Dashboard The CORGI Job Dashboard or "CORGI dashboard" consists of a front-end microservice and a backend microservice. The CORGI Dashboard acts mainly as a "queue" of jobs to be processed by the Enki Concourse pipeline. 1. Backend - written using Python and the [FastAPI ASGI Framework](https://fastapi.tiangolo.com/). The backend API is used by the front-end and Enki pipelines to create, retrieve, or update job information. 2. Frontend - written using [svelte](https://svelte.dev/) and acts as the main dashboard interface of the CORGI system. You can see the list of jobs, create jobs, or abort a job that's in progress. Shows information pertaining to errors and status. ### [Enki](https://github.com/openstax/enki) Enki produces the Concourse pipeline configuration files that are used to build book artifacts. Book artifacts can be pdf files, web preview links, docx, etc. Concourse uses "workers" that watch for new jobs created using the CORGI job dashboard. The Concourse pipeline reads the jobs from the CORGI backend and updates the job status and upon completion updates the job with links to the respective book artifact that was produced. The full explanation of Enki it is out of scope for this documentation. To learn more please reference the [Enki README.md](https://github.com/openstax/enki/blob/main/README.md) within the project. --- ## Development Internals ### Installing Docker * Install [Docker](https://docs.docker.com/install/). * Install [Docker Compose](https://docs.docker.com/compose/install/). > **A note for Mac and PC users** > After installing Docker, navigate to the Docker Desktop GUI preferences and increase the `Memory` value to at least `8GiB`. > [Here's where you can find the Docker Desktop GUI settings](https://docs.docker.com/docker-for-windows/#resources) ## Local development **General Workflow** 1. Start stack with or without GitHub OAuth (see below) 1. Visit http://localhost/ to see the frontend 1. Make modifications 1. Wait for automatic reload **NOTE**: You might get 502 when visiting http://localhost/ at first: this is normal. Wait a few seconds and try again. It's recommended that you initialize your corgi environment before starting by running: ```bash source corgi ``` in the root of the repository. This will initialize your environment with helpful aliases. With your environment initialized: 1. You can run `corgi` without prefixing it with `./` 1. `npm` will be prefixed with your `frontend` directory (you can run npm from anywhere) 1. Your PS1 should be updated to include (corgi) ### GitHub OAuth Disabled (Leashed Mode) To start corgi without GitHub OAuth, in the project root directory, run: ```bash ./corgi start-leashed ``` or ```bash ./corgi start ``` then you can initialize the database with sample data by running ```bash ./corgi create-jobs ``` In leashed mode, corgi only uses data that is stored locally. Currently, that data is stored in `backend/app/tests/unit/data`. **NOTE:** By default, you will only be able to use `tiny-book` as the repository and `book-slug1` as the book when queuing jobs. This data can be updated: see [Run backend unit tests](#run-backend-unit-tests) for more information. ### GitHub OAuth Enabled **Prerequisites** * You will need access to a GitHub OAuth app. You can [create your own](https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app) if you do not have one already. * Create an env file, `backend/app/app/core/.env`, with two entries: 1. `GITHUB_OAUTH_ID` (your oauth app client id) 2. `GITHUB_OAUTH_SECRET` (your oauth app client secret) Build/start the stack with the Corgi script: ```bash ./corgi start-dev ``` View the API Docs here: * http://localhost/docs (Swagger) * http://localhost/redoc (ReDoc) To check the logs run: ```bash ./corgi logs backend/frontend/etc. ``` ### Functions Patched in Leashed Mode Leashed mode attempts to patch any `app.github.api` function functions starting with `get_`. It looks for the associated mock functions in `backend/app/tests/unit/init_test_data.py`. ### Hot Reloading Using the development stack, the Svelte frontend is rebuilt inside the container as you make changes: no restarts required. The page should reload automatically as well. The same is true for the backend: as you make modifications, the backend server should reload with your latest changes. ### Run backend unit tests To run unit tests: ```bash cd backend/app poetry run pytest tests/unit ``` The unit tests use vcr to store response from GitHub. To update the test data: ```bash cd backend/app poetry run pytest tests/unit --init-test-data --github-token "" ``` ### Run integration and UI tests To run the tests execute: ```bash ./corgi ui-tests # Run UI tests on staging # IMPORTANT: no trailing slash ./corgi ui-tests https://corgi-staging.ce.openstax.org # Run UI tests on localhost (see note) ./corgi ui-tests http://host.docker.internal fake-token ``` **NOTE**: `host.docker.internal` refers to the host machine's `localhost` instead of the container's (see [documentation](https://docs.docker.com/desktop/networking/#use-cases-and-workarounds-for-all-platforms) for more info). The ui tests will still fail if CORGI is running in leashed mode because of how the the github api calls are patched. If you want the ui tests to work locally, you will need to configure the OAuth credentials accordingly and run CORGI in dev mode. ### How to develop UI tests Playwright UI tests are stored in `backend/app/tests/ui`. ### Clear the database Start the stack as described above Run the reset-db command that is contained in the `manage.py` file. docker-compose exec backend python manage.py reset-db ### Migrations Automatic migration files can be generated for the database. After a change is made you'll want to create a new revision. Make sure the stack is running in development mode. Enter the backend Docker container: docker-compose exec backend bash Create a revision: docker-compose exec backend alembic revision --autogenerate -m "added new column X" A new revision will be added in the [migration](./backend/app/migrations/versions) folder. Open the file and ensure the changes are what you expect. Do the migration: docker-compose exec alembic upgrade head ### Load testing for the backend (Under Construction) *UNDER CONSTRUCTION* Load testing with Locust.io is in the directory `./backend/app/tests/performance/` Please look at the [README](./backend/app/tests/performance/README.md) in this directory on how to run load tests locally and for production systems. ## Sphinx Documentation ### View the Docs For our documentation we use [Sphinx-docs](https://www.sphinx-doc.org/en/master/) and lives in the [./docs](./docs) directory. If you are currently running the entire stack you should be able to see the documentation by visiting [http://localhost:8000](http://localhost:8000). The documentation is configured to watch for changes and re-build the documentation. This allows developers the ability to preview their documentation changes as they make them. If you would like to run the documentation without the entire stack running you can do so by running: docker-compose up docs ### Editing The Docs Edits are done in restructured text (rst). While editing, you can check the logs by running ``` $ ./corgi docs logs ``` or ``` $ ./corgi docs logs -f ``` If edits have been made to the Navigation and are not reflected, re-build the docker image: ``` $ ./corgi build $ ./corgi start [stack-name] ``` ### Auto-generated docs Additional documentation is automatically pulled from this README and added to the sphinx docs when the docs service is running and when corgi.readthedocs.io is deployed. Each level 2 header (##) becomes a separate page in the sphinx docs. Links, like [README.md](./README.md), are resolved to github links relative to the markdown file's parent directory. This documentation is stored in [docs/auto](./docs/auto) and this directory should not be modified directly. It is saved as markdown and converted with the m2r2 sphinx extension (see [conf.py](./docs/conf.py) for more information). When the files are generated, they are numbered so that the glob result will match the order the headers occurred in within the README. The `docs` docker image uses [watchdog](https://pypi.org/project/watchdog/) to automatically update the auto-generated documentation as you update [README.md](./README.md). There is a pre_build step in [.readthedocs.yaml](./.readthedocs.yaml) that generates the documentation during deployment. ## Releasing The documentation is located in the [Releasing CORGI article](https://openstax.atlassian.net/wiki/spaces/CE/pages/1256521739/Releasing+CORGI) in our Confluence documentation. ## Deploying Web Hosting Pipeline The documentation is located in the [How to Deploy Web Hosting Pipeline article](https://openstax.atlassian.net/wiki/spaces/CE/pages/573538307/Deploying+the+web-hosting+pipeline) in our Confluence Documentation. ## Managing Webhosting Pipeline Versions CORGI supports running three concurrent Enki webhosting pipelines simultaneously, each pinned to a different image version. This lets the team roll out a new version gradually while keeping older pipelines running for in-progress content. ### Slots The three slots are: | Position | Label | Purpose | |----------|---------|--------------------------------------| | 0 | Newest | Latest version — new jobs go here | | 1 | Second | Previous version — still active | | 2 | Oldest | Oldest active version | ### Using the Pipeline Versions dialog The dialog is accessible from the **Pipeline Versions** button on the CORGI home page (admin login required). - **Current CORGI Version** — displayed at the top. Shown in green when it already matches the Newest slot, red when it does not. - **Slot rows** — each slot shows the currently saved version (greyed out with strikethrough if you have changed it) and a dropdown to pick a new version. - **Promote Latest** — sets the Newest slot to the most recent available tag and shifts the previous Newest and Second values down. Disabled when the current CORGI version is already in the Newest slot. - **Save Changes** — asks for confirmation, then writes the new slot values to the database. Enabled even when nothing has changed (in which case it is a no-op). Available tags come from the intersection of Docker Hub and GitHub tags for `openstax/enki`, so only versions that were properly released appear in the dropdowns. ### API - `GET /api/pipeline-version/` — returns the current slot configuration (unauthenticated). Polled by the Concourse metapipeline to detect version changes. - `PUT /api/pipeline-version/` — replaces all slots (admin only). Rejects requests with duplicate version strings. - `GET /api/version/tags/openstax::enki` — returns recent available tags (authenticated users). Accepts `count` (max 25) and `pattern` query parameters. ## Attribution A lot of the structure and ideas for this service come from Tiangolo's [full-stack-fastapi-postgres](https://github.com/tiangolo/full-stack-fastapi-postgresql) project with additional supporting software and ideas from [authlib](https://docs.authlib.org/en/latest/client/fastapi.html). Thanks Tiangolo and Authlib devs! ## Authentication and Authorization ### How authentication works CORGI uses GitHub OAuth for login. After a successful OAuth flow, the backend creates an encrypted session cookie (using [Fernet](https://cryptography.io/en/latest/fernet/) symmetric encryption) that stores the user's GitHub OAuth token and role. The session lasts 8 hours, matching the lifetime of the OAuth token itself, so both expire at the same time. The `SESSION_SECRET` environment variable is the encryption key. It is randomly generated on each deployment, which automatically invalidates all active sessions on deploy — no explicit logout or session cleanup is needed. ### Roles Role assignment is based on GitHub team membership, checked at login time: | Role | Assigned to | Access | |---------|------------------------------------------------------|------------------------------------------------| | `ADMIN` | Members of `ce-tech`, `ce-admins`, `content-managers` | All endpoints including admin-only writes | | `USER` | Everyone else who logs in | Read/authenticated endpoints (e.g. tag listing)| `USER` is the default role for anyone not on an admin team — there is no separate "no access" state for logged-in users. ### Login with GitHub Personal Access Token (PAT) For situations where it is difficult or impossible to login with a username and password, there is an alternative way to login with a GitHub PAT. To utilize this functionality: **Note:** the same restrictions apply to users regardless of login method (i.e. you do not gain additional permissions by logging in with a token). 1. Create a GitHub PAT with at least the `read:user`, `read:org`, and `repo` scopes. 2. Make a request to `/api/auth/token-login` with an additional header: `Authorization: Bearer ` 3. Get the session cookie from the `set-cookie` header in the response. 4. Make additional response with the session cookie. ### Example ```python import os import requests # Note: This example assumes you are running CORGI locally my_token = os.environ["TOKEN"] # Don't hardcode secrets ;) with requests.session() as session: response = session.get( "http://localhost/api/auth/token-login", headers={"Authorization": f"Bearer {my_token}"}) assert response.cookies.get("session", None) is not None, \ "Could not get session cookie" # Now your cookie will be used to make requests that require a valid user # session jobs = session.get("http://localhost/api/jobs") print(jobs.json()) ``` ## Generating an ERD The corgi CLI supports generating an ERD from database data. The auto-generated ERD, as well as a small description, replaces the README section labeled `## CORGI ERD`. ### Running the script Assuming you have the CORGI stack running and you are in the root of the repository, you can generate an ERD like this: ```bash # Create some sample data if you have not already ./corgi create-jobs # Generate the ERD and update the README ./corgi create-erd ``` ### How it works This will run the [generate-erd](./scripts/generate-erd.py) script which attempts to crawl all database entities, starting with `Jobs` entity, and collect information about their fields and relationships. From this, the script generates an ERD using mermaid diagram syntax. An awk script replaces the section in the README with the newly generated ERD. ### Limitations (TODO?) * Does not specify which fields are primary or foreign keys * When an optional field is None, it cannot infer the data type (uses `opt` as placeholder type) * Annoyingly, in mermaid ERD syntax `relationship-label` is [required](https://mermaid.js.org/syntax/entityRelationshipDiagram.html#entities-and-relationships). Since that is non-trivial to auto-generate, it is set to "" (empty string). ## CORGI ERD [comment]: <> (BEGIN AUTO-GENERATED SECTION - CORGI ERD) This section describes the data model used by CORGI. For more information about how to read this diagram, see [Crow's foot notation](https://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model#Crow's_foot_notation). ```mermaid erDiagram ApprovedBook { int book_id datetime created_at datetime updated_at int consumer_id int code_version_id } Book { int id str slug str uuid str style int edition int commit_id } BookJob { int job_id int book_id opt artifact_url } CodeVersion { int id str version datetime created_at datetime updated_at } Commit { int id str sha datetime timestamp int repository_id } Consumer { int id str name datetime created_at datetime updated_at } JobTypes { int id str name datetime created_at datetime updated_at str display_name } Jobs { int id str git_ref int user_id int status_id datetime created_at datetime updated_at int job_type_id opt error_message str worker_version } PipelineVersion { int position datetime created_at datetime updated_at int code_version_id } Repository { int id str name str owner } RepositoryPermission { int id str name } Status { int id str name datetime created_at datetime updated_at } User { int id str name str avatar_url } UserRepository { int user_id int permission_id int repository_id } Book ||--|{ ApprovedBook : "" CodeVersion ||--|{ ApprovedBook : "" Consumer ||--|{ ApprovedBook : "" Commit ||--|{ Book : "" Book ||--|{ BookJob : "" Jobs ||--|{ BookJob : "" Repository ||--|{ Commit : "" JobTypes ||--|{ Jobs : "" Status ||--|{ Jobs : "" User ||--|{ Jobs : "" CodeVersion ||--|{ PipelineVersion : "" Repository ||--|{ UserRepository : "" RepositoryPermission ||--|{ UserRepository : "" User ||--|{ UserRepository : "" ``` [comment]: <> (END AUTO-GENERATED SECTION - CORGI ERD)