# Dev notes This is a development guide for the govcy-express-services project. ## Release management Details on release management can be found in the [release document](./RELEASE.md). ## Local development ### Installation 1. Clone the repository: ```sh git clone git@github.com:gov-cy/govcy-express-services.git ``` 2. Navigate to the project directory: ```sh cd govcy-express-services ``` 3. Install dependencies: ```sh npm install ``` ### `secrets/.env` sample for local development Create a `secrets/.env` file (see example below): ```dotenv SESSION_SECRET=12345678901234567890123456789012345678901234567890 PORT=44319 CYLOGIN_ISSUER_URL=https://aztest.cyprus.gov.cy/cylogin/core/.well-known/openid-configuration CYLOGIN_CLIENT_ID=your-CYLOGIN-client-id CYLOGIN_CLIENT_SECRET=your-CYLOGIN-client-secret CYLOGIN_SCOPE=openid cegg_profile your.scope CYLOGIN_REDIRECT_URI=https://localhost:44319/signin-oidc CYLOGIN_CODE_CHALLENGE_METHOD=S256 CYLOGIN_POST_LOGOUR_REIDRECT_URI=https://localhost:44319/ ALLOW_SELF_SIGNED_CERTIFICATES=false NODE_ENV=development # Debug or not ------------------------------- # In production set this to false DEBUG=true # DSF Gateway --------------------------- DSF_API_GTW_CLIENT_ID=your-DSF-API-gateway-client-id DSF_API_GTW_SECRET=your-DSF-API-gateway-secret DSF_API_GTW_SERVICE_ID=your-DSF-API-gateway-service-id # Notification API URL DSF_API_GTW_NOTIFICATION_API_URL=https://127.0.0.1/api/v1/NotificationEngine/simple-message # SERVICES stuf------------------------------- # SERVICE: test TEST_SUBMISSION_API_URL=http://localhost:3002/success TEST_SUBMISSION_API_CLIENT_KEY=12345678901234567890123456789000 TEST_SUBMISSION_API_SERVIVE_ID=123 TEST_SUBMISSION_DSF_GTW_KEY=12345678901234567890123456789000 TEST_ELIGIBILITY_1_API_URL=http://localhost:3002/success TEST_ELIGIBILITY_2_API_URL=http://localhost:3002/success # Unit TEST USER TEST_USERNAME=testuser TEST_PASSWORD=******** ``` To generate the SESSION_SECRET, run: `node -e "console.log(require('crypto').randomBytes(32).toString('hex'));"` #### Create certs for local development Make sure to have certs for local development in the root of your project folder (see [install notes](./INSTALL-NOTES.md#create-certs-for-local-development)) ### non secret environment variables Create `.env.development`, `.env.staging`, and `.env.production` files in the root folder of the project. Here's an example for `.env.development`: ```dotenv MATOMO_SITE_ID=51 MATOMO_URL=//wp.matomo.dits.dmrid.gov.cy ``` --- ## Data layer The application uses Express session to store data. Here's a an overview of the session structure: ### Data Overview This is an overview of the data stored in the session: ```javascript { cookie: { ... }, // Session cookie configuration csrfToken: "string", // CSRF protection token siteData: { ... }, // Service-specific data user: { ... } // Authenticated user information } ``` ### Detailed Breakdown #### 1. Cookie Configuration ```javascript "cookie": { "originalMaxAge": 3600000, // 1 hour expiry "expires": "2025-04-03T06:35:22.026Z", "secure": true, // HTTPS only "httpOnly": true, // No JS access "path": "/", "sameSite": "lax" // CSRF protection } ``` #### 2. CSRF Token ```javascript "csrfToken": "ld524iof94w6vxifrpvny5i4bc1mo76l" // Session-wide CSRF token ``` #### 3. Site Data ```javascript { "siteData": { "[siteId]": { "loadData": {}, // Data used to load the site when presaved "inputData": { // Input data of the site "[pageUrl]": { // Pages level - e.g., "page1" "formData": {}, // Form data per page. For multiple things pages, array of item objects "multipleDraft": {}, // Temporary item data during multiple things "add" flow "validationErrors": { // Page-level validation "errors": {}, "formData": {}, "errorSummary": [] } } }, "eligibility": { // Site eligibility cached results "TEST_ELIGIBILITY_1_API_URL": { // Eligibility 1 "result": { // results "Succeeded": true, "ErrorCode": 0, "ErrorMessage": null, "Data": {} }, "timestamp": 1749755264663 // timestamp }, "TEST_ELIGIBILITY_2_API_URL": { "result": { "Succeeded": true, "ErrorCode": 0, "ErrorMessage": null, "Data": {}, }, "timestamp": 1749755362834 } } "submissionErrors": { // Site-level validation "errors": {}, "errorSummary": [] }, "submissionData": { // Site level successful submission data "submissionUsername" : "", // User's username "submissionEmail" : "", // User's email "submissionData": {}, // Raw data as submitted by the user in each page "submissionDataVersion": "", // The submission data version "printFriendlyData": [], // Print friendly data "rendererData" :{}, // Renderer data of the summary list "rendererVersion": "", // The renderer version "designSystemsVersion": "", // The design systems version "service": { // Service info "id": "", // Service id "title": {} // Service title multilingual object }, "referenceNumber": "", // Reference number } } } } ``` For a sample of submission data see at [README's sample submission data](README.md#submission-data) #### 4. User Data ```javascript "user": { "sub": "0000000950496523", // Subject identifier "name": "User1", // Display name "profile_type": "Individual", // User type "client_ip": "127.0.0.1", // IP address "unique_identifier": "0000XXXXXX", // User ID "email": "user@example.com", "phone_number": "99XXXXXX", "id_token": "xxxxxx", // CY Login tokens "access_token": "xxxxxx", "refresh_token": null } ``` ### Notes - Each section serves a specific purpose: - `Cookie`: Manages session lifetime and security - `CSRF Token`: Protects against cross-site request forgery. Protects against cross-site attacks. Valid for the whole session. - `Site Data`: Stores form data, validation, and submissions - `User Data`: Stores authenticated user information from CY Login - Data is managed through the data layer abstraction (`govcyDataLayer.mjs` more on that below) , which provides methods to: - Initialize data structures - Store and retrieve form data - Handle validation errors - Manage submission data ### Session Example The application uses a data layer abstraction to manage all session data: For example: ```json { "cookie": { // cookie data "originalMaxAge": 3600000, "expires": "2025-04-03T06:35:22.026Z", "secure": true, "httpOnly": true, "path": "/", "sameSite": "lax" }, "csrfToken": "ld524iof94w6vxifrpvny5i4bc1mo76l", // csrf token // site data "siteData": { // SITE: `nsf-2` "nsf-2": { "loadData": { // Data used to load the site when presaved "submissionId": "f97c091b-caa6-4e69-a515-f57043a704b0", "serviceId": "f11077f3-72c7-4eec-ab01-f07d9efed08d", "referenceValue": "0000000924107836", "submissionTs": "2025-08-07T17:05:26+03:00", "status": 0, "statusTs": "2025-08-07T17:05:34+03:00", "userId": "0000000924107836", "userType": 1, "userName": "", "userIdentifier": "0001013317", "userEmail": "", "submissionData": "{\"index\":{\"formData\":{\"certificate_select\":[\"birth\",\"permanent_residence\"]}}}", "dataStatus": 0 }, // INPUT DATA "inputData": { // SITE: `nsf-2`, PAGE: `bank-details` "bank-details": { // FORM DATA submitted "formData": { "IBAN": "CY12345678900001225", "SWIFT": "12345678" }, // PAGE validation errors "validationErrors": { // validation errors with messages "errors": { // Error on-> site: `nsf-2`, page: `bank-details`, element: `IBAN` "IBAN": { "id": "IBAN", "message": { "en": "Enter the IBAN", "el": "Εισαγάγετε το IBAN" }, "pageUrl": "" }, // Error on-> site: `nsf-2`, page: `bank-details`, element: `SWIFT` "SWIFT": { "id": "SWIFT", "message": { "en": "Enter the SWIFT", "el": "Εισαγάγετε το SWIFT" }, "pageUrl": "" } }, // Data submitted that produced the error on site: `nsf-2`, page: `bank-details` "formData": { "IBAN": "", "SWIFT": "" }, "errorSummary": [] } }, // SITE: `nsf-2`, PAGE: `answer-bank-boc` "answer-bank-boc": {}, // Multiple things page "qualifications": { "formData": [ // form data in an array { "institution": "Manchester university", "title": "MSc Computer Science", "year": "2005", "proof": { "sha256": "mock-sha256-hash", "fileId": "mock-file-id" } } ], "multipleDraft": {}, // Multiple things page draft when adding and uploading "validationErrors": { // this page has validation errors while adding "add": { "errors": { "institution": { "id": "institution", "message": { "el": "Το όνομα ιδρύματος πρέπει να αποτελείται μόνο από γράμματα, αριθμούς και ορισμένους άλλους χαρακτήρες", "en": "The institute name must consist only of letters, numbers and some other characters", "tr": "" }, "pageUrl": "" } }, "formData": { "institution": "*/*-/", "title": "", "year": "", "proof": "" }, "errorSummary": [] } } } }, // ELIGIBILITY "eligibility": { // Site eligibility cached results "TEST_ELIGIBILITY_1_API_URL": { // Eligibility 1 "result": { // results "Succeeded": true, "ErrorCode": 0, "ErrorMessage": null, "Data": {} }, "timestamp": 1749755264663 // timestamp }, "TEST_ELIGIBILITY_2_API_URL": { "result": { "Succeeded": true, "ErrorCode": 0, "ErrorMessage": null, "Data": {}, }, "timestamp": 1749755362834 } } }, // SITE: `dsf-plugin-v3` "dsf-plugin-v3": { // INPUT DATA "inputData": { "data-entry-checkboxes": { "formData": { "certificate_select": "permanent_residence", "_csrf": "ld524iof94w6vxifrpvny5i4bc1mo76l" } }, "data-entry-radios": {} }, // SITE LEVEL SUBMISSION ERRORS "submissionErrors": { "errors": { "data-entry-radiosmobile_select": { "id": "mobile_select", "message": { "el": "Επιλέξετε αν θέλετε να χρησιμοποιήσετε το τηλέφωνο που φαίνεται εδώ, ή κάποιο άλλο", "en": "Choose if you'd like to use the phone number shown here, or a different one", "tr": "" }, "pageUrl": "data-entry-radios" }, "data-entry-selectorigin": { "id": "origin", "message": { "el": "Επιλέξετε χώρα καταγωγής", "en": "Select country of origin", "tr": "" }, "pageUrl": "data-entry-select" }, "data-entry-textinputmobile": { "id": "mobile", "message": { "el": "Εισαγάγετε τον αριθμό του κινητού σας", "en": "Enter your mobile phone number", "tr": "" }, "pageUrl": "data-entry-textinput" } }, "errorSummary": [] } }, "site3": { // INPUT DATA "inputData": {}, // SITE LEVEL SUBMISSION ERRORS "submissionErrors": {}, "submissionData": { } //see sample above } }, // USER DATA "user": { "sub": "0000000950496523", "name": "User1", "profile_type": "Individual", "client_ip": "127.0.0.1", "unique_identifier": "0000XXXXXX", "email": "user@example.com", "phone_number": "99XXXXXX", "id_token": "xxxxxx", "access_token": "xxxxxx", "refresh_token": null } } ``` ### govcyDataLayer The [govcyDataLayer.mjs](./src/utils/govcyDataLayer.mjs) file provides a centralized abstraction for managing session data. It abstracts session handling, ensuring a structured and reusable approach for storing and retrieving data. #### Purpose - Centralizes session data management. - Simplifies handling of form data, validation errors, and submissions. - Ensures session structure is initialized and maintained. #### Example Usage 1. Storing Form Data ```js import * as dataLayer from "../utils/govcyDataLayer.mjs"; dataLayer.storePageData(req.session, "site1", "page1", { field1: "value1", field2: "value2" }); ``` 2. Retrieving Validation Errors ```js import * as dataLayer from "../utils/govcyDataLayer.mjs"; const validationErrors = dataLayer.getPageValidationErrors(req.session, "site1", "page1"); ``` More examples in the [govcyDataLayer.mjs](./src/utils/govcyDataLayer.mjs) file. --- ## Important flows ### 📝📥❌✅ Page post, validation checks and page view flow 1. Form submission -> `govcyFormsPostHandler` 2. Check field validations - If **validation passes**: - Form data stored in session under `siteData[siteId].inputData[pageUrl].formData` - User redirected to next page - If **validation fails**: - Errors stored in session under `siteData[siteId].inputData[pageUrl].validationErrors` - Form data preserved for repopulation - User redirected back to form with `#errorSummary-title` in URL 3. On page load -> `govcyPageHandler`: - Checks for validation errors - if **validation errors exist** - Populates form with data from validation errors - Displays error messages - Displays error summary - Clears errors after display - if **no validation errors exist** - Populates form if prepopulated data exists - Displays form ### 👀 Review page generation 1. User reaches review page 2. System: - Collects all form data from session - Formats data for display - Generates summary sections - Adds change links to each section ### 👀📥 Review page post flow 1. User submits review page -> `govcyReviewPostHandler` 2. System validates all pages: - Loops through each page in service - Gets stored form data from session - Validates each form element - Collects validation errors per page 3. If **validation errors exist**: - Stores errors in `siteData[siteId].submissionErrors` - Redirects back to review page with error summary - Displays error messages with links to relevant pages 4. If **validation passes**: - Prepares submission data - Generates reference number - Creates print-friendly data format - Stores submission in session under ` siteData[siteId].submissionData` - Clears the pages data from session. - Redirects to success page ### 🧩 MultipleThings Flow (Add / Edit / Delete / Hub) This pattern is used when a service needs to collect multiple entries of the same structure (e.g. academic qualifications, dependents, addresses). Each entry is stored as an object inside an array in the session under: ```js req.session.siteData[siteId].inputData[pageUrl].formData = [ {...}, {...}, ... ] ``` More at [REAMDE.md - Multiple things](README.md#multiple-things-pages-repeating-group-of-inputs) ### 🧱 File operations with multipleThings - The `/upload`, `/view-file`, and `/delete-file` routes in single mode are disabled for multipleThings pages. - File actions for multipleThings pages must use the indexed paths: - `POST /apis/:siteId/:pageUrl/multiple/add/upload` - `POST /apis/:siteId/:pageUrl/multiple/edit/:index/upload` - `GET /:siteId/:pageUrl/multiple/edit/:index/view-file/:elementName` - `POST /:siteId/:pageUrl/multiple/edit/:index/delete-file/:elementName` ### ✅ Task list internals - `govcyTaskListHandler` (GET) renders GOV.CY compliant task-list pages. It copies the page definition, injects CSRF/form controls (using `govcyResources.csrfTokenInput`), and feeds renderer sections with the output of `computeTaskListStatus`. Nested task lists are supported because the helper keeps a `visitedPages` set to prevent circular references. - `govcyFormsPostHandler` includes a dedicated branch for `page.taskList`. It recomputes all task statuses, and if any item is still pending it stores a synthesized `validationErrors.errorSummary` payload (per-page links + localized text) before redirecting back with `#errorSummary-title`. - Optional behaviours come directly from JSON config: `taskList.linkToContinue` enables the GOV.CY error-summary “continue anyway” link, while `showSkippedTasks` toggles visibility of pages skipped by conditions. - Multiple-things hubs flag themselves as “posted” (`dataLayer.setPagePosted`) even when `min=0`. The status helper interprets `(min:0, posted:true)` as “started”, so optional hubs can reach `COMPLETED` once validations pass (even with an empty list). - Custom pages report their status via `setCustomPageTaskStatus`. `computePageTaskStatus` checks the custom store first and avoids running standard validators when a custom page is encountered. - Whenever these middlewares detect an invalid configuration (missing `nextPage`, undefined `taskPages`, etc.) they delegate to `handleMiddlewareError`. This keeps Express error handling consistent across GET and POST flows, and it is the preferred way to bubble up configuration issues from new middleware. --- ## Logging configuration The application logs in the console with the following levels: `error`: System errors and crashes `warn`: Validation failures, auth issues `info`: Request completion, submissions `debug`: Form processing, session data Use the govcyLogger utility to configure logging levels. For example: ```js import { logger } from "../utils/govcyLogger.mjs"; logger.debug(`No pageUrl provided for siteId: ${siteId}`, req); logger.info("404 - Page not found.", err.message, req.originalUrl); // Log the error ``` You can set the `DEBUG` environment variable to `true` to enable debug logging. ---- ## API Integration The project integrates with external APIs for form submissions. The `govcyApiRequest` utility handles API communication with retry logic. ### Example Usage ```javascript import { govcyApiRequest } from "../utils/govcyApiRequest.mjs"; const response = await govcyApiRequest("post", "https://api.example.com/submit", submissionData); ``` ---- ## Testing To run tests, use the `npm test` command. This will run all the tests. To run all the tests both the mockAPI and the server need to be started. You can run individual tests with: - `npm run test:unit`: Runs unit tests - `npm run test:integration`: Runs integration tests. Needs mockAPI to be started. - `npm run test:package`: Runs package tests and checks the intallability. - `npm run test:functional`: Runs functional tests with puppeteer. Needs the server to be started. - `npm run coverage`: Runs all tests and generates a coverage report. - `npm run coverage:report`: Generates a coverage report. - `npm run coverage:badge`: Generates a coverage badge. To add new tests, create a new test file in the `test` directory, using the naming convention `testName.test.mjs`. See examples in the [tests](./test) directory. ### Mock API Server A mock API server is included for testing API integrations. It listens on port `3002` and simulates various API responses based on the request URL. To start the mock server: ```sh npm run start:mock ``` Example endpoints: - `/success`: Simulates a successful submission. - `/error102`: Simulates an error response with code 102. - `/error103`: Simulates an error response with code 103. - `/invalid-key`: Simulates a bad request. ## Project Structure Generated by https://gitdiagram.com/ ```mermaid flowchart TD %% Frontend Layer Browser["Browser (User)"]:::frontend govcyFS["govcy-frontend-renderer & govcy-design-system"]:::frontend PublicAssets["Static Assets (src/public)"]:::frontend %% Configuration & Env ConfigStore["Configuration Store (data/*.json & express-service-shema.json)"]:::config EnvVars["Environment Variables (.env*, secrets/.env)"]:::config %% External Services CYLogin["CY Login (OIDC)"]:::external DSFElig["DSF Eligibility API"]:::external DSFTempSave["DSF Temp-Save API"]:::external DSFSubmit["DSF Submission API"]:::external DSFFile["DSF File Services (upload/download/delete)"]:::external %% Express Server Entry ExpressEntry["Express Entry Points"]:::internal %% Express Server Middleware Layers subgraph "Express Server" ExpressEntry subgraph "Middleware Layers" subgraph Authentication A1["cyLoginAuth"]:::internal end subgraph "Config & Language" C1["govcyConfigSiteData"]:::internal C2["govcyCsrf"]:::internal C3["govcyLanguageMiddleware"]:::internal end subgraph "Eligibility & Data" E1["govcyServiceEligibilityHandler"]:::internal E2["govcyLoadSubmissionData"]:::internal end subgraph "File Handling" F1["govcyFileUpload"]:::internal F2["govcyFileViewHandler"]:::internal F3["govcyFileDeleteHandler"]:::internal end subgraph "Form Handling" H1["govcyFormsPostHandler"]:::internal end subgraph "Page Rendering" R1["govcyRoutePageHandler"]:::internal R2["govcyPageHandler"]:::internal R3["govcyPageRender"]:::internal end subgraph "Review & Success" RS1["govcyReviewPageHandler"]:::internal RS2["govcyReviewPostHandler"]:::internal RS3["govcySuccessPageHandler"]:::internal end subgraph "Logging & Error" L1["govcyHttpErrorHandler"]:::internal L2["govcyLogger"]:::internal L3["govcyRequestTimer"]:::internal end end Session["Session Store"]:::internal end %% Data Flows Browser -->|HTTPS GET/POST| ExpressEntry Browser -->|GET assets| PublicAssets Browser --> govcyFS ExpressEntry --> A1 A1 -->|OIDC redirect| CYLogin A1 --> C1 C1 --> C2 C2 --> C3 C3 --> E1 E1 -->|eligibility checks| DSFElig E1 --> E2 E2 -->|GET temp-save| DSFTempSave E2 --> F1 F1 -->|proxy upload| DSFFile F2 -->|proxy download| DSFFile F3 -->|proxy delete| DSFFile C3 --> H1 H1 -->|PUT temp-save| DSFTempSave H1 -->|session update| Session H1 --> R1 R1 --> R2 R2 --> R3 R3 --> Browser R3 --> govcyFS R3 --> RS1 RS1 --> RS2 RS2 -->|submission POST| DSFSubmit RS2 --> RS3 RS3 --> Browser ExpressEntry -->|load config| ConfigStore ExpressEntry -->|read/write| Session %% Click Events click ExpressEntry "https://github.com/gov-cy/govcy-express-services/blob/main/src/index.mjs" click A1 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/cyLoginAuth.mjs" click C1 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyConfigSiteData.mjs" click C2 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyCsrf.mjs" click C3 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyLanguageMiddleware.mjs" click E1 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyServiceEligibilityHandler.mjs" click E2 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyLoadSubmissionData.mjs" click F1 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyFileUpload.mjs" click F2 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyFileViewHandler.mjs" click F3 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyFileDeleteHandler.mjs" click H1 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyFormsPostHandler.mjs" click R1 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyRoutePageHandler.mjs" click R2 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyPageHandler.mjs" click R3 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyPageRender.mjs" click RS1 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyReviewPageHandler.mjs" click RS2 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyReviewPostHandler.mjs" click RS3 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcySuccessPageHandler.mjs" click L1 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyHttpErrorHandler.mjs" click L2 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyLogger.mjs" click L3 "https://github.com/gov-cy/govcy-express-services/blob/main/src/middleware/govcyRequestTimer.mjs" click PublicAssets "https://github.com/gov-cy/govcy-express-services/tree/main/src/public" click govcyFS "https://github.com/gov-cy/govcy-express-services/blob/main/src/resources/govcyResources.mjs" click ConfigStore "https://github.com/gov-cy/govcy-express-services/blob/main/express-service-shema.json" click EnvVars "https://github.com/gov-cy/govcy-express-services/blob/main/.env.development" %% Styles classDef frontend fill:#fddc5c,stroke:#e69a00,color:#000 classDef internal fill:#a0c4ff,stroke:#3d5a80,color:#000 classDef config fill:#d3d3d3,stroke:#666,color:#000 classDef external fill:#caffbf,stroke:#2f8f2f,color:#000 ```