# @memberjunction/core-entities

Strongly-typed entity classes, singleton engine caches, and domain logic for every entity in MemberJunction's core metadata schema. This package is the primary data-access layer that all MJ applications -- server and client -- depend on to interact with the MJ data model in a type-safe, validated way.

All entity subclasses in this package are **auto-generated by CodeGen**. Manual edits to files under `src/generated/` will be overwritten. Custom business logic is layered on top via extended classes in `src/custom/`.

## Installation

```bash
npm install @memberjunction/core-entities
```

## Overview

The package is structured into four layers that build on each other:

```mermaid
flowchart TD
    subgraph Generated["Generated Layer (CodeGen)"]
        ZOD["Zod Schemas<br/>Runtime validation"]
        ENT["272 Entity Subclasses<br/>Typed getters/setters"]
    end

    subgraph Custom["Custom Layer (Hand-Written)"]
        EXT["Extended Entities<br/>Business logic overrides"]
        RPE["Resource Permissions<br/>Access control"]
    end

    subgraph Engines["Engine Layer (Singletons)"]
        UIE["UserInfoEngine"]
        UVE["UserViewEngine"]
        DE["DashboardEngine"]
        CME["ComponentMetadataEngine"]
        AME["ArtifactMetadataEngine"]
        FE["FileStorageEngine"]
        EE["EncryptionEngineBase"]
        CE["ConversationEngine"]
        ME["MCPEngine"]
        TTC["TypeTablesCache"]
        KHE["KnowledgeHubMetadataEngine"]
    end

    subgraph Extraction["Artifact Extraction"]
        AER["Extract Rules<br/>Type definitions"]
        AEX["ArtifactExtractor<br/>Rule resolution + execution"]
    end

    ENT --> EXT
    ZOD --> ENT
    EXT --> Engines
    ENT --> Engines
    AER --> AEX

    style Generated fill:#2d6a9f,stroke:#1a4971,color:#fff
    style Custom fill:#2d8659,stroke:#1a5c3a,color:#fff
    style Engines fill:#7c5295,stroke:#563a6b,color:#fff
    style Extraction fill:#b8762f,stroke:#8a5722,color:#fff
```

### Generated Entity Classes

CodeGen scans the MemberJunction database schema and produces:

- **Zod schema objects** for every entity, providing runtime validation with field-level descriptions, related-entity annotations, and value-list constraints.
- **Entity subclasses** extending `BaseEntity<T>` with strongly-typed getter/setter properties for every column, typed `Load()` methods, and JSDoc annotations including SQL types, defaults, and foreign key references.

There are currently **272 generated entity classes** covering approximately 78,000 lines of code in a single `entity_subclasses.ts` file.

#### AIAgentNote Consolidation Schema (v5.30.x+)

Migration `V202604260056__v5.30.x__Memory_Consolidation_Schema.sql` adds five columns to `AIAgentNote` to support the Memory Manager consolidation pipeline. The generated `MJAIAgentNoteEntity` getters/setters reflect these in `entity_subclasses.ts`:

| Column | Type | Purpose |
|---|---|---|
| `ConsolidatedIntoNoteID` | uniqueidentifier (FK → AIAgentNote.ID, nullable) | When this note has been folded into a successor consolidation, points at the successor — preserves provenance after revocation |
| `ConsolidationCount` | int | Number of times this note has itself absorbed clusters; capped at `maxConsolidationCount=3` for drift prevention |
| `DerivedFromNoteIDs` | nvarchar(max) JSON array of UUIDs, nullable | For consolidated notes, the list of original sources the LLM merged into this one — enables anchored-mode drilling when a cluster reaches the cap |
| `ProtectionTier` | nvarchar (Immutable / Protected / Standard / Ephemeral) | Modulates decay rate and consolidation eligibility |
| `ImportanceScore` | float | Composite score driving Ebbinghaus decay and outlier auto-promotion |

Server-side persistence and vector-store invariant maintenance for these fields lives in [`@memberjunction/core-entities-server`](../MJCoreEntitiesServer/README.md#mjaiagentnoteentityserver-v530x). The pipeline that produces them is documented in [`@memberjunction/ai-agents`](../AI/Agents/README.md#consolidation-pipeline) and [`specs/001-memory-consolidation/spec.md`](../../specs/001-memory-consolidation/spec.md).

### Extended Entity Classes

Several entities require custom business logic beyond what CodeGen can generate. Extended classes use `@RegisterClass` to override the generated base class in MJ's class factory:

| Extended Class | Entity Name | Purpose |
|---|---|---|
| `UserViewEntityExtended` | User Views | Filter/column JSON parsing, WHERE clause generation, smart filter AI stub, permission checks |
| `DashboardEntityExtended` | Dashboards | Default config initialization, permission-based validation on save/delete |
| `EntityEntityExtended` | Entities | Auto-sets `AutoUpdateDescription` when description is manually changed |
| `EntityFieldEntityExtended` | Entity Fields | Prevents modification of database-reflected properties, auto-sets `AutoUpdateDescription` |
| `ComponentEntityExtended` | MJ: Components | Syncs derived fields from Specification JSON, keeps `spec` property in sync |
| `TemplateEntityExtended` | Templates | Content/param management, template input validation |
| `ScheduledActionEntityExtended` | Scheduled Actions | Auto-generates CronExpression from schedule type settings |
| `ListDetailEntityExtended` | List Details | Composite key handling, duplicate detection on save |
| `EnvironmentEntityExtended` | MJ: Environments | Exposes static `DefaultEnvironmentID` constant |
| `ResourcePermissionEntityExtended` | Resource Permissions | Notification workflow on permission request/approval |

### Singleton Engines

Engine classes extend `BaseEngine<T>` to provide cached, singleton access to groups of related entities. They load data once and keep it in memory, with auto-refresh when underlying data changes.

```mermaid
flowchart LR
    subgraph App["Application Code"]
        SVC["Services / Components"]
    end

    subgraph Engines["Singleton Engines"]
        UIE["UserInfoEngine"]
        DE["DashboardEngine"]
        RPE["ResourcePermissionEngine"]
    end

    subgraph DB["Database"]
        MJD["MJ Schema"]
    end

    SVC -->|".Instance"| Engines
    Engines -->|"Config() loads once"| DB
    Engines -->|"Cached data"| SVC

    style App fill:#2d6a9f,stroke:#1a4971,color:#fff
    style Engines fill:#7c5295,stroke:#563a6b,color:#fff
    style DB fill:#64748b,stroke:#475569,color:#fff
```

### Artifact Extraction System

A metadata-driven system for extracting structured attributes from artifact content using declarative JavaScript-based rules with hierarchical inheritance through parent artifact types.

## Key Features

- **272 strongly-typed entity classes** covering all MemberJunction core schema entities
- **Zod runtime validation** with field-level constraints and value list enums
- **Extended entity classes** with permission checks, workflow logic, and JSON state parsing
- **10 singleton engine caches** eliminating redundant database queries (including KnowledgeHubMetadataEngine for vector/content metadata)
- **Resource permission engine** with role-based access control (View/Edit/Owner levels)
- **Dashboard permission engine** with owner/direct/category permission hierarchy
- **User view engine** with filter-to-SQL conversion and smart filter AI support
- **Artifact extraction** with hierarchical rule inheritance and sandboxed execution
- **Encryption engine base** for field-level encryption metadata caching

## Usage

### Working with Entity Objects

Always use `Metadata.GetEntityObject<T>()` to create entity instances -- never instantiate entity classes directly. This ensures the MJ class factory resolves the correct (potentially extended) subclass.

```typescript
import { Metadata } from '@memberjunction/core';
import { UserEntity, ApplicationEntity } from '@memberjunction/core-entities';

const md = new Metadata();

// Load an existing record
const user = await md.GetEntityObject<UserEntity>('Users');
await user.Load('some-user-id');
console.log(user.Email);
console.log(user.FirstName);

// Create a new record
const app = await md.GetEntityObject<ApplicationEntity>('Applications');
app.NewRecord();
app.Name = 'My Application';
app.Description = 'A custom application';
await app.Save();
```

### Loading Collections with RunView

Use `RunView` with generics for loading collections of entity records:

```typescript
import { RunView } from '@memberjunction/core';
import { ActionEntity } from '@memberjunction/core-entities';

const rv = new RunView();
const result = await rv.RunView<ActionEntity>({
    EntityName: 'Actions',
    ExtraFilter: `Status='Active'`,
    OrderBy: 'Name',
    ResultType: 'entity_object'
});

if (result.Success) {
    for (const action of result.Results) {
        console.log(action.Name, action.Status);
    }
}
```

### Using the UserViewEntityExtended

The extended User View entity parses JSON state columns into structured objects and generates SQL WHERE clauses:

```typescript
import { Metadata } from '@memberjunction/core';
import { UserViewEntityExtended, ViewInfo } from '@memberjunction/core-entities';

// Load a view by name using the convenience class
const view = await ViewInfo.GetViewEntityByName('Active Contacts');

// Access parsed column and filter info
const columns = view.Columns;       // ViewColumnInfo[]
const filters = view.Filter;        // ViewFilterInfo[]
const sortInfo = view.ViewSortInfo;  // ViewSortInfo[]

// Check user permissions
if (view.UserCanEdit) {
    view.Name = 'Updated View Name';
    await view.Save(); // Auto-regenerates WhereClause from FilterState
}
```

### Using Singleton Engines

All engines follow the same pattern: get the singleton instance, call `Config()` to load data, then access cached properties.

```typescript
import {
    UserInfoEngine,
    DashboardEngine,
    ResourcePermissionEngine
} from '@memberjunction/core-entities';

// UserInfoEngine -- notifications, favorites, settings, applications
const userEngine = UserInfoEngine.Instance;
await userEngine.Config(false, contextUser);
const notifications = userEngine.UnreadNotifications;
const favorites = userEngine.UserFavorites;

// Debounced settings (batches rapid updates)
userEngine.SetSettingDebounced('ui-preference/theme', 'dark');

// DashboardEngine -- dashboards, categories, permissions
const dashEngine = DashboardEngine.Instance;
await dashEngine.Config(false, contextUser);
const accessible = dashEngine.GetAccessibleDashboards(userId);
const canEdit = dashEngine.CanUserEditDashboard(dashboardId, userId);

// ResourcePermissionEngine -- role-based resource access
const permEngine = ResourcePermissionEngine.Instance;
await permEngine.Config(false, contextUser);
const level = permEngine.GetUserResourcePermissionLevel(resourceTypeId, recordId, user);
// Returns 'View' | 'Edit' | 'Owner' | null
```

### Artifact Extraction

Extract structured attributes from artifact content using declarative rules:

```typescript
import { ArtifactExtractor } from '@memberjunction/core-entities';

// Resolve extract rules with inheritance from parent types
const rules = ArtifactExtractor.ResolveExtractRules([
    childArtifactType,   // Most specific (overrides parent rules by name)
    parentArtifactType   // Least specific
]);

// Extract attributes from content
const result = await ArtifactExtractor.ExtractAttributes({
    content: '{"subject": "Q4 Campaign", "body": "..."}',
    extractRules: rules,
    throwOnError: false,
    timeout: 5000
});

// Access extracted values
const name = ArtifactExtractor.GetStandardProperty(result.attributes, 'name');
const description = ArtifactExtractor.GetStandardProperty(result.attributes, 'description');

// Serialize for database storage
const serialized = ArtifactExtractor.SerializeForStorage(result.attributes);
```

### Encryption Metadata

The `EncryptionEngineBase` provides cached access to encryption keys, algorithms, and key sources:

```typescript
import { EncryptionEngineBase } from '@memberjunction/core-entities';

const engine = EncryptionEngineBase.Instance;
await engine.Config(false, contextUser);

// Validate a key is usable
const validation = engine.ValidateKey(keyId);
if (!validation.isValid) {
    console.error(validation.error);
}

// Get full key configuration (key + algorithm + source)
const config = engine.GetKeyConfiguration(keyId);
```

## API Reference

### Generated Entity Classes

Every generated entity class provides:

| Member | Description |
|---|---|
| `Load(ID, relationships?)` | Load a record by primary key |
| `Save(options?)` | Persist changes to the database |
| `Delete(options?)` | Delete the record |
| `NewRecord()` | Initialize a blank record for creation |
| `Validate()` | Run Zod schema validation |
| Typed getters/setters | One per database column, with JSDoc annotations |

### UserViewEntityExtended

| Member | Type | Description |
|---|---|---|
| `Filter` | `ViewFilterInfo[]` | Parsed filter state from JSON |
| `Columns` | `ViewColumnInfo[]` | Parsed column configuration from JSON |
| `ViewSortInfo` | `ViewSortInfo[]` | Parsed sort state |
| `OrderByClause` | `string` | SQL ORDER BY clause |
| `ViewEntityInfo` | `EntityInfo` | Metadata for the view's entity |
| `ParsedDisplayState` | `ViewDisplayState \| null` | View mode and display preferences |
| `UserCanEdit` | `boolean` | Whether current user can edit this view |
| `UserCanView` | `boolean` | Whether current user can view this view |
| `UserCanDelete` | `boolean` | Whether current user can delete this view |
| `SetDefaultsFromEntity(e)` | method | Initialize columns from entity metadata |
| `UpdateWhereClause()` | method | Regenerate WHERE clause from filter state |

### ViewInfo (Static Utility)

| Method | Description |
|---|---|
| `GetViewsForUser(entityId?, contextUser?)` | Load all views for a user |
| `GetViewID(viewName)` | Look up a view ID by name |
| `GetViewEntity(viewId, contextUser?)` | Get a cached view by ID |
| `GetViewEntityByName(viewName, contextUser?)` | Get a cached view by name |

### UserInfoEngine

| Member | Description |
|---|---|
| `UserNotifications` | Notifications for current user (newest first) |
| `UnreadNotifications` | Unread notifications only |
| `Workspaces` | User's workspaces |
| `UserApplications` | Installed applications (sorted by sequence) |
| `UserFavorites` | Favorited records (newest first) |
| `UserRecordLogs` | Recent record access history |
| `UserSettings` | User settings key-value pairs |
| `GetSetting(key)` | Get a setting value by key |
| `SetSetting(key, value)` | Create or update a setting |
| `SetSettingDebounced(key, value)` | Debounced setting update for rapid UI changes |
| `FlushPendingSettings()` | Force-save all debounced settings |
| `InstallApplication(appId)` | Install an application for the user |
| `DisableApplication(appId)` | Disable (but not delete) an application |
| `CreateDefaultApplications()` | Create records for default-for-new-user apps |
| `NotificationPreferences` | User's notification preferences |

### DashboardEngine

| Member | Description |
|---|---|
| `Dashboards` | All cached dashboards |
| `DashboardCategories` | All dashboard categories |
| `DashboardPartTypes` | Dashboard widget/part types |
| `GetDashboardPermissions(id, userId)` | Get effective permissions (owner/direct/category) |
| `CanUserReadDashboard(id, userId)` | Check read access |
| `CanUserEditDashboard(id, userId)` | Check edit access |
| `CanUserDeleteDashboard(id, userId)` | Check delete access |
| `GetAccessibleDashboards(userId)` | All dashboards user can read |
| `GetSharedDashboards(userId)` | Dashboards shared with user (not owned) |
| `GetAccessibleCategories(userId)` | Categories user can access |

### ResourcePermissionEngine

| Member | Description |
|---|---|
| `ResourceTypes` | All resource type definitions |
| `Permissions` | All resource permission records |
| `GetResourcePermissions(typeId, recordId)` | Get all permissions for a resource |
| `GetUserResourcePermissionLevel(typeId, recordId, user)` | Get highest permission level (View/Edit/Owner/null) |
| `GetUserAvailableResources(user, typeId?)` | Get all resources a user can access |

### UserViewEngine

| Member | Description |
|---|---|
| `AllViews` | All cached views |
| `GetViewById(id)` | Look up view by ID |
| `GetViewByName(name)` | Look up view by name |
| `GetViewsForEntity(entityId)` | Views for a specific entity |
| `GetViewsForCurrentUser()` | Views owned by current user |
| `GetAccessibleViewsForEntity(entityId)` | Owned + shared views with permission checks |
| `GetDefaultViewForEntity(entityId)` | Default view for an entity |

### ArtifactExtractor

| Method | Description |
|---|---|
| `ResolveExtractRules(typeChain)` | Resolve rules with hierarchical inheritance |
| `ExtractAttributes(config)` | Execute extraction rules against content |
| `SerializeForStorage(attributes)` | Convert attributes to database-storable format |
| `DeserializeFromStorage(stored)` | Convert stored attributes back to runtime objects |
| `GetStandardProperty(attributes, prop)` | Find a standard property value (name/description/etc.) |

### ConversationEngine

Centralized, reactive cache for conversations, conversation details (messages), and peripheral data such as agent runs. `ConversationEngine` is the single source of truth for conversation data across all UI consumers (chat area, sidebar, overlay, etc.), replacing per-component caching that previously lived in multiple scattered locations.

The engine is user-scoped and environment-filtered, so it manages its own caching instead of using `BaseEngine.Load()` bulk-load pattern.

```typescript
import { ConversationEngine } from '@memberjunction/core-entities';

// Initialize once at app startup
await ConversationEngine.Instance.Config(false, contextUser);

// Load conversations for the current user and environment
await ConversationEngine.Instance.LoadConversations(environmentId, contextUser);

// Subscribe to reactive conversation list changes
ConversationEngine.Instance.Conversations$.subscribe(conversations => {
    console.log(`${conversations.length} conversations loaded`);
});

// Load and cache details (messages) for a specific conversation
const details = await ConversationEngine.Instance.LoadConversationDetails(conversationId, contextUser);

// Instant cache reads (no DB round-trip)
const cached = ConversationEngine.Instance.GetCachedDetails(conversationId);

// CRUD operations that automatically update the reactive list
await ConversationEngine.Instance.CreateConversation('New Chat', environmentId, contextUser);
await ConversationEngine.Instance.PinConversation(conversationId, true, contextUser);
await ConversationEngine.Instance.ArchiveConversation(conversationId, contextUser);

// Cache management
ConversationEngine.Instance.InvalidateConversation(conversationId);
ConversationEngine.Instance.ClearCache();
```

Source: [`src/engines/conversations.ts`](src/engines/conversations.ts)

### KnowledgeHubMetadataEngine

Caches all Knowledge Hub-related metadata in a single singleton, providing fast lookups for entity documents, vector indexes, vector databases, content sources, content types, content source types, and content file types. Uses `BaseEngine` for automatic caching and entity-event auto-refresh.

```typescript
import { KnowledgeHubMetadataEngine } from '@memberjunction/core-entities';

const khEngine = KnowledgeHubMetadataEngine.Instance;
await khEngine.Config(false, contextUser);

// Cached data access (no DB round-trip)
const activeDocuments = khEngine.GetActiveEntityDocuments();
const entitiesWithDocs = khEngine.GetEntitiesWithDocuments(); // For dropdowns
const doc = khEngine.GetEntityDocumentById(docId);
const docsForEntity = khEngine.GetEntityDocumentsForEntity('Contacts');
const vectorIndex = khEngine.GetVectorIndexById(indexId);
const vectorDB = khEngine.GetVectorDatabaseById(dbId);

// Raw cached arrays
khEngine.EntityDocuments;    // All entity documents
khEngine.VectorIndexes;      // All vector indexes
khEngine.VectorDatabases;    // All vector databases
khEngine.ContentSources;     // All content sources
khEngine.ContentTypes;       // All content types
khEngine.ContentSourceTypes; // All content source types (Web, RSS, etc.)
khEngine.ContentFileTypes;   // All content file types (.pdf, .html, etc.)
```

Source: [`src/engines/knowledgeHubMetadata.ts`](src/engines/knowledgeHubMetadata.ts)

### Other Engines

| Engine | Purpose |
|---|---|
| `KnowledgeHubMetadataEngine` | Caches entity documents, vector indexes/databases, content sources/types/file types |
| `ConversationEngine` | Reactive cache for conversations, messages, and agent runs (user-scoped) |
| `EncryptionEngineBase` | Caches encryption keys, algorithms, and key sources |
| `FileStorageEngine` | Caches file storage accounts and providers |
| `MCPEngine` | Caches MCP servers, connections, and tools |
| `ComponentMetadataEngine` | Caches component definitions, libraries, and registries |
| `ArtifactMetadataEngine` | Caches artifact type definitions |
| `TypeTablesCache` | Caches entity relationship display components |

## Permission Model

The package implements a layered permission model used across views, dashboards, and resources:

```mermaid
flowchart TD
    subgraph Check["Permission Resolution Order"]
        direction TB
        OWN["1. Ownership Check<br/>Record owner gets full access"]
        ADM["2. Admin Check<br/>System admins get full access"]
        DIR["3. Direct Permission<br/>User-specific grants"]
        CAT["4. Category Permission<br/>Inherited from categories"]
        ROL["5. Role Permission<br/>Granted through user roles"]
    end

    OWN -->|"Not owner"| ADM
    ADM -->|"Not admin"| DIR
    DIR -->|"No direct grant"| CAT
    CAT -->|"No category grant"| ROL
    ROL -->|"No role grant"| DENY["Access Denied"]

    style Check fill:#2d6a9f,stroke:#1a4971,color:#fff
    style DENY fill:#64748b,stroke:#475569,color:#fff
```

Permission levels are hierarchical: **Owner** > **Edit** > **View**. The engine returns the highest applicable level across all sources (direct user grants and role-based grants).

## View Filter Architecture

The `UserViewEntityExtended` converts structured filter JSON into SQL WHERE clauses:

```mermaid
flowchart LR
    subgraph Input["Filter State (JSON)"]
        FS["{ logic: 'and',<br/>filters: [...] }"]
    end

    subgraph Process["UserViewEntityExtended"]
        PARSE["Parse FilterState<br/>JSON column"]
        GEN["GenerateWhereClause()<br/>Recursive group processing"]
        SQL["SQL WHERE clause"]
    end

    subgraph Modes["Filter Modes"]
        STD["Standard Filter<br/>UI-driven FilterState"]
        SMART["Smart Filter<br/>AI-generated from prompt"]
        CUSTOM["Custom Filter<br/>Admin-set WhereClause"]
    end

    FS --> PARSE --> GEN --> SQL
    Modes --> GEN

    style Input fill:#b8762f,stroke:#8a5722,color:#fff
    style Process fill:#2d6a9f,stroke:#1a4971,color:#fff
    style Modes fill:#2d8659,stroke:#1a5c3a,color:#fff
```

Supported filter operators: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `startswith`, `endswith`, `contains`, `doesnotcontain`, `isnull`, `isempty`, `isnotnull`, `isnotempty`. Filters support arbitrary nesting with `and`/`or` logic groups.

## Important: CodeGen and This Package

This package's `src/generated/entity_subclasses.ts` file is **entirely auto-generated** by the MemberJunction CodeGen system. Key implications:

- **Never edit generated files** -- changes will be overwritten on the next CodeGen run.
- **Custom logic goes in `src/custom/`** -- create an extended class with `@RegisterClass` to override the generated base.
- **Entity names matter** -- some entities use the `MJ: ` prefix (e.g., `MJ: Components`, `MJ: Environments`). Always verify names in the generated file's `@RegisterClass` decorators.
- **Zod schemas are authoritative** -- they reflect the exact database constraints including value lists, nullable fields, and default values.

## Dependencies

- [@memberjunction/core](../MJCore/readme.md) -- Base entity classes, metadata system, RunView, BaseEngine
- [@memberjunction/global](../MJGlobal/README.md) -- `@RegisterClass` decorator, global utilities
- [@memberjunction/ai](../AI/README.md) -- AI integration types (used by some engine classes)
- [@memberjunction/interactive-component-types](../InteractiveComponents/README.md) -- ComponentSpec type for component entity extensions
- [zod](https://zod.dev/) -- Runtime schema validation

## Related Packages

- [@memberjunction/core](../MJCore/readme.md) -- Framework foundation this package builds on
- [@memberjunction/core-entities-server](../MJCoreEntitiesServer/README.md) -- Server-side extensions of entities defined here (e.g., smart filter AI implementation)
- [@memberjunction/server](../MJServer/README.md) -- Server that uses these entities for API operations

## Contributing

See the [MemberJunction Contributing Guide](../../CONTRIBUTING.md) for development setup and guidelines.