# @fluid-example/app-integration-external-data
This example demonstrates how data from an external data source (e.g. a work tracking system) might be integrated with Fluid to enable more-real-time collaboration. For example, to allow collaborators to see proposed changes updating live before committing them back to the database.
Please note that the ideas explored here are experimental and under development. They are not yet recommended for broad use in production. When this changes, we will update the documents accordingly.
## Scenario
This example demonstrates a scenario in which the "source of truth" of the customer data lives in a service that is external to the Fluid service. Customers can then:
1. Import the external data into a Fluid collaboration session.
2. Export data from a Fluid collaboration session back to the source of truth.
3. Sync updates between Fluid and the source of truth in as close to real-time as the scenario allows.
In this case, the Fluid collaboration session serves as a "drafting surface" in which clients collaborate to create a draft of the data and then send the saved data back to the source of truth for long term storage.
## Strategy overview
In order to accomplish the goals above, we have split up the responsibilities into a few different pieces:
### External Data Service
Many services that would hold the "source of truth" data offer explicit commit style interfaces (e.g. via REST call or similar) which are not well suited to rapid updates.
However, they often expose third-party integration via REST APIS for querying and manipulating data, as well as webhooks for watching updates to the data.
This repo contains a service that mocks the external "source of truth" data server. This mock service offers a REST API collection and webhook interfaces in `./src/mock-external-data-service`. The API requests served by this "external" service are the following:
1. POST `/register-for-webhook?externalTaskListId=XXX`. Required body parameters: `url` (string). Registers the sender's URL to receive notifications when the external task-list data changes. Currently, the Customer Service registers its `/external-data-webhook` endpoint here to be called when data changes.
2. GET `/fetch-tasks/:externalTaskListId`: Fetches the task list from the external data store. Called by the Fluid client.
3. POST `/set-tasks/:externalTaskListId`: Updates external data store with new tasks list (complete override). Called by the Fluid client.
Find the details of the API in the [External Data Service README](./src/mock-external-data-service/README.md)
### Customer Service
Next we need a customer service that functions as the intermediary between the External Data Service and the Fluid Service. This server is responsible for authenticating to the external service on the customer's behalf. It registers to the External Data Service webhooks and listens for incoming changes. It also acts as a translation layer, translating to and from the External Data Service expected format and the Fluid Service's expected format.
In this example, the Customer Service contains the following endpoints:
1.POST `/register-session-url`. Required body parameters: `containerUrl` (string), `externalTaskListId` (string). Creates an entry in the Customer Service of the mapping between the container and the external resource id. It then calls the External Service's `/register-for-webhook` endpoint to call it's own `/external-data-webhook?externalTaskListId=XXX` endpoint (details below), in order to notify the Fluid Service containers subscribed to that externalTaskListId of the corresponding change.
2. POST `/external-data-webhook`. Required querystring parameters: `externalTaskListId`(string). This gets called by the External Data Service when there's been a change to the data and causes the Customer Service to in turn call the `/broadcast-signal` endpoint in the Fluid Service. Note that this is a route established by the Customer Service to be used exclusively as a subscription endpoint for the external data service's webhook and should be considered a private implementation detail to the service. Customers may choose to implement this differently.
Find the details of the API in the [Customer Service README](./src/mock-customer-service/README.md).
### Fluid Service
The `broadcast-signal` endpoint is new and still under construction. We will update it here once it is available to use.
1. POST `/broadcast-signal`. Required body parameters: `containerUrl` (string), `externalTaskListId` (string), `taskData`(ITaskData). This endpoint is called by the Customer Service when it needs the Fluid Service to notify the Fluid clients that there has been a change to the upstream data. The body must contain the `containerUrl`, from which we can extract a few pieces relevant to the Fluid service: the `socketStreamUrl`, the `containerId` (sometimes known as the `documentId`), and the `tenantId`. The Fluid service will broadcast a signal of type `SignalType.RuntimeSignal` to the clients to alert them of the change. In this way, it "echoes" the webhook call from the External Data Service to the Customer Service.
This example uses the tinylicious driver to stub out what changes will be necessary in the odsp-driver. The prototype of the full signal and driver flow can be seen in this [`dev/external-data-prototyping` branch to main comparison](https://github.com/microsoft/FluidFramework/compare/main...dev/external-data-prototyping).
#### Fluid Client
On receiving the signal, the clients (or elected leader client) can then send a fetch call to retrieve the information and display it to screen by making a call to the external data server's GET `/fetch-tasks` endpoint.
The client can then display the diff on the screen and the users can choose how to reconcile the changes.
Once the changes are reconciled, the collaboration session can continue as expected, and when the collaboration session is ready to be closed, the clients can simply Save Changes to write back to the External Data Source by making a request to the External Data Server's POST `/set-tasks` endpoint.
### Functional Flows
#### Set up on client joining a collaboration session
#### Data changes on External Service
#### Collaboration session ends
### Concepts of data in this repository
A few useful concepts to understand in implementing conflict resolution in this app:
Task - This is the unit that can be edited and attributed to an author. It is also the level at which conflict resolution currently takes place. The task holds knowledge of the local edits as well as the external edits and can display one or both to the screen.
TaskList - This can be compared to a "board" that holds all of the tasks. It is the larger visible entity to show up in a component within the app. The app stores two types of TaskLists - a "draft" version and a "saved version". More on these below.
TaskList - This can be compared to a "board" that holds a list of related tasks; for example, the results of a particular query. It is the larger visible entity to show up in a component within the app. The app stores two types of TaskLists - a "draft" version and a "saved version". More on these below.
ITaskData - This is similar to Task except that it is the External Data Services model of the task.
ITaskListData - This is similar to TaskList except that it is the External Data Services model of the taskList.
SavedData - Data that comes in fresh from the external data source is first stored in a [SharedMap](https://fluidframework.com/docs/data-structures/map/) known as "SavedData".
DraftData - Local collaboration state between the Fluid clients is stored in a SharedMap known as "DraftData". This is known as draft data because we are treating the Fluid collaboration session as a drafting surface.
## Getting Started
Complete these steps to run the example:
1. Run `corepack enable` to enable [Corepack](https://nodejs.org/docs/latest-v16.x/api/corepack.html).
2. From the `FluidFramework` root directory, run `pnpm install`.
3. From the `FluidFramework` root directory, run `pnpm run build:fast --nolint`.
- To build only this package, add the package name to the command:
`pnpm run build:fast --nolint @fluid-example/app-integration-external-data`
4. Run `pnpm start` from this directory.
5. Open in a web browser.
To run the example with SharePoint, complete these steps:
1. Follow the [webpack-fluid-loader instructions](https://github.com/microsoft/FluidFramework/blob/main/examples/utils/webpack-fluid-loader/README.md#sharepoint) to get authentication credentials.
2. Run `pnpm start:spo` or `pnpm start:spo-df` from this directory.
3. Open in a web browser.
## Testing
```bash
npm run test:playwright
```
To run the tests in a visible browser with the Playwright inspector:
```bash
npm run test:playwright -- --headed --debug
```
## Contribution Guidelines
You can [contribute](https://github.com/microsoft/FluidFramework/blob/main/CONTRIBUTING.md) to Fluid Framework in these ways:
- Answer questions in [GitHub Discussions](https://github.com/microsoft/FluidFramework/discussions).
- [Submit bug reports](https://github.com/microsoft/FluidFramework/issues) and help verify fixes.
- Review [source code changes](https://github.com/microsoft/FluidFramework/pulls).
- [Contribute bug fixes](https://github.com/microsoft/FluidFramework/blob/main/CONTRIBUTING.md).
For detailed instructions, read the [repo documentation](https://github.com/microsoft/FluidFramework/blob/main/docs/content/Home.md).
This project follows the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information, read the [Code of Conduct frequently asked questions](https://opensource.microsoft.com/codeofconduct/faq/).
For questions or comments, contact .
This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services.
Use of these trademarks or logos must follow Microsoft’s [Trademark & Brand Guidelines](https://www.microsoft.com/trademarks).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
## Help
Read the [Fluid Framework documentation](https://fluidframework.com/docs/) for information about Fluid Framework concepts and APIs.
To request information that the documentation does not contain, [create an issue](https://github.com/microsoft/FluidFramework/blob/main/docs/content/Contributing/Submitting-Bugs-and-Feature-Requests.md).
## Trademark
This project may contain Microsoft trademarks or logos for Microsoft projects, products, or services.
Use of these trademarks or logos must follow Microsoft's [Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.