Slack
- Join the discussion with the community on Slack [here](https://join.slack.com/t/coral-sql/shared_invite/zt-s8te92up-qU5PSG~spK33ovPPL5v96A)!
## Modules
**Coral** consists of following modules:
- **Coral-Hive**: Parses HiveQL (and Spark SQL, which is largely Hive-compatible) into Coral IR.
- **Coral-Trino**: Generates Trino SQL from Coral IR. Parsing Trino SQL into Coral IR is in progress.
- **Coral-Spark**: Generates Spark SQL from Coral IR (typically also valid HiveQL).
- **Coral-Dbt**: Applies Coral transformations to dbt models.
- **Coral-Incremental**: Rewrites a query into an incremental form for view maintenance.
- **Coral-Schema**: Derives the Avro schema of a view from its logical plan and the Avro schemas of its base tables.
- **Coral-Spark-Plan** [WIP]: Converts Spark plan strings into an equivalent logical plan.
- **Coral-Visualization**: Renders Coral SqlNode and RelNode trees to an image.
- **Coral-Service**: Exposes Coral via REST APIs (see [Coral-as-a-Service](#Coral-as-a-Service) for more details).
- **Coral-Benchmark** [WIP]: Cross-dialect translation correctness framework. Verifies translations via IR round-trip, target-engine EXPLAIN, and result-set comparison against real engines.
## Version Upgrades
This project adheres to semantic versioning, where the format x.y.z represents major, minor, and patch version upgrades. Consideration should be given to potential changes required when integrating different versions of this project.
**Major version Upgrade**
A major version upgrade represents a version change that introduces backward incompatibility by removal or renaming of classes.
**Minor version Upgrade**
A minor version upgrade represents a version change that introduces backward incompatibility by removal or renaming of methods.
Please carefully review the release notes and documentation accompanying each version upgrade to understand the specific changes and the recommended steps for migration.
## How to Build
Clone the repository:
```bash
git clone https://github.com/linkedin/coral.git
```
Build:
**Please note that this project requires Python 3 and Java 8 to run.** Set `JAVA_HOME` to the home of an appropriate version and then use:
```bash
./gradlew clean build
```
or, set the `org.gradle.java.home` gradle property to the Java home of an appropriate version as below:
```bash
./gradlew -Dorg.gradle.java.home=/path/to/java/home clean build
```
## Contributing
The project is under active development and we welcome contributions of different forms.
Please see the [Contribution Agreement](CONTRIBUTING.md).
## Resources
### Blog Posts
- [Coral: A SQL translation, analysis, and rewrite engine for modern data lakehouses](https://engineering.linkedin.com/blog/2020/coral), LinkedIn Engineering Blog, December 10, 2020.
- [Transport: Towards Logical Independence Using Translatable Portable UDFs](https://engineering.linkedin.com/blog/2018/11/using-translatable-portable-UDFs), LinkedIn Engineering Blog, November 14, 2018.
- [Dali Views: Functions as a Service for Big Data](https://engineering.linkedin.com/blog/2017/11/dali-views--functions-as-a-service-for-big-data), LinkedIn Engineering Blog, November 9, 2017.
### Talks
- [ViewShift: Dynamic Policy Enforcement for Every Data Lake](docs/talks/coral-viewshift.pdf), Databricks Data + AI Summit, June 10, 2025.
- [Harnessing Coral and Iceberg for Advanced Incremental View Maintenance](docs/talks/coral-incremental-view-maintenance.pdf), Iceberg Summit, May 2024.
- [Incremental View Maintenance with Coral, DBT, and Iceberg](https://www.slideshare.net/walaa_eldin_moustafa/incremental-view-maintenance-with-coral-dbt-and-iceberg), Iceberg Meetup, May 11, 2023.
- [Coral: A SQL translation and rewrite engine for modern data lakes](https://cdmsworkshop.github.io/2022/Proceedings/InvitedTalks/Abstract_WalaaEldinMoustafa.pdf), CDMS Workshop @ VLDB 2022, September 2022.
- [Coral & Transport UDFs: Building Blocks of a Postmodern Data Warehouse](https://www.slideshare.net/walaa_eldin_moustafa/coral-transport-udfs-building-blocks-of-a-postmodern-data-warehouse-229545076), Facebook HQ, February 28, 2020.
### Cited-by
- [SQL Telemetry & Intelligence – How we built a Petabyte-scale Data Platform with Fabric](https://blog.fabric.microsoft.com/en-us/blog/sql-telemetry-intelligence-how-we-built-a-petabyte-scale-data-platform-with-fabric), Raki Rahman, Microsoft Fabric Blog, December 16, 2025.
- [How Uber Migrated from Hive to Spark SQL for ETL Workloads](https://www.uber.com/blog/how-uber-migrated-from-hive-to-spark-sql-for-etl-workloads/), Kumudini Kakwani, Suprit Acharya, Nimesh Khandelwal, Akshayaprakash Sharma, Chintan Betrabet, and Aayush Chaturvedi, Uber Engineering Blog, June 12, 2025.
- [Data Guard: A Fine-grained Purpose-based Access Control System for Large Data Warehouses](https://arxiv.org/abs/2502.01998), Khai Tran et al., ICDE 2026 (arXiv:2502.01998), February 4, 2025.
- [OpenIVM: a SQL-to-SQL Compiler for Incremental Computations](https://arxiv.org/abs/2404.16486), Ilaria Battiston, Kriti Kathuria, and Peter Boncz, SIGMOD 2024 (arXiv:2404.16486), April 25, 2024.
## Coral-as-a-Service
**Coral-as-a-Service** or simply, **Coral Service** is a service that exposes REST APIs that allow users to interact with Coral without necessarily coming from a compute engine. Currently, the service supports an API for query translation between different dialects and another for interacting with a local Hive Metastore to create example databases, tables, and views so they can be referenced in the translation API. The service can be used in two modes: remote Hive Metastore mode, and local Hive Metastore mode. The remote mode uses an existing (already deployed) Hive Metastore to resolve tables and views, while the local one creates an empty embedded Hive Metastore so users can add their own table and view definitions.
### API Reference
#### /api/translations/translate
A **POST** API which takes JSON request body containing following parameters and returns the translated query:
- `sourceLanguage`: Input dialect (e.g., spark, trino, hive -- see below for supported inputs)
- `targetLanguage`: Output dialect (e.g., spark, trino, hive -- see below for supported outputs)
- `query`: SQL query to translate between two dialects
- [Optional] `rewriteType`: Type of Coral IR rewrite (e.g, incremental)
#### /api/catalog-ops/execute
A **POST** API which takes a SQL statement to create a database/table/view in the local metastore
(note: this endpoint is only available with Coral Service in local metastore mode).
### Instructions to use with examples
1. Clone [Coral repo](https://github.com/linkedin/coral)
```bash
git clone https://github.com/linkedin/coral.git
```
2. From the root directory of Coral, access the coral-service module
```bash
cd coral-service
```
3. Build
```bash
../gradlew clean build
```
#### To run Coral Service using the **local metastore**:
4. Run
```bash
../gradlew bootRun --args='--spring.profiles.active=localMetastore'
```
#### To run Coral Service using the **remote metastore**:
4. Add your kerberos client keytab file to `coral-service/src/main/resources`
5. Appropriately replace all instances of `SET_ME` in `coral-service/src/main/resources/hive.properties`
6. Run
```
../gradlew bootRun
```
You can also specify a custom location of `hive.properties` file through `--hivePropsLocation` as follows
```
./gradlew bootRun --args='--hivePropsLocation=/tmp/hive.properties'
```
Then you can interact with the service using your [browser](#coral-service-ui) or the [CLI](#coral-service-cli).
### Coral Service UI
After running `../gradlew bootRun --args='--spring.profiles.active=localMetastore'` (for local metastore mode)
or `../gradlew bootRun` (for remote metastore mode) from coral-service module, configure and start the UI.
Please note: The backend service runs on port 8080 (by default) and the web UI runs on port 3000 (by default).
#### To configure environment variables:
1. Create a `.env.local` file in the frontend project's root directory
2. Copy over the template from `.env.local.example` into the new `.env.local` file
3. Fill in the environment variable values in `.env.local`
#### Install the required packages in the frontend directory:
```bash
npm install
```
#### Now you can start the Coral Service UI by running:
```bash
npm run dev
```
Once compiled, the UI can be accessed from the browser at http://localhost:3000.
