import { useWorkspace, useWorkspaceRole } from "../../lib/context"; import { EntitlementStatus } from "../Billing/EntitlementStatus"; import { useEntitlements } from "../../lib/entitlements"; import { get } from "../../lib/useApi"; import { DestinationConfig, FunctionConfig, StreamConfig } from "../../lib/schema"; import React, { useEffect, useState } from "react"; import { SomeZodObject, z } from "zod"; import { ConfigurationObjectLinkDbModel } from "../../prisma/schema"; import { useRouter } from "next/router"; import { assertTrue, getLog, requireDefined } from "juava"; import { Alert, Button, Input, InputNumber, Radio, Switch, Tooltip } from "antd"; import { BaseBulkerConnectionOptions, getCoreDestinationType } from "../../lib/schema/destinations"; import { confirmOp, copyTextToClipboard, feedbackError, feedbackSuccess } from "../../lib/ui"; import FieldListEditorLayout, { EditorItem } from "../FieldListEditorLayout/FieldListEditorLayout"; import { DataLayoutType } from "@jitsu/protocols/analytics"; import { Activity, Copy } from "lucide-react"; import { useBilling } from "../Billing/BillingProvider"; import { hasIdentityStitching, IDENTITY_STITCHING_FUNCTION_ID, IDENTITY_STITCHING_OFF_WARNING, } from "../../lib/shared/plan-features"; import styles from "./ConnectionEditorPage.module.css"; import { Htmlizer } from "../Htmlizer/Htmlizer"; import { FunctionsSelector } from "../FunctionsSelector/FunctionsSelector"; import { Expandable } from "../Expandable/Expandable"; import { useStoreReload } from "../../lib/store"; import { DestinationSelector } from "../Selectors/DestinationSelector"; import { SourceSelector } from "../Selectors/SourceSelector"; import { FunctionVariables } from "../FunctionsDebugger/FunctionVariables"; import { EditorToolbar } from "../EditorToolbar/EditorToolbar"; import JSON5 from "json5"; import { toURL } from "../../lib/shared/url"; import { BackButton } from "../BackButton/BackButton"; import { JitsuButton } from "../JitsuButton/JitsuButton"; const log = getLog("ConnectionEditorPage"); type ConnectionOptionsType = Partial & { [key: string]: any }; type EditorProps = { value?: T; disabled?: boolean; onChange: (value: T) => void; }; type EditorComponent = React.FC & P>; const DataLayoutEditor: EditorComponent = props => ( props.onChange(val.target.value)} >
{(props.fileStorage || props.value === "passthrough") && (
Original
Keep original event structure.
)}
Single Table{!props.fileStorage ? (recommended) : <>}
The data is written into a table events. Field names and naming convention are same as in Segment
Segment
This data layout emulates segment tables: pages, identify, etc
Legacy Jitsu
Table name events. Field names are as in Jitsu 1.0
); export const BatchOrStreamEditor: EditorComponent = ({ value, disabled, onChange, limitations, }) => { const [streamModeState, setStreamModeState] = useState<"locked" | "disabled" | "allowed">("allowed"); const [streamModeCaveats, setStreamModeCaveats] = useState(); useEffect(() => { if (limitations?.streamModeLocked) { setStreamModeState(value !== "stream" ? "locked" : "allowed"); setStreamModeCaveats(limitations?.streamModeLocked); } else if (limitations?.streamModeDisabled) { setStreamModeState("disabled"); setStreamModeCaveats(limitations?.streamModeDisabled); } else { setStreamModeState("allowed"); setStreamModeCaveats(undefined); } }, [limitations, value]); return ( onChange(e.target.value)} >
Stream {streamModeState === "locked" && ( )}
{streamModeCaveats}
Batch
); }; export const SyncFrequencyEditor: EditorComponent = ({ value, disabled, onChange, }) => ( onChange(v as number)} /> ); type TextEditorComponent = EditorComponent; type SwitchComponentType = EditorComponent; const TextEditor: TextEditorComponent = ({ rows, value, onChange, disabled, className }) => { if (rows && rows > 1) { return ( onChange(e.target.value)} /> ); } else { return ( onChange(e.target.value)} /> ); } }; export const SwitchComponent: SwitchComponentType = ({ value, disabled, onChange, className }) => { return ( onChange(checked)} /> ); }; function hasZodFields(zod: SomeZodObject, ...fields: (keyof ConnectionOptionsType)[]) { for (const f of fields) { if (!zod.shape.hasOwnProperty(f)) { return false; } } return true; } function ConnectionEditor({ streams, destinations, links, functions, }: { streams: StreamConfig[]; destinations: DestinationConfig[]; links: z.infer[]; functions: FunctionConfig[]; }) { const router = useRouter(); const existingLink = router.query.id ? links.find(link => link.id === router.query.id) : undefined; assertTrue(streams.length > 0, `Streams list is empty`); assertTrue(destinations.length > 0, `Destinations list is empty`); const [loading, setLoading] = useState(false); const workspace = useWorkspace(); const role = useWorkspaceRole(); const canEdit = role.editEntities; const billing = useBilling(); // JITSU-228: Identity Stitching is Enterprise-only. Resolved by the server // rather than from useBilling(), which is unavailable on an EE install // without Firebase, where the server still enforces. Self-hosted consoles have // no plans and the endpoint allows everything there. const entitlements = useEntitlements(); const identityStitchingPlanTooLow = entitlements.identityStitching === false; const [dstId, setDstId] = useState(existingLink?.toId || destinations[0].id); const [srcId, setSrcId] = useState(existingLink?.fromId || streams[0].id); const destination = requireDefined( destinations.find(d => d.id === dstId), `Destination ${dstId} not found in ${destinations.map(d => d.id).join(", ")}` ); const destinationType = getCoreDestinationType(destination.destinationType); const connectionOptionsZodType = destinationType.connectionOptions; // parse() fills the schema's 60m default for an absent `frequency`. That's right // for a new connection (the value is persisted on save), but an existing link // saved without `frequency` really runs on bulker's fallback period // (BATCH_RUNNER_DEFAULT_PERIOD_SEC, 1m in cloud) — keep the field empty so the // editor shows that instead of a value that was never stored. const [connectionOptions, setConnectionOptions] = useState(() => { const parsed = connectionOptionsZodType.parse(existingLink?.data || {}); if (existingLink && (existingLink.data as any)?.frequency == null) { parsed.frequency = undefined; } return parsed; }); //Once destination is changed, we need to update default connection options const updateConnectionOptions = (selectedDestination: string) => { const destination = requireDefined( destinations.find(d => d.id === selectedDestination), `Destination ${selectedDestination} not found in ${destinations.map(d => d.id).join(", ")}` ); const newConnectionOptions = getCoreDestinationType(destination.destinationType).connectionOptions.parse({}); log.atDebug().log("Updating connection options to", newConnectionOptions); setConnectionOptions(newConnectionOptions); }; const updateOptions: (patch: Partial) => void = patch => { log.atDebug().log("Patching connection options with", patch, " existing options", connectionOptions); setConnectionOptions({ ...connectionOptions, ...patch }); }; const [limitations, setLimitations] = useState({}); const reloadStore = useStoreReload(); useEffect(() => { if (connectionOptionsZodType) { const description = connectionOptionsZodType.description; if (description) { try { const obj = JSON.parse(description); setLimitations(obj.limitations || {}); } catch (e) { console.error(e); } } else { setLimitations({}); } } }, [connectionOptionsZodType]); const usesBulker = destinationType.usesBulker || destinationType.id === "webhook"; const configItems: EditorItem[] = [ { name: "Enabled", component: ( { updateOptions({ disabled: !enabled }); }} /> ), }, { name: existingLink ? "Select Source" : "Source", documentation: "Select destination to connect", component: ( ), }, { name: existingLink ? "Select Destination" : "Destination", documentation: existingLink ? "You can't change destination of existing connection. Please create a new one" : "Select destination to connect", component: ( { setDstId(id); updateConnectionOptions(id); }} /> ), }, ]; if (hasZodFields(connectionOptionsZodType, "mode")) { configItems.push({ name: "Sync Mode", documentation: ( <>

stream mode sends data to destination as soon as it is received from source, however it's not scalable and not going to work for more that ~10 events per second.

batch mode sends data to destination in batches. It's scalable and recommended for most cases.

), component: ( <> updateOptions({ mode })} /> {destinationType.id === "webhook" && destination.signatureMethod && destination.signatureMethod !== "none" && connectionOptions.mode === "batch" && ( )} ), }); } if (hasZodFields(connectionOptionsZodType, "mode") && connectionOptions.mode === "batch") { configItems.push({ name: "Sync Frequency", component: ( updateOptions({ frequency })} /> ), }); configItems.push({ name: "Max Batch Size", documentation: ( <> Maximum number of events to send in one batch. If there are more events in queue than 'Batch Size', events will be sent as a sequence of batches with provided size. ), component: ( updateOptions({ batchSize: batchSize ?? 10000 })} /> ), }); } if (hasZodFields(connectionOptionsZodType, "dataLayout")) { configItems.push({ documentation: <>Data layout defines how data is written to database, name: "Data Layout", component: ( {" "} { if (existingLink) updateOptions({ dataLayout }); else { if (dataLayout === "jitsu-legacy") { updateOptions({ dataLayout, primaryKey: "eventn_ctx_event_id", timestampColumn: "_timestamp", keepOriginalNames: true, }); } else { updateOptions({ dataLayout, primaryKey: connectionOptionsZodType.parse({}).primaryKey || "message_id", timestampColumn: "timestamp", }); } } }} value={connectionOptions.dataLayout || "segment-single-table"} /> ), }); } if (hasZodFields(connectionOptionsZodType, "primaryKey")) { configItems.push({ group: "Advanced", name: "Primary Key", documentation: ( <> The unique field that should present in any event. In case of duplicated values, the new event will either replace the old one or be dropped as 'failed' event depending on 'Deduplicate' option. Don't change this field unless you know what you are doing.

Keep in mind that for data warehouses, all field names get translated to snake_case. For example,{" "} messageId get translated to message_id. ), component: ( {" "} { updateOptions({ primaryKey, ...(primaryKey === "" ? { deduplicate: false } : {}) }); }} /> ), }); } if (hasZodFields(connectionOptionsZodType, "deduplicate")) { configItems.push({ name: "Deduplicate", documentation: ( <> Deduplicate events with repeated values of 'Primary Key'. If enabled, the new event with the same 'Primary Key' value will replace the old one. If disabled, the new event will be dropped as 'failed' event. ), component: (
{ let functions = connectionOptions.functions ?? []; if (!deduplicate) { // remove user recognition function when deduplication is disabled functions = functions.filter(f => f.functionId !== IDENTITY_STITCHING_FUNCTION_ID); } updateOptions({ deduplicate, functions }); }} /> {destinationType.id === "bigquery" && (
Deduplication relies on reading data from the database. Using the Deduplication feature may therefore lead to higher BigQuery costs. These additional costs increase linearly with the frequency of syncs.
)}
), }); } if (hasZodFields(connectionOptionsZodType, "timestampColumn")) { configItems.push({ group: "Advanced", documentation: ( <>The field that represents timestamp. Depending on database, the field might be used to partition data ), name: "Timestamp Column", component: ( {" "} { updateOptions({ timestampColumn }); }} /> ), }); } if ( hasZodFields(connectionOptionsZodType, "deduplicateWindow") && (destinationType.id === "bigquery" || destinationType.id === "redshift" || destinationType.id === "snowflake") ) { configItems.push({ group: "Advanced", documentation: ( <> To optimize query performance, Jitsu detects the timestamp range of each incoming batch and checks for duplicates only within that range in the historical data.

This setting limits the maximum date range over which deduplication is applied. It can reduce processing time and costs when ingesting batches with a wide range of timestamps, by skipping deduplication for older records. ), name: "Deduplicate Window", component: ( updateOptions({ deduplicateWindow: deduplicateWindow ?? 365 })} /> ), }); } if (hasZodFields(connectionOptionsZodType, "functions", "deduplicate") && !limitations?.identityStitchingDisabled) { // JITSU-228: a connection that already has Identity Stitching on keeps it — // only switching it on is gated, matching assertIdentityStitchingAllowed on // the server. That leaves a grandfathered connection free to turn it off, // and the server will refuse to turn it back on afterwards — so turning it // off on such a plan asks first. const identityStitchingOn = hasIdentityStitching(connectionOptions); // What the stored connection has, not the form: the server compares against // the stored connection, so switching it off and back on before Save is a // no-op there. Locking on the form state instead would grey the switch the // instant it was turned off and leave the mistake impossible to undo. const identityStitchingSaved = hasIdentityStitching(existingLink?.data); const identityStitchingLocked = entitlements.identityStitching !== true && !identityStitchingSaved; configItems.push({ group: "Advanced", documentation: ( <> Identity Stitching Function retroactively updates data rows of anonymous user events with userId and traits as soon as user sings in. For correct work 'Deduplicate' option must be enabled. ), name: "Identity Stitching", component: (
{identityStitchingPlanTooLow && !identityStitchingSaved && ( // Inline, not in `documentation`: that is rendered behind a help // icon by DocumentedLabel, so a locked user would see a greyed-out // switch and no reason for it — and no hover at all on touch.
Available on the Enterprise plan.{" "} Contact sales {" "} to enable it.
)} {entitlements.identityStitching === null && !identityStitchingSaved && ( )} { // Only on an explicit "plan too low": with the lookup unknown (null) // nothing is known about whether it could be turned back on. if (!ur && identityStitchingOn && identityStitchingPlanTooLow) { if (!(await confirmOp(IDENTITY_STITCHING_OFF_WARNING))) { return; } } const f = (connectionOptions.functions ?? []).filter( f => f.functionId !== IDENTITY_STITCHING_FUNCTION_ID ); if (ur) { f.push({ functionId: IDENTITY_STITCHING_FUNCTION_ID, }); } updateOptions({ functions: f }); }} />
), }); } if (hasZodFields(connectionOptionsZodType, "schemaFreeze") && existingLink) { configItems.push({ group: "Advanced", documentation: ( <> By default, Jitsu automatically creates table columns for any new properties in events. If your table is already in desired state and you want to control further table schema changes you can enable{" "} Schema Freeze
With Schema Freeze enabled Jitsu will no longer apply changes to table schema, but incoming data for any properties that don't have corresponding columns will be stored in _unmapped_data column in JSON format. ), name: "Schema Freeze", component: ( { updateOptions({ schemaFreeze: sf }); }} /> ), }); } if (hasZodFields(connectionOptionsZodType, "keepOriginalNames") && connectionOptions.dataLayout !== "jitsu-legacy") { configItems.push({ group: "Advanced", documentation: ( <> By default, Jitsu translate all event properties to snakeCase, e.g.: myPropertyName is translated to my_property_name.
Enable this option if you want to keep original names ), name: "Keep Original Names", component: ( {" "} { const opts = { keepOriginalNames: sf } as any; if (sf && connectionOptions.primaryKey === "message_id") { opts.primaryKey = "messageId"; } else if (!sf && connectionOptions.primaryKey === "messageId") { opts.primaryKey = "message_id"; } updateOptions(opts); }} /> ), }); } if (hasZodFields(connectionOptionsZodType, "clickhouseSettings") && destinationType.id === "clickhouse") { configItems.push({ group: "Advanced", name: "Clickhouse Settings", documentation: ( <> Clickhouse Settings {" "} are used to configure Clickhouse connection. Format is key=value separated by new line. This settings will be merged with Parameters configured in destination. ), component: ( { updateOptions({ clickhouseSettings }); }} /> ), }); } if (hasZodFields(connectionOptionsZodType, "events")) { configItems.push({ documentation: ( <> Events that trigger tag insertion. Base events page, identify, group,{" "} track, and custom event names used with track call can be provided.
One per line ), name: "Events Allowlist", component: ( { updateOptions({ events }); }} /> ), }); } if (hasZodFields(connectionOptionsZodType, "hosts")) { configItems.push({ documentation: ( <> Hosts on witch tag will be inserted. You can use a wildcard as in *.domain.com. To include domain and all subdomain add two entries: *.domain.com and domain.com
One per line
Use * for all hosts ), name: "Hosts Allowlist", component: ( { updateOptions({ hosts }); }} /> ), }); } // if (hasZodFields(connectionOptionsZodType, "multithreading")) { // configItems.push({ // group: "Advanced", // documentation: ( // <> // Use multiple threads to send data to the endpoint. Faster, but doesn't guarantee order of events. Recommended // on high volumes of data. // // ), // name: "Multithreading", // component: ( // { // updateOptions({ multithreading: m }); // }} // /> // ), // }); // } return (

{(existingLink ? "Edit" : "Create") + " connection"}

{existingLink && (
, href: "#", onClick: () => { copyTextToClipboard(existingLink.id); feedbackSuccess("Copied to clipboard"); }, }, { title: "Logs", icon: , href: toURL(`/${workspace.slugOrId}/data`, { query: JSON5.stringify({ activeView: usesBulker ? "bulker" : "function", viewState: { [usesBulker ? "bulker" : "function"]: { actorId: existingLink.id } }, }), }), }, ].filter(Boolean) as any } className="mb-4" />
)}
Functions} hideArrow={false} caretSize="1.5em" contentLeftPadding={false} > s.id === srcId) || streams[0]} destination={destinations.find(d => d.id === dstId) || destinations[0]} selectedFunctions={connectionOptions.functions} onChange={enabledFunctions => { const nonUdfs = (connectionOptions.functions ?? []).filter(f => !f.functionId.startsWith("udf.")); const enabledF = enabledFunctions.map(f => ({ functionId: `udf.${f.id}` })); if (nonUdfs.length > 0) { enabledF.push(...nonUdfs); } updateOptions({ functions: enabledF }); }} /> Environment Variables} hideArrow={false} caretSize="1.5em" contentLeftPadding={false} >
Provided variables can be used inside functions via process.env object. For example:{" "} process.env.DEBUG
updateOptions({ functionsEnv })} />
{existingLink && ( { if (await confirmOp("Are you sure you want to unlink this site from this destination?")) { setLoading(true); try { await get(`/api/${workspace.id}/config/link`, { method: "DELETE", query: { id: existingLink.id }, }); await reloadStore(); feedbackSuccess("Successfully unliked"); router.back(); } catch (e) { feedbackError("Failed to unlink site and destination", { error: e }); } finally { setLoading(false); } } }} > Delete )}
{ setLoading(true); try { await get(`/api/${workspace.id}/config/link`, { body: { ...(existingLink ? { id: existingLink.id } : {}), fromId: srcId, toId: dstId, type: "push", data: connectionOptions, }, }); await reloadStore(); if (router.query.backTo) { router.push(`/${workspace.slugOrId}${router.query.backTo}`); } else { router.back(); } } catch (error) { feedbackError(`Can't link destinations`, { error }); } finally { setLoading(false); } }} > Save
); } export default ConnectionEditor;