Embedded Destination

Build a custom connector that looks and feels just like a native destination

Overview

The Embedded Destination framework allows Hightouch users to develop their own custom data connector and use it as an extension of Hightouch's existing library.

If you are looking to send data to a private API or a SaaS tool that Hightouch doesn't support, you can use Hightouch to send data to your own connector in a specific format, while continuing to use all of our alerting, reliability, and error handling systems.

To implement the destination, Hightouch will need users to provide information on what objects, modes, and fields are available in the destination, as well as how data is synced.

Example destinations

Our team has produced some example destinations for reference.

• Slack: Sends messages formatted with Liquid to a Slack channel as rows are added to the query results
• Mailchimp: Adds and updates contacts in a Mailchimp audience list

Setup

Embedded destinations in Hightouch are implemented by providing information of two steps:

• Sync configuration: Hightouch needs information on what objects, modes, and fields are available, as well as custom forms
• Sync execution: Hightouch sends data that needs to be synced to a method on your connector, which should format and send the data to the destination and return a success or error response

Sync configuration

Hightouch uses the following methods to render the configuration form.

These methods can be optionally implemented and used in any combination:

• list_objects: Retrieves a list of objects available in the destination (for example, users, organizations, etc.)
• list_modes: Retrieves a list of modes available in your implementation (for example, upsert, update, mirror, etc.)
• list_fields: Retrieves a list of object fields, including identifiers and custom fields
• list_options: Retrieves a JSON schema of custom configuration fields, which will be rendered into a form based on react-jsonschema-form
• validate: This method is called when a user attempts to save the configuration form, and is used to display errors that aren't handled automatically by default or by JSON schema

Example 1: Objects, modes, fields, and options

When all the methods are implemented and all the concepts are used in conjunction, there is a hierarchy where objects must be selected first before modes are shown, and modes must be selected before fields and options are shown.

This is because the mode options shown are conditional on the object selected, for example, "upsert" and "update" can be enabled for the "user" object, but only "update" is enabled for the "organizations" object.

This will render a form similar to the example above.

Example 2: Options only

When only list_options is implemented, Hightouch will render a form based on the JSON schema. Note that all formats such as dropdowns, arrays, text areas are available. In this case, only the list_options method is implemented.

The other methods such as list_objects aren't implemented, and therefore selecting an object is not required.

Sync execution

The Embedded Destination acts as a transformation and parsing layer between Hightouch and any API. Hightouch calls the following methods when a sync is run, either by schedule or manual run.

When a sync is run, Hightouch first calls for information on sync behavior, then sends sync requests of each operation (add, change, remove) with the data and configuration in batches of a specified size:

• behavior: Retrieves behavior settings such as batch size, which operations to send, etc. for the sync run
• add: This method is called with a batch of raw data from rows that have been added since the last sync run, as well as configuration details
• change: This method is called with a batch of raw data from rows that have changed since the last sync run, as well as configuration details
• remove: This method is called with a batch of raw data from rows that have been removed since the last sync run, as well as configuration details

Each of the add, change, remove methods should return a response with any errors encountered when sending the data to the destination API.

API reference

Implement the following methods in a JSON-RPC server available via a public URL.

The server can be implemented with packages such as:

• Jayson: A JSON-RPC server for node.js
• json-rpc: A JSON-RPC implementation written on Python

list_objects

This method should return the objects that are available to sync to in the destination API.

Request body

list_objects has no request body since it's the highest level method.

Response sample

{
  "objects": [
    { "label": "User", "id": "user", "description": "A user." },
    {
      "label": "Organization",
      "id": "organization",
      "description": "An organization."
    }
  ]
}

Properties of an object

list_modes

This method should return the sync modes that are available for a specific object (or no object).

Request body

Response sample

{
  "modes": [
    {
      "id": "upsert",
      "label": "Upsert",
      "description": "Pushes new records and updates records that change in your source."
    }
  ]
}

Properties of a mode

list_fields

This method should return the mapping fields that are available for a specific object and mode (or no object and mode).

Request body

Response sample

{
  "fields": [
    {
      "id": "email",
      "label": "Email Address",
      "type": "STRING",
      "identifier": true
    },
    {
      "id": "first_name",
      "label": "First Name",
      "type": "STRING",
      "required": true
    },
    {
      "id": "last_name",
      "label": "Last Name",
      "type": "STRING",
      "required": true
    }
  ]
}

Properties of a field

Field types

Hightouch supports the following field types. Hightouch will not automatically cast the value to this type, nor will it validate whether the data in the row is of that type. it's generally just for suggestion and display purposes.

STRING;
ENUM;
NUMBER;
BOOLEAN;
DATE;
DATETIME;
EMAIL;
OBJECT;
ARRAY;
REFERENCE;
UNKNOWN;

list_options

This method should return the schema and uiSchema of the form that you would like to render. Hightouch uses React JSON Schema Form to build a form out of a semi-standard JSON Schema.

If your schema at the highest level is an object, we will spread the properties into the configuration, how

Response sample

{
  "options": {
    "schema": {
      "type": "object",
      "properties": {
        "channel": {
          "title": "Channel",
          "description": "Channel that messages are sent to.",
          "type": "string",
          "enum": [1, 2, 3],
          "enumNames": ["#general", "#engineering", "#marketing"]
        },
        "message": {
          "label": "Message",
          "description": "Template for messages in Liquid format.",
          "type": "string"
        }
      }
    },
    "uiSchema": {
      "message": {
        "ui:widget": "textarea"
      }
    }
  }

Properties of options

validate

This method receives a JSON of the configuration and should return an array of errors. If there is no error, it should return an empty array.

Request body

Request sample

{
  "configuration": {
    "object": "user",
    "mode": "upsert",
    "identifierMapping": [{ "from": "user_email", "to": "email" }],
    "mappings": [
      { "from": "user_fn", "to": "first_name" },
      { "from": "user_ln", "to": "last_name" }
    ],
    "customMappings": [{ "from": "user_ltv", "to": "ltv" }],
    // the following properties come from custom configuration from list_options
    "archiveOnRemove": true
  }
}

Response sample

{
  "errors": ["If first name is mapped, last name must also be mapped."]
}

behavior

This method should return the operations that Hightouch should send, as well as the batch size Hightouch should send at.

Request body

behavior has no request body.

Response sample

{
  "batchSize": 100,
  "operations": {
    "add": true,
    "change": true,
    "remove": false
  },
  "skipInitialRun": false
}

add change remove

This method should send the batch of data forward to the destination API and return an array of rejected rows if any errors occurred.

Request body

Request sample

{
  "idColumn": "user_email",
  "configuration": {
    "object": "user",
    "mode": "upsert",
    "identifierMapping": [{ "from": "user_email", "to": "email" }],
    "mappings": [
      { "from": "user_fn", "to": "first_name" },
      { "from": "user_ln", "to": "last_name" }
    ],
    "customMappings": [{ "from": "user_ltv", "to": "ltv" }],
    // the following properties come from custom configuration from list_options
    "archiveOnRemove": true
  },
  "rows": [
    {
      "user_email": "alice@hightouch.com",
      "user_fn": "Alice",
      "user_ln": "Doe",
      "user_ltv": 0
    },
    {
      "user_email": "bob@hightouch.com",
      "user_fn": "Bob",
      "user_ln": "Doe",
      "user_ltv": 10
    }
  ]
}

Response sample

{
  "rejectedRows": [
    { "id": "alice@hightouch.com", "reason": "Couldn't insert user" }
  ]
}

Properties of a rejectedRow