Search documentation...

K
ChangelogBook a demoSign up

Kafka

Overview

Hightouch supports streaming events from your existing Kafka cluster in realtime.

Simply configure your Kafka connection in the Hightouch UI, and Hightouch will automatically extract, transform, and load your data into Hightouch Events. Your data can then be synced to your warehouse or streamed to our catalog of realtime destinations.

Setup

To get started, create an event source, select Kafka and follow the steps below.

Choose connection type

Hightouch can connect directly to Kafka over the public internet or via an SSH tunnel. Since data is encrypted in transit via TLS, a direct connection is suitable for most use cases. You may need to set up a tunnel if your Kafka instance is on a private network or virtual private cloud (VPC).

Direct connection vs. SSH tunnel

Hightouch supports both standard and reverse SSH tunnels. To learn more about SSH tunneling, refer to Hightouch's tunneling documentation.

Configure your source

Enter the following required fields into Hightouch:

  • Broker URL: The hostname:port for your Kafka broker(s).
  • Topic: The topic to consume events from. Messages in this topic must be in JSON format.
  • Authentication: We support multiple authenetication methods, check your cluster configuration to find the suitable method.

Message format

Hightouch Events requires Kafka messages be encoded as JSON.

Event Structure

All events are ingested into Hightouch as track events. The Kafka message key, value, and headers are exposed as custom properties on the track event. Metadata such as topic, partition, and offset are also exposed in the context.kafka object.

For example, the following Kafka message:

{
  "partition": 1,
  "offset": 123,
  "timestamp": 1721659050648,
  "key": "k1",
  "value": {
    "user_id": "user_123",
    "amount": 123.45
  },
  "headers": [
    {
      "key": "h1",
      "value": "abc"
    }
  ]
}

Will be ingested into Hightouch as:

{
  "type": "track",
  "event": "Kafka Event",
  "properties": {
    "key": "k1",
    "value": {
      "user_id": "user_123",
      "amount": 123.45
    },
    "headers": {
      "h1": "abc"
    }
  },
  "context": {
    "kafka": {
      "topic": "my-topic",
      "partition": 1,
      "offset": "123"
    }
  },
  "timestamp": "2024-07-22T14:52:37.648Z"
}

Field mappings

You can configure how Hightouch transforms messages from your Kafka topic to Hightouch track events. Mappings can be configured for the following fields:

  • event
  • userId
  • anonymousId
  • messageId
  • timestamp

You can also transform your events using Functions in order to standardize the schema or enrich its contents.

Schema Enforcement

Event Contracts can be attached to Kafka sources just like any other Event Source.

Tips and troubleshooting

If you encounter an error or question not listed below and need assistance, don't hesitate to . We're here to help.

Ready to get started?

Jump right in or a book a demo. Your first destination is always free.

Book a demoSign upBook a demo

Need help?

Our team is relentlessly focused on your success. Don't hesitate to reach out!

Feature requests?

We'd love to hear your suggestions for integrations and other features.

Last updated: Nov 5, 2024

On this page

OverviewSetupChoose connection typeConfigure your sourceMessage formatEvent StructureField mappingsSchema EnforcementTips and troubleshooting

Was this page helpful?