> ## Documentation Index > Fetch the complete documentation index at: https://docs.sync.cdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Apache Kafka You can use the Apache Kafka connector from the CData Sync application to move data from any supported source to the Apache Kafka destination. To do so, you need to add the connector, authenticate to the connector, and complete your connection. **Note:** With this connector, you can authenticate either to Apache Kafka or to Azure Event Hubs. ## Add the Apache Kafka Connector To enable Sync to use data from Apache Kafka, you first must add the connector, as follows:
  1. Open the **Connections** page of the Sync dashboard.
  2. Click **Add Connection** to open the **Select Connectors** page.
  3. Click the **Destinations** tab and locate the **Apache Kafka** row.
  4. Click the **Configure Connection** icon at the end of that row to open the **New Connection** page. This action opens the **Add Connection** dialog. **Note:** If the **Configure Connection** icon is not available, click the **Download Connector** icon to install the Apache Kafka connector.
  5. Enter a name for your connection in the **Add Connection** dialog.
  6. Click **Add** to open the **Settings** tab for your connector.
For more information about installing new connectors, see [Connections](https://docs.sync.cdata.com/26.4/en/connections/connections.md). ## Authenticate to Apache Kafka After you add the connector, you need to set the required properties. **Bootstrap Servers:** Enter the address of the Apache Kafka Bootstrap servers to which you want to connect. CData Sync supports authenticating to Apache Kafka in several ways. Select your authentication method below to proceed to the relevant section that contains the authentication details. * [**None**](#none) (default) * [**Plain**](#plain) * [**SCRAM**](#scram) * [**SCRAM-SHA-512**](#scram-sha-512) * [**Kerberos**](#kerberos) ### None To connect without authentication, select **None** for **Auth Scheme**. No additional properties are required. ### Plain To connect with a plain-text login, specify these properties: ### SCRAM To connect with SCRAM (SHA-256) credentials, specify these properties: * **Auth Scheme:** Select **SCRAM**. * **User:** Enter the username that you use to authenticate to your Apache Kafka account. * **Password:** Enter the password that you use to authenticate to your Apache Kafka account. ### SCRAM-SHA-512 To connect with the SCRAM (SHA-512) credentials, specify these properties: * **Auth Scheme:** Select SCRAM-SHA-512. * **User:** Enter the username that you use to authenticate to your Apache Kafka account. * **Password:** Enter the password that you use to authenticate to your Apache Kafka account. ### Kerberos To connect with Kerberos, specify these properties: * **Auth Scheme:** Select **Kerberos**. * **Kerberos SPN:** Enter the service principal name (SPN) for the Kerberos domain controller. * **Kerberos Service Name:** Enter the name of the Kerberos service with which you want to authenticate. * (Optional) **Kerberos Keytab File:** Enter the keytab file that contains your pairs of the Kerberos principals and encrypted keys. * (Optional) **Use Kerberos Ticket Cache:** Select **True** to use a ticket cache with the logged-in user instead of a keytab file. The default value is **False**. ## Authenticate to Azure Event Hubs After you add the connector, you need to set the following required properties to connect to Azure Event Hubs: For **Bootstrap Servers,** enter the address of the Apache Kafka bootstrap servers to which you want to connect. CData Sync supports authenticating to Apache Event Hubs in several ways. Select your authentication method below to proceed to the relevant section that contains the authentication details. * [**SSL Certificate**](#ssl-certificate) * [**Azure Active Directory**](#azure-active-directory) * [**Azure Managed Service Identity**](#azure-managed-service-identity) * [**Azure Service Principal**](#azure-service-principal) * [**Azure Service Principal Certificate**](#azure-service-principal-certificate) ### SSL Certificate To connect with a Secure Sockets Layer (SSL) client certificate, specify these properties: * **Auth Scheme:** Select **SSLCertificate**. * **SSL Client Cert:** Enter the SSL client certificate that is used to validate to the Apache Kafka broker. * **SSL Client Cert Type:** Select the format of the SSL client certificate that is used to connect to the Apache Kafka broker: * **JKSFILE** * **PFXFILE** * **PEMKEY\_FILE** (default) * **PEMKEY\_BLOB** * (Optional) **SSL Client Cert Password:** Enter the password that is used to decrypt the SSL client certificate. ### Azure Active Directory To connect with Azure Active Directory (AD) credentials, specify the following properties: * **Azure Tenant:** Enter the Microsoft Online tenant that is used to access data. If you do not specify a tenant, Sync uses the default tenant. * **OAuth Client Id:** Enter the client Id that you were assigned when you registered your application with an OAuth authorization server. * **OAuth Client Secret:** Enter the client secret that you were assigned when you registered your application with an OAuth authorization server. ### Azure Managed Service Identity To leverage Azure Managed Service Identity (MSI) when CData Sync is running on an Azure virtual machine, select **Azure MSI** for **Auth Scheme**. No additional properties are required. ### Azure Service Principal To connect with an Azure service principal and client secret, set the following properties: To obtain the OAuth client Id and client secret for your application:
  1. Log in to the [Azure portal](https://portal.azure.com).
  2. In the left navigation pane, select **All services**. Then, search for and select **App registrations**.
  3. Click **New registrations**.
  4. Enter an application name and select **Any Azure AD Directory - Multi Tenant**.
  5. After you create the application, copy the application (client) Id value that is displayed in the **Overview** section. Use this value as the OAuth client Id.
  6. Navigate to the **Certificates & Secrets** section and select **New Client Secret** for the application.
  7. Specify the duration and save the client secret. After you save it, the key value is displayed.
  8. Copy this value because it is displayed only once. You will use this value as the OAuth client secret.
  9. On the **Authentication** tab, make sure to select **Access tokens (used for implicit flows)**.
### Azure Service Principal Certificate To connect with an Azure service principal and client certificate, set the following properties: To obtain the OAuth certificate for your application:
  1. Log in to the [Azure portal](https://portal.azure.com).
  2. In the left navigation pane, select **All services**. Then, search for and select **App registrations**.
  3. Click **New registrations**.
  4. Enter an application name and select **Any Azure AD Directory - Multi Tenant**.
  5. After you create the application, copy the application (client) Id value that is displayed in the **Overview** section. Use this value as the OAuth client Id.
  6. Navigate to the **Certificates & Secrets** section and select **Upload certificate**. Then, select the certificate to upload from your local machine.
  7. Specify the duration and save the client secret. After you save it, the key value is displayed.
  8. Copy this value because it is displayed only once. You will use this value as the OAuth client secret.
  9. On the **Authentication** tab, make sure to select **Access tokens (used for implicit flows)**.
## Authenticate to GCP Kafka After you add the connector, you need to set the following required properties to connect to GCP Kafka: For **Bootstrap Servers,** enter the address of the Apache Kafka bootstrap servers to which you want to connect. CData Sync supports authenticating to GCP Kafka in several ways. Select your authentication method below to proceed to the relevant section that contains the authentication details. * [**OAuth JWT**](#oauth-jwt) * [**GCP Instance Account**](#gcp-instance-account) * [**AWS Workload Identity**](#aws-workload-identity) ### OAuth JWT To connect with a Apache Kafka account using OAuth JWT, specify the following properties: * **Auth Scheme:** Select **OAuthJWT**. * **OAuth Client Id:** Enter the client Id that you were assigned when you registered your application with an OAuth authorization server. * **OAuth Client Secret:** Enter the client secret that you were assigned when you registered your application with an OAuth authorization server. * **OAuth JWT Cert:** Enter your JSON web tokens (JWT) certificate store. * **OAuth JWT Cert Type:** Enter the type of key store that contains your JWT certificate. The default type is **PEMKEY\_BLOB**. * **OAuth JWT Cert Password:** Enter the password for the OAuth JWT certificate that is used to access a password-protected certificate store. If the certificate store does not require a password, leave this property blank. * (Optional) **OAuth JWT Cert Subject:** Enter the subject of your OAuth JWT certificate. ### GCP Instance Account When you run CData Sync on a GCP virtual machine, Sync can authenticate by using the service account that is tied to the virtual machine. Select **GCPInstanceAccount** for **Auth Scheme** to use that account. No additional properties are required. ### AWS Workload Identity To connect with AWS Workload Identity federation credentials, specify the following properties: ## Extracting Metadata from Topics Sync models Apache Kafka topics as tables and messages as rows. The **Type Detection Scheme** property controls how Sync reads schema information from each topic. Details about **Type Detection Scheme** are provided in [Complete Your Connection](#complete-your-connection). ### Schema Registry The schema registry contains a list of topics that have registered schemas. The list of tables and columns are simply read directly from the schema registry. To connect to a service with a schema registry, specify the following properties: * **Bootstrap Servers:** Enter the server (host name or IP address) and port (in the format *Server:Port*) of the Apache Kafka bootstrap servers. * **Type Detection Scheme:** Select the detection scheme that you want to use. For complete details about using this scheme, see [Complete Your Connection](#complete-your-connection). * (Optional) **Schema Merge Mode:** Select the mode by which you want to merge multiple schemas. * **None** (default) * **Simple** #### Confluent Schema Registry When you connect to Confluent Cloud, the registry URL corresponds to the Schema Registry endpoint value that is located in **Schemas > Schema Registry > Instructions**. The Confluent schema registry supports several authentication options. Typically, Confluent Cloud deployments require that you set the **Registry Auth Scheme** property to **Basic**, along with a registry user and registry password. To find your user and password, navigate to **Schemas > Schema Registry > API Access** and locate the access-key and secret-key values. On-premises deployments might not require authentication. In these configurations, you should set **Registry Auth Scheme** to **None**. These deployments might require SSL client certificates also. For that, you need to set the **SSL Certificate** registry auth scheme as well as the **Registry Client Cert** and **Registry Client Cert Type** options. #### Amazon Web Services (AWS) Glue Schema Registry When you connect to AWS Glue, the registry URL corresponds to the Amazon Resources Name (ARN) value of the registry. The AWS Glue schema registry only supports the **Basic** registry auth scheme. You should set **Registry User** and **Registry Password**, respectively, to the access key and secret key of a user that has access to the registry. ## Topic Routing You can control how source tables map to Apache Kafka topics when you create a job that uses the Apache Kafka destination. To do so, select one of the following options in the **Add Job** dialog when you create a job: * **One topic per source table:** Each source table is written to its own topic. Sync automatically names the topic using the format *SchemaName*\_*TableName*. If the source does not have a schema, the topic name is simply the table name. * **All data to a single named topic:** All source data for a table is written to one topic. When you select this option, a **Topic** field is displayed, as shown below. You can either enter a topic name in the field or search and select a topic, if a topic list is available. ![Add Job dialog showing the Topic field for a single named topic](https://mintcdn.com/cdata-sync/4njnJgdQs2eFHAEE/26.4/images/Apache-Kafka-Destination_topic-routing_1.png?fit=max\&auto=format\&n=4njnJgdQs2eFHAEE\&q=85\&s=4936454faa241768715c3c7451da15b7) > [!NOTE] > > If a target topic does not exist when the job runs, Sync either creates it automatically or returns an error, depending on your broker configuration. ## Message Partitioning In Sync, new Apache Kafka destination jobs automatically partition messages by the source table's primary key. This behavior controls how messages are distributed across partitions in the target topic. * **Single primary key:** Sync uses the primary key value as the Apache Kafka message key. All messages with the same key value are written to the same partition, so downstream consumers see changes for a given key in the order they were written. * **No primary key:** Sync distributes messages across partitions in a round-robin pattern (each message goes to the next partition in sequence). No Apache Kafka message key is assigned. > [!NOTE] > > * Automatic key-based partitioning does not support tables with composite (multiple-column) primary keys. To partition by a single key, update your query to include only one primary key column. > * Existing Apache Kafka destination jobs (prior to release 26.3) are not affected by this change. ## Complete Your Connection To complete your connection: 1. Specify the following properties: * **Type Detection Scheme:** Select the detection scheme that you want to use. * (Default) **RowScan**—Scans messages from the topic to infer column names and data types. No additional configuration is required. * **SchemaRegistry**—Reads schema definitions from a Confluent-compatible schema registry. When you select this option, additional schema registry properties appear. For this scheme, specify the following properties: * **Registry Url:** Enter the URL to the server for the schema registry. When you specify this property, Sync reads the Apache Kafka schema from the server. * **Registry Service:** Select the Schema Registry service that you want to use for working with topic schemas. * **Confluent** (default) * **AWSGlue** * **Registry Auth Scheme:** Select the scheme that you want to use to authenticate to the schema registry. * (Default) **None:** This scheme specifies that no authentication is used. * **Basic:** For this scheme, specify the following properties: * **Registry User:** Enter the username that you use to authenticate with the server that you specified earlier for **Registry Url**. * **Registry Password:** Enter the password that you use to authenticate with the server that you specified earlier for **Registry Url**. * **SSLCertificate:** This scheme specifies that SSL client-certificate authentication should be used. For this setting, you must specify the following: * **Registry Client Cert:** Enter the TLS/SSL client certificate store for SSL client authentication (two-way SSL) with the schema registry. * (Optional) **Registry Client Cert Password:** Enter the password for your TLS/SSL client certificate. * (Optional) **Registry Client Cert Subject** Enter the subject of your TLS/SSL client certificate. * **Use SSL:** Select whether you want to enable Secure Sockets Layer (SSL). By default, the **Enable** checkbox is not selected.
  1. Define advanced connection settings on the **Advanced** tab. (In most cases, though, you should not need these settings.)
  2. If you authenticate with AzureAD, click **Connect to Apache Kafka** to connect to your Apache Kafka account.
  3. Click **Create & Test** to create your connection.
## More Information For more information about interactions between CData Sync and Apache Kafka, see [Apache Kafka Connector for CData Sync](https://cdn.cdata.com/help/SKN/synch/default.htm). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.