Avro

​

Add the Avro Connector

​

Connect to Avro

• Local
• Amazon S3
• Azure Blob Storage
• Azure Data Lake Storage
• Azure Files
• Box
• Dropbox
• FTP
• Google Cloud Storage
• Google Drive
• HDFS
• HTTP
• IBM Object Storage Source
• OneDrive
• Oracle Cloud Storage
• SFTP
• SharePoint Online

​

Local

• Connection Type: Select Local.
• URI: Enter your local directory path, one file per table (for example, C:/FolderPath).

​

Amazon S3

​

Azure Blob Storage

• Connection Type: Select Azure Blob Storage.
• URI: Enter the name of your container and blob in the URI text box (for example, azureblob://MyContainer/MyBlob).
• Azure Storage Account: Enter the name of your Azure storage account.

​

Azure Data Lake Storage

​

Azure Files

• Connection Type: Select Azure Files.
• URI: Enter the name of your Azure file share and the name of the resource (for example, azurefile://FileShare/RemotePath).
• Azure Storage Account: Enter the name of your Azure storage account.

​

Box

• Connection Type: Select Box.
• URI: Enter the pathname for the file system and the name of the folder that contains your Avro files (for example, box://Folder1).
• Auth Scheme: Select OAuth. provides an embedded OAuth application with which to connect. No additional properties are required.

​

Dropbox

• Connection Type: Select Dropbox.
• URI: Enter the path to a folder that contains Avro files (for example, dropbox://Folder1).
• Auth Scheme: Select OAuth. provides an embedded OAuth application with which to connect. No additional properties are required.

​

FTP

• Connection Type: Select either FTP or FTPS.
• URI: Enter the address of the server, followed by the path to the folder that you want to use as the root folder, as shown in these examples:
  • FTP: ftp://localhost:990/Folder1
  • FTPS: ftps://localhost:990/Folder1
• Auth Scheme: Use the default scheme, Basic.
• User: Enter the username that you use to authenticate to your FTP or FTPS server.
• Password: Enter the password that you use to authenticate to your FTP or FTPS server.

​

Google Cloud Storage

• Connection Type: Select Google Cloud Storage.
• URI: Enter the name of the bucket and folder that contains your Avro files (for example, gs://Bucket/RemotePath).
• Project Id: Enter the identifier for the project to which you want to connect.

​

Google Drive

• Connection Type: Select Google Drive.
• URI: Enter the pathname for the file system and the name of the folder that contains your Avro files (for example, gdrive://Folder1).

​

Hadoop Distributed File System (HDFS)

• Connection Type: Set this to HDFS or HDFS Secure.
• URI: Enter the path to a folder that contains your Avro files, as shown in these examples:
  • HDFS: webhdfs://Host:Port/RemotePath
  • HDFS Secure: webhdfss://Host:Port/RemotePath

​

HTTP

• Connection Type: Select either HTTP or HTTPS.
• URI: Enter the path to a folder that contains your Avro files, as shown in these examples:
  • HTTP: http://RemoteStream
  • HTTPS: https://RemoteStream

​

IBM Object Storage Source

• Connection Type: Select IBM Object Storage Source.
• URI: Enter the path to a folder that contains your Avro files (for example, ibmobjectstorage://Bucket1/RemotePath).
• Api Key: Enter the API key for your IBM Object Storage source. To obtain this key:
  1. Log in to your IBM Cloud account.
  2. Navigate to the Platform API Keys page.
  3. Click Create an IBM Cloud API Key (middle right corner) to create a new API key. This action opens a dialog box.
  4. Specify a name for the API key. Then, click Create.
     Save the API key somewhere because you cannot access it again from the dashboard.
• Region: Enter the region for your IBM instance (for example, eu-gb).

​

OneDrive

• Connection Type: Select OneDrive.
• URI: Enter the path to a folder that contains your Avro files (for example, onedrive://RemotePath).

​

Oracle Cloud Storage

• Connection Type: Select OracleCloudStorage.
• URI: Enter the name of your bucket and your folder (for example, os://Bucket/RemotePath).
• Region: Enter the hosting region for your S3-like web services.
• Oracle Namespace: Enter your Oracle cloud namespace.
• Auth Scheme: Use the default scheme, HMAC.
• Access Key: Enter your Oracle Cloud access key.
• Secret Key: Enter your Oracle Cloud secret key.

​

SFTP

• Connection Type: Select SFTP.
• URI: Enter the address of your server, followed by the path to your root folder (for example, sftp://Server:Port/RemotePath).
• Auth Scheme: Use the default scheme, SFTP.

​

SharePoint Online

• Connection Type: Select either SharePoint Online REST or SharePoint Online SOAP.
• URI: Enter the path to a folder that contains your Avro files:
  • SharePoint Online REST: sprest://RemotePath
  • SharePoint Online SOAP: spr://RemotePath
• Storage Base URL: Enter the base URL for the SharePoint site. The following are examples of valid URLs:
  If the domain for this option ends in -my (for example, ) then you might need to connect through OneDrive.

​

Complete Your Connection

1. Specify the following properties:
   • Data Model: Select the data model to use for parsing Avro documents and generating the database metadata. The default data model is Document.
   • OAuth Grant Type: Select the grant type for the OAuth flow. The default type is CODE.
   • Aggregate Files: Specify whether you want to aggregate all the files in the URI directory into a single result. The default value is False.
2. Define advanced connection settings on the Advanced tab. (In most cases, though, you should not need these settings.)
3. If you authenticate with , click Connect to to connect to your account.
4. Click Create & Test to create your connection.

​

More Information