Supported File Formats
When writes data to , you can choose the file format for the exported data. The following file formats are supported for the source:- (Default) CSV — Plain text comma-separated values.
- Avro — A row-based binary format that supports schema evolution.
- Parquet — A columnar storage format that is optimized for analytics.
Prerequisites
If you want to authenticate with a Google Cloud service account (for silent authentication or delegated organization-wide access), do the following before configuring your connection in :- Create a service account in Google Cloud and grant it the required permissions on your bucket or project.
- For OAuth JWT authentication, register a custom OAuth application and download the certificate file (.p12 or .pfx).
Add the Google Cloud Storage Connector
Authenticate to Google Cloud Storage
After you add the connector, you need to set the required properties.-
File Format: Select the file format that you want to use: Delta Parquet, CSV (default), Avro, and Parquet.
The Delta Parquet and Apache Iceberg file formats are not supported for source connectors. If you select either of these formats, generates an error.
- URI: Enter the path to the resource location for your file format.
- Project Id: Enter the identifier (Id) of the project where your instance resides.
- OAuth (default)
- OAuth PKCE
- OAuth JWT
- GCP Instance Account
OAuth
To connect with OAuth custom credentials, specify the following properties:- Auth Scheme: Select OAuth.
- OAuth Version: Select the version of OAuth that you want to use. The default version is 2.0.
- (Optional) Scope: Specify the scope of your access to the application.
- (Optional) OAuth Authorization URL: Enter the OAuth authorization URL for the OAuth service.
- (Optional) OAuth Access Token URL: Enter the URL from which to retrieve the access token.
- (Optional) OAuth Refresh Token URL: Enter the URL from which to refresh the OAuth token.
OAuth PKCE
To connect with the OAuth PKCE extension, specify the following properties:- Auth Scheme: Select OAuthPKCE.
- (Optional) OAuth Client Id: Enter the client identifier (Id) that you were assigned when you registered your application with an OAuth authorization server.
OAuth JWT
GCP Instance Account
Complete Your Connection
To complete your connection:-
Specify the following properties:
For all file formats:
-
(Optional) Project Id: Enter the identifier (Id) of the project where your Google Cloud Storage instance resides.
This property is required for the Avro file format, and it is set in Authenticate to Google Cloud Storage.
- FMT: Enter the format that you want to use to parse all text files. The default format is CsvDelimited.
- Aggregate Files: Select whether you want to aggregate all the files that are located in the URI directory and that have the same schema into a single table named AggregatedFiles. By default, the Enable checkbox is not selected.
- Include Column Headers: Select whether you want to obtain column headers from the first lines of the specified files. By default, the Enable checkbox is already selected.
- Data Model: Select the data model that you want to use to parse documents for your format and to generate the database metadata. The default data model is Document.
- Aggregate Files: Select whether you want to aggregate all the files that are located in the URI directory and that have the same schema into a single table named AggregatedFiles. By default, the Enable checkbox is not selected.
-
(Optional) Project Id: Enter the identifier (Id) of the project where your Google Cloud Storage instance resides.
- Define advanced connection settings on the Advanced tab. (In most cases, though, you should not need these settings.)
- If you authenticate with OAuth and OAuthPKCE, click Connect to Google Cloud Storage to connect to your Google Cloud Storage account.
- Click Create & Test to create your connection.