tKuduOutput properties for Apache Spark Batch

Use an existing configuration

Select this check box and in the Component List drop-down list, select the desired connection component to reuse the connection details you already defined.

Server connection

Click the [+] button to add as many rows as the Kudu masters you need to use, each row for a master.

Then enter the locations and the listening ports of the master nodes of the Kudu service to be used.

This component supports only the Apache Kudu service installed on Cloudera.

For compatibility information between Apache Kudu and Cloudera, see the related Cloudera documentation:Compatibility Matrix for Apache Kudu.

Schema and Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

• Built-In: You create and store the schema locally for this component only.

• Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this option to view the schema only.

• Change to built-in property: choose this option to change the schema to Built-in for local changes.

• Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

Kudu table

Enter the name of the table to be created, changed or removed.

Action on table

Select an operation to be performed on the table defined.

• None: No operation is carried out.

• Drop and create table: The table is removed and created again.

• Create table: The table does not exist and gets created.

• Create table if does not exist: The table is created if it does not exist.

• Drop table if exist and create: The table is removed if it already exists and created again.

Action on data

Select an action to be performed on data of the table defined.

• Insert: Add new entries to the table. If duplicates are found, the Job stops.

• Update: Make changes to existing entries.

• Upsert: Update the record with the given reference. If the record does not exist, a new record would be inserted.

• Delete: Remove entries corresponding to the input flow.

Replicas

Enter, without double quotation marks, the replication factor of this table to create copies of your table and its tablets.

For further information about Kudu tablets and Kudu replication policies, see Distribution and Fault Tolerance.

Hash partitions

When you are creating a Kudu table, it is recommended to define how this table is partitioned. By default, your table is not partitioned.

1. Click the [+] button to add the columns based on which rows are hash-partitioned, for example, add a host column and a metric column.

   These columns must exist in your data and in the schema you have defined in tKuduOutput.

2. In Number of buckets, enter, without double quotation marks, the number of the buckets to be used to store the partitions. These buckets are created on the fly.

At runtime, rows are distributed by hash value in one of those buckets. If you leave this Hash partitions table empty, hash partitioning is not applied during the creation of the table.

For further information about hash partitioning in Kudu, see Hash partitioning.

Range partitions

When you are creating a Kudu table, it is recommended to define how this table is partitioned. By default, your table is not partitioned.

1. Click the [+] button to add the primary key columns based on which rows are partitioned to contiguous segments, for example, add a time column.

   These columns must exist in your data and in the schema you have defined in tKuduOutput.

2. In N. Partition, enter, without double quotation marks, a number to be used as the ID number of the partition to be created. For example, enter 1 to create Partition 1.
3. In Lower boundary and Upper boundary, enter the boundaries between which rows are partitioned. The upper boundary is excluded and only the data between the boundaries (including the lower boundary) is written to the Kudu table.

   For example, if you are partitioning your data based on a time column of Date type and the format of your time data is yyyy-mm-dd, you can set the lower boundary to be 2016-01-01 by entering, without double quotation marks, TalendDate.parseDate("yyyy-MM-dd", "2016-01-01") and do the same to set the upper boundary to be 2018-01-01, then only rows with dates between the two boundaries enter this partition.

At runtime, rows of these columns are distributed using the values of the columns you have added to this Range partitions table. If you leave this table empty, range partitioning is not applied during the creation of the table.

For further information about hash partitioning in Kudu, see Range partitioning.

Die on error

Select the check box to stop the execution of the Job when an error occurs.

Usage rule

This component is used as an end component and requires an input link.

Spark Connection

This connection is effective on a per-Job basis.