Skip to main content

Component-specific settings for tKafkaInputAvro

The following table describes the Job script functions and parameters that you can define in the setSettings {} function of the component.

Function/parameter Description Mandatory?

USE_EXISTING_CONNECTION

Set this parameter to true and specify the name of the relevant connection component using the CONNECTION parameter to reuse the connection details you already defined.

No

BROKER_LIST

Specify the addresses of the broker nodes of the Kafka cluster to be used, in the form of "\"host1:port1,host2:port2,...\"".

This parameter works when the Kafka cluster version is Kafka 0.9.2.1 or higher.

Yes

AUTO_OFFSET_RESET

Specify the starting point from which the messages of a topic are consumed. Acceptable values:

  • SMALLEST
  • LARGEST (default)

No

KAFKA_TOPIC

Specify the name of the topic from which this component receives the feed of messages.

Yes

GROUP_ID

Specify the name of the consumer group to which you want the current consumer to belong.

This consumer group will be created at runtime if it does not exist at that moment.

Yes

KAFKA_MAX_RATE_PER_ PARTITION_CHECK

If needed, set this parameter to true and use the KAFKA_MAX_RATE_PER_PARTITION parameter to specify the maximum number of messages per batch to send for processing.

By default, this parameter is set to false, which means that the component tries to read all the available messages in one second into one single batch before sending it, potentially resulting in Job that stops responding in case of a huge quantity of messages.

No

USE_HTTPS

Set this parameter to true enable SSL or TLS encryption of the connection, and use the HTTPS_SETTING parameter to specify the tSetKeystore component that you use to define the encryption information.

No

USE_KRB

If the Kafka cluster to be used is secured with Kerberos, set this parameter to true and use the following parameters to define the related security information:

  • JAAS_CONF: specify the path to the JAAS configuration file to be used by the Job to authenticate as a client to Kafka.

  • KRB_SERVICE_NAME: specify the primary part of the Kerberos principal you defined for the brokers when creating the broker cluster.

    For example, for the principal kafka/kafka1.hostname.com@EXAMPLE.COM, the value of this parameter is kafka.

  • SET_KINIT_PATH: Kerberos uses a default path to its kinit executable. If you have changed this path, set this parameter to true and use the KINIT_PATH parameter to specify the custom access path.

  • SET_KRB5_CONF: Kerberos uses a default path to its configuration file, krb5.conf (or krb5.ini in Windows) for Kerberos 5 for example. If you have changed this path, set this parameter to true and use the KRB5_CONF to specify the custom access path to the Kerberos configuration file.

No

KAFKA_CONSUMER_ PROPERTIES {}

If you need to use custom Kafka consumer configuration, include in this function one or more sets of the following parameters to specify the property or properties to be customized. Then at runtime, the customized property or properties will override the corresponding ones used by the Studio.

  • PROPERTY: Type in the name of the property.
  • VALUE: Type in the new value of the property.

No

USE_HIERARCHICAL

Set this parameter to true to map the specified binary (including hierarchical) Avro schema to the flat schema defined in the schema of the current component. If the Avro message to be processed is flat, set this parameter to false (default). With this parameter set to true, you need to specify the Avro schema file and define the mappings.

  • SCHEMA_FILENAME: Specify the file which defines the schema of the Avro data to be processed.
  • MAPPING {}: Include in this function one or more sets of the parameter to define the mappings between the schema columns of the current component and the data stored in the hierarchical Avro message to be handled:
    • SCHEMA_COLUMN: Specify the schema column.
    • NODE_PATH: Specify the JSON path pointing to the data to be read from the Avro message.

No

LABEL

Use this parameter to specify a text label for the component.

No

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here