Skip to main content Skip to complementary content

Setting up monitoring and auditing systems for Talend JobServer

Configure a transfer system for stats and trace messages and an audit logging system to record key operations performed on Talend JobServer.

Configuring stats and trace message transfer for Talend JobServer

You can specify a port through which Talend Studio fetches the latest stats and trace messages from Talend JobServer for Jobs being executed remotely.

About this task

The stats and trace messages feature is a development level feature.

Generally, it is recommended to disable the stats and trace messages transfer parameter for production environments, to avoid an excessive usage of memory and to protect against connections from outside if the network connections from outside are not already blocked.

Using real-time statistics may result in excessive memory usage on JobServer side. If you observe ZeroMQ related memory leaks, disable stats and trace messages on JobServer. With that, real-time statistics on Talend Administration Center will be unavailable.

Procedure

  1. Go to the directory <root>\conf, where <root> is the Talend JobServer path, and open the TalendJobServer.properties file to edit it.
  2. In the line dedicated to the configuration of the message transfer port, specify a port number. 
    org.talend.remote.jobserver.server.TalendJobServer.PROCESS_MESSAGE_PORT=<port_number>

    The default port is 8555. You can specify any port that is available in the system.

  3. To enable stats and trace message transfer, set the following parameter to true. 
    org.talend.remote.jobserver.server.TalendJobServer.ENABLED_PROCESS_MESSAGE=true
    If the Talend JobServer is deployed on the same machine with the Talend Studio, you can set this parameter to false to disable the service and save your port resources.
  4. Save your changes and restart the Talend JobServer so that the configuration takes effect.

Configuring audit logs for Talend JobServer

Starting from Talend JobServer version 8.0.2.20250110_1323_patch (zip name: Patch_20250114_TPS-6002_v4-8.0.1), you can activate audit logging to track operations triggered by client applications such as Talend Studio or Talend Administration Center.

This allows for monitoring and auditing of critical activities. Audit logs record the following operations:
  • Receiving a Job
  • Deploying a Job
  • Running a Job
  • Killing a running Job execution
  • Verifying if a user is authorized for an operation
  • Getting the useful CRC(Cyclic Redundancy Check) in case of errors before deployment

About this task

A logger for audit logs has been pre-configured in the log4j2.xml file:
<Logger name="org.talend.audit" level="info" additivity="false">
   <AppenderRef ref="Audit"/>
</Logger>
This procedure shows:
  • How to activate or deactivate audit logs.
  • How to update the audit appender used by this logger to specify the storage of audit logs.

Procedure

  1. Activate or deactivate the audit log using the org.talend.remote.jobserver.commons.config.JobServerConfiguration.AUDIT_LOG_LEVEL property in the <jobserver_path>/conf/TalendJobServer.properties file.
    The values to be used are:
    • 0 to deactivate audit logs
    • 1 to activate audit logs
  2. Configure whether audit logs are written to a dedicated file or included with other logs.
    • Recommended: to write audit logs in a separate file with specific file pattern, in the log4j2.xml file, update the fileName and the filePatern attributes in the RollingFile appender called Audit. For example: 
      <RollingFile name="Audit" fileName="jobserver_audit.log"
   filePattern="jobserver_audit-%d{MM-dd-yyyy}.log.gz" ignoreExceptions="false">
      <!-- Default log layout skipped for brevity -->
</RollingFile>
      With this configuration, Talend JobServer writes audit logs to the specified file jobserver_audit.log.
    • To include audit logs with other logs in jobserver.log, keep the default configuration of the Audit appender. The default configuration reads as follows:
      <RollingFile name="A1" fileName="jobserver.log" filePattern="jobserver-%d{MM-dd-yyyy}.log.gz"
  ignoreExceptions="false">
    <!-- skipped for brevity -->
</RollingFile>

<RollingFile name="Audit" fileName="jobserver.log" filePattern="jobserver-%d{MM-dd-yyyy}.log.gz"
 ignoreExceptions="false">
   <PatternLayout pattern="%d %-5p %c{1} 
   - %m%n %X{userId,jobId,utcTime,jobServerId,clientIP,projectId,failed}%n"/>
   <ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
   <Policies>
      <SizeBasedTriggeringPolicy size="1000KB"/>
   </Policies>
   <DefaultRolloverStrategy max="100"/>
</RollingFile>

Results

By default, audit logs use the MDC format: %X{userId,jobId,utcTime,jobServerId,clientIP,projectId,failed}.

You can retrieve the MDC fields to create a custom log layout such as JSON with named fields.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here