Log Listener

The Log Listener allows the user to get all the log information, which typically will also be available in tesb.log, into the Event Logging. This means that the Log Listener is the primary listener for the Event Logging Agent.

The pax logging, also used by Talend Runtime, is used as the entry point for the Event Logging Log Listener. This way, the user can also use the standard pax logging configuration to configure which logs should be sent to the Log Listener and which logs should not, like for any other log appender.

log4j2.rootLogger=INFO, out, osgi:VmLogAppender

The Log Listener will expose a second pax logging appender named eventloglisteneraudit where all log events sent through this appender will be set to audit=true and thus, would have the highest priority of all audit configurations. In general, if an Event has the audit flag set to true, no other Talend components or mappings will change it back to false. But if the audit flag is false, it might be set to true later on, by additional mapping options. The eventloglisteneraudit can be used in the pax logging configuration and would automatically make all events, sent via this appender, 'Audit' events.

The Log Listener is implemented using Apache Camel, via the paxlogging: component, and provides a configurable Category Mapper processor with a Direct-VM communication to the Talend Event Logging Agent.

The paxlogging: component will have a fixed configuration, to expose eventloglistener as appender name: paxlogging:eventloglistener.

And in case the logging framework supports MDC attributes, the MDC attributes are also taken into the event as additional metadata. This would also allow custom and business code logs to contain a business correlation ID or other important metadata in a structured form, for example, using MDC.put(“CorrelationID”, "abc").

The "Log to Event" conversion step will transform the log message format into the Event format, where the Event Structure is as completely filled as possible, and the additional metadata (including the MDC attributes) is transformed in the Event Customer Information (Key, Value) list.

The Category Mapper allows the user to define an Event Logging category in a configurable way to the log message by package name from where the message comes from, and if messages from this package should be treated as audit or non-audit messages.

The Log Listener uses the org.talend.eventlogging.listener.log.cfg configuration file with the following parameters:

# Default category for events
category.default=system

# Define key in MDC attributes which will be used to get EventCategory
category.attribute=eventCategory
# Define key in MDC that specify audit
audit.attribute=watchThis

# Category mapping configuration, for example, if a log message comes from 
# package org.apache.cxf, it will be mapped to the service category
category.mapping.org.apache.cxf = service
category.mapping.org.apache.cxf.rt.security = security(audit=true)
category.mapping.org.apache.wss4j = security(audit=true)
category.mapping.org.talend.esb.sts=security(audit=true)

The configuration is primarily for the Category Mapper and will be used as follows:

1. The first priority to define which EventCategory is assigned to the current log event is defined by an MDC attribute. The name of this attribute can be configured under category.attribute. By default, it is eventCategory. If this attribute exists, the Category will be set to the value of this attribute.

2. If the attribute does not exist, the Category Mapper will look for the category.mapping. configuration. It will build a tree in memory, will go from the leaves to the root of this tree to find the closest node to the given package name, as the log4j log appender package based filter configuration would do, and will apply the category and audit flag definition as defined in the configuration file, but only if the audit attribute is not already set to true and no overwrite of true is allowed for the audit flag. Once the audit flag is true, it will stay true.

3. If the audit.attribute parameter of the log event has a custom value and the audit setting is currently set to false, the audit setting will automatically be activated and it will take the custom value referred to in the audit.attribute parameter.

Example:

• In the example above, if a log message comes from package org.apache.cxf.binding.soap, it will be mapped to the service category, with audit implicitly set to false (audit=false), as org.apache.cxf is the matching node configuration.

• If a log message comes from org.apache.cxf.rt.security.saml, it will be mapped to the security category and marked as an audit message by the additional audit definition after the category: (audit=true).

  By default, if audit is not explicitly defined after the category, event will be set to audit=false.

• If the audit setting of the event was not yet set to true, and its attribute contains a custom value, for example: watchThis, the audit will automatically be set to true.

After this step, the log message is sent to the Event Logging Agent in a synchronous way (Direct-VM) to allow the agent to do a short pre-processing before the event is stored into a local buffer within the agent for final processing and sending to the backend.