Configuring persistent logging in the Remote Engine Gen2 installed locally or on your private VPC

By default, the logs of all your executions are persisted in Apache Livy but you have the possibility to disable or re-enable them manually in order to avoid the overhead of logging.

This configuration only applies for users who installed Remote Engine Gen2 on premises or on their private VPC (AWS or Azure).

Procedure

Go to the Remote Engine Gen2 installation directory.
Open the following file to edit it:

<engine_directory>/default/.env if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.

<engine_directory>/eap/.env if you are using the engine as part of the Early Adopter Program.
Edit the following line to disable or enable logs:
- To disable logs:
```
LIVY_JOB_LOG_PERSISTENCE_ENABLE=false
```
- To enable logs:
```
LIVY_JOB_LOG_PERSISTENCE_ENABLE=true
```
Save the file.
In the docker-compose.yml file, configure the Livy volume to be used. This file is found in the default or eap folder depending on the data center you are using.
Example
Add the following section to specify the volume:
```
livy:
    image: ${LIVY_IMG}
    networks:
      default:
        aliases:
          - livy
    environment:
      ...
    volumes:
       - <HOST_FOLDER_PATH>:/tmp/livy-jobs-logs:rw
```
Then change the value of <HOST_FOLDER_PATH> to /tmp/livy-jobs-logs, for example.
This configuration creates the log host folder with the read and write permissions and mounts it in the Livy Docker container.
Restart your Remote Engine Gen2 to take your changes into account.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here