Configuring an HTTPS connection between Talend Data Preparation, Streams Runner and Spark Job Server

Securing the connections between Talend Data Preparation, Streams Runner and Spark Job Server requires editing their corresponding configuration files.

Any security configuration in the Streams Runner configuration file should be done at the end of the file, in the Append section, after the Include section, to avoid being overwritten.

The first step will be to configure Spark Job Server as a service in HTTPS. Then, you will need to enable SSL communication between Streams Runner and Spark Job Server running in HTTPS. After that, you will configure Streams Runner as a service in HTTPS, and finally, enable SSL communication between Talend Data Preparation and Streams Runner running in HTTPS.

Before you begin

Talend Data Preparation has been configured as a service in HTTPS. For more information, see Configuring an HTTPS connection for Talend Data Preparation.
You have generated a certificate for Talend Data Preparation and added it to your Web browser truststore.
Spark Job Server and Streams Runner are installed and running.

Procedure

To secure the Spark Job Server service in HTTPS, open the <Spark_Job_Server_installation_path>/settings.sh configuration file.
Set the value of the security_ssl_enabled parameter to on.

Edit the values of the security_path_to_keystore and security_keystore_password parameters to set the path and password of your keystore file containing the certificate for Spark Job Server.

security_ssl_enabled=on
# DO NOT CHANGE
# SECURITY PATH TO KEYSTORE
# Required : No
# Env variable : SECURITY_PATH_TO_KEYSTORE
security_path_to_keystore=<path_to_keystore>
# DO NOT CHANGE
# SECURITY KEYSTORE PASSWORD
# Required : No
# Env variable : SECURITY_KEYSTORE_PASSWORD
security_keystore_password=<password>

After restarting the service, Spark Job Server will be running in HTTPS.

To enable SSL communication between Streams Runner and Spark Job Server running in HTTPS, you can either:
- Use the JOBSERVER_TRANSPORT_PROTOCOL environment variable with the following command: export JOBSERVER_TRANSPORT_PROTOCOL=https.
- Edit the <Streams_Runner_installation_path>/conf/application.conf configuration file and set the value of the app.svc.jobserver.protocol parameter to https.
Using the environment variable will override the application.conf configuration.
To add the Spark Job Server certificate, or its Certificate Authority to the Streams Runner truststore, add the following lines to the <Streams_Runner_installation_path>/conf/application.conf file, according to the file format used for your truststore.
- For .pem files:
```
  play.ws.ssl {
    trustManager = {
      stores = [
        { type = "PEM", path = "/path/to/pem_file" }
      ]
    }
  }
```
- For .jks files:
```
  play.ws.ssl {
    trustManager = {
      stores = [
        { type="JKS", path="/path/to/truststore", password="<password>"}
      ]
    }
  }
```
For more information, see the Play documentation.
To secure the Streams Runner service in HTTPS, define the path and password of its certificate by editing the following lines of the <Streams_Runner_installation_path>/conf/application.conf file:
```
play.server.https.keyStore.path = <path_to_keystore>
play.server.https.keyStore.password = <password>
```
Edit the two following lines to set the HTTPS port and disable the HTTP port:
```
https.port=9443
http.port=disabled
```
9443 is the default port value for the HTTPS connection.

Information noteWarning: Any play.server.http.port=<port> configuration will conflict with the http.port=disabled configuration and the port will not be disabled.
After restarting the service, Streams Runner will be running in HTTPS.
To enable SSL communication between Talend Data Preparation and Streams Runner running in HTTPS, retrieve the Streams Runner certificate, or its Certificate Authority, and add it to the Talend Data Preparation truststore using the following command:
keytool -import -trustcacerts -alias <cert-alias> -file <streams_runner_certificate.crt> -keystore <truststore.jks>

In the <Data_Preparation_Path>/config/application.properties file, add the following properties to set the truststore:

tls.trust-store=/path/to/<truststore.jks>
tls.trust-store-password=<trust-store_password>

false to disable hostname verification
tls.verify-hostname=true

Restart Talend Data Preparation.

Results

Your Talend Data Preparation instance running in HTTPS can now communicate with Streams Runner and Spark Job Server, also running with a secured HTTPS connection.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here