Skip to main content Skip to complementary content
  • New archived content: Talend MDM, Talend Data Catalog 8.0, and Talend 7.3 products reached their end of life in 2024. Their documentation was moved to the Talend Archive page and will no longer receive content updates.

Adding the latest Big Data Platform dynamically

In the Studio, if there is no support for the Big Data Platform you want to use, follow the procedure explained below to add this distribution yourself to make it available to the Studio.

In the current Studio version, you can use this procedure to add the Cloudera and the Hortonworks distributions only. This procedure uses Cloudera to demonstrate how to add a dynamic distribution to the Studio.

With this feature, you are empowered with more agility and flexibility to use a Cloudera or Hortonworks version that was not released the moment your Talend Studio was released, by simply adding this version yourself through several clicks.

The dynamic distributions added this way are generally minor versions of a Talend-certified major release of your distribution. Talend relies on the distribution vendors' compatibility statements to ensure the compatibility of the Studio with these minor versions and, by this measure, provides official support for the use cases that can be produced on these minor versions as well as on the Talend-certified versions. For further information about the Talend-certified distribution versions and Talend general support policy about the certified and the compatible versions, see Talend Installation Guide.

  • Dynamic distributions for HDP 3.x are in technical preview.

  • On the version list of the distributions, some versions are labelled Builtin. These versions were added by Talend via the Dynamic distribution mechanism and delivered with the Studio when the Studio was released. They are certified by Talend, thus officially supported and ready to use.

Before you begin

Prerequisites:

  • One of the Talend solutions with Big Data.

Procedure

  1. In the Integration perspective, click File > Edit project properties to open the Project settings dialog box.
  2. Expand the General node and click Dynamic distribution settings to open its view.
  3. From the Distribution drop-down list, select Cloudera.
  4. Set up your local Nexus repository to store the dynamic distribution jar files to be downloaded.
    While not mandatory, this step allows other users or other Studio instances to download the same jar files much faster.
    1. Set up a proxy on your local Nexus repository and link this proxy to the dedicated Talend proxy: https://talend-update.talend.com/nexus/content/groups/dynamicdistribution/.
      The credentials to be used to connect to this Talend proxy are:
      • Username: studio-dl-client
      • Password: studio-dl-client

      When you create your local proxy, you need to define the credentials specific to this local proxy. For an example about how to create a Nexus proxy, see Proxy settings from the Nexus documentation.

    2. On the Dynamic distribution settings view, select the Override default setup check box to activate the Repository field.
    3. In the Repository field, enter the URL of your local proxy and the credentials defined for this proxy.
  5. Click the Dynamic distribution setup button to open the dynamic distribution configuration wizard.
  6. Select the Create new dynamic configuration radio button and click Refresh to display, on the Version drop-down list, the Cloudera versions that are available in the connected Cloudera repository.
  7. Select the Cloudera version for which you want to generate the configuration to be used by the Studio.
  8. Click Finish.

    The Studio starts to retrieve the configuration files for this distribution from the Cloudera repository. This retrieval may take a while.

    Once done, the [Dynamic distribution setup] wizard is automatically closed to bring you back to the Dynamic distribution settings view. The newly generated "dynamic" distribution for the version you previously selected is displayed on the Version list.

  9. You can repeat the operations to add more versions if needs be. Otherwise, click OK to close the Project settings dialog box.

Results

You can then use this new version the same way as you use the built-in distributions provided along with the Studio. You can:

  • Set up the connection to this dynamic distribution in the Repository and reuse this connection in Talend Jobs.

  • Directly use this dynamic distribution in your Jobs. If you build your Job to generate executable files in a zip and need to run the executable files on Windows, do not use the .bat script but use the .ps1 script.

Although you can usually export a Job with its dependencies such as a connection defined in the Repository, the connection to a dynamic distribution cannot be exported the same way. If you need to export such a connection, see Export or import the configuration of a dynamic Big Data platform distribution.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here