Skip to main content Skip to complementary content

Retrieving Microsoft Azure web applications, filtering them and creating gists

This scenario aims at helping you set up and use connectors in a pipeline. You are advised to adapt it to your environment and use case.

Example of a pipeline created from the instructions below.

Before you begin

Procedure

  1. Click Connections > Add connection.
  2. In the panel that opens, select the type of connection you want to create.

    Example

    Azure OAuth Client credentials
  3. Select your engine in the Engine list.
    Information noteNote:
    • It is recommended to use the Remote Engine Gen2 rather than the Cloud Engine for Design for advanced processing of data.
    • If no Remote Engine Gen2 has been created from Talend Management Console or if it exists but appears as unavailable which means it is not up and running, you will not be able to select a Connection type in the list nor to save the new connection.
    • The list of available connection types depends on the engine you have selected.
  4. Select the type of connection you want to create.
    Here, select HTTP Client.
  5. Fill in the connection properties and URL address of the server to be invoked as described in HTTP Client properties, check the connection and click Validate: In this example, the https://graph.microsoft.com/v1.0/ service listing Microsoft Azure apps is invoked.
    Configuration of a new HTTP Client connection.
    1. Base URL: https://graph.microsoft.com/v1.0/
    2. Authentication type: Select OAuth 2.0.
    3. Flow: Select Client credentials.
    4. Authentication mode: Select FORM.
    5. Token endpoint: Enter https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token.
    6. Client ID and Client secret: Enter the Azure web application ID and client secret.
    7. Additional parameters: Enter https://graph.microsoft.com/.default as the scope attribute.
  6. Click Add dataset to validate your connection and create the corresponding dataset.
  7. In the Add a new dataset panel, name your dataset.

    Example

    Azure apps
  8. Fill in the required properties to invoke the service and click View sample to see a preview of your dataset sample:
    Configuration of a new HTTP client dataset.
    1. Type: Select Batch as you want to invoke the service only once.
    2. HTTP method: Select GET.
    3. Path: Enter Applications so that the final URL will be https://graph.microsoft.com/v1.0/Applications.
    4. Disable the Parameters option.
    5. Enable the Query parameters option and add these query parameters: 
      ("$select", "appId,displayName,createdDateTime,publisherDomain")

      and

      ("$filter", "createdDateTime ge 2022-12-01T00:00:00.000Z")
    6. Disable the Request headers and Request body options.
    7. Response body format: Select JSON.
    8. Extract Sub-part of the JSON: Enter .value to retrieve records from this JSON field.
    9. Returned content: select Body.
  9. Click Validate to save your dataset.
  10. Click Connections > Add connection.
  11. In the panel that opens, select the type of connection you want to create.

    Example

    Git PAT
  12. Select your engine in the Engine list.
  13. Fill in the connection properties and URL address of the server to be invoked as described in HTTP Client properties, check the connection and click Add dataset: In this example, the https://api.github.com service is invoked.
    Configuration of a new HTTP Client connection.
    1. Base URL: https://api.github.com
    2. Authentication type: Select Bearer token.
    3. Bearer Access token: Enter the personal access token you have previously created.
  14. Click Add dataset to validate your connection and create the corresponding dataset.
  15. In the Add a new dataset panel, name your dataset.

    Example

    GIST CREATE
  16. Fill in the required properties to invoke the service and click View sample to see a preview of your dataset sample:
    Configuration a new HTTP client dataset.
    1. HTTP method: Select POST.
    2. Path: Enter gists so that the final URL will be https://api.github.com/gists.
    3. Disable the Parameters and Query parameters options.
    4. Enable the Request headers option and add these lines for the Main query: 
      ("Accept", "application/vnd.github+json")

      and

      ("X-GitHub-Api-Version", "2022-11-28")
    5. Enable the Request body option, select JSON and paste these lines: 
      {
  "description":"Azure application {.input.displayName}",
  "public":false,
  "files":{
     "AzureApp_{.input.appId}.xml":{"content":"<?xml version=\"1.0\"?>\n<application>\n  <id>{.input.appId}</id>\n  <name>{.input.displayName}</name>\n  <createdDate>{.input.createdDateTime}</createdDate>\n  <domain>{.input.publisherDomain}</domain>\n</application>"
          }
     }
}
    6. Answer body format: Select TEXT.
    7. Returned content: select Body.
  17. Click Validate to save your dataset.
  18. Click Add pipeline on the Pipelines page. Your new pipeline opens.
  19. Give the pipeline a meaningful name.

    Example

    Retrieve Azure apps & filter them on creation date
  20. Click ADD SOURCE and select your source dataset, Azure apps, in the panel that opens.

    You can see a sample of the retrieved Microsoft Azure apps.

    Sample of the Microsoft Azure apps data.
  21. Click the ADD DESTINATION item and select the destination dataset, GIST CREATE, in the panel that opens.
  22. On the top toolbar of Talend Cloud Pipeline Designer, click the Run button to open the panel allowing you to select your run profile.
  23. Select your run profile in the list (for more information, see Run profiles), then click Run to run your pipeline.

Results

Your pipeline is being executed, the Azure apps that were invoked from the Microsoft Graph service have been filtered and sent to GitHub GIST where XML content for each application has been created:
XML content of two Azure apps sent to GitHub GIST.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here