Skip to main content Skip to complementary content

Removing unnecessary fields from flight records

A pipeline with a local source, a Data cleansing processor, a Field remover processor, and a test destination.

Before you begin

  • You have previously added the dataset holding your source data.

    Download the file: field_remover-fields.csv. It contains a local dataset with a list of flights and their information (aircraft call sign, flight number, ICAO 24 aircraft address, flight registration, origin, destination, etc.).

  • You also have created the connection and the related dataset that will hold the processed data.

    Here, a Test destination dataset.

Procedure

  1. Click Add pipeline on the Pipelines page. Your new pipeline opens.
  2. Give the pipeline a meaningful name.

    Example

    Remove fields from flight records
  3. Click ADD SOURCE to open the panel allowing you to select your source data, here a local file containing flight information.
    Preview of a data sample with flight records.

    You can see in the sample that a lot of flight numbers are missing.

  4. Select your dataset and click Select in order to add it to the pipeline.
    Rename it if needed.
  5. Click Plus and add a Data cleansing processor to the pipeline. The configuration panel opens.
  6. Give a meaningful name to the processor.

    Example

    mark empty flight numbers as N/A
  7. Configure the processor:
    1. Select Fill cells with value in the Function name list.
    2. Select or enter .number in the Fields to process list.
    3. Select Value in the Use with list and enter N/A in the Value field, as you want to add N/A in places where flight numbers are not listed.
  8. Click Save to save your configuration.

    Look at the preview of the processor to compare your data before and after the cleansing operation.

    Preview of the Data cleansing processor after replacing empty flight numbers with N/A text.
  9. Click Plus and add a Field remover processor to the pipeline. The configuration panel opens.
  10. Give a meaningful name to the processor.

    Example

    remove unnecessary fields
  11. To remove the fields that you do not need, here the ICAO 24 aircraft addresses, the ICAO aircraft type codes and the flight day timestamp, configure the processor like this:
    1. Select or enter .icao24 in the Field to remove list.
    2. Add a new element and select .typecode in the Input list.
    3. Add a new element and select .day in the Input list.
  12. Click Save to save your configuration.

    Look at the preview of the processor to compare your data before and after the deletion operation.

    Preview of the Field remover processor after removing records.

    The unnecessary fields are removed.

  13. Click ADD DESTINATION and select the dataset that will hold your cleansed data.
    Rename it if needed.
  14. On the top toolbar of Talend Cloud Pipeline Designer, click the Run button to open the panel allowing you to select your run profile.
  15. Select your run profile in the list (for more information, see Run profiles), then click Run to run your pipeline.

Results

Your pipeline is being executed, the data is cleansed and the unnecessary fields are deleted according to the conditions you have stated.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here