Shuffling data values to restrict the use of actual sensitive data
With the tDataShuffling component, you can shuffle sensitive information to replace it with other values for the same column from a different row, allowing production data to be safely used for purposes such as testing and training.
This scenario applies only to Talend Data Management Platform, Talend Big Data Platform, Talend Real-Time Big Data Platform, Talend MDM Platform, Talend Data Services Platform, Talend MDM Platform and Talend Data Fabric.
- The tFixedFlowInput component to generate personal data including credit card numbers.
- The tDataShuffling component to shuffle original data and replace values with other values for the same column from a different row.
- The tFileOutputExcel component to output the shuffled dataset.
Prerequisites: Further restricting the use of sensitive data
When shuffling data, it is still advised to mask sensitive data. Remember also to consider relationships between the columns when shuffling data and make sure the original dataset cannot be reconstructed.
In this scenario, last names and first names are grouped together but the email adresses are not in the same group. Consequently, the email column does not relate to the lname and fname columns. Since the email column usually contains information about first names and last names, it may help attackers to reconstruct the original data.
Additionally, the address1, city and email columns are not in any group, so they were not shuffled. This means it is possible to infer, for example, that Robert Damstra lives at 1619 Stillman Court, Lynnwood.
-
To avoid the use of real credit card numbers, you can mask credit card numbers using the tDataMasking component.
-
To avoid the identification of customers with their email addresses, you can mask email addresses using the tDataMasking component.
-
To make it more difficult to read real addresses, you can add the address1 and city columns in other groups.
Setting up the Job
Procedure
- Drop the following components from the Palette onto the design workspace: tFixedFlowInput, tDataShuffling and tFileOutputExcel.
- Connect the three components together using the Main links.
Configuring the input component
Procedure
-
Double-click tFixedFlowInput to open its
Basic settings view in the Component tab.
-
Create the schema through the Edit Schema
button.
In the open dialog box, click the [+] button and add the columns that will hold the initial input data: customer id, credit_card, lname, fname, mi, address1, city, state_province, postal_code, country, phone and email.
Configuring the tDataShuffling component
Procedure
-
Double-click tDataShuffling to display the
Basic settings view and define the component
properties.
-
Click the Advanced settings tab.
In the Partitioning columns table, click the [+] button to add one row.The Job will shuffle the original data rows sharing the same value for the partitioning columns.In the above example, the component is configured to apply the shuffling process to the rows sharing the same value for the country column.
Configuring the output component and executing the Job
Procedure
-
Double-click the tFileOutputExcel component
to display the Basic settings view and define
the component properties.
-
Right-click the output component and select Data
Viewer to display the shuffled data.
tDataShuffling outputs shuffled data. tDataShuffling shuffles values within the first group of columns (credit_card) and within the second group of columns (lname, fname and mi).The shuffling process only applies to the rows sharing the same values for the country column, as defined in the component advanced settings.Sensitive personal information in the input data has been shuffled but data still looks real and consistent. The shuffled data is still usable for purposes other than production.