Upgrading and uninstalling Data Movement gateway

This release resolves the following issues:

Security

Oracle Java Standard Edition 17.0.8, that ships with Data Movement gateway has several known vulnerabilities. This patch replaces 17.0.8 with 17.0.12, thereby resolving the issues.

Salesforce source

During CDC, for each change, multiple INSERT operations would be added to the associated _ct table (instead of only one). This would significantly increase the number of records in the _ct table.

This release resolves the following issues:

Snowflake JDBC Driver vulnerability

On August 14th 2024, Snowflake issued a security advisory about Snowflake JDBC drivers >= 3.2.6 & <= 3.19.1, some of which were used with previous versions of Data Movement gateway. The vulnerability was patched in Snowflake JDBC Driver 3.20, which is the version that is now installed with the driver installation utility. Existing customers should upgrade their driver at the earliest opportunity, by following the Updating the driver procedure described in Driver setup.

Oracle source

• When using redo event 11.22, missing INSERTs would occur when processing multiple INSERTs on a compressed page that was not compressed prior to the INSERTs.
• In rare scenarios, incorrect parsing of DELETE evens in the redo log record would generate a "The Redo Log DELETE event contains an unknown structure" warning followed by various issues.

SAP Extractor source

The instructions in the YAML file were updated to reflect the correct version of the SAP Java Connector.

Snowflake target

When using Data Movement gateway to connect to Snowflake target via a proxy, the connection would fail with the following error:

500 Failed to connect to Data Movement Gateway

Resolved issues

• Oracle Source

  • Missing INSERTs would sometimes occur in multiple INSERT operations when using redo event 11.22.

  • After upgrading Oracle 19g to the July 2024 patch, UPDATE operations would sometimes not be captured, and the following warning would be shown:

    A compressed row cannot be parsed

• Snowflake Target

  When the task settings were configured to create the control table schema, the task would fail with the following error:

  Failed to delete directory

• Scheduled CDC

  When a task was scheduled to run periodically, it would sometimes fail with following error:

  The task stopped abnormally

• Stuck transformation and storage tasks

  Transformation and storage tasks would sometimes remain in a Queued state for an excessively long time.

• Metadata manager

  Tasks would fail when using the use_manipulation_pk_for_apply feature flag with Store Changes replication.

• Prolonged timeout resulting in issues with SaaS application sources

  Extended the S3 timeout to 6 hours to prevent issues resulting from prolonged timeouts such as losing the token to download the files.

Enhancements

• Added Key Pair authentication to the Snowflake target connector

  Key/pair provides a more robust authentication method than user/password for connecting to Snowflake with your service accounts. This approach is recommended for workloads such as data loading (replication or landing tasks) and transformations.

• Support for retrieving metadata from multiple tables in parallel

  In previous versions, refreshing the metadata on an existing dataset or a newly added dataset would sometimes fail with an error. This enhancement ensures that metadata can be retrieved from multiple tables in parallel without any issue.

Resolved issues

• DECIMAL data type wrongly mapped to STRING in Google BigQuery

  When a source table contained a column with a DECIMAL data type - for example, DECIMAL (38, 20) - preparing the storage task on Google BigQuery would fail with the following error (excerpt):

  Column <n> in <table name> has incompatible types: STRING, BIGNUMERIC at [93:1]

  The issue was resolved by mapping the source DECIMAL data type to DECIMAL in Google BigQuery.

• Schema rule error

  After making changes to an existing schema rule in a data task, the following error would occur:

  QRI SQL error not implemented

• Missing /QTQVC/QRI table from the SAP transport

  When preparing a landing task which connected to a SAP Application source, the task would complete successfully, but the following error would be reported in the repsrv.log log file:

  Invalid object name 'hk1./QTQVC/QRI'

Enhancements

• Support for metadata changes in replication and lake landing tasks without needing to drop and recreate all tables

  Previously, when a metadata change occurred, all tables would be dropped and recreated even if the metadata change did not affect all tables. Now, only the changed tables will be dropped and recreated, thereby improving performance.

• Support for automatic loading of new tables in replication tasks

  Tables created in the source database during the replication data task that match the include pattern, will now be automatically captured during CDC (Change data capture).

• Microsoft SQL Server (MS-CDC) source enhancements

  • Timestamp and transaction ID preservation

    Previously, when reading multiple transactions from Microsoft SQL Server (MS-CDC), Qlik Talend Data Integration would preserve the transaction ID and timestamp of the first transaction only. On the target, this gave the appearance of the records being part of a single transaction. Now, Qlik Talend Data Integration will preserve the original transaction ID and timestamp for each individual record. This benefits customers who wish to leverage the Transaction ID and Timestamp header columns in Change tables.

  • Transaction ID bytes order

    Previously, the bytes in the transaction ID were encoded in reverse order. From this version, the bytes will be encoded in the correct order. Information Customers who would rather preserve the existing behavior can do so using internal parameters. For details, please contact Qlik Support.

• Support for caching SHA-2 pluggable authentication with MySQL

  Qlik Talend Data Integration now supports caching SHA-2 pluggable authentication (caching_sha2_password) when working with MySQL sources or targets, either on-premises or in the cloud. In previous versions, only Native Pluggable Authentication (mysql_native_password) was supported.

• Amazon Redshift data type mapping changes

  BYTES and BLOB data types will now be mapped to VARBYTE on Amazon Redshift instead of VARCHAR.

Newly certified platforms, endpoints and versions

• Data sources

  • PostgreSQL 16.x

  • MySQL 8.1
  • IBM DB2 for z/OS 3.1

• Target platforms

  • PostgreSQL 16.x
  • MySQL 8.1
  • Databricks 14.3 LTS

• Drivers

  • SQL Server ODBC Driver 18.3

End of support

Support for the following database versions has been discontinued:

• All Oracle versions and drivers earlier than Oracle 19.x
• Microsoft SQL Server 2014
• MySQL 5.7
• PostgreSQL 11
• IBM DB2 for LUW 10.5
• IBM DB2 for z/OS: z/OS 2.3

Resolved issues

• SAP (Application) source

  When a captured cluster document change deleted all rows of all its captured tables, missing DELETE operation and unnecessary assertion messages would be encountered.

• Security vulnerabilities

  Updated Microsoft Authentication Library for Java (MSAL4J) and Bouncy Castle to versions without known vulebrabilities.

• Snowflake target

  • The task would sometimes fail when using Snowflake internal storage.
  • The task would fail when the target schema name was Japanese Katakana.

• Oracle source

  When resuming a task with an Oracle source, the task would continue to wait for a deleted Archived Redo Log instead of failing with an appropriate error.

Enhancements

• Snowflake metadata schema

  From this version, the Snowflake metadata schema (for staged files) will be created if it does not exist.

Resolved issues

• After a change was made to a Rename dataset rule (that concatenated the table name with the schema name) and a View was added to the landing task, the Prepare operation would fail with the following error:

  QRI SQL error not implemented.

• Updated the java_file_factory component to a version without any known vulnerabilities.
• Updated org.postgresql:postgresql to a version without any known vulnerabilities.
• When moving data to SQL Server, the Prepare operation would fail when index names exceeded 128 characters. The issue was resolved by altering the logic to create shorter index names.

• When moving data from SQL Server, a missing column name in the table definitions would cause an infinite notification loop in the repsrv.log file, with the following message:

  mssql_resolve_sqlserver_table_column_attributes(...) failed to find column

Enhancements

• Parallel metadata retrieval

  Added support for concurrently retrieving the metadata for multiple tables.

• Improved monitoring performance

  The monitoring information for landing and replication tasks will now be updated every 10 seconds (instead of every 60 seconds), providing a more precise indication of their current status.

• Newly supported region

  Japan is now supported as a Qlik Cloud tenant region.

Resolved issues

• The retry interval between the data gateway and Qlik Cloud would continually increase, but never be reset (unless the service was restarted).
• When moving data from a SaaS application source, tables would sometimes enter an error state during reload.

• Tasks with a MySQL source would sometimes fail with the following error during CDC:

  Read next binary log event failed; mariadb_rpl_fetch error 0 Error reading binary log.

• Previously, CDC audit events would only be logged for landing tasks. Now, they will be logged for replication tasks ask well.
• When moving data from SQL Server (MS-CDC), tasks with numerous tables would sometimes take several hours to start.

• When the source table contained CLOB columns and the "Limit LOB size" value exceeded 10240, replication to Snowflake would fail with the following error:

  Invalid character length: 0

Support for installing Data Movement gateway on Red Hat 9.x

Customers can now install Data Movement gateway on Red Hat 9.x or on any corresponding and compatible Linux distribution.

Setting up Data Movement gateway

Changes to Data Movement gateway service commands

The commands for stopping, starting and checking the status of the Data Movement gateway service have changed.

Data Movement gateway service commands

Changes to Amazon Redshift data type mapping

• The BOOLEAN data type, which was mapped to VARCHAR(1) in Amazon Redshift, will now be mapped to BOOLEAN.
• The BYTES and BLOB data types, which were mapped to VARCHAR(1) in Amazon Redshift, will now be mapped to VARBINARY (length).

Newly supported databases, database versions, and driver versions

This section lists the newly supported databases, database versions, and driver versions.

• Newly supported data source versions and editions

  The following data source versions are now supported:

  • Azure Database for MySQL - Flexible Server (Supported via the MySQL source connector)
  • MariaDB 10.4 - 10.11 (previously 10.4 and 10.5)

• Newly supported target data platforms and editions

  The following data target versions are now supported:

  • Azure Database for MySQL - Flexible Server (Supported via the MySQL target connector)
  • Databricks: Databricks 13.3 LTS and Serverless SQL Warehouse

• Newly supported SAP HANA driver version

  Customers with a SAP HANA source who want to install Data Movement gateway on Red Hat Linux 9.x, must install SAP HANA ODBC 64-bit Driver version 2.0.19 or later.

End-of-support database versions

This section provides information about end-of-support database versions.

• Support for the following data source versions has been discontinued:

  • Oracle 11.x
  • SAP HANA 1.0

Resolved issues

• Installing Data Movement gateway without providing a server password would not allow the tenant and proxy URLs to be configured in one command.
• Moving data from a Salesforce (SaaS application) data source would print a large number of redundant warnings, thereby impacting data loading performance.
• When retrieving changes for a SaaS application data source, if an error occurred retrieving changes for one of the tables, that table would be suspended and removed from the pipeline. Now, when encountering an error, the task will try to retrieve the changes up to three times before suspending the table.

New data lake targets for Replication projects

We are continuing to expand the supported targets for the Replication project in Qlik Cloud Data Integration, in addition to Amazon S3, you can now choose Azure Data Lake Storage (ADLS) and Google Cloud Storage (GCS) for data lake delivery, as Parquet, JSON or CSV file formats.

Resolved issues:

• IBM DB2 for LUW source connector

  The connection to the IBM DB2 for LUW would fail when the size of files needed for the connection (such as the SSL client certificate and the keystore file) exceeded 4 KB.

• Driver for IBM DB2 for LUW and IBM DB2 for z/OS source connectors

  The DB2 driver installation would fail when using the driver installation utility.

Support for Microsoft Fabric as a data pipeline target

Microsoft Fabric joins the ever-expanding list of data warehouses that can be used as targets in data pipeline projects.

Resolved issues:

Updated the Snowflake driver version in the driver installation utility.

Expanded Support for Private Connections

A private connection can be used to ensure your data traffic remains secure and compliant. It simplifies both network management and security of your VPC (Virtual Private Cloud) without the need to open inbound firewalls ports, proxy devices or routing tables. Qlik Cloud Data Integration already supports private connections to Snowflake, Microsoft SQL Server and Amazon S3 data pipeline targets. With this release, customers can additionally utilize private connections when moving data to Databricks, Microsoft Azure Synapse, Google BigQuery, and Amazon Redshift.

PostgreSQL-based data sources

• All PostgreSQL-based data sources - Multirange data type support: The following multirange data types are now supported with all PostgreSQL-based data sources (on-premises and cloud).

  • INT4MULTIRANGE
  • INT8MULTIRANGE
  • NUMMULTIRANGE
  • TSMULTIRANGE

• AWS Aurora Cloud for PostgreSQL data source - Non-superuser role support: The user specified in the PostgreSQL connector no longer needs to have the superuser role to move data from an AWS Aurora Cloud for PostgreSQL data source. This is especially useful for organizations with corporate security policies that prevent them from granting superuser access to non-privileged users.

Microsoft Azure SQL (MS-CDC) data source: Geo-replication support

Customers moving data from a Microsoft Azure SQL (MS-CDC) data source can now use a Geo Replica database.

Oracle data source: HSM (Hardware Security Module) support

Qlik Cloud Data Integration now supports reading data from Oracle encrypted tablespaces and encrypted columns during CDC.

New regions support

Qlik Cloud Data Integration now supports tenants in DE and UK regions.

Newly supported versions and third-party software

This section lists newly supported database and driver versions.

• Newly supported data source versions. The following data source versions are now supported:

  • Microsoft SQL Server 2022
  • Oracle 21c
  • PostgreSQL 15.x
  • DB2 13.1 (when working with IBM DB2 for z/OS)
  • IBM DB2 for iSeries 7.5

• Newly supported target data platform versions. The following data target versions are now supported:

  • Databricks (Cloud Storage): Databricks 12.2 LTS and Databricks SQL Serverless

• Driver versions. The following ODBC driver versions are now supported:

  • IBM Data Server Client 11.5.8 for IBM DB2 for z/OZ and IBM DB2 for LUW
  • Simba ODBC driver 3.0.0.1001 for Google Cloud BigQuery
  • MySQL ODBC Unicode Driver 64-bit 8.0.32

Microsoft SQL Server driver - Required upgrade

Customers moving data to or from Microsoft SQL Server need to upgrade their SQL Server ODBC driver version to 18.x or later. Note that continuing to use SQL Server ODBC Driver 17.x might result in data errors. Upgrading the driver can be done using either the driver installation utility or manually. For instructions, see Driver setup.

End-of-support database versions

This section provides information about end-of-support database versions.

• End-of-support data source versions. Support for the following data source versions has been discontinued:

  • PostgreSQL 10
  • MySQL 5.6
  • MariaDB 10.3

• End-of-support target data platform versions. Support for the following data target versions has been discontinued:

  • Databricks 9.1 LTS

2022.11.74

August 15, 2023

Replication support

We take a cloud-first approach that enables rapid innovation and adoption. However, that does not mean that we are cloud-only. As part of our continued commitment to improving the long-term value for our customers, we are pleased to announce the release of a new use case for Replication when creating data projects. The new Replication use case is in addition to the existing ability to create data pipelines for all your data integration needs, such as Data Warehouse modernization.

The Replication project supports real-time data replication from supported data sources to a supported target.

Starting with this latest release, the first target that supports replicating data in real time, is Microsoft SQL Server. In addition to supporting Microsoft SQL Server on-premises, the following cloud providers are also supported:

• On-premises
• Amazon RDS
• Google Cloud
• Microsoft Azure (Microsoft Azure Managed Instance and Microsoft Azure Database)

Microsoft Azure Synapse Analytics target - Required driver upgrade

Customers moving data to Microsoft Azure Synapse Analytics need to upgrade their SQL Server ODBC driver version to 18.x or later. Note that continuing to use SQL Server ODBC Driver 17.x might result in data errors. Upgrading the driver can be done using either the driver installation utility or manually. For instructions, see Driver setup.

New "Load data from source" option for SaaS application sources

A new Load data from source option was introduced, which allows customers to read their data directly from the source during Full Load, instead of using the cached data.

For more information on this option including use cases, see Landing settings.

Updated CA certificates

Data Movement gateway 2022.11.74 includes updated CA certificates, which are needed for authenticating the Qlik Cloud tenant. The updated CA certificates also provide support for the Ireland and Frankfurt regions. Therefore, customers with Qlik Cloud tenants in Ireland or Frankfurt who want to use Qlik Cloud Data Integration must upgrade to this version.

Resolved issue

Tasks landing data from an Oracle source would sometimes fail when a wide table contained unused or unsupported columns, or LOB columns that were not replicated.

2022.11.70

June 28, 2023

Improved driver installation utility for SAP clients

In previous versions, customers needed to run the "source arep_login.sh" command when installing SAP clients. From this version, it is no longer necessary to run this command.

Updated CA certificates

This version includes updated CA certificates, which are needed for authenticating the Qlik Cloud tenant.

Resolved issues

• When a replication task on Data Movement gateway failed and recovered automatically, the recovered state would not be communicated to the Landing data asset in Qlik Cloud.

• End-to-end encryption for Data Movement gateway would not be enabled by default, and was controlled by runtime flags.

2022.11.63

May 2, 2023

Driver installation utility

This version introduces a driver installation utility that eliminates the need to manually install and configure drivers. The new utility shortens the installation process while significantly reducing the possibility of user error. When the utility is run, the required driver is downloaded automatically, if possible, and installed. If the driver cannot be downloaded (DB2 drivers require login, for example), all you need to do is download the driver, copy it to a dedicated folder on the Data Movement gateway machine, and run the utility.

For an example of using the driver installation utility to install a PostgreSQL driver, see Prerequisites

Snowflake cloud data platform

The Snowflake connector now supports 4-byte emoji characters.

Support for Azure Database for PostgreSQL - Flexible Server as a data source

The PostgreSQL connector can now move data from Azure Database for PostgreSQL - Flexible Server.

Support for Cloud SQL for PostgreSQL as a data source

The PostgreSQL connector can now move data from Cloud SQL for PostgreSQL.

Newly supported data source versions

This version introduces support for the following new data source versions:

• PostgreSQL 14
• DB2 (for IBM DB2 for z/OS) 12.1
• IBM DB2 for z/OS 2.5

Newly supported cloud data platform versions

Databricks 11.3 LTS

Discontinued support for the following data source versions

The following data source versions are no longer supported:

• DB2 (for IBM DB2 for z/OS) 11
• PostgreSQL 9.6
• Microsoft SQL Server 2012
• MariaDB 10.2

Resolved issues

This version resolves the following issues:

• Oracle data source: When stopping and resuming a task, the task would sometimes fail with a “Failed to set stream position on context” error.
• SAP Application source: Changes would not be captured during the landing task.