System Settings: Level Control
Level Control allows users to control the level at which specified data is managed in the metadata catalog. Consider a typical use case for Level Control functionality: Data in a quarterly report file may no longer be required though the metadata is still relevant—the entity containing the records can be demoted from Managed to Addressed where the metadata can be refreshed (though load) as needed or promoted as necessary back to Managed. Promotion and demotion are user-driven actions; a user changes level for a source or entity through grid drop-down level change (screenshot below). Note that Addressed or Registered entities can be published and those entities are temporarily loaded on-demand in order to extract metadata, statistics, profile and sample data and automatically returned to original level.
Level |
Load type definition |
---|---|
Managed |
Data. Objects are fully ingested with full data profiled and validated, generating an extensive catalog of information around the objects and data. |
Registered |
Statistics. Objects exist or have been added to the lake independent of Qlik Catalog or have been demoted from Managed status within Qlik Catalog. Analysis of Registered objects produces summary statistics (profiling and validation information) and metadata about the data is retained. Sample data can be produced (user selected option). |
Addressed |
Metadata. Users are able to see database schema metadata on accessible platforms within the enterprise network without having to move it into the lake. |
Users can:
- Promote data from Addressed to Registered|Managed or from Registered to Managed.
- Demote data from Managed to Registered|Addressed or from Registered to Addressed.
Default level is controlled in three ways:
- Upon Ingest: Inherited from source or set when adding new entities to an existing source
- Source Grid: Users can control object level by checking one or more sources or entities and selecting Change Level
- Inclusion in a process (dataload, dataflow, publish) where level is temporarily promoted and then reset to original level
For more information on setting Level Control in the source module, see Changing Level Control and load logs.
Change Level
Administrative rights are required to access and modify Level Control (governed by access mask) accessed in the administration panel.
If Lock level to only Managed is checked, Default level for Sources is disabled.
If Lock level to only Managed is unchecked, Default level for Sources can be locked to only allow the selected level to be used. This lock and setting can be toggled.
Lock level to only Managed: If this setting has been unlocked to allow Registered and Addressed levels for entities, it cannot be relocked. The default setting for objects in Qlik Catalog is Managed (full data ingest and management). Therefore if administrators do not change anything on this screen, application behavior will not change and all objects will continue to be Managed. When Lock is checked, level is set for all sources and entities to Managed. This setting disables the ability to change level for Source Connections.
Default level for new sources (Top Level Managed Control is Un-checked):
- Managed: If selected (default), Qlik Catalog will keep all entity data (on distribution tables and file system) and profiling and validation metadata gathered upon ingest. If unlocked, users with permissions can change level for individual objects in grid.
- Registered: If selected, Qlik Catalog will keep only sample data along with the profile data and schema metadata but will delete all the other data from distribution tables and files system that may have imported as part of the data ingestion process. If unlocked, users with permissions can change level for individual objects in grid.
- Addressed: If selected, Qlik Catalog will only gather schema metadata upon ingest or if the object is demoted from Managed or Registered, Qlik Catalog will remove all data from distribution tables and file system gathered on the data ingestion process, including sample data and profiling statistics. If unlocked, users with permissions can change level for individual objects in grid.
Level |
Options |
---|---|
Lock level to only Managed |
Once unlocked, this action is not reversible (items cannot be relocked). When lock is on, all sources and entities are managed, and level cannot be modified:
|
Level |
Description |
Managed Full ingest, tightly coupled to all features/capabilities within Qlik Catalog. Data load log type: Data |
Qlik Catalog makes an internal copy of the external source data Sample data can be produced (user specified inheritable core_env.properties: default.record.sampling.probability) Good, Bad, Ugly partitioned record tables maintained in a table view (such as Hive) |
Registered Data has been added into data lake independent of Qlik Catalog OR demoted from Managed. Data load log type: Statistics |
Qlik Catalog will not retain a new copy of the data. When the object is ingested initially Qlik Catalog creates sample data, generates profile statistics and schema metadata Sample data can be produced if sampling probability meets threshold (user selected option in core_env.properties: default.record.sampling.probability) Good, Bad, Ugly tables NOT populated/maintained in a table view Can be used to execute prepare and publish jobs if sample data is produced On demotion from Managed, system removes the data and table partitions On promotion from Addressed, system accesses the source, executes dataload to create sample data, generates profile statistics and schema metadata |
Addressed Source has been defined and analyzed outside of Qlik Catalog OR a Managed/Registered object has been demoted. Data load log type: Metadata |
Schema metadata is acquired and maintained Fields metadata is updated when fields are added or deleted, or changes are made to data type, business name/description, index and properties they are updated in external and internal entities Can be used for creating prepare dataflows and publish jobs where they are temporarily promoted before being returned to their original level Information note
If a field name is changed, previous custom properties will not migrate On demotion to Addressed, only schema metadata is maintained: data in file system and table views do not exist or are no longer maintained in Qlik Catalog. Sample data and profiling statistics are also not retained upon demotion to Addressed. |