Understanding the Data Locations

Most application data is obviously located inside your database server, you are responsible for regular backup of such database. Upgrading your application may also upgrade the associated database content (database schema, stored procedures, indexes and of course data). It is important to understand to understand that part of the Talend Data Catalog software is implemented as database stored procedures like tracing the lineage. Therefore, a given version of Talend Data Catalog corresponds to a version of the MM software in the application server (tomcat) and a version of the MM software in the database (e.g. tomcat) Therefore make sure you always backup your database before any upgrade.

Furthermore, the upgrade process may take several hours (on large repositories) and also need extra space for temp data during the migration. Therefore, make sure the database has at least 20% free space.

Finally, it is also important to understand that the software installation directory (known as $MM_HOME in this document) also contains some critical application data and application setup customizations that have to be taken into account in your backup or upgrade process, including:

• postgresql/data
  ‎which contains the actual PostgreSQL database data only on Windows and only when configured with the Setup utility (in the "Database Server" tab).
• data
  ‎which contains other application data, including:
  • download/
    • MIMB/ contains third-party download software packages.
  • files/mimb/contains files for upgrade of MIMB Web Services
  • files/mm/ contains files for MM activities:
    • analytics/ contains the generated analytics data files.
    • backups/ contains model backups for model comparison purpose.
    • operations/ contains the operation execution generated files such as a troubleshooting package or a backup ready to download. As these generated files can be very large, it is critical to execute the operation "Delete operation logs and files" either manually (MANAGE > Repository, right click on at the repository root) or automatically (MANAGE > Schedules).
    • sessions/ contains any temporary files for user login session.
  • logs/ for the log files:
    • search/ contains the search query log files.
    • tomcat/ contains the tomcat log files.
  • MIMB/
    • cache/ contains bridge execution files organized as BridgeId/ImportId/mir_Version,nativeVersion
    • parameters/ contains bridge execution parameter files (only for some bridges like DI/ETL with runtime variables) organized as BridgeId/ImportId/Parameters
• search/ contains the lucene search engine index files from previous versions prior to Solr upgrade (they will be automatically fully rebuilt from scratch if the folder is empty which will take a lot of time)
• sessions/ contains files for sessions of MIMB Web Services.
• solr/ contains the Solr search engine index files.
• temp/ for any temporarily files (including from the MIMB bridges)
• upgrade / contains files for upgrade of MM Web Services
• webapps/ for application server (tomcat) cache

Remember that the actual location of this Talend Data Catalog Application Server data directory can be configured with the Setup utility (in the "Application Server" tab).

• conf which contains the MM configuration / customizations organized as follows:
  • conf.properties contains file containing most customizations defined with the Setup utility (in the "Application Server" tab)
  • ModelBridgeList.xml contains the list of enabled bridges and their names
  • resources/ directory containing any User Interface Customizations, in particular MM.properties and MetadataExplorer.xml.
  • Template/ contains the default template files of all the above files/directories, including a potentially new updated ModelBridgeList.xml or MetadataExplorer.xml after cumulative patches.
• tomcat/conf
  ‎with the tomcat.properties file containing the tomcat port and memory customizations defined the Setup utility in the "Application Server" tab, and the keystore file containing the tomcat SSL certificates defined with the Setup utility (in the "Application Server" tab).
• jre/lib/security
  ‎which also contains some SSL customizations defined with the Setup utility (in the "Application Server" tab). It is recommended to not reuse such directory, but rather reinstall the SSL keys with the Setup utility.