Recommended configuration
|
Description |
Location |
Configuration property |
Default/Minimal value |
Recommended value |
|---|---|---|---|---|
|
Maximum number of database connections in the Quartz connection pool |
Talend Administration Center configuration file |
WEB-INF/classes/quartz.properties: org.quartz.dataSource.QRTZ_DS.maxConnections |
30 |
org.quartz.threadPool.threadCount + 3 |
|
Maximum number of concurrent Jobs handled by the Scheduler |
Talend Administration Center configuration file |
WEB-INF/classes/quartz.properties: org.quartz.threadPool.threadCount |
30 |
MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS |
|
Maximum database connections for Talend Administration Center (apart from Quartz) |
Talend Administration Center configuration file |
WEB-INF/classes/configuration.properties: hibernate.c3p0.max_size |
32 |
MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS |
|
Defines the period between each remote Job check |
Talend Administration Center database table configuration |
scheduler.conf.taskStatusRefreshTime |
1 |
|
|
Defines the size of thread pool which checks the latest executions at startup |
Talend Administration Center database table configuration |
dashboard.conf.taskExecutionsHistory.threadPoolSize |
10 |
( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS ) / 25 |
|
Defines the size of thread pool which checks all the tasks at startup |
Talend Administration Center database table configuration |
scheduler.conf.simultaneousThreadsForStatusRefresh |
5 |
MAX_CONCURRENT_TASK_EXECUTIONS / 50 |
|
Defines the number of threads to improve Job execution time when many Jobs are running simultaneously |
Talend Administration Center database table configuration |
remoteDataRetriever.threadPool.size |
30 | The same value as the peak number of Jobs running simultaneously. For more information, see Improving Job execution time section. |
|
Defines the number of maximum opened files for database process |
Host of database server |
Maximum opened files: For example, under Linux set the MySQL configuration property "open_files_limit" and ensure that the system file limit is >= to the formula on the right |
(depends on operating system) |
( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS ) x 3 |
|
Defines the number of maximum connections allowed to the database |
Database server |
Max connections: For example, set the MySQL configuration property "max_connections = 10000" |
(depends on database vendor) |
(org.quartz.dataSource.QRTZ_DS.maxConnections + hibernate.c3p0.max_size) x 1.2 |
|
Defines the maximum number of concurrent connections accepted by the JobServer |
JobServer configuration file |
conf/TalendJobServer.properties: org.talend.remote.server.MultiSocketServer. MAX_CONCURRENT_CONNECTIONS |
1000 |
MAX_CONCURRENT_JOBS_EXECUTIONS x 2 |
| Variable | Description |
|---|---|
|
MAX_CONCURRENT_JOBS_EXECUTIONS |
Maximum expected number of concurrent executed Jobs on the Talend JobServer side |
|
MAX_CONCURRENT_LOGGED_USERS |
Maximum expected number of concurrent logged users (Talend Administration Center + Talend Studio) on Talend Administration Center side |
|
MAX_CONCURRENT_PLAN_EXECUTIONS |
Maximum expected number of concurrent plan executions on Talend Administration Center side |
|
MAX_CONCURRENT_TASK_EXECUTIONS |
Maximum expected number of concurrent task executions on Talend Administration Center side |
You might encounter performance issues if the properties are not properly configured.
The following properties have the biggest influence on performance and memory consumption. By default their values are:
- hibernate.c3p0.max_size=32
- org.quartz.threadPool.threadCount = 30
- org.quartz.dataSource.QRTZ_DS.maxConnections = 30
Default values work if you have less than 10 tasks running at the same time.
If you have more tasks you can find optimal org.quartz.threadPool.threadCount using the following formula:
org.quartz.threadPool.threadCount = 3 x peak quantity of Jobs running at the same time
You don't need too many redundant threads because each thread consumes memory.
The number of connections depends on how fast Talend Administration Center is interacting with the database server and on how many Jobs are running at the same time.
The total number of database connections = org.quartz.threadPool.threadCount + hibernate.c3p0.max_size
The total number of database connections cannot be more than what is configured on the server, for example in MySQL it is 150. The optimal number can also be calculated using the following formula:
org.quartz.dataSource.QRTZ_DS.maxConnections = 3 x peak quantity of Jobs running at the same time
hibernate.c3p0.max_size=3 x peak quantity of Jobs running at the same time
In some cases, if the connection to the database has no delay, you can use the following formula:
org.quartz.dataSource.QRTZ_DS.maxConnections = 2 x peak quantity of Jobs running at the same time
hibernate.c3p0.max_size=2 x peak quantity of Jobs running at the same time