Dafne: Created page with "== Managing System Load and Parallel Processing == When processing a large number of tasks, system resources may become temporarily overloaded. In database environments, th..."

2025-03-25T18:45:09Z

Created page with "== Managing System Load and Parallel Processing == When processing a large number of tasks, system resources may become temporarily overloaded. In database environments, th..."

New page

== Managing System Load and Parallel Processing ==

When processing a large number of tasks, system resources may become temporarily overloaded. In database environments, this typically occurs due to excessive I/O operations, insufficient memory, or high CPU usage. Overload is defined as a situation where a combination of concurrently running tasks completes slower than if they were executed sequentially or with a lower degree of parallelism.

Overloading is generally undesirable and should be identified using database server and operating system monitoring tools. By continuously analyzing execution patterns, it is possible to determine which tasks require exclusive execution or should run only when no other high-resource tasks are active.

The **DWFC framework** provides several tools to manage system load:

* **Global parallelism limit settings**
* **Parallelism limits at the group/connection level**
* **Load limits at the group/connection level**
* **Exclusive dependencies between tasks**

These methods can be combined for flexible workload management.

== Global Parallelism Limit ==

The global parallelism limit is considered every time a task attempts to start. The system evaluates the number of currently running tasks and, if the global limit is reached, prevents additional tasks from starting until one of the active tasks completes—regardless of whether it finishes successfully or fails.

This limit is configured in the application administration panel under **Settings > ParallelLimit**. The current value is displayed on the right side of the screen, and administrators can adjust it as needed. After confirming the change, the new limit becomes active.

This is the simplest method of workload management and is suitable for environments where tasks impose a similar level of load on system resources.

== Parallelism Limit at the Group/Connection Level ==

A more refined method of managing workload is by setting parallelism limits for specific task groups or database connections. These limits define the maximum number of tasks that can run simultaneously within a group or connection.

Each group or connection limit is configured independently. If both a group-level and a connection-level limit apply to a task, the stricter limit is enforced.

For example:
* A task belongs to a group with a parallelism limit of 5, where 3 tasks are currently running.
* The same task is assigned to a connection with a parallelism limit of 6, where all 6 available slots are occupied.
* Since the connection limit is reached, the task will not start, even though the group limit allows it.

In such cases, the system evaluates other tasks based on priority (see [[Task Prioritization]]).

This method is suitable for moderately complex workloads where resource bottlenecks are infrequent or as an initial step before implementing more advanced load management strategies.

== Load Limit at the Group/Connection Level ==

If tasks impose varying loads on system resources, a **weight value** can be assigned to each task. This is configured in the task's **Advanced Settings > Weight** field.

The weight represents the proportion of total system load consumed by a task within a group or connection. For example:
* A task has a weight of 20, and the total allowed load for its connection is set to 100.
* This means the task consumes 20% of the total available resources for that connection.

Weight values are manually assigned by administrators and do not automatically adjust based on actual resource consumption. If a task's resource demands change significantly, its weight must be manually updated.

Before launching a task, the system evaluates the cumulative load of currently running tasks. If running the new task would exceed the defined limit, it is postponed, and other tasks are considered based on priority (see [[Task Prioritization]]).

Typical weight assignments in a Data Warehouse (DWH) environment can be estimated based on the number of records processed:

* Tables with **≤1,000 records**: 1–5% load
* Tables with **≤100,000 records**: 10–30% load
* Larger tables: Require individual evaluation, typically set around 40%

These estimates apply to basic ETL scenarios without complex transformations or large table joins. For more complex workflows, the load should be assessed by manually running the task and monitoring system performance.

== Exclusive Dependencies Between Tasks ==

Exclusive dependencies help prevent conflicts caused by resource contention, such as table locks. This mechanism ensures that two tasks requiring exclusive access to the same resource do not run simultaneously.

For example:
* Two ETL tasks attempt to insert data into the same target table.
* On some databases, this may cause performance degradation or deadlocks.
* In systems like **Sybase IQ**, simultaneous inserts can cause task failures, requiring a retry.

To enforce an exclusive dependency, add a dependency to a task and change its type to **Exclusive**. This ensures that the dependent task will not start until the exclusive task has completed execution.

---

By effectively managing task execution limits and dependencies, DWFC optimizes system resource usage while preventing overload-related performance issues.

Resource management - Revision history

Dafne: Created page with "== Managing System Load and Parallel Processing == When processing a large number of tasks, system resources may become temporarily overloaded. In database environments, th..."