Parallelism

Multiday Processing in DWH

Overview

During daily processing, unexpected failures may occur due to extract unavailability, software or hardware failures, or other issues. Depending on the nature of the failure, recovery may take several days. In such cases, it is essential to catch up on the delayed processing as quickly as possible. Additionally, multiple days of data processing may be required after corrective loads, where certain data is retroactively adjusted, necessitating the reprocessing of dependent tables for an extended period.

In a typical DWH architecture, processing is often sequential, meaning that the next day’s processing can only start once the current day’s processing is complete. However, a more detailed analysis often reveals opportunities for parallel execution of some tasks, independent of strict chronological order. To maximize the benefits of parallel processing, the following concepts should be implemented:

• Snapshot-based staging: The staging layer should support multiple snapshots simultaneously, allowing ETL processes to work with versioned tables.
• Full extracts in staging: Even if source systems provide incremental data, converting these into full extracts for staging is beneficial. This applies mainly to entity data or balance-related data rather than transactional data.

Implementing these recommendations allows for greater parallelism in processing, which, when properly managed, significantly reduces downtime and accelerates data recovery. Without parallel execution, certain tasks remain idle due to unmet dependencies, leading to underutilization of available resources. In multiday processing, tasks from different days can be initiated concurrently, prioritizing older pending tasks first. If no task from the oldest pending day can be executed, the next available task from subsequent days is selected, provided dependencies and resource limits are met.

Configuring Parallel Execution

Not all task types can be executed in parallel or out of sequence. Various levels of control allow for managing parallel execution:

Scenario-Level Control

The maximum number of concurrent executions for a given scenario is configured via the Max parallel attribute. Setting a higher value permits multiple executions to run simultaneously without waiting for previous ones to finish. For scheduling, the scenario query must return a new processing date or iteration even if the current processing is still in progress.

Task-Level Control with Multiday Mode

The Multiday mode attribute on tasks or task groups further refines execution control. The available options are:

• Group Setting: Applicable at the task level, where the task inherits the group’s settings.
• Parallel: Allows the task to run for multiple dates in any order, typically used for full extracts in a snapshot-based stage or versioned tables in higher layers.
  

  Parallel execution
  • Parallel execution of a task across multiple processing iterations.

• Serial: The task can run before other tasks from previous iterations are completed, but it cannot start until its own execution from previous iterations has finished. This mode is commonly used for incremental extracts and transactional data.
  • Serial execution of a task across multiple processing iterations.
    

    Serial execution

• Consumed: Similar to Serial mode but with an additional restriction that dependent tasks must have completed in previous iterations before execution can proceed. This is useful when processing is divided into transformation and historization phases, ensuring data consistency before historization.
  • Execution of a task across multiple iterations only after dependent tasks have completed their output processing.
    

    Execution after all previous instance finished

• Fully Processed: The most conservative mode, where the task is executed only after all previous days’ processing has been fully completed. This mode is used when the task depends on a significant portion of the data model.
  • Execution of a task only after complete processing of all previous days.

Benefits of Multiday Processing

By allowing parallel execution where possible, multiday processing ensures optimal resource utilization and faster data recovery. The correct configuration of dependencies and execution modes helps maintain data integrity while minimizing processing time.