Recovery after an outage

This page describes catching up when processing has been interrupted for one or more days — after an infrastructure failure, an extended source system outage, or a period when the executor was stopped.

The goal is to return to the current processing date as quickly as the dependencies and resources allow, without corrupting the data on the way.

Before restarting

Establish the state before resuming anything:

1. What is the last completed processing date? Use the History of a specific scenario... quick filter in the Monitor console to see the recent runs of the scenario.
2. Are there processings left in an incomplete state? A processing that was active when the executor stopped will have tasks in Invalid status — the executor could not record how they ended.
3. Is the source data available for the missed days? Catching up is pointless until the extracts are in place.
4. Is the cause resolved? Restarting into the same failure produces a second outage to clean up.

Dealing with Invalid tasks

Invalid means the executor stopped while the task was running or prepared. It says nothing about whether the work completed on the target system.

For each Invalid task, establish what actually happened on the target before acting:

• if the work did not complete, use Repeat
• if it did complete, use Set complete so that dependent tasks are released without doing the work twice
• where the connection provider supports it, Repair may be the appropriate action instead

Repeating a task that in fact completed can duplicate loaded data. This check is worth the time it takes.

How the catch-up date is chosen

For a daily scenario, the behaviour after a gap depends on the scenario's schedule base:

Effect of the schedule base after an interruption

Setting	Behaviour
Previous run	The next processing is planned for the day following the last completed one. No gaps are left in the processed dates: after a week's pause, the next run is for six days ago, and the scenario works forward day by day until it catches up.
Calendar/realtime	The next processing is planned for the current day. The missed days are left as a gap in the processed dates.

This is the single most important thing to know before restarting. With Previous run, the system will catch up on its own but will work through every intervening day. With Calendar/realtime, it resumes today and the missed days must be handled deliberately — normally by reprocessing them.

Check the setting before restarting rather than after; see Planning.

Letting the catch-up run in parallel

Working through several days one after another is slow. Dafne can process multiple dates concurrently, which is what makes a catch-up practical.

Two settings control this:

• the scenario's Max parallel — how many processings of that scenario may be active at once. During a catch-up this is often raised temporarily.
• each task's multiday mode — whether that task may run for several dates at the same time

Multiday modes

Mode	Behaviour during catch-up
Parallel	Runs for multiple dates in any order. Suitable for full extracts into a snapshot-based stage and for versioned tables.
Serial	May start before other tasks from previous dates finish, but not before its own execution for previous dates has finished. Used for incremental extracts and transactional data.
Consumed	As Serial, but also requires the dependent tasks of previous dates to have completed. Used where transformation and historisation are separate phases.
Fully processed	Runs only after all previous days are completely processed. The most conservative mode.

Tasks set to Fully processed serialise the whole catch-up at that point in the flow. If a catch-up is progressing far more slowly than expected, this is usually why. See Parallelism.

When multiple dates are active, Dafne prioritises the oldest pending processing. If no task from the oldest day can run, it takes the next available task from a later day. See Prioritization.

Restarting

1. Resolve any Invalid tasks, as above.
2. Confirm the scenario's schedule base and Max parallel are set appropriately for the catch-up.
3. Resume or start the executor — see Executor control.
4. Watch the first planned processing to confirm it picks the date you expect. If it does not, stop and re-check the schedule base before letting it run further.

During the catch-up

• Expect a higher level of parallelism than usual, and watch the target systems for overload. If the catch-up is causing contention, lower the parallelism or load limits rather than letting tasks fail — see Resource management.
• Do not raise Max parallel far beyond what the target database can sustain. Beyond a certain point a higher degree of parallelism exhausts the database engine's resources and execution slows down overall.
• Keep the regular daily run in mind. Once the catch-up reaches the current date, both compete for the same resources.

Handling the gap under Calendar/realtime

If the scenario is set to Calendar/realtime, processing resumes at today's date and the missed days remain unprocessed. Close the gap with a reprocessing run over the affected interval, selecting the tasks that need to be rebuilt and giving the run a distinctive iteration code.

See also

• Parallelism
• Reprocessing data
• Executor control
• Planning