Investigating a stalled processing
This page covers a specific and common situation: a processing is active, no task is failing, but nothing is running. The Running counter is zero while the Waiting counter is not.
A stalled processing is almost always waiting for something rather than broken. The job is to find out what.
Contents
Use the waiting reasons
Select any waiting task in the Monitor console. If the task is blocked, a pending icon appears on the right of the detail panel. Opening it lists the waiting reasons — the specific conditions that task is still waiting on.
This is the first thing to check. It answers directly what the rest of this page works out indirectly.
The usual causes
The executor is not running
Check the executor status button in the filter bar. If it reads Paused, no new tasks will be started although running ones continue. If it reads Stopped or Stopping, nothing will start at all.
The task action menu also reports this: it shows Scheduler stopped and disables the actions. See Executor control.
The processing is paused
A single processing can be paused while the executor keeps running other processings normally. Its status in the Active processings panel reads Paused, and the task action menu shows Processing not active.
Resume it from the processing actions menu; see Controlling a processing.
The tasks are waiting on dependencies
Select a waiting task and open the Backward tab. It lists what the task is waiting for, with each entry's current status. Click through to the entry that has not completed, and repeat, until you reach the task at the head of the chain.
What you find at the head is usually one of:
- a task in Error that has stopped the chain — see Handling failed tasks
- a task On hold that somebody held earlier and did not release
- a group dependency that is waiting for every task in a group, where one member has not finished
Remember that a group dependency is effectively a dependency between every combination of tasks in the two groups, so one unfinished member blocks everything downstream of the group. See Dependencies.
A dependency cycle
Tasks in a dependency cycle can never start, because their dependencies will never all be satisfied.
Dafne checks for cycles when a processing is planned. A cycle found at that point is reported. However, if dependencies are changed while a processing is already planned, the new cycle is not detected, and the affected tasks will wait indefinitely. This has to be resolved manually — correct the dependency definition, then release the blocked tasks.
This is the reason for not changing dependencies during a run; see Daily supervision.
Resource limits are reached
If tasks are ready to run but none start, and no dependency is outstanding, the limit is a resource one. Dafne will not start a task that would exceed:
- the global parallelism limit
- the parallelism limit on the group or connection
- the load limit on the group or connection, evaluated against the weights of the tasks already running
Where a group limit and a connection limit both apply, the stricter one wins. A task can therefore be blocked by a connection that is fully occupied even though its group has capacity. See Resource management.
An exclusive dependency produces the same symptom for a single pair of tasks: the second will not start while the first is running.
The scenario has not triggered
If the processing has not appeared at all, rather than stalling, the scenario has not triggered yet. The Active processings panel shows waiting scenarios with the time they will start.
For an SQL-based scenario, the panel also shows the polling count, the time and status of the last polling, the value the planning query returned, and the error message if the polling failed. That is usually enough to tell whether the query is returning no date, returning a date that has already been processed, or failing outright.
Also check the generic constraints — scenario validity dates and the Not before / Not after window — and, for a daily scenario, whether all previous instances have completed. See Planning.
The maximum parallel count is reached
A scenario will not plan a new instance beyond its Max parallel setting. If earlier dates are still open, the next one waits. This is expected behaviour, not a fault; either let the open runs finish or raise the limit deliberately. See Parallelism.
Checking a single task's time window
A task can carry its own Not before / Not after window. Where one is set, it appears in the Info tab of the task detail. A task inside a running processing will wait until its window opens.
