Reprocessing data
Reprocessing runs a selected set of tasks over a range of processing dates. It is the standard response to a corrective load: source data for a past period has been restated, and the tables derived from it have to be rebuilt for every affected day.
Reprocessing creates a processing instance for every day in the selected interval. It does not modify the original processings for those dates.
Contents
When to use it
Use reprocessing when:
- data has been corrected retroactively and the dependent tables must be rebuilt over a period
- a defect in a transformation has been fixed and the affected days have to be recalculated
- a subset of tasks — not a whole scenario — has to run again across several dates
Use Reopen instead when the correction concerns a single finished processing and should stay attached to it.
The Reprocess definition dialog
The dialog has a selection area at the top and two task lists below.
| Field | Description |
|---|---|
| Scenario | The scenario the tasks belong to. Type to search; pick one from the list. |
| Group | Optional. Narrows the available task list to one group. |
| Date range | Start and end date. A processing instance is generated for every day in this interval. |
| Iteration | A code distinguishing this run from the regular one for the same dates. Maximum 20 characters. |
The Available tasks list shows the tasks matching the scenario and group selected, and can be filtered further by name. Click a task to move it to Selected tasks; click an entry there to remove it.
Run becomes available once the selection is valid. Cancel discards it.
The iteration code
The iteration code matters more than it appears. A processing is identified by its scenario, date and iteration, and a processing cannot be created for a scenario and date that already has the same iteration code. Give the reprocessing run a distinctive code — something that identifies the reason, such as the ticket reference — rather than reusing a code already present.
Iteration codes are compared as text, not as numbers or dates. Where the order of iterations matters, pad them so that string ordering matches the intended order: use 09 and 10 rather than 9 and 10. See Processing.
Selecting the right tasks
The dialog runs exactly the tasks selected. It does not automatically pull in the tasks that depend on them.
Before running:
- select the task that rebuilds the corrected data, then use the Forward tab in the Monitor console on that task to see what depends on it
- include every downstream task whose output would otherwise be inconsistent with the corrected data
- check the multiday mode of the tasks involved — this determines whether the days can be processed concurrently or must run in order
Selecting too few tasks leaves the warehouse internally inconsistent, which is harder to detect than an outright failure.
How the days are executed
Each day in the interval becomes its own processing, and the days are subject to the normal rules:
- the scenario's Max parallel setting limits how many of them can be active at once
- each task's multiday mode determines whether it may run for several dates concurrently, must run in date order, or must wait for previous days to complete
A reprocessing run over a long interval with tasks set to Fully processed will run essentially in sequence, and will take correspondingly longer. Understanding this before starting avoids the impression that the run has stalled. See Parallelism.
Monitoring the run
The generated processings appear in the Active processings panel like any others, identified by their iteration code. They can be paused, resumed and have tasks added individually; see Controlling a processing.
For a long interval, use a scenario or iteration filter to separate the reprocessing run from the regular daily run, which will continue alongside it.
Before starting a large reprocessing
- Confirm the corrected source data is actually in place. Reprocessing against uncorrected data wastes the run.
- Consider the load. A reprocessing run competes with the regular processing for the same connections and resource limits. Where the interval is long, either run it outside the main processing window, or lower its priority so it yields to the daily run. See Prioritization and Resource management.
- Decide the iteration code in advance and record it, so the run can be identified later.
