Run a Pipeline on a Specific Commit¶
Sometimes you need to see the result of merging different commits to analyze and identify correct combinations and potential flows in data collection and processing. Or, you just want to rerun a failed job manually.
Pachyderm enables you to run your pipelines with old commits or in different handpicked combinations of old commits that are stored in separate repositories or branches. This functionality is particularly useful for cross-pipelines, but you can also use it with other types of pipelines.
For example, you have branches A
, B
, and C
. Each of these branches has had commits on it at various times, and each new commit triggered a new job from the pipeline.
Your branches have the following commit history:
- Branch
A
has commitsA1
,A2
,A3
, andA4
. - Branch
B
has commitsB1
andB2
. - Branch
C
has commitsC1
,C2
, andC3
.
For example, you need to see the result of the pipeline with the combination of data after A4
, B1
, and C2
were committed. But none of the output commits were triggered on this particular combination.
To get the result of this combination, you can run the pachctl run pipeline cross-pipe
command.
Example
pachctl run pipeline cross-pipe A4 B1 C2
This command triggers a new job that creates a commit on the pipeline's output branch with the result of that combination of data.
Because A4
is the head of branch A
, you can also omit the A4
commit in the command and specify only the C2
and B1
commits:
Example
pachctl run pipeline cross-pipe C2 B1
Pachyderm automatically uses the head for any branch that did not have a commit specified. The order in which you specify the commits does not matter. Pachyderm knows how to match them to the right place.
Also, you can reference the head commit of a branch by using the branch name.
Example
pachctl run pipeline cross-pipe A B1 C2
This behavior implies that if you want to re-run the pipeline on the most recent commits, you can just run pachctl run pipeline cross-pipe
.
If you try to use a commit from a branch that is not an input branch, pachctl
returns an error. Similarly, specifying multiple commits from the same branch results in error.
You do not need to run the pachctl run pipeline cross-pipe
command to initiate a newly created pipeline. Pachyderm runs the new pipelines automatically as you add new commits to the corresponding input branches.