Managing Pipeline Artifacts

A pipeline may produce an artifact, such as compiled binaries, minified JavaScript, a code that has been tested, or a text file. These artifacts are consumed by the succeeding pipelines. If multiple pipelines are producing an artifact that is going to be consumed by the next pipeline in the workflow, you can choose which preceding pipeline should be the source of that artifact. This is demonstrated in the following example:

box: alpine

build:
  steps:

echoa:
  steps:
    - script:
        name: echo AA
        code: sleep 5 && echo 'AA' > forward.txt

echob:
  steps:
    - script:
        name: echo BB
        code: sleep 8 && echo 'BB' > forward.txt

readfile-a:
  steps:
    - script:
        name: read forward.txt
        code: cat echoa/forward.txt

readfile-b:
  steps:
    - script:
        name: read forward.txt
        code: cat echob/forward.txt

In this example, the pipeline build fans out to the following two pipelines: echoa and echob. Each of the pipelines, echoa and echob produce an artifact called forward.txt containing the text AA and BB respectively. The pipelines echoa and echob fan into the following two pipelines: readfile-a and readfile-b.

You can specify whether the pipelines readfile-a and readfile-b obtain the artifact forward.txt from echoa or echob by collecting and storing the artifact in that pipeline’s directory, for example <faninPipelineName1>/ and <faninPipelineName2>/, and then referencing it in the required pipeline.

Using the Workflow Editor to create fan-in connections

Fan-in connections come in handy when you want a particular pipeline to depend on specific preceding pipeline(s) to complete executing and also select the source of the artifact it is going to consume.

In the following example, the system-test pipeline is waiting for the unit-test and integration-test pipelines to finish, and is also going to consume artifacts from both the pipelines. Establishing fan-in connection

To establish a fan-in connection using the Workflow Editor:

  1. Click a pipeline on which the successive pipeline is depending. For example, integration-test.
  2. Click the pipeline that you want to fan into. For example, system-test.

By default, only the artifact of the pipeline that originally preceded is made available to the pipeline that now has a fan-in connection. If you want the artifacts of additional or all the previous pipelines to be made available - or remove any fan-in sources, click the edit icon on the pipeline you that are currently configuring. The Manage source pipeline dialog box is displayed.

  1. On the Manage source pipelines dialog box:
    1. Select the pipelines that should finish executing before this pipeline starts.
    2. Select a pipeline which would be the source of an artifact for this pipeline. You can either select a particular pipeline, or you can select the all pipelines option. If you select the all pipelines option, then the artifacts from all the preceding pipelines will be stored in a separate sub-folder.
  2. Click Update to save the configuration and to return to the Editor.