How to Read Pipelines

A pipeline is a representations of a workflow. Oxford Dictionary defines workflow as the sequence of industrial, administrative, or other processes through which a piece of work passes from initiation to completion.

Stuff that makes up a pipeline workflow.

Concourse CI uses only a handful of abstract concepts to help you implement the most advanced workflows. Grab a piece of paper and draw the following relationships between concepts to absorb its mental model once and for all.

  • The highest-level concept and a container for a workflow is a pipeline.
  • A pipeline is made up out of a set of jobs.
  • Each job contains one or more steps. Steps are typically sequential, but can be executed in parallel.
  • Steps may accept multiple inputs or produce multiple outputs. Anything that can be represented on a filesystem as a directory with files in it can serve as input or output of a step.
  • When inputs are passed between steps within a job they can remain just that: inputs/outputs. For passing inputs/outputs between jobs, you must use resources. A resource is an input/output set whose state is retrieved/stored externally by a job, e.g. a git repo or an S3 object.
  • A good way to think about a resource is as a feed of versions. Resource Versions are instantiations of the resource. For example, a path to a Github repo could be defined as a resource, but a git repo checked out at a specifc commit is a resource version

How to read a pipeline

Visually, a pipeline is a set of linked jobs (green rectangles) with progress of work flowing through resources (black rectangles) from left to right (build -> test -> deploy), Example pipeline

  • A solid line leading from a resource into the job means that resource triggers the execution of the job. For example, a job will trigger on every push to the git repo.
  • A dashed line from a resource into a job means that a resource might be used by some of the steps but it's not triggering the job.
  • If a resource links two jobs, the job on the right requires the job on the left to successfully complete execution, before it can proceed with a given version.

Click through to the build job: The build job

Controls, from the top right:

  • Encircled plus sign = start a new build of this job the latest inputs. Clicking this button will result in an integer build number in the bottom left corner, e.g. 2
  • Semi-encircled dotted plus = start a new build of this job without checking for new inputs. Use inputs from the recently completed build. Clicking this button results in a subscripted build number, e.g. 2.1