Why Use Concourse CI?

By Camille Spain

Concourse CI is suitable for building, testing, and deploying the most complex software. Concourse CI pipelines are stable and scalable, feature excellent performance, and make builds reproducible and debuggable. Additionally, Concourse CI supports advanced security practices and can be deployed in air-gapped environments.

Handle Complexity

Achieve a high level of software quality and deliver it reliably. As the size and complexity of a piece of software grows, the more challenging successful CI/CD becomes. Reducing complexity isn't always an option, as in the case of organizations that must adhere to a range of government regulations. Concourse CI provides the capabilities to handle complex codebases effectively.

Concourse CI is designed to promote loose coupling and compositional patterns by providing small, abstracted pieces of functionality. This makes it flexible enough to create any pipeline required without resorting to breakage-prone workarounds.

A different approach to build system inputs and outputs makes building software with a large dependency graph easier. Exercise fine-grained control over inputs by pinning a particular commit, tag, or branch so that the product is always built with the correct dependencies, instead of just the latest. Anything that can be represented as a file can be an input and Concourse CI accepts multiple inputs, rather than a single repo or commit. Concourse CI can also produce multiple arbitrary outputs, including a commit, an s3 artifact, VMWare OVA file, or a deployment.

Stability and Scalability

Concourse CI can sustain high deployment frequency and low lead times. As software complexity increases, the need for robust, scalable pipelines increases. Difficulties with scaling can cause the CI/CD system to become a major bottleneck. Concourse CI is designed to scale without encountering stability issues.

Designed as a distributed system, horizontally scaling Concourse CI is trivial. It can be run as a 12 Factor App. The system state is restricted to the database, and the web nodes coordinate via the database to manage work distribution. Workers and web nodes are stateless processes, which makes adding and removing them to accommodate the workload simple.

Detailed metrics provide insight into workload distribution and system health, so the operator can monitor system stability and knows when and how the build system needs to be scaled. Workload distribution can be configured with container placement strategies that alter container and volume placement logic across the worker cluster to optimize infrastructure usage and prevent certain workers from becoming overloaded.

High Performance

Reduce overall build time by as much as 80%. Concourse CI's caching system means pre-built artifacts are downloaded only once and intermediate artifacts can be cached to speed up repeated runs of a task. Additionally, each run of a build will only rebuild the parts that changed.

Reproducible and Debuggable Builds

Concourse CI improves productivity and keeps the value of your CI system from deteriorating. Poor build reproducibility leads to wasted time retrying erroneous failed builds and uncertainty about whether test failures are real. Inadequate debugging tools exacerbate build quality problems. Concourse CI maintains high build quality with less effort by providing effective debugging tools and making deft use of containerization technology.

Computer environments are complex and rapidly changing, which can make build reproducibility difficult to achieve. Concourse CI handles this problem by running every build step in a container and creating fresh containers and volumes each time a build is run, so the system is always in a known state. Reusing containers from build to build can cause unexpected behavior and failed builds. Superior version control keeps all inputs and outputs within the build system tightly controlled and predictable. Additionally, each task can be configured as a pure function that declares its own image and manages all of its own dependencies. This declarative model avoids the hazards of managing dependencies and state on workers.

Debugging flaky builds is both possible and easy with Concourse CI. Failed builds can be rerun with the same set of inputs in new containers via the fly CLI or web UI. Individual containers can be troubleshot with the fly CLI intercept command, which opens an interactive shell in the chosen container. The visualization provided by the web UI makes locating and understanding problems more intuitive.

Advanced Security

Concourse CI can reduce an organization's attack surface. Inadequately secured CI/CD systems pose significant security risks. Leaked secrets can give threat actors access to important infrastructure. Stolen user account credentials can lead to supply chain attacks. Third-party code in CI plugins can contain security vulnerabilities. Concourse CI mitigates those risks in a number of ways.

Built around well-abstracted functionality, Concourse CI's design makes it more secure by default. There is no need to install plugins from third-party vendors to achieve even complex pipelines. Reliance on abstractions, rather than more code, means fewer opportunities for vulnerabilities to be introduced into the CI/CD system.

Granular user access control reduces the potential for compromised build system security. Concourse CI adheres to the principle of least privilege with configurable RBAC (role-based access control) and team authorization namespaces to limit developer access to only what's required to perform their job. Open Policy Authorization integration is available for organizations that require a custom RBAC system or wish to enforce organization-specific policies.

Compromised secrets and user account credentials are a common attack vector. Concourse CI employs a variety of methods to restrict access to secrets and credentials or keep them out of the build system entirely. Any portion of the pipeline that is likely to contain credentials is encrypted prior to entering the database, reducing the likelihood of a successful attack even if the build system database is breached. Concourse CI variables prevent credentials from being persisted in the build system or ending up in source control, make credential rotation effortless, and allow integration with many credential managers, including Vault, LastPass, AWS Secrets Manager, Conjur, and others. OAuth login is convenient and improves security by keeping user account credentials out of the build system.

For organizations that need to protect particularly sensitive information or that are subject to stringent regulatory requirements, Concourse CI can be deployed in air-gapped environments or entirely within a VPN.

No Vendor Lock-In

Concourse CI is open source software based on open standards. This gives users the freedom to leave their current hosted Concourse CI provider at any time without having to switch to a different CI solution.

Hosted Concourse CI

Hosted Concourse CI can reduce engineering department frustration. No infrastructure to contend with means teams can focus on delivering features, instead of managing the build system. Concourse CI does have a steeper learning curve initially, which is why choosing the right hosted Concourse CI provider is important. Providers that offer consulting and other support, like pipeline migrations, can ease the transition.