Triggering Multiple Pipelines for One Repository

[Jonny Carter]

I have a repository that builds and deploys several submodules. These submodules share some common code defined in shared modules within the repository.

I'm trying to un-complicate the current setup, which has multiple other repositories with distinct pipeline definitions that manually clone the source repo and run a determined series of build steps.

The way we currently trigger the other pipelines is using the trigger pipeline pipe (source at https://bitbucket.org/atlassian/trigger-pipeline/src/master/).

I can use the same pipe to trigger pipelines within the same repository. That helps some, because I can more easily reuse pipeline steps using YAML anchors and it shaves some time off the build process.

The trouble I have is that if I kick off more than two pipelines, only two will show up on the pull request, even if they are all pipelines within the same repository on the same branch. For pull requests that cause any of the submodule builds to fail, I definitely don't want the PR to be mergeable.

I can tell the trigger-pipeline pipe to wait for the result from the other pipeline, but this effectively doubles the amount of build minutes I pay for, which is needlessly expensive.

I have searched, but can't find out if this is a documented limitation of Bitbucket Pipelines or just a quirk of the UI. Is there some setting somewhere that I can use to allow more than 2 pipeline results to be associated with a pull request? I have tried passing the BITBUCKET_PR_ID, BITBUCKET_PR_DESTINATION_BRANCH, BRANCH_NAME, and other relevant variables to the child pipeline using the `PIPELINE_VARIABLES` variable defined by trigger-pipeline, but I still only get the top two hits.

Is there some more scalable way to associate lots of pipelines back to one pull request than the trigger-pipeline pipe? I'd happily put them all in nested `parallel` blocks, but that isn't really an option in Bitbucket Pipelines.

[Theodora Boudale]

Hi Jonny,

The builds that are displayed in the PR depend on the build statuses associated with the last commit of the PR's source branch. There is no hard limit of 2 build statuses, it can be just one, it can also be multiple; it depends on what pipeline definitions are running and also how the builds get triggered.

That being said, pipelines can update the existing build status of a commit instead of creating a new one (e.g. this happens with custom builds, see feature request: https://jira.atlassian.com/browse/BCLOUD-23256). I went ahead and created a feature request for creating a separate build status for each pipeline that runs on a branch/commit:

• https://jira.atlassian.com/browse/BCLOUD-23605

You can add your vote to that feature request (by selecting the link Vote for this issue) to increase the chances of this being implemented. You are more than welcome to leave feedback, and you can also add yourself as a watcher (by selecting the link Start watching this issue) if you'd like to be notified via email on updates.

You also raised another issue:

For pull requests that cause any of the submodule builds to fail, I definitely don't want the PR to be mergeable.

I can tell the trigger-pipeline pipe to wait for the result from the other pipeline, but this effectively doubles the amount of build minutes I pay for, which is needlessly expensive.

I assume that the submodule pipelines are defined in in the bitbucket-pipelines.yml of each submodule repo, and so they run on the submodule repo? If so, and if you'd like to keep this setup, then using the WAIT variable is the only way.

However, you could work around this the following way:

Instead of running the submodule builds in the submodule repos, you could run them in the main repo. You could define a custom pipeline for each submodule. In each custom pipeline, you could clone the respective submodule and then run your tests there. You could also disable cloning of the main repo for these pipelines, if it's not needed for the submodule builds.

We have the following blog post that explains how to clone another repo in pipelines (you'll need to set up authentication):

• https://www.atlassian.com/blog/bitbucket/cloning-another-bitbucket-repository-in-bitbucket-pipelines

One thing that is missing from this article is that an SSH client needs to be installed in the Docker image you use a build container, so you can clone via SSH.

Please feel free to reach out if you have any questions.

Kind regards,
Theodora

[Aron Gombas _Midori_]

@Jonny Carter I don't have a real answer, but maybe two hints to investigate further:

1. Dynamic Pipelines is a mechanism to better orchestrate your builds if you don't mind to develop a small-ish Forge app for that purpose. It may worth the effort!
2. Pipeline Forms is a Bitbucket Cloud app to trigger builds with user-picked parameters, periodically, etc. It doesn't require programming skills.

Maybe one of these will be helpful.

[Jonny Carter]

I can imagine that Dynamic Pipelines could do it. I haven't been willing to futz with it because I'm categorically against buying in too hard to any particular CI system. My view is CI systems should be commodity products with very low vendor lock-in. Dynamic Pipelines smell like more commitment and maintenance burden than I really wanna take on for this need.

It also seems like the kind of workflow Bitbucket should be able to support out of the box. All the competitors (GitLab, GitHub, heck, even Bitbucket DC which has no built-in CI) can support showing more than two CI builds on a PR.