How can we fail builds to prevent linter warnings increasing?

[James Cheese]

Hi! We're currently migrating some of our builds over to Bitbucket Pipelines and looking at retiring our old Jenkins/Octopus setup, with the goal of reducing the infrastructure we're maintaining and flattening the CI processes down a bit. We've got some basic build scripts set up that can run our tests, linters, build and deployment - had some "learning experiences" but nothing insurmountable.

We've hit a bit of a snag when it comes to one of our processes at the moment - the short version of the rule is "your branch should introduce no new Eslint warnings".

The longer version is that we've inherited legacy code, and are slowly working down a "not small" number of warnings from existing code. As such, we've got a policy that developers should be leaving code no worse than they found it - so we've got a reasonably strict policy of ensuring warnings can't increase.

Any suggestions on how to implement this when building with Bitbucket Pipelines?

With Jenkins, we've used the Warnings plugin (https://plugins.jenkins.io/warnings-ng/), which is configured to track when individual warnings were introduced by comparing against the commit where a given branch started against the current build commit. This works really well, but I'm aware might not be doable without a decent amount of infrastructure behind it, so something like a running tally of the number of warnings and making sure a given branch/PR doesn't increase it.

Any ideas would be much appreciated :)

[James Cheese · Accepted solution]

Following on from the above, the approach we ended up going with was to use `eslint-plugin-diff` set up with a pipeline hook on the `pull-requests` start condition. This allows us to check if there are any warnings on code that's been touched in the branch being merged back down.

We then parse the output from eslint in a Node script, submitting a Code Insights report that shows in both the Pipeline and the PR, and set an exit code of 1 to fail the build if any new warnings were introduced.

Sources:

• https://github.com/paleite/eslint-plugin-diff
• https://support.atlassian.com/bitbucket-cloud/docs/pipeline-start-conditions/#Pull-Requests
• https://developer.atlassian.com/cloud/bitbucket/rest/api-group-reports/#api-repositories-workspace-repo-slug-commit-commit-reports-reportid-put

[Theodora Boudale]

Hi James!

Just to give you some context, Pipelines builds run in Docker containers. For every step of your build, a Docker container starts (the build container) using the image you have specified in your bitbucket-pipelines.yml file. The repo is cloned there and then the commands of that step's script (from the yml file) are executed. When the step is finished, the container gets destroyed.

It is not possible to install any plugins in Pipelines, but you can use command line tools if they are preinstalled in the Docker image that you use as a build container, or if you install them during the build by adding the necessary commands in a step's script in the yml file.

I did a quick search on the internet and I found the following third-party command-line alternative to the warnings plugin:

• https://github.com/melexis/warnings-plugin

There may be others available as well, this was one of the first few results of my search. You could perhaps explore this to see if it does what you want.

We have a guide for troubleshooting Pipelines builds locally with Docker. You can use it to reproduce a build locally and try this tool. You just need to have Docker installed on your computer to follow this guide.

• https://confluence.atlassian.com/bbkb/troubleshoot-failed-bitbucket-pipelines-locally-with-docker-1318882094.html

Comparing the number of warnings generated during a Pipelines build with the number of warnings generated in a previous build is a bit more tricky. You can store info (like the number of warnings) in a file that you upload to the Downloads section of the repo (and that you can download during a build and parse). If you want to keep track of the number of warnings for multiple commits, you could store pairs of commit hash and the number of warnings.

If you have the info you want, failing a build can be achieved by adding in the yml file an if statement that runs exit 1 if the number of warnings is increased.

Please feel free to reach out if you have any questions.

Kind regards,
Theodora

[James Cheese]

Hi Theodora,

Thanks for the response! Already made a good bit of progress, and as you've mentioned - comparing the number of warnings is the awkward part. Managed to put together a script to parse the linter warnings and fire them through to Code Insights to get them represented nicely.

The suggestion about dropping into Downloads is an interesting one - as you say, storing the number of warnings and commit hash looks like a decent start.

The other thing that's piqued my curiosity a little is the display on the Reports screen within a Pull Request - the annotations list on there does show an option to filter by "This pull request" - this might actually cover our basic requirements - given failing the build is mostly a way to prevent PRs going through... Just a little bit more manual than ideal - can't see a way to bind something appearing in the "This pull request" to blocking a PR.

It did raise an interesting line of thought though - I'm assuming that this works by comparing the list of annotations to lines that were changed in the PR? Whilst it'd be great to just pull that straight from the API, that could be an alternative to storing values (ie: run a git diff against the master branch and discard/ignore anything that's not covered in the diff). Something like lint-diff might do the job?

https://github.com/grvcoelho/lint-diff

Anyway, thanks again for the information - some bits there that'll certainly help! Will see how far I can get on this, but looks like I should have something workable at least.

Thanks,

James

[James Cheese]

Hi Theodora,

Sorry if this reply comes through multiple times.

Thanks for the response! Already made a good bit of progress, and as you've mentioned - comparing the number of warnings is the awkward part. Managed to put together a script to parse the linter warnings and fire them through to Code Insights to get them represented nicely.

The suggestion about dropping into Downloads is an interesting one - as you say, storing the number of warnings and commit hash looks like a decent start.

The other thing that's piqued my curiosity a little is the display on the Reports screen within a Pull Request - the annotations list on there does show an option to filter by "This pull request" - this might actually cover our basic requirements - given failing the build is mostly a way to prevent PRs going through... Just a little bit more manual than ideal - can't see a way to bind something appearing in the "This pull request" to blocking a PR.

It did raise an interesting line of thought though - I'm assuming that this works by comparing the list of annotations to lines that were changed in the PR? Whilst it'd be great to just pull that straight from the API, that could be an alternative to storing values (ie: run a git diff against the master branch and discard/ignore anything that's not covered in the diff). Something like lint-diff might do the job?

https://github.com/grvcoelho/lint-diff

Anyway, thanks again for the information - some bits there that'll certainly help! Will see how far I can get on this, but looks like I should have something workable at least.

Thanks,

James

[Theodora Boudale]

Hi James,

These are good ideas!

If it is possible to get the number of warnings only in the diff with master branch, that might eliminate the need to keep track of warnings in different commits.

About annotations, I'm unsure about the filter "This pull request". I tried creating a few reports and annotations via our API, but I can't see that filter in my reports. You can reach out via a support ticket if you'd like more info on that (via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs").

Blocking a PR's merge based on Code Insights is not available in Bitbucket's merge checks at the moment. We have a feature request that you can vote for if you'd be interested in that:

• https://jira.atlassian.com/browse/BCLOUD-20547

It may be possible to do this if you add your own merge check with Forge (please keep in mind this is still in beta):

• https://bitbucket.org/blog/beta-custom-merge-checks-in-bitbucket-cloud
• https://community.atlassian.com/t5/Bitbucket-Cloud-x-Forge-EAP/Introducing-Custom-merge-checks-powered-by-Forge/ba-p/2563447

Otherwise, you could fail the build and use the merge check about failed builds.

Kind regards,
Theodora