How do the caches work for Pipeline Runners?

[ntmat]

Hi,

I'm running a group of tests with the Runner feature of Bitbucket Pipelines. Inside the runners I have steps to install pip requirements for testing and have specified pip as a cache for the runners steps. However the runners don't seem to be pulling these requirements from cache on subsequent runs.

Can anyone point out where I'm going wrong in the high level configuration below:

pull-requests:
  "**":
    - parallel:
        - step:
            name: solution_testing_0
            runs-on:
              - self.hosted
              - linux
              - solution.testing
            caches:
              - pip
            script:
              - bash install_system_requirements.sh  # Installs all pip requirements
              - bash run_test.sh --group 0  # Runs subset of tests
        - step:
            name: solution_testing_1
            runs-on:
              - self.hosted
              - linux
              - solution.testing
            caches:
              - pip
            script:
              - bash install_system_requirements.sh  # Installs all pip requirements
              - bash run_test.sh --group 1  # Runs subset of tests

How are caches supposed to be configured for pipeline runners?

[Justin Thomas · Accepted solution]

@ntmat Caches work the same way for both cloud and self-hosted runners. Be aware that the performance improvement from caches is different between cloud and self-hosted runners. Self-hosted runners are located in your data center, so it might take more time to pull the cache from Bitbucket. 

[ntmat]

@Justin ThomasThanks for your reply. I can confirm that the cloud and self-hosted runners use the cache as expected, I needed to delete the existing cache so the runner would re-upload and use the new pip cache.

I see your point regarding performance, what is the alternative? Would you recommend using a docker:dind service to startup a local docker image with all the necessary dependencies preinstalled to avoid cache download from Bitbucket for self.hosted runners?

[Justin Thomas]

Unfortunately, there is no easy way around it. Using a docker:dind service is a workaround and can help, but I think in long term it might complicate the setup. It depends on the performance impact.

Cheers,

Justin