pipeline: how to create a dependency between steps

[Admin Admin]

Hello,

 is there a way to make the "deploy" step wait for the "build" step?

issue:

 the download of the docker image for scp pipe takes 40 seconds and the actual copy of the artifacts - 2.

It would be good to have all the downloading done while "build" is in progress.

bitbucket_pipelines.yml

image: atlassian/default-image:3

pipelines:
  default:
        - step:
            name: Build and Test
            caches:
              - node
            script:
              - rm -rf node_modules/
              - rm package-lock.json
              - npm install
              - npm run build
            artifacts:
              - node_modules/**
              - .nuxt/**
              - package.json
        - step:
            name: Deploy npm_modules artifacts using SCP
            deployment: test
            script:
               - pipe: atlassian/scp-deploy:1.2.1
                 variables:
                   USER : $DEP_USER
                   SERVER : $DEP_SERVER
                   REMOTE_PATH : '/tmp/test/'
                   LOCAL_PATH : 'node_modules/'
                   DEBUG: 'true'

[Theodora Boudale · Accepted solution]

Hi @Admin Admin,

It is not possible to configure what you are asking, non-parallel steps are always executed sequentially and the previous step needs to complete for the next one to start. Parallel steps would not be a solution for your use case, as it is not possible for parallel steps to use artifacts by steps in the same parallel set.

What you could do to speed up the process would be to use a docker cache, this way the pipe's Docker image will be cached and then used in subsequent builds. You can modify the second step as follows:

- step:
    name: Deploy npm_modules artifacts using SCP
    deployment: test
    caches: 
      - docker
    script:
      - pipe: atlassian/scp-deploy:1.2.1
        variables:
        USER : $DEP_USER
        SERVER : $DEP_SERVER
        REMOTE_PATH : '/tmp/test/'
        LOCAL_PATH : 'node_modules/'
        DEBUG: 'true'

The cache will be saved after the first successful build and subsequent builds will use the cache for the pipe's Docker image. Any cache which is older than 1 week will be cleared automatically and repopulated during the next build:

https://support.atlassian.com/bitbucket-cloud/docs/cache-dependencies/

If cloning the repo during a pipelines build takes a long time, you could try adjusting the depth of the clone; the default is 50, but if you don't need that you could specify a lower value as follows in the yaml file

clone:
   depth: 5 # include the last five commits

Kind regards,
Theodora

[Admin Admin]

@Theodora Boudale 

 not what i was _hoping_ to hear. But thank you for clarifying the question for me.

[Theodora Boudale]

You are very welcome, please feel free to reach out if you ever need anything else.

Kind regards,
Theodora

[Pramodh M]

Hi @Admin Admin 

It's already dependent, If your step fails, succeeding steps will not be executed!!

There's a configuration to run steps in parallel

https://bitbucket.org/blog/speed-build-parallel-steps-pipelines

Thanks,
Pramodh

[Admin Admin]

i do NOT want the dependent step to fail.

I'd like the dependent step to wait until the artifacts are created...

[Pramodh M]

Hi @Admin Admin 

Since the Image where you are building the npm project, and SCP is happening in another pipe. This should go one after the other.

Thanks,
Pramodh

[Admin Admin]

@Pramodh M,

 I maybe not be clear, apologies.

I'd like to kick off the deployment step before the artifacts in the build step are ready.

What I'm trying to do is :

1. step build and step deployment: downloads repos and dockers simultaneously

2. build is building;

3. deployment is waiting for the build to finish before starting the deployment.

[Pramodh M]

Yes @Admin Admin 

I understood, Since we are using Pipe to transfer the file in the step, the docker image will be pulled at the moment in time.

If you are using a self-hosted runner there is a workaround, but with Atlassian, runners it's not efficient (You could try having npm build && docker pull image-name to keep the image before you run the next step where it uses docker image)

Thanks,
Pramodh