How to get a pipeline to fetch (or be sent) a file(s) from a remote server

[Adam Brunt]

Hi all,

I am well versed with using the atlassian/sftp-deploy pipe for sending files to a remote server but - unless I am missing something obvious - that only PUTs files it can't be used to GET them. Again, I might be forgetting something obvious, but is there anyway I can get a file(s) sent to the pipeline - ideally to the artefacts folder for the current run ?

Basically, in one step we SSH onto a remote server and generate some files (on that server) which we then want to be available to a subsequent step in the pipeline. The files would NOT be available publicly via cURL or an FTP server. Is this possible ? Maybe I am just missing some knowledge about what an SSH connection can/can't do i.e can you fetch a file over an SSH connection ?

[Adam Brunt]

Hi @Theodora Boudale

That's very helpful.

I am assuming we run

scp $USER@$SERVER:/path/to/somefile.txt $BITBUCKET_CLONE_DIR/artifacts

inside a step in the bitbucket-pipeline.yaml file ?

We don't use SSH keys; for all our instances of using atlassian/ssh-run:0.4.1 we are passing a value for SSH_KEY (and using a secured, base64-encoded, repository variable) so can you clarify exactly what you mean by:

you will need to create the private key in this step as well and then you can pass its path and name

[Theodora Boudale]

Hi Adam,

If the base64-encoded value of the private SSH key is stored in a variable named MY_SSH_KEY, then you can create the private key file from this variable by adding this command in your bitbucket-pipelines.yml file (in the script of the step where you want to excute the scp command):

- (umask 077 ; echo $MY_SSH_KEY | base64 --decode > ~/.ssh/id_rsa)

The scp command also needs to be added in the script of the same step in the bitbucket-pipelines.yml file. I suggest adjusting it as follows:

- scp -i ~/.ssh/id_rsa $USER@$SERVER:/path/to/somefile.txt $BITBUCKET_CLONE_DIR/artifacts

to ensure that the private SSH key file you create with the previous command is offered.

Another thing to check is if your server's public IP address or domain name is added in the Known hosts. If you open the Repository settings for this repo from our website > section Pipelines - SSH Keys, you will see the option Known hosts on this page. If your server's host address is not already added there, you can add it and fetch the fingerprint.

Kind regards,
Theodora

[Adam Brunt]

Thanks again @Theodora Boudale that works perfectly

[Theodora Boudale]

Hi Adam,

This should be possible with scp. For example:

scp $USER@$SERVER:/path/to/somefile.txt $BITBUCKET_CLONE_DIR/artifacts

where /path/to/somefile.txt is the path and the name of the file on the remote server. The file will then be copied in a directory named artifacts inside the clone directory.

In order to use an SCP command, an SSH client needs to be installed in the Docker image you use as a build container (for the step where you want to fetch files).

You can check that locally if you have Docker installed on your computer. For example, if you use atlassian/default-image:4 as a build container for this step, you run locally

docker run -it --entrypoint=/bin/bash atlassian/default-image:4

Then, when you are inside the container, you run

ssh -V

If an SSH client is installed, you will see something like the following:

OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022

If an SSH client is not installed, you will need to either install it during the build or use a different Docker image with an SSH client preinstalled.

Since you SSH into this server, I assume that SSH keys are already set up. If you use SSH keys configured in Repository settings > SSH keys, you can also adjust the command as follows to ensure this specific SSH key is offered:

scp -i /opt/atlassian/pipelines/agent/ssh/id_rsa $USER@$SERVER:/path/to/somefile.txt $BITBUCKET_CLONE_DIR/artifacts

If you are not using the SSH keys configured in Repository settings but a variable that stores the base64-encoded value of the SSH key, you will need to create the private key in this step as well and then you can pass its path and name with the -i argument to the scp command.

Kind regards,
Theodora