Automate online site-backups for Jira and Confluence without programming

[#1]

Here I'll introduce a way to automate site backups of Jira Cloud and Confluence Cloud with Jira Automation. 

(0) Why site-backup is important

Atlassian doesn't officially provide data rollback based on customer requests. So it's essential to take site backup regularly. For example, the backup will help you restore data when any of the site data is lost by a user's interaction. Refer to the following part in Data storage FAQ for more details:

Atlassian cloud sites don't support the use of backup data to roll back changes.

First, let me clarify what is covered by this article. The approach introduced here is only good as a temporary workaround as there are some constraints, such as:

• Disclaimer: This approach is not officially supported and is currently maintained by the community. Consider voting for CLOUD-6498 to show your interest in official support.
• Prerequisite: You need to subscribe to at least one Jira family product; meaning it's unable to follow this instruction if you only with Confluence subscription as we'll utilize Jira Automation feature which is only available for Jira. Now Confluence is going to have Automation as well.
• Importantly, in this approach, we skip downloading the created site backups to any local machine simply because it's better than nothing. It is recommended for sure to download site-backups if you can. To do so, use the bash scripts listed below:
  • Backup script for Jira Cloud
  • Backup script for Confluence Cloud

(1) Preparation

First things first, you always need to have credentials when you do something important:

1. Issue an API token of the site-admins account at https://id.atlassian.com/manage/api-tokens
2. Encode your credentials "$EMAIL_ADDRESS:$API_TOKEN" with base64 as (1-2) Example below. Or go with any online tool like Base64 Encode. Make sure not to include a line break at the end.
   • 

(1-2) Example - How to encode your API token

$ echo -n "site-admin@example.com:123e4567-e89b-12d3-a456-426614174000" | base64
c2l0ZS1hZG1pbkBleGFtcGxlLmNvbToxMjNlNDU2Ny1lODliLTEyZDMtYTQ1Ni00MjY2MTQxNzQw
MDA=

Besides, for those using IP Allowlisting, include the egress IP ranges introduced at IP addresses and domains for Atlassian cloud products #Outgoing Connections.

(2) Create a rule for Confluence site-backup

Although you can start with either Jira or Confluence, we'll look into how to configure a rule for Confluence here for convenience.

1. Set up the schedule
   • Add the content of (2-1) below to Cron expression
   • Select "simply run the conditions and actions without providing issues" for When rule executes
   • 
2. Add an action - Send web requests
   • Webhook URL: {{baseUrl}}/wiki/rest/obm/1.0/runbackup
   • Headers:
     • Authorization: Basic <The output we got from (1-2)>
   • Webhook body: refer to (2-2)
   • 
3. Add another action - Log action
   1. To be able to keep track of what we got with the webhook request, specify {{webhookResponse}} in Log message
4. Name the rule "Site backup - Confluence Cloud" and save the setting
5. Do a test run with [Run rule] button
   • The log then will be available in the Audit log
   • 
6. Check the download link
   • 

(2-1) Cron expression

Each backup should be performed after a 48-hour interval at least. We here are going to schedule the execution twice a week on Monday and Thursday. Also, make sure to avoid maintenance windows for your site. For example, suppose your company sits in EST, specify 9 am in UTC:

0 0 9 ? * 1,4

(2-2) Webhook body

{"cbAttachments":"true" }

(3) Create a rule for Jira site backup

I'll put the diff in settings as it's almost the same with Confluence.

1. Schedule - same
2. Action - Send web request
   • Webhook URL: {{baseUrl}}/rest/backup/1/export/runbackup
   • Webhook body: refer to (3-1)
3. Remaining procedures - same
   • 

(3-1) Cron expression

Same here, on Monday and Thursday at 9 am in UTC:

0 0 9 ? * 1,4

(3-2) Webhook body

{"cbAttachments":"true", "exportToCloud":"true"}

(4) Constraints

Other than noted above, there are several points you should be aware of:

• Jira has an upper bound on rule executions per subscription plan. That means you will not get a backup if exceed the limit. So consider upgrading to Premium, especially if you already have a respectable number of existing global automation rules.
• The backup link will be available for 14 days for Jira and 7 days for Confluence. The link then will be replaced by the subsequent one. That means you will only be able to retrieve the backup file until the next backup ends. So make sure to download the backup file as soon as you find any data loss; otherwise, the backup link will be replaced with the one taken after the data loss.

[Veera]

Great job, @K. Yamamoto!!! 

[Taranjeet Singh]

very useful information indeed. Thanks!

[ITS.Elvadata]

is it possible to create a backup and download the file automatically via a Windows Server ?

Thanks

[K. Yamamoto]

Yes, it is possible. Refer to atlassianlabs/automatic-cloud-backup and try the PowerScript or Python script.

[ITS.Elvadata]

thx, i will try it :-)

[Thomas Kringstad]

Thanks, looks like this works.. but I have one question...in the Log Action is the correct input {webhookResponse}} or {webhookResponse.body}} 

In the png you have written .body but not in the text...

[Deleted user]

Any idea how to change the limit of the backups triggered from 48 hours to let's say every hour?

[Nivruti Vora]

@[deleted]I don't think it's possible(depending on what you want to backup). Check https://confluence.atlassian.com/jirakb/how-to-automate-backups-for-jira-cloud-applications-779160659.html

From the article: "You can perform data-only backups—excluding attachments, avatars, and logos—one after the other, without a delay. When you're creating a backup that includes attachments, avatars, and logos, though, you need to wait 48 hours from the when you last created a backup." It's a limitation!

[K. Yamamoto]

@Thomas Kringstad - As the response is different between Jira and Confluence, I'd recommend you to use {{webhookResponse}} for both to avoid the confusion. It will include all the headers and body content.

[K. Yamamoto]

@[deleted] - As Nivruti kindly suggested, there's no option to change the backup frequency limit at the moment. Consider to vote for these feature requests:

• https://jira.atlassian.com/browse/CLOUD-6498
• https://jira.atlassian.com/browse/CLOUD-6617

If you are working on something that requires frequent backups such as data migration, reach out to https://support.atlassian.com/contact/ so that Atlassian Support can temporarily increase the quota.

[Deleted user]

@K. Yamamoto - I have an open case with support, thank you!

[Jeffrey O'Hara]

Configured the backups for JIRA Cloud instance using Automation Rules and it works great. 

However, I would like to download the backups locally since they're overwritten. 

Configured the Jira-backup.sh script from atlassianlabs/automatic-cloud-backup link above. However, I'm receiving the error "Basic auth with password is not allowed on this instance". I see this link to setup oath for JIRA Cloud https://developer.atlassian.com/cloud/jira/platform/jira-rest-api-oauth-authentication/?_ga=2.101848449.109137584.1587588476-1332438393.1568149174. How can I tell which authorization I'm using? I don't have SAML authentication configured. 

[Christopher Harris]

Even with all these steps, it still seems that one has to manually download the backup file.  What exactly is being accomplished with the above steps?  Simply generating a backup file, this is achievable with one click in backup manager.  What we really need is automatic backups, end to end.  But what we really really need, is Jira to build in the ability to recovery deleted data at least for some period of time.

[K. Yamamoto]

@Jeffrey O'Hara - Create an API token at https://id.atlassian.com/manage-profile/security/api-tokens and specify it as API_TOKEN.

[K. Yamamoto]

@Christopher Harris - I understand your concern and do agree with you. The workaround introduced in this article is far from the ideal experience. Please go with other solutions like below:

• Backup script for Jira Cloud
• Backup script for Confluence Cloud

what we really really need, is Jira to build in the ability to recovery deleted data at least for some period of time. 

As this feature is not yet implemented, I'd advise you to add yourself as a watcher to the feature request tickets below to receive any updates and add any comments you think are necessary and/or important to the feature request. I've also added a private comment linking this case to the feature request for our dev's review.

• https://jira.atlassian.com/browse/CLOUD-6498
• https://jira.atlassian.com/browse/CLOUD-6617

In addition, more information about how Atlassian implements New Features can be found in Implementation of New Features Policy.

[KP]

Awesome, @K. Yamamoto !

[Darryl Lee]

Hi, I've setup the Automation Rule for backups and it has run twice now (and the logs show that it was successful). But each time I've gone to the Backup manager page to look (usually a day or so after the rule runs), I do not see a download link.

I thought that backups are supposed to be kept for 7 days.

I just manually ran the rule, and the "Create backup for cloud" button is grayed out, so I assume a backup is in process, but I see no progress bar, and I suspect I will have the same issue again of no download link when the backup is complete.

Any ideas?

[Ooof, I realized I forgot to output the {{webhookResponse}} to the Audit log. Perhaps that will have a URL for download/progress that I'm missing. Still odd about the link not showing up in the Backup manager page. This is for Jira, BTW.]

[Darryl Lee - CCC]

Ah, I think the trick was that I forgot this in the Webhook body: 

"exportToCloud":"true"

INTERESTINGLY... the backup was created, and it is supposedly Server compatible, and I *DID NOT* have to delete my Next-Gen project (which isn't an option for us).

It was a pain to find the download though. I had to hit up this URL:

> https://{MYHOST}.atlassian.net/rest/backup/1/export/lastTaskId

Which provided the TaskId of the last backup I kicked off. So then I did:

> https://{MYHOST}.atlassian.net/rest/backup/1/export/getProgress?taskId={LASTTASKID}

Which showed this result

{
status: "Success",
description: "Cloud Export task",
message: "Completed export",
result: "export/download/?fileId=MYFILEID",
progress: 100,
exportType: "SERVER",
}

So then, the final download link is:

> https://${INSTANCE}/plugins/servlet/export/download/?fileId=MYFILEID

And... proof being in the pudding, I was able to successfully import this backup file to a Server instance. Woot!

UPDATE
After the import, I couldn't login, which is a known issue after importing from Cloud. So, per the instructions here, I restarted with the atlassian.recovery.password property set.

For some reason, restarting with this property set triggered Jira to do an immediate reindex.

Unfortunately this reindex failed with 1 error:

2020-09-24 14:58:46,125-0700 JIRA-Bootstrap ERROR      [c.a.jira.upgrade.LicenseCheckingUpgradeService] An exception occurred when running to call a callable method, this shouldn't happen.

java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.atlassian.jira.index.IndexingFailureException: Indexing completed with 1 errors

Finally though, after another stop and restart, Jira did successfully start up and I was able to login with the recovery_admin account/password. But meh, no data. Ok, will continue to dig into it.

[Zack Foster]

Thank you @K. Yamamoto , this was extremely helpful!

A very minor CRON nit I found while implementing:

0 0 9 ? * 1,4

In the article, you describe this as 9AM UTC Monday and Thursday. The CRON numeric schedule actually runs Sunday-Saturday, so your expression actually equates to Sunday (1) and Wednesday (4) best I can tell.

Thank you again for the article.

Warm regards~

[Steffen Müller]

Hey, I just created a CLI tool with Opsgenie integration to manage Atlassian cloud backups. It creates, downloads and manages (as in "removes old") your backups and is configured with an easy config file. If things go wrong, Opsgenie alerts are created (optional).

It's open source and I'd be happy if it helps you as well, take a look at https://bitbucket.org/addcraftio/atlascloud-backup

I wrote it to scratch my own itch because I wanted to have Opsgenie monitoring and backup rotation on our offsite Synology NAS -- and got carried away a little. The tool is used in production for all our addcraft cloud instances.

Hope this helps out a little, @Christopher Harris & @K. Yamamoto!

[Danny Grenzowski _Rewind_]

Thanks for posting this guide, it's very helpful. I noticed there were some comments, such as @Christopher Harris, asking about a more comprehensive solution, including restore functionality for Jira Cloud.

My team is working on a fully automated backup-and-restore product for Jira Cloud - Rewind Backups. If you interested, please let me know! I'd love to chat with Jira customers who are looking for this type of solution.

[Ludvig Hede]

@Darryl Lee - CCCI'm reading your posts here regarding a download link with high interest. I'm curious how you automatically fetched the link and how you knew when the link was ready for download. How did you implement this?

[Phillip C]

The download link for Jira doesn't show but the job runs successfully. Any reason why this would be?

However, Confluence shows the download link and works fine as well.

I've tested with and without the 

"exportToCloud":"true"

 Also, there is no web page for https://{MYHOST}.atlassian.net/rest/backup/1/export/lastTaskId

Doesnt load, doesn't do anything - (dev tools in browser showing web page status code 204)

[K. Yamamoto]

UPDATE: Free Apps for Jira and Confluence

It's been a while since I've published this article. Here, I'm thrilled to announce that here we have free apps providing backup automation, available for Jira and Confluence.

• Admin Kit for Jira Cloud - Atlassian Marketplace
• Admin Kit for Confluence Cloud - Atlassian Marketplace

Not like the approach introduced in this article, the apps are not relying on Jira Automation. So you can now automate the backups without having any Jira license. Besides, the other advantage of the apps is backup history. Although the apps don't keep the local copy of backup files, the download links will be available even after creating the subsequent backups.

Please note, the apps are in beta and not officially supported. I am looking forward to hearing your feedback.

[Ali Alnakeeb]

thanks, however, the basic authentication method is being deprecated, this article could be updated to latest status