Test Environment

[Jen Hamilton]

How difficult is it to setup a test environment; and how to transfer to production

[Mike Rathwell · Accepted solution]

Jen,

I'm not sure if you're doing this in JIRA Cloud or on a server. If Cloud... can't help you there. If it's on your own server I do this all the time myself and it is, with JIRA, surprisingly easy. The steps I take are:

1. Build a brand new, totally empty JIRA server of the version you want (same as production or whatever you want to try to update to).
2. Once you get it up and happy and can log into and see the empty JIRA instance, shut JIRA down.
3. On your source production system, run the XML backup
4. Copy the <jira-data> directory from the source production in its entirety to your target test system replacing the <jira-data> directory created by the base JIRA instance install
5. Move the export created from <jira-data>/export to <jira-data>/import
6. Change the ownership of the directory and contents to whatever user you run JIRA as. If you don't you'll get an epic failure to start.
7. Start the target test system JIRA instance. You still don't have your clone back quite yet
8. Import the file you just exported to the system
9. When the system settles down you have a clone of the system

NOTES:

1. The above assumes that you are running the database on the same server as your JIRA instance, the host you are pointing to is "localhost" or "127.0.0.1". If you have a FQDN for your datasource even if it is local to your JIRA instance OR you point at an external DB server CHANGE THE DB CONFIG IN <jira-data> BEFORE YOU START JIRA. Other wise you'll bork your production data.
2. If JIRA has a failure at one of these points, it's possible it's just unhappy with how things came over and can't sort itself out. This isn't a consistent failure; i have it so it doesn't fail at all or fails at different points of the steps above (from intial start to import, to final restart). To fix it at any of these points it fails and then just be able to carry on:
   1. Stop JIRA
   2. Delete the contents of:

      1. <jira-home>/caches directory

      2. <jira-home>/plugins/.osgi-plugins directory

      3. <jira-home>/plugins/.bundled-plugins directory

      4. <jira-install>/work directory

   3. Restart JIRA

When it's time to transfer to production... perhaps it's an awful hack but when I am happy with the migration and it's flawless after testing... I do the following:

1. Take that last point in time backup of current production
2. STOP the JIRA instance to keep people from making changes even if I've communicated to the community the server is being updated so leave it alone from X time to Y time.
3. Shut down the server after you have the <jira-data> directory copied over
4. Re-assign the IP address that was being used on the old production to the new production server.
5. Do the migration steps above.
6. When it's done, the new server is automagically... in production. If Something Bad happens on the migration, roll back is as simple as restarting the old server.

You CAN do this with a DNS Alias change but I found that the IP address swap had the new JIRA server come up without having to re-establish application links to anything and Crowd just happily authenticated users to it without having to reconnect the Crowd server as its user authority. With the DNS alias change you have to reconnect applications and tell Crowd about the new server.

This approach is also great if you want to slide a new OS and/or Database server under your JIRA application. For.... reasons... I've used this approach to go from CentOS and MySQL to Windows and PostgreSQL and to where it is now, Debian with PostgreSQL

Let me know if this helps

mike

[Jen Hamilton]

Thanks Mike, we need it for cloud but this is a great reference to have.  Much appreciated.

[Nic Brough -Adaptavist-]

For Cloud, there's no concept of "test" - a Cloud system is a Cloud system.  You can use it for whatever you want! 

So, if you set up two Cloud accounts, one production and one test, you can work in them separately, but your options to transfer between them are very limited - you'll be doing config and setup mostly by hand (or merging in local dev systems before re-uploading)

[Jen Hamilton]

Got it, thanks so much Nic you are right on point 

[Michael Poersch]

Short and very good description to setup a test environment. Two additional steps from my point of view:

1. Remove incoming mail handler (I changed server to localhost in XML-File)
2. Disable users to log in to test environment (for example remove them from group "jira-software-users")

[Platform]

I'm about to set up a test environment, this is a great help. Thankyou. 
How is licencing handled? Do I need to just keep applying for trial keys?

[Nic Brough -Adaptavist-]

No, that will be shut down.  If you need a Cloud "test" system, you will have to pay for it.  If you want a Server test system, use a "developer" licence, which is associated with your main server licence.

[Platform]

Ah perfect thanks!

[vrejbusiness]

Mike Rathwell thx for such simple and step by step explenation.

[acast2]

@Mike Rathwell I have a similar situation but with some differences, so I wanted to get your input on this.

I have an existing test environment and production environment for Jira/Confluence but this was managed by someone else not at our company anymore, so now I'm taking over and I found that our test server was running version 7.8 but our production was running 7.6.  Way behind on versions and EOL as well, so I'm needing to bring us back to the latest version but found that our existing SQL server where Jira DB is located is on a separate server and it will not support the latest version of Jira, so I have to upgrade both sides. My environment is all virtual and running Windows.

I wanted to start clean with a test environment that has Windows Server 2019 for OS for Jira and SQL Server 2019 on SQL server with matching OS as well, but due to production having different server OS/SQL version it's not that simple, so this is what I've done so far. By the way, I'm following the Atlassian documentation for creating a test environment, upgrade, etc.

I created two new test servers, one for Jira and one for SQL. I installed 7.6 on new test server because I want to have the same version of Jira as production on this server to do an in-place upgrade using Jira installer.  The end goal is that once the upgrade goes well in test then I'm going to run the upgrade on production and once production is on the latest version of Jira matching test then move/migrate over to a new production Jira and SQL server that matches test, so that both environments match and moving forward future upgrades would be much easier.

I also read posts from others in this community suggesting to clone the production VM, change the base URL along with other things for test environment and then do the in-place upgrade on that which also makes sense but then test won't be running current OS for Windows Server, which means I'll need to still migrate/move to new servers later.

[Mike Rathwell]

@acast2 , to my tiny mind the thought of cloning off the VM is valid if you aren't looking at a platform version change. It simplifies things and makes a quick/easy way getting a snapshot of full production to use for test. I like your thoughts on proving out one bit at a time starting with the platform.

However... if it was me doing it.... 

The only bits you really need are the mated pair of the Jira home directory and database snapshot (any method you can move somewhere else and have it light up) to clone off your instance. 

1. Build an empty target environment comprised of your desired OS/DB versions
2. Install your target Jira version and then stop Jira
3. Import your source DB dump to the new database
4. Copy the Jira home directory directly over the blank created by doing the install.
5. Start it up.

The two minor release version leap is small enough that it will just start, do any migration tasks it thinks it needs to do and leave you with the cleanest possible new environment. I have long since moved on to containerized Jira on Linux where an upgrade literally is stop the current one; apply the new version image, start it up.

Another suggestion; if you have maintenance paid or had it paid long enough to get to 7.13, I would make that your target. That is the last 7.x version before Jira 8 and was very stable (it was a LTS version). Going to Jira 8 is a Good Idea but there is some work you would need to do to get there. However, 7.6 to 7.13 would be a plug and play upgrade.

[acast2]

@Mike Rathwell I agree, the VM clone part would work best when there are no platform changes, so I'll definitely keep that in mind for future upgrades once I've cleaned up this mess.

In regards to your steps, I've completed 1 and 2, but it's number 3 that gave me some issues. So just to be clear, I do take my production database and restore it to override the existing clean install one right? not restore it as a new database that I then point the test Jira dbconfig.xml file to?

If your answer is override, then I'll try again no problem.

As for your suggestion to go to 7.13 LTS, we have a community license so technically we have no support at the moment until we get to a supported version. I only get support for the upgrade procedures.

I know it's more work, but I've done most of it already in building new servers for production upgrade as well, so after running the upgrade checks within Jira it tells me I'm clear to upgrade to 8.13.9 LTS. 

Once I get there then I'll proceed all the way to the latest 8.18.x version.

With help from people like you and Atlassian support I'll get there soon enough...let me know...thanks

[Mike Rathwell]

Oh... didn't think about that... You SHOULD be able to point only the dbconfig file at the target. A couple of things that can bork you:

• Make sure the user in the dbconfig file exists and has admin privileges on the jira database.
• Make sure when you copy in the home directory from your source to the target that the permissions on the copy are such that the Jira service can read/write that filesystem. I have been bit by THAT one before (less of a deal on Windows but always a deal on Linux)

I have tended to make my leaps from LTS to LTS rather than latest and/or greatest UNLESS there is something post-LTS I need or want. That way, the maintenance point releases are the least amount of fuss to upgrade to and takes the least amount of qualification for each version. 

[Mike Rathwell]

BTW, the 7.x to 8.x assessor might tell you good to go BUT there are API changes between the two. If you have any Groovy scripts or script fragments in JMWE, those may well break (notably ComponentAssessor moved)

[acast2]

I hope that's true because pointing to dbconfig file at the target database is much easier than override it which requires a few extra settings in SQL to accomplish...I figured that one out yesterday the hard way  :-)

As for the couple of things you mentioned to watch out for...yes the first one is covered in their documentation so I was aware of it and have it covered because the SQL DB user for the database is the same in test as production.

The second one however is not one I'm clear on...I'm logged into the server as a local/domain admin and I'm the one initiating the copy/paste of the Jira directories and when I look at both of them...the source directories and the current existing test ones, the local Administrator and local Administrator group is what's on the filesystem permissions as having full access.

I haven't read anything from Atlassian mentioning that the Jira user has to have permissions on those folders? I have the install configured to run as a service and it's using the local system account per defaults so the only Jira user is the one for the database and that's a SQL user not a Windows user.

Like you said, it's less of a deal on Windows, so hopefully that's why I haven't been bit by that one.  Ok, going to attempt what we discussed...I'll report my status as I go...thanks again

[acast2]

In regards to your previous comment about going to 8.x yes I agree and see what you're saying as the documentation also backs up what you're saying but luckily for our setup we don't have any major customizations or issues that would affect us, so we'll see what happens...like I said if I run into any issues then support will assist where needed.

Worst case scenario, I can always upgrade like you said up until the latest 7.x LTS and then go from there a little at a time.  I also get what you're saying about not always needing to be at the latest version unless a specific feature justifies it...definitely will note that and consider it since upgrading every two years is much easier than every few months...thanks  ;-p

[acast2]

@Mike Rathwell well, I changed the dbconfig to point to test SQL DB that I restored from production server, copied the directories over and the same thing happened...when I started Jira it takes me to the setup mode screen where I get to pick the database I wish to connect to, which again I didn't expect since I've already made the changes so it knows what to point to.

I don't believe this is normal or is it?

I followed this step:

6. Replicate Jira

To replicate Jira, make a copy of your Jira installation and point it to your test database.

1. Copy your entire production installation directory to your test server. 
2. Copy your entire production home directory to your test server. 
3. Edit <installation-directory>/atlassian-jira/WEB-INF/classes/jira-application.properties to point to your test home directory. 
   CLUSTER For DC, make this change on every node.

4. Edit <home-directory>/dbconfig.xml or  <installation-directory>/server.xml (older versions) to point to your test database. 

And it still takes me to the Jira Setup screen where it asks to set it up for me or I'll set it up myself.  I choose I set it up myself and put in the info to the SQL DB server and it then gives me an error that it's not an empty database.  This is where I get stuck which is frustrating me.

I'm done for tonight, but hope you can help some more...thanks!

[acast2]

@Mike Rathwell just an update for you since you've helped me the most, even more than Atlassian support has helped.

After reconfiguring the test server from having two volumes (C and D) down to one with just the default C: the replications steps for Jira worked a lot better even though some of the SQL commands from the steps in the "Creating a test environment for Jira" completed with errors or not at all, but the majority of them did work.

I was now able to start Jira in test environment with the data from production and continued with other configurations to make sure its suitable for a test environment such as making sure mail server isn't configured, application links, etc.

The only thing the test server doesn't have is the Jira Service Desk installed which our production server does have. So my next question is although the documentation states that test should be a replica of production, would I need to install/enable the Jira Service Desk in test environment before running upgrade or can this step be optional and I can continue with the testing of the Jira upgrade?

I know that based on the documentation Jira Service Desk can be upgraded from within Jira after the upgrade is completed, so that makes me think that in a test environment maybe it's not needed just to test an upgrade of Jira?

[Mike Rathwell]

@acast2 I think I would tend to get JSD installed up front before you do any further work on the JSW part. The JSD projects act and behave completely differently and I would be worried that the upgrade mucking in the database might bork some of the JSD projects at least in configurations. You CAN upgrade it later but before you do much more twiddling, I would make sure your instance understands JSD.

One of those things that "might" be alright but still gives me an uneasy feeling.

[acast2]

Ok, I have that feeling as well so that's why I asked about it...just like to hear it from someone else sometimes...thanks.

[Merve Nur Bas]

Hi @Mike Rathwell

many thanks for these awesome instructions for Jira. Is the approach for Confluence and Bitbucket rather similar, if you have got any experiences regarding that matter?
(I know the post is like 4 years old, but is suits so well :)). 

My best

Merve

[Mike Rathwell]

Hi @Merve Nur Bas ,

I can't speak, to Bitbucket as I have never run one of those BUT it is largely the same for Confluence. The only delta is that it is a different cache blast for Confluence. In the Confluence case:

<confluence-home>/bundled-plugins
<confluence-home>/plugins-cache
<confluence-home>/plugins-osgi-cache
<confluence-home>/plugins-temp
<confluence-home>/bundled-plugins_language

[Merve Nur Bas]

Thanks a lot! Really nice of you replying to an ancient post :) 

Have a nice weekend!

My best

Merve