Slow performance in Docker

[Artesa]

Hi all, I would like to ask you about Confluence performance in Docker.

On my server I allready have installed Jira SD (locally, not in docker) and local postgres DB server. I run Docker with the latest Confluence version which is connected to a local database server (shared with Jira SD). My problem is that if you want to do anything with Confluence it´s extremely slow except login page. I tried to increase JVM memory size, but without any luck.

Jira works normally without any performance issues.

Of course I was check performance monitor with TOP command and docker perfomance monitor, and there is no unusual CPU or memory load.

Thank you very much for your support.

[Boris MBOUMEHANG · Accepted solution]

Hi Artesa,

I personnaly never deployed Confluence in a Docker, but, I guess you used the official Atlassian's Docker Hub. I also guess that you have installed Confluence 6.13 or later in your docker (because the images used on the Atlassian's Docker Hub for Confluence Server are built on the AdoptOpenJDK images, which was not supported prior to Confluence 6.13; See Confluence 6.13 Release Notes for more details).

If this is your case, then I think you can ask the Atlassian support to help you investigate on this performance issue. 

Thanks and good luck ;)

Best regards,
Boris

[Artesa]

Hi Boris, I use latest docker container from docker hub: https://hub.docker.com/r/atlassian/confluence-server/

Version of my Confluence is 7.0.1

[Boris MBOUMEHANG]

Hi Artesa,

Thank you for your feedback. Then, I think it could be either related to your container's environment (for example, you could check if there are not container's isolation issue between your containers lying on the same Kernel ressources, with the command perf), or to your Confluence setting (try these best practices for performance troubleshooting).

I hope this will help. Otherwise, as I previously suggested, feel free to contact the Atlassian Support.

Thanks and good luck :)

Boris

[Artesa]

Hi Boris, I have allready installed the perf tool, but I don´t know if this tool told me somethik new, because it looks like as TOP command. There is no unusual performance.

Thank you for your support.

[Boris MBOUMEHANG]

Hi Artesa, 

You are welcome. I'm sorry that the perf tool does not show more details than the TOP command. 
Did you have the opportunity to follow the best practices for performance troubleshooting tools ?

If so and you still encounter issues, then, I will advise you to contact the Atlassian Support; they will certainly be able to help you :)

Wish you a good luck! 
Boris 

[Artesa]

Hi Boris, when I want to contact support, it redirect me to community, there is no email contact or something like that because we have starter license (now for testing).

[Boris MBOUMEHANG]

Hi Artesa, 

Oh I see, sorry about that. It would have been great to have somebody from the Atlassian Team to help you on this case.

Could you please provide the following information:

• What about the best practices for performance troubleshooting tools ? Did you try to follow them in order to do your debug? 
• Which operating system is running under your Docker container?
• Dould you try to run your Confluence Server container on another virtual machine (a one only running that container would be great), just to eliminate the case of a possible kernel ressource's problem?

Thanks
Boris

[Artesa]

Hi Boris, I will try make another virtual machine and then run postgres and confluence container on this new virtual machine.

We use Centos 7.

About debug tool, I was used only machine performance monitor and docker performance monitor.

Thanks a lot.

[Boris MBOUMEHANG]

Hi Artesa, 

Thanks for your answer. In fact, trying to run your container on a seperate virtual machine could help you identify the root cause of these performance issues. 

In addition, @Mike Rathwell (another Community Leader) gave a more complete answer on this case: 

The "generic" Docker image is good for a POC and has a lot of stuff in it but I didn't use that one for production other than plagiarizing the elements that got it started so I didn't have to write 'em myself. Where I went from there:

• Use my own base image with the latest OpenJDK supported for the version. Note: I tried on my test upgrade to Jira 8 to use JDK11 Openj9; it was snappy but saw some oddities in the log file pointing at different UI handling that put me back to Hotspot VM... shame... Eclipse VM is noticeably faster

• UseAlpinefor the base image and addonlythe things needed for your image. Anything added, use the no cache option for packages to keep the image small and clean

• My Dockerfile adds some extra variables to the setenv.sh scripts I can use at runtime to set Xms and Xmx along with other JAVA_OPTS to tune for the environment I am running in. Also, it's an old "recommendation" but I set the heap size pretty big (5GB on my prod) and set Xms and Xmx the same 5GB to minimize some of the GC events.

I hope this will help. 

Thank you and thanks to Mike for his feedback. 
Boris

[Boris MBOUMEHANG]

The end of his message :

If it's helpful, here is my Confluence Dockerfile (same image in test and prod; provision the run env at runtime) that is on the adoptopenjdk/openjdk8:alpine base:

ARG DOCKER_REGISTRY
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
# Setup useful environment variables
ARG JDBC_VERSION
ARG CONF_VERSION
ENV CONF_HOME /var/atlassian/confluence
ENV CONF_INSTALL /opt/atlassian/confluence
COPY jira.cer /tmp
COPY confluence.cer /tmp
ENV JAVA_CACERTS $JAVA_HOME/jre/lib/security/cacerts
ENV CERTIFICATE $CONF_HOME/certificate
# Install Atlassian Confluence and hepler tools and setup initial home
# directory structure.
RUN set -x \
&& apk --no-cache add bash xmlstarlet tomcat-native graphviz fontconfig ttf-ubuntu-font-family msttcorefonts-installer openssl apr apr-util apr-dev curl ttf-dejavu perl procps wget ca-certificates \
&& update-ms-fonts \
&& fc-cache -f \
&& ln -s /usr/lib/libfontconfig.so.1 /usr/lib/libfontconfig.so \
&& ln -s /lib/libuuid.so.1 /usr/lib/libuuid.so.1 \
&& ln -s /lib/libc.musl-x86_64.so.1 /usr/lib/libc.musl-x86_64.so.1 \
&& mkdir -p "${CONF_HOME}" \
&& chmod -R 700 "${CONF_HOME}" \
&& chown bin:bin "${CONF_HOME}" \
&& mkdir -p "${CONF_INSTALL}/conf" \
&& curl -LsS "https://www.atlassian.com/software/confluence/downloads/binary/atlassian-confluence-${CONF_VERSION}.tar.gz" | tar -xz --directory "${CONF_INSTALL}" --strip-components=1 --no-same-owner \
&& chmod -R 700 "${CONF_INSTALL}/conf" \
&& chmod -R 700 "${CONF_INSTALL}/temp" \
&& chmod -R 700 "${CONF_INSTALL}/logs" \
&& chmod -R 700 "${CONF_INSTALL}/work" \
&& chown -R bin:bin "${CONF_INSTALL}/conf" \
&& chown -R bin:bin "${CONF_INSTALL}/temp" \
&& chown -R bin:bin "${CONF_INSTALL}/logs" \
&& chown -R bin:bin "${CONF_INSTALL}/work" \
&& rm -f "${CONF_INSTALL}/confluence/WEB-INF/lib/postgres"*"jar" \
&& curl -o "${CONF_INSTALL}/confluence/WEB-INF/lib/postgresql-${JDBC_VERSION}.jar" "https://jdbc.postgresql.org/download/postgresql-${JDBC_VERSION}.jar" \
&& echo -e "\nconfluence.home=$CONF_HOME" >> "${CONF_INSTALL}/confluence/WEB-INF/classes/confluence-init.properties" \
&& xmlstarlet ed --inplace \
--delete "Server/@debug" \
--delete "Server/Service/Connector/@debug" \
--delete "Server/Service/Connector/@useURIValidationHack" \
--delete "Server/Service/Connector/@minProcessors" \
--delete "Server/Service/Connector/@maxProcessors" \
--delete "Server/Service/Engine/@debug" \
--delete "Server/Service/Engine/Host/@debug" \
--delete "Server/Service/Engine/Host/Context/@debug" \
"${CONF_INSTALL}/conf/server.xml" \
&& touch -d "@0" "${CONF_INSTALL}/conf/server.xml" \
&& sed -i \
-e "/-Dconfluence.context.path/a # The variable below has been added to the Docker image build scripts for other values set by deploy environment" \
-e "/-Dconfluence.context.path/a CATALINA_OPTS=\"\${LOCAL_CONF_JVM_ARGS} \${CATALINA_OPTS}\"" \
-e "s/Xms1024m/Xms\${LOCAL_CONF_MINIMUM_MEMORY}/" \
-e "s/Xmx1024m/Xmx\${LOCAL_CONF_MAXIMUM_MEMORY}/" \
"${CONF_INSTALL}/bin/setenv.sh" \
&& touch -d "@0" "${CONF_INSTALL}/bin/setenv.sh" \
&& ${JAVA_HOME}/bin/keytool -importcert -keystore ${JAVA_CACERTS} -storepass changeit -alias jira.company.com -noprompt -file /tmp/jira2018.cer \
&& ${JAVA_HOME}/bin/keytool -importcert -keystore ${JAVA_CACERTS} -storepass changeit -alias test.jira.company.com -noprompt -file /tmp/jira2018.cer \
&& ${JAVA_HOME}/bin/keytool -importcert -keystore ${JAVA_CACERTS} -storepass changeit -alias *.jira.company.com -noprompt -file /tmp/jira2018.cer \
&& ${JAVA_HOME}/bin/keytool -importcert -keystore ${JAVA_CACERTS} -storepass changeit -alias confluence.company.com -noprompt -file /tmp/confluence2018.cer \
&& ${JAVA_HOME}/bin/keytool -importcert -keystore ${JAVA_CACERTS} -storepass changeit -alias test.confluence.company.com -noprompt -file /tmp/confluence2018.cer \
&& ${JAVA_HOME}/bin/keytool -importcert -keystore ${JAVA_CACERTS} -storepass changeit -alias *.confluence.company.com -noprompt -file /tmp/confluence2018.cer \
&& chown bin:bin "${JAVA_CACERTS}"
COPY docker-entrypoint.sh /
RUN chmod go+x /docker-entrypoint.sh
# Use the default unprivileged account. This could be considered bad practice
# on systems where multiple processes end up being executed by 'bin' but
# here we only ever run one process anyway.
USER bin:bin
# Expose default HTTP connector port.
EXPOSE 8090
# Set volume mount points for installation and home directory. Changes to the
# home directory needs to be persisted as well as parts of the installation
# directory due to eg. logs.
VOLUME ["${CONF_HOME}", "${CONF_INSTALL}/logs", "${CONF_INSTALL}/temp", "${CONF_INSTALL}/work"]
# Set the default working directory as the Confluence home directory.
WORKDIR ${CONF_HOME}
ENTRYPOINT ["bash", "/docker-entrypoint.sh"]
# Run Atlassian Confluence as a foreground process by default.
CMD ["/opt/atlassian/confluence/bin/catalina.sh", "run"]h 

[Artesa]

Hi Borris, so if I understand it right, it is not great idea to use docker for Confluence or Jira or all together for production? 

If not, I will not lose my time and install it on the whole system.

[Boris MBOUMEHANG]

Hi Artesa, 
Indeed, the docker images for Confluence and Jira are not recommanded for production deployements, as mentionned in the documentation of Confluence Server on Docker Hub. 

These Confluence Docker images are presented as a technical preview, and not recommended for critical production deployments. However if you are interested in deploying with containers we would be interested in hearing your feedback.

https://hub.docker.com/r/atlassian/confluence-server/ 

So, it would be great if you deployed Confluence Server on a Virtual machine directly. 

Thanks.
Boris

[Artesa]

OK, I will do it.

Thank you so much.

[Boris MBOUMEHANG]

You are welcome :)

[Aileen Tan]

Hi Borris,

Currently using Docker containers 

JIRA 8.11.1 (cptactionhank)

Conflucence 7.4.3 

Bitbucket 7.5.0 

Allocated 4 vCPU and 8GB RAM for instance in hyper-v

Keep encountering surge in CPU usage  and Memory exceed 8GB, Atlassian products is rather unstable , need constant reboot and reloading JIRA and confluence in production. Docker is not suitable for production environment? 

Kindly provide a link for ground up installation in VM for atlassian product thank a lot 

[Boris MBOUMEHANG]

Hi Aileen, 

I am sorry you encounter performance issues while running Confluence, Jira and Bitbucket on Docker containers. As I said in the comments above, the deployment of these products on Docker is not recommanded for critical production environment (see https://hub.docker.com/r/atlassian/confluence-server/).

To install them on VM, you can use the documentation below : 

• Confluence : https://confluence.atlassian.com/doc/confluence-installation-guide-135681.html
• Jira : https://confluence.atlassian.com/adminjiraserver/installing-jira-applications-938846823.html
• Bitbucket : https://confluence.atlassian.com/bitbucketserver/bitbucket-server-installation-guide-867338382.html

I hope this will help. 

Kind regards,
Boris