Internal gadgets title are appearing as __MSG_gadget.xxxxxxxx

[Akshatha]

Most of the internal gadgets title are appearing as __MSG_gadget.xxxxxxxx. 

I could see below errors in the logs (JIRA 7.10.1)

WARN [o.a.h.impl.auth.HttpAuthenticator] NEGOTIATE authentication error: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm)

WARN akshatha 824x269x1 /plugins/servlet/applications/versions-licenses [c.a.u.application.marketplace.ApplicationMarketplaceQueriesImpl] Marketplace product query failed: com.atlassian.marketplace.client.MpacException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Would be helpful if I get suggestions on how to debug. 

[Alexis Robert]

Hi @Akshatha , 

this is most likely because you are using a SSL certificate that is not trusted by the JVM. You need to import it by following this documentation : https://confluence.atlassian.com/kb/how-to-import-a-public-ssl-certificate-into-a-jvm-867025849.html

Let me know if this helps, 

--Alexis

[Akshatha]

Hi Alexis,

JIRA is being accessed as HTTP and not HTTPS and is installed as a service from a Installer which runs with its own jre.

Will the above solution still be valid?. If yes please could u let me know how google.com:443 can replaced with Jira ip address (12.2.21.9) and running on port 8080 in below command (referring to the link provided).

openssl s_client -connect google.com:443 -servername google.com:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > public.crt

[Alexis Robert]

Hi @Akshatha , 

if Jira is accessed with HTTP, then you probably have an issue in your configuration because it looks like it's expecting HTTPS.

Do you want HTTP or HTTPS ? I'm not sure what you are trying to achieve.

[Akshatha]

Hi @Alexis Robert 

I want to access JIRA with HTTP protocol. Please could you let me know where I can expect the configuration issue?

Present JIRA (which is facing gadget issue) is installed in linux server using installer and running as a service, I tried installing a new JIRA instance on a different location o different port in the same server, the new vanilla JIRA instance also has same issue.

[Alexis Robert]

Hi @Akshatha , 

can you show me the content of your server.xml file ? It should be located in the atlassian-jira-software-xxx-standalone/conf/ folder. This way we can have a look for any errors in the configuration.

[Akshatha]

Hi Alexis

Below is the server.xml 

<!--...-->

<Server port="8005" shutdown="SHUTDOWN"><Listener className="org.apache.catalina.startup.VersionLoggerListener"/><Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"/><Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"/><Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"/><Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"/><Service name="Catalina">

<!--...--><Connector port="8080" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" enableLookups="false" maxHttpHeaderSize="8192" protocol="HTTP/1.1" useBodyEncodingForURI="true" redirectPort="8443" acceptCount="100" disableUploadTimeout="true" bindOnInit="false"/>

<!--...-->

<!--...-->

<!--...-->

<!--...-->

<!--...-->

<!--...-->

<Engine name="Catalina" defaultHost="localhost"><Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true"><Context path="" docBase="${catalina.home}/atlassian-jira" reloadable="false" useHttpOnly="true"><Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction" factory="org.objectweb.jotm.UserTransactionFactory" jotm.timeout="60"/><Manager pathname=""/><JarScanner scanManifest="false"/></Context></Host><Valve className="org.apache.catalina.valves.AccessLogValve" pattern="%a %{jira.request.id}r %{jira.request.username}r %t "%m %U%q %H" %s %b %D "%{Referer}i" "%{User-Agent}i" "%{jira.request.assession.id}r""/></Engine>

</Service>

</Server>

Thanks 

[Alexis Robert]

Hi @Akshatha , 

looks good to me. Is your base URL configured as HTTP and not HTTPS ? 

[Akshatha]

It is configured as HTTP.

[Alexis Robert]

Can you check your logs and look for lines starting with ERROR ? The one you attached on your first post is only a WARNING and might not be related to your issue

[Akshatha]

I have attached errors i could find in catalina.out. 

I tried running JIRA in safe mode but still the issue exists.

2019-11-13 06:31:08,539 Caesium-1-2 ERROR ServiceRunner [c.a.j.p.devstatus.provider.DefaultDevSummaryPollService] Response from 'http://xx.xx.xx.xx:8085/rest/jira-dev/latest/dev-status-pull-changes?sequenceNo=1569230649445&maxResults=2000' has HTTP status '403 Forbidden'. Response:
<HTML><HEAD>^M
<TITLE>Appliance Error</TITLE>^M
</HEAD>^M
<BODY>^M
<FONT face="Helvetica">^M
<big><strong></strong></big><BR>^M
</FONT>^M
<blockquote>^M
<TABLE border=0 cellPadding=1 width="80%">^M
<TR><TD>^M
<FONT face="Helvetica">^M
<big>Appliance Error (configuration_error)</big>^M
<BR>^M
<BR>^M
</FONT>^M
</TD></TR>^M
<TR><TD>^M
<FONT face="Helvetica">^M
Your request could not be processed because of a configuration error: "Account is locked out."^M
</FONT>^M
</TD></TR>^M
<TR><TD>^M
<FONT face="Helvetica">^M
^M
</FONT>^M
</TD></TR>^M
<TR><TD>^M
<FONT face="Helvetica" SIZE=2>^M
<BR>^M
</FONT>^M
</TD></TR>^M
</TABLE>^M
</blockquote>^M
</FONT>^M
</BODY></HTML>^M

2019-11-13 06:41:18,445 Caesium-1-1 ERROR ServiceRunner [c.a.j.p.devstatus.provider.DefaultDevSummaryPollService] com.atlassian.sal.api.net.ResponseException: org.apache.http.conn.ConnectTimeoutException: Connect to proxy.xxx.xx.com:8080 [proxy.zz.zz.com/xx.xx.xx.xx] failed: connect timed out
at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:108)
at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:131)
at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor.execute(ApplicationLinkRequestAdaptor.java:58)
at com.atlassian.applinks.oauth.auth.OAuthRequest.execute(OAuthRequest.java:58)
at com.atlassian.jira.plugin.devstatus.provider.source.applink.ApplicationLinkDataProvider.getPollResponse(ApplicationLinkDataProvider.java:311)

2019-11-06 17:00:06,923 HealthCheck:thread-1 ERROR ServiceRunner [c.a.t.j.healthcheck.support.GadgetFeedUrlHealthCheck] An error occurred when performing the Gadget feed URL healthcheck
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 

2019-11-06 13:36:24,342 http-nio-8080-exec-2 ERROR anonymous 816x2x1 - xx.xx.xx.xx,xx.xx.xxx.xx /rest/capabilities [c.atlassian.instrumentation.Instrument] Unable to snapshot thread local operations (implementation of OpTimerFactory is not a ThreadLocalOpTimerFactory): null


2019-11-06 14:55:39,164 localhost-startStop-2 ERROR [c.a.s.core.lifecycle.DefaultLifecycleManager] LifecycleAware.onStop() failed for component with class 'com.onresolve.scriptrunner.runner.diag.DiagnosticsManagerImpl' from plugin 'com.onresolve.jira.groovy.groovyrunner'
java.lang.NullPointerException
at net.java.ao.EntityManager.putStatementParameters(EntityManager.java:991)
at net.java.ao.EntityManager.deleteWithSQL(EntityManager.java:513)
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.deleteWithSQL(EntityManagedActiveObjects.java:116)
at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.deleteWithSQL(TenantAwareActiveObjects.java:281)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 2 filtered


2019-11-06 15:28:53,416 ThreadPoolAsyncTaskExecutor::Thread 9 ERROR [o.e.g.b.e.i.dependencies.startup.DependencyWaiterApplicationContextExecutor] Expecting state (RESOLVING_DEPENDENCIES) not (DEPENDENCIES_RESOLVED) for context [NonValidatingOsgiBundleXmlApplicationContext(bundle=com.atlassian.streams.core-plugin, config=osgibundle:/META-INF/spring/*.xml)]; assuming an interruption and bailing out

2019-11-12 17:35:59,037 http-nio-8080-exec-6 ERROR [c.a.j.web.servlet.InternalServerErrorServlet] {errorId=b1628161-afa8-4dad-8274-cc037aaa142d, interpretedMsg=, cause=com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.NullPointerException, stacktrace=com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.NullPointerException

2019-11-12 17:36:52,690 http-nio-8080-exec-25 ERROR [c.a.s.core.lifecycle.DefaultLifecycleManager] LifecycleAware.onStop() failed for component with class 'com.onresolve.scriptrunner.runner.diag.DiagnosticsManagerImpl' from plugin 'com.onresolve.jira.groovy.groovyrunner'
java.lang.NullPointerException

2019-11-12 17:38:52,426 http-nio-8080-exec-2 ERROR [o.a.c.c.C.[.[localhost].[/].[file-server]] Servlet.service() for servlet [file-server] in context with path [] threw exception
java.lang.NullPointerException

2019-11-12 17:35:58,307 http-nio-8080-exec-6 ERROR anonymous 1055x10837x3 1dq37yk xx.xx.x.1xx,xx.xx.xxx.xx/s/-hch7dk/710002/6411e0087192541a09d88223fb51a6a0/7.8.0/_/download/resources/com.atlassian.auiplugin:internal-iconfont-v2/fonts/adgs-icons.woff [o.a.c.c.C.[.[localhost].[/].[file-server]] Servlet.service() for servlet file-server threw exception

2019-11-12 17:35:58,306 http-nio-8080-exec-13 ERROR anonymous 1055x10839x4 1dq37yk xx.xx.xx.xx,xx.xx.xxx.xx/s/daf31ca686a683137da87cefe7cba74c-CDN/-ounpsz/710002/6411e0087192541a09d88223fb51a6a0/7.10.1/_/download/batch/com.atlassian.jira.jira-header-plugin:newsletter-signup-tip/com.atlassian.jira.jira-header-plugin:newsletter-signup-tip.js [o.a.c.c.C.[.[localhost].[/].[file-server]] Servlet.service() for servlet file-server threw exception

[Alexis Robert]

Hi @Akshatha , 

you can have a look at . this KB : https://confluence.atlassian.com/jirakb/how-to-fix-gadget-titles-showing-as-__msg_gadget-813697086.html/?_ga=2.73076308.696304845.1573393740-1867168444.1563630283

Also, can you try from the server to call its own url ? You can try this : 

• From the Jira server itself, Jira is unable to communicate with itself via its Base URL (Run curl -v <base_url> to verify)

Let me know if this works, 

--Alexis

[Akshatha]

Yes I am able to below is the response for the curl command

cur[root@linuxsever logs]# curl -v http://x.xx.x.xx:8080
* About to connect() to proxy proxy.zz.zz.com port 8080 (#0)
* Trying xx.xx.x.xx....
* Connected to proxy.zz.zz.com (xx.xx.x.xx) port 8080 (#0)
* Proxy auth using Basic with user 'username'
> GET http://x.xx.x.xx:8080/ HTTP/1.1
> Proxy-Authorization: Basic MTmnbzQ5MzpTsdteIf0QNDAyMDE5
> User-Agent: curl/7.29.0
> Host: x.xx.x.xx:8080
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NEGOTIATE
< Proxy-Authenticate: NTLM
* Authentication problem. Ignoring this.
< Proxy-Authenticate: BASIC realm="auth1"
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
* HTTP/1.1 proxy connection set close!
< Proxy-Connection: close
< Set-Cookie: BCSI-CS-97998adcf3234244d12304=2; Path=/
< Connection: close
< Content-Length: 847
<
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (authentication_failed)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your credentials could not be authenticated: "General authentication failure due to bad user ID or authentication token.". You will not be permitted access until your credentials can be verified.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
This is typically caused by an incorrect username and/or password, but could also be caused by network problems.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
* Closing connection 0
  

Thanks

[Alexis Robert]

I can see that curl is using a proxy : 

* Connected to proxy.zz.zz.com (xx.xx.x.xx) port 8080 (#0)
* Proxy auth using Basic with user 'username'

You should also configure the proxy in Jira, as described here : https://confluence.atlassian.com/jirakb/configure-an-outbound-proxy-for-use-in-jira-server-247857187.html

And then restart the application. If this doesn't work, you can try changing your /etc/hosts file to point your Jira DNS to the local IP :

127.0.0.1 my.jira.com 

[Akshatha]

Hi Alexis 

I guess curl was going through the proxy because proxy details (http_proxy, https_proxy) was configured in /etc/profile.d/proxy.sh. On removing the file, curl command gives below error

[root@linuxserveraks]# curl -v http://x.x.x.x:8080/
* About to connect() to x.x.x.x port 8080 (#0)
* Trying x.x.x.x...
* Connection timed out
* Failed connect to x.x.x.x:8080; Connection timed out
* Closing connection 0
curl: (7) Failed connect to x.x.x.x:8080; Connection timed out

 And as mentioned in above blog, I had already configured the proxy in setenv.sh below is the configuration.

JVM_SUPPORT_RECOMMENDED_ARGS="-Dhttp.proxyHost=proxy.zz.zz.com -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.zz.zz.com -Dhttps.proxyPort=8080 -Dhttp.proxyUser=username -Dhttp.proxyPassword=password -Dhttps.proxyUser=username -Dhttps.proxyPassword=password -Dhttps.nonProxyHosts=localhost -Dhttp.nonProxyHosts=localhost"

Kindly let me know if u find error in above configuration

[Alexis Robert]

OK so you can try bypassing the proxy by calling the internal IP directly :

change your /etc/hosts file to point your Jira DNS to the local IP :

127.0.0.1 my.jira.com 

[Akshatha]

Hi Alexis,

We have not configured DNS for JIRA, and we are using IP to access JIRA instance (http://xx.xx.xx.xx) 

As per your suggestion, i have modified /etc/hosts like below

127.0.0.1 xx.xx.xx.xx and restarted JIRA but issue still exists.

[Alexis Robert]

OK so this is actually a network issue, you should ask your network administrator to change the routing so that the Jira instance does not have to go through the proxy to call its own IP.

[Akshatha]

Hi Alexis

Okay we will check with the network team but if its a proxy call issue then wont that affect on all gadgets. Beacuse few gadgets appear correctly, while few are not.

[Alexis Robert]

Some gadgets need to probably call themselves and others not ... but I see your point.

[Akshatha]

I wanted to confirm if its actually the proxy issue or are we missing something.. :(

[Alexis Robert]

You should try removing the proxy settings in Jira, and ask your IT team to change the network rules so that the server can call itself.