Invalid character found in the request target on Confluence behind nginx

[Tran Tien dung]

Hi everyone,

Currently I'm using Confluene 6.10.2 behind nginx. I have some pages with the page name including character '>' could not accessible, the error is:

HTTP Status 400 – Bad Request

---

Type Exception Report

Message Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986

Description The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).

Exception

java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
	org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:474)
	org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:294)
	org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:764)
	org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1388)
	org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.lang.Thread.run(Thread.java:748)

Note The full stack trace of the root cause is available in the server logs.

---

Apache Tomcat/9.0.10

But when I access the page bypassing the reverse proxy, it's ok, so it could be a problem in nginx.

I read logs in Confluence and nginx, but did not find any stranges, please advise me how to fix the issue.

[Tran Tien dung]

 Removing "/confluence" in proxy_pass line solved my issue. Even my confluence instance already configured context path /confluence. I don't know why I have to change it, these links with invalid character seen worked in the past.

    location /confluence {
        ...
        proxy_pass http://localhost:8090;
    }

[Henrique Bittencourt]

Hi Tran!

I'm not a Nginx pro, but, doing a little bit of digging into some internet threads I just found Nginx ignores some characters when those are passed on the headers:

The ignore_invalid_headers directive allows to control whether only headers matching the rules documented will be handled and passed to upstream servers by nginx, or any header will do. 

A possible workaround is to set the ignore_invalid_headers to off in the Nginx conf file:

• https://stackoverflow.com/questions/35193451/nginx-discards-invalid-headers-when-ignore-invalid-headers-is-set-to-off

It is worth testing!

Best Regards.

[Tran Tien dung]

Thanks Henrique, I tried as the link suggest, but no luck, the issue still persist. :(

[Markus Gut]

Since all of the workarounds described above, I added

relaxedPathChars="[]|" relaxedQueryChars="[]|{}^\`"<>"

to the server.xml, as proposed in JiraKB (https://confluence.atlassian.com/jirakb/changing-server-xml-to-handle-requests-with-special-characters-958453799.html). Now there is no more Tomcat error, instead Confluence throws a similar error:

Cause

java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in path at index 25: /display/HT/Ultra+Series+|+DHI-XVR8816S
    at com.github.kristofa.brave.servlet.ServletHttpServerRequest.getUri(ServletHttpServerRequest.java:27)

caused by: java.net.URISyntaxException: Illegal character in path at index 25: /display/HT/Ultra+Series+|+DHI-XVR8816S
    at java.net.URI$Parser.fail(URI.java:2848)

Stack Trace:[hide]

java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in path at index 25: /display/HT/Ultra+Series+|+DHI-XVR8816S
 at com.github.kristofa.brave.servlet.ServletHttpServerRequest.getUri(ServletHttpServerRequest.java:27)
 at com.atlassian.confluence.web.filter.ZipkinTracingFilter$1.lambda$create$0(ZipkinTracingFilter.java:38)
 at com.github.kristofa.brave.http.HttpServerRequestAdapter.getSpanName(HttpServerRequestAdapter.java:41)
 at com.github.kristofa.brave.ServerRequestInterceptor.handle(ServerRequestInterceptor.java:47)
 at com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:56)
 at com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:52)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
 at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:59)
 at com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:32)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at com.atlassian.core.filters.encoding.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:39)
 at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:42)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
 at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
 at org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:206)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
 at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
 at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
 at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:770)
 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
 at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.URISyntaxException: Illegal character in path at index 25: /display/HT/Ultra+Series+|+DHI-XVR8816S
 at java.net.URI$Parser.fail(URI.java:2848)
 at java.net.URI$Parser.checkChars(URI.java:3021)
 at java.net.URI$Parser.parseHierarchical(URI.java:3105)
 at java.net.URI$Parser.parse(URI.java:3063)
 at java.net.URI.<init>(URI.java:588)
 at com.github.kristofa.brave.servlet.ServletHttpServerRequest.getUri(ServletHttpServerRequest.java:25)
 ... 49 more

Any other ideas?

[Rilwan Ahmed]

This is due to Tomcat increased their security and no longer allows raw square brackets in the query string in newer versions

[Mathy Vanvoorden]

This was also reported as a bug here:

https://jira.atlassian.com/browse/CONFSERVER-57582

The suggested workaround works for me, just add

relaxedQueryChars="[,]"

to your connector in server.xml

[Tran Tien dung]

Unfortunately, this workaround not work in my issue. :(

Thanks you for let me know the bug related RSS feed.

[Shannon S]

Tran Tien dung,

Can you confirm that you do NOT receive the 400 error when you're not using the NGINX setup? 

If you do not, I would recommend reviewing this article to ensure your setup was completed as Confluence expects it to be:

• How to use NGINX to proxy requests for Confluence

Regards,

Shannon

[Tran Tien dung]

Hi Shannon,

Yes, bypassing nginx is ok and I configured nginx definiely the same this article.

Thanks.

[Shannon S]

Hi Tran,

Alright, thank you for confirming that. One more thing you can try is to enable access logs to all URLs.

Uncomment this line in confluence/WEB-INF/classes/log4j.properties:

log4j.category.com.atlassian.confluence.util.AccessLogFilter=INFO

And then add the alternative filter:

An alternative filter: For troubleshooting purposes, often it is useful to capture all accesses to Confluence. To do this use the following filter mapping in confluence/WEB-INF/web.xmlinstead of the above:

<filter-mapping>
   <filter-name>AccessLogFilter</filter-name>
   <url-pattern>/*</url-pattern>
</filter-mapping>

You'll want to first see what errors you see while behind NGINX, and then while bypassing it, and compare the two. This will help you to understand what the issue is. Be sure to use the same Confluence page URL when testing.

 At this point we can confirm that it's not an issue with Confluence, but your NGINX setup. It may be that NGINX doesn't support the language or characters you're using. In general, we do not provide support for NGINX, so you may find it useful to contact NGINX directly.

Regards,

Shannon

[Tran Tien dung]

Hi Shannon,

Yes, definely from Nginx, I enable a access log as your advise, I can see any requests from clients to Confluence except a page name with character ">", seen like nginx  that request, it not even go to confluence upstream.

I try to contact nginx directly, thanks for your link.

[Shannon S]

You're welcome, Tran. 

What about the solution that was proposed to your question on StackOverflow?

Probably you'll need to add a rewrite rule.

• Nginx Rewrite to Replace a single Character
• Nginx Doc - Creating Rewrite Rules

if ($request_uri ~ ^(/.*)[>](.*)$) { 
    return 301 $1%3E$2;
}
if ($request_uri ~ ^(/.*)[<](.*)$) { 
    return 301 $1%3C$2;
}

You may want to place this inside a location block to limit the scope. See this caution on the use of if.

Did that help?

Shannon

[Tran Tien dung]

Hi Shannon,

I tried that but no luck. In fact, nginx return a correct uri, so no need this rewrite rule, nginx just somehow reject the request.

Thanks.

[Shannon S]

Alright, I thought I'd just check that.

If you follow-up with that response let them know it didn't work, and perhaps they can help to find another solution, otherwise NGINX does have their own support and community.

Once you're able to resolve the issue you can come back here and let us know how it was solved. Even though we don't support NGINX issues here I'm sure we still have users coming here for similar issues and they would find your answer useful!

Regards,

Shannon

[Claus Westerkamp]

I confirm 400 is thrown regardless accessing the application port or webserver port

imho it is no nginx issue