Artifact sharing failure

[Adam Lins]

I have a build plan with stages and tasks running on a Remote Agent (Windows Server 2008 R2). I wanted to share an output executable as an artifact with another task in the same plan. The artifact can be seen, but cannot be copied from the remote agent to the host: there is a socket write error.

Communication between the host and remote agent has been secured with SSL certificates per the Atlassian documentation. The remote agent can communicate with the host OK: the plan works up until the point of sharing the artifact. What could be going wrong? Do I have to specify some other parameter in the wrapper to convince the artifact copy to take place over HTTPS instead of SSL? Do I need to open another port somewhere?

{code}

INFO | jvm 1 | 2012/08/12 17:10:52 | 2012-08-12 17:10:52,662 INFO 1-BAM::10.1.10.116::Agent:pool-3-thread-1 AbstractArtifactManager Publishing Client executables for PCB-PCBP-BUILDUI-20: 1 file(s) matching iCelero Internet Acceleration.exe in directory D: Users bamboo-agent-home xml-data build-dir PCB-PCBP-BUILDUI .. PCB-PCBP-CHKOUT IWP Win32 Release

INFO | jvm 1 | 2012/08/12 17:10:53 | 2012-08-12 17:10:53,176 ERROR 1-BAM::10.1.10.116::Agent:pool-3-thread-1 HttpMessageSender Exception occurred when posting message com.atlassian.bamboo.v2.build.agent.messages.ArtifactPublishMessage@262ad8e1. Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset by peer: socket write error

INFO | jvm 1 | 2012/08/12 17:10:53 | javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset by peer: socket write error

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.SSLSocketImpl.checkWrite(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.AppOutputStream.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.BufferedOutputStream.flushBuffer(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.BufferedOutputStream.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:119)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.ChunkedOutputStream.write(ChunkedOutputStream.java:178)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.writeFatalException(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.writeObject(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.v2.build.agent.remote.sender.HttpMessageSender$1.writeRequest(HttpMessageSender.java:59)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:495)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:1973)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:993)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.v2.build.agent.remote.sender.HttpMessageSender.send(HttpMessageSender.java:87)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteArtifactManager.publishFileSet(RemoteArtifactManager.java:45)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.build.artifact.AbstractArtifactManager.publish(AbstractArtifactManager.java:95)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.build.artifact.BuildArtifactPostProcessor.call(BuildArtifactPostProcessor.java:59)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.performCustomBuildProcess(ExecuteBuildTask.java:161)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.call(ExecuteBuildTask.java:87)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:204)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:103)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:109)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$2.run(NamedThreadFactory.java:50)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.lang.Thread.run(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset by peer: socket write error

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.Alerts.getSSLException(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.SSLSocketImpl.handleException(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.SSLSocketImpl.handleException(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.AppOutputStream.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.BufferedOutputStream.flushBuffer(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.BufferedOutputStream.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:119)

INFO | jvm 1 | 2012/08/12 17:10:53 | at org.apache.commons.httpclient.ChunkedOutputStream.write(ChunkedOutputStream.java:178)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream$BlockDataOutputStream.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.nio.channels.Channels$WritableByteChannelImpl.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at com.atlassian.bamboo.v2.build.agent.messages.ArtifactTransferPayload.writeObject(ArtifactTransferPayload.java:83)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.lang.reflect.Method.invoke(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectStreamClass.invokeWriteObject(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.writeSerialData(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.writeObject0(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.writeSerialData(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.io.ObjectOutputStream.writeObject0(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | ... 20 more

INFO | jvm 1 | 2012/08/12 17:10:53 | Caused by: java.net.SocketException: Connection reset by peer: socket write error

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.net.SocketOutputStream.socketWrite0(Native Method)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.net.SocketOutputStream.socketWrite(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at java.net.SocketOutputStream.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.OutputRecord.writeBuffer(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.OutputRecord.write(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.SSLSocketImpl.writeRecordInternal(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)

INFO | jvm 1 | 2012/08/12 17:10:53 | ... 40 more

INFO | jvm 1 | 2012/08/12 17:10:53 | 2012-08-12 17:10:53,223 INFO 1-BAM::10.1.10.116::Agent:pool-3-thread-1 HttpMethodDirector I/O exception (java.io.IOException) caught when processing request: Exception occurred when posting message com.atlassian.bamboo.v2.build.agent.messages.ArtifactPublishMessage@262ad8e1. Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset by peer: socket write error

{code}

[Adam Lins · Accepted solution]

We used nginx; once it was upgraded to support HTTP 1.1, Artifact Copying worked like a charm. Thanks to Atlassian Support who helped resolve this.

[Pascal]

I had exactly the same problem and started looking into configuring "chunkin" for our Nginx install. Then I found that from Nginx 1.3.9 onward, this "chunkin" configuration is not necessary anymore, since "chunked_transfer_encoding" is now on by default. We were running Nginx 1.4.6. So this was not the problem. Then I found that another build plan was returning a "413 Request Entity Too Large" error, which seemed strange, since out "client_max_body_size" was set to 20M and none of the artifacts were larger than this. Nevertheless, I increased the "client_max_body_size" to 800M and lo and behold: it worked. Turns out that Bamboo packed the artifacts into one package with a total size of 46MB. 

Then I gave the build plan with the initial problem another try. That plan was building a far bigger solution. Turns out that increasing the "client_max_body_size" worked for this as well: the packed artifacts had a total size of 650MB!

[Gavin Bunney]

I'm having the same problem with our nginx -> bamboo setup. The artifact copier is failing to publish artifacts from our remote agent.

What was the change you made in the nginx configure Adam that enabed the HTTP 1.1 support? We have added 'proxy_http_version 1.1' but still receiving the same error.

Thanks!

[Adam Lins]

There is some downloading and rebuilding to be done to install the chunked encoding support into the NGinx application. I assume that you have that built and installed?

Once that is done, the key is to add the chunked encoding directives to the NGinx sites-enabled configuration file (lines 11--15 below).

server {
    listen   443;
    server_name  bamboo.yourdomain.com;
    ssl  on;
    ssl_certificate         /etc/nginx/ssl/yourdomain.com.crt;
    ssl_certificate_key     /etc/nginx/ssl/yourdomain.com.key;
    ssl_protocols  TLSv1 SSLv3;
    ssl_ciphers RSA:!EXP:!NULL:+HIGH:+MEDIUM:-LOW;
    ssl_prefer_server_ciphers   on;
    
    chunkin     on;
    error_page 411 = @my_411_error;
    location @my_411_error {
        chunkin_resume;
    }
    
    location / {
        proxy_pass      http://bamboo.yourdomain.com:8085/;
        proxy_redirect http://bamboo.yourdomain.com:8085/ /;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

[Gavin Bunney]

Excellent thanks Adam! Adding chunked encoding support seems to have fixed the problem.

[Adam Lins]

I answered the question with my comment.

[Przemek Bruski]

Which agent server URL are you using? Does it point to Bamboo directly, or some fronting server like nginx?