Wednesday, December 19, 2012

How to use WSO2 ESB VFS transport to transfer large files..


We can use VFS transport in WSO2 ESB as described in the sample [1]. Apart from that Supun which was a project manger of WSO2 ESB has written and article[2] on that. These articles and samples provides a great help on using VFS transport to transfer files.

But when using that configurations for transferring large files (Greater that 500 MB), i got an Out Of Memory exceptions[3] and some other errors[4].

When looking in to the problem more deeply, i could found the solution for the problem. The issue was with the message builder class that i have used and missing a property. We need to use the message builder "org.apache.axis2.format.BinaryBuilder" for this. Apart from that  we need to include the property "ClientApiNonBlocking" in the proxy configuration.

As a sample i have defined a custom content type "chs/binary".

Entries in axis2.xml


<messageFormatters>
        ...    
        <messageFormatter contentType="chs/binary"
        class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
        ...
</messageFormatters>
    <messageBuilders>
        ...
        <messageBuilder contentType="chs/binary"
        class="org.apache.axis2.format.BinaryBuilder"/>
        ...
   </messageBuilders>

Then we need to have a sample VFS proxy as :

 <proxy xmlns="http://ws.apache.org/ns/synapse"
       name="FileProxy"
       transports="vfs"
       startOnLoad="true"
       trace="disable">
    <description/>
    <target>
       <inSequence>
          <log level="custom">
             <property name="FileProxy" value="Processing file"/>
          </log>
          <property name="OUT_ONLY" value="true"/>
          <property name="ClientApiNonBlocking"
                   value="true"
                  scope="axis2"
                  action="remove"/>
          <send>
             <endpoint name="FileEpr">
                <address uri="vfs:file:////home/shammi/file-out"/>
             </endpoint>
          </send>
       </inSequence>
    </target>
    <parameter name="transport.vfs.Streaming">true </parameter>
    <parameter name="transport.PollInterval">15 </parameter>
    <parameter name="transport.vfs.ActionAfterProcess">MOVE </parameter>
    <parameter name="transport.vfs.FileURI">file:///home/shammi/file-in </parameter>
    <parameter name="transport.vfs.MoveAfterProcess">file:///home/shammi/file-original </parameter>
    <parameter name="transport.vfs.MoveAfterFailure">file:////home/shammi/file-failure </parameter>
    <parameter name="transport.vfs.Locking">enable </parameter>
    <parameter name="transport.vfs.FileNamePattern">.*.zip|.*.test </parameter>
    <parameter name="transport.vfs.ContentType">chs/binary </parameter>
    <parameter name="transport.vfs.ActionAfterFailure">MOVE </parameter>
 </proxy>

    

With this proxy , i could use VFS transport to transfer a file with the size 1 GB with out any issue. I have made the points bold which needs to pay more attention..

Cheers.. Thats it......





[1]http://docs.wso2.org/wiki/pages/viewpage.action?pageId=15471427
[2]http://wso2.org/library/articles/2011/01/wso2-esb-example-file-processing
[3]2012-12-19 11:49:04,797 [-] [Framework Event Dispatcher]  WARN PollTableEntry transport.vfs.FileURI parameter is missing in the proxy service configuration
2012-12-19 11:50:08,896 [-] [vfs-Worker-3] ERROR NativeWorkerPool Uncaught exception
java.lang.OutOfMemoryError: Java heap space
at org.apache.commons.io.output.ByteArrayOutputStream.needNewBuffer(ByteArrayOutputStream.java:124)
at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:155)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1263)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1236)
at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:360)
at org.apache.axis2.format.BinaryBuilder.processDocument(BinaryBuilder.java:72)
at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:558)
at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:312)
at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:158)
at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:107)
at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)


[4]

2012-12-19 13:59:58,033 [-] [Axis2 Task] ERROR VFSTransportSender IO Error while creating response file : file:///home/shammi/wso2/foo/file-out/response.xml
org.apache.axis2.AxisFault: Error serializing binary content of element : {http://ws.apache.org/commons/ns/payload}binary
at org.apache.axis2.format.BinaryFormatter.writeTo(BinaryFormatter.java:66)
at org.apache.synapse.transport.vfs.VFSTransportSender.populateResponseFile(VFSTransportSender.java:235)
at org.apache.synapse.transport.vfs.VFSTransportSender.sendMessage(VFSTransportSender.java:173)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:627)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.commons.vfs2.FileNotFoundException: Could not read from "file:///home/shammi/wso2/foo/file-in/a.zip" because it is a not a file.
at org.apache.commons.vfs2.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1316)
at org.apache.commons.vfs2.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:397)
at org.apache.synapse.transport.vfs.FileObjectDataSource.getInputStream(FileObjectDataSource.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.axis2.format.ManagedDataSourceFactory$DataSourceManager.invoke(ManagedDataSourceFactory.java:91)
at $Proxy15.getInputStream(Unknown Source)
at javax.activation.DataHandler.writeTo(DataHandler.java:290)
at org.apache.axis2.format.BinaryFormatter.writeTo(BinaryFormatter.java:64)
... 7 more
Caused by: java.io.FileNotFoundException: /home/shammi/wso2/foo/file-in/a.zip (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:120)
at org.apache.commons.vfs2.provider.local.LocalFile.doGetInputStream(LocalFile.java:210)
at org.apache.commons.vfs2.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1308)
... 17 more
2012-12-19 14:00:26,458 [-] [Timer-7]  WARN TimeoutHa

1 comment:

  1. Helpful article. Did you test files above 1Gb? Any downside/drawback to making your configuration file properties the default option? Could the ESB recognized the filesize (from the http header) and fail gracefully or auto-adjust the processing behavior?

    ReplyDelete