How to use WSO2 ESB VFS transport to transfer large files..

We can use VFS transport in WSO2 ESB as described in the sample [1]. Apart from that Supun which was a project manger of WSO2 ESB has written and article[2] on that. These articles and samples provides a great help on using VFS transport to transfer files.

But when using that configurations for transferring large files (Greater that 500 MB), i got an Out Of Memory exceptions[3] and some other errors[4].

When looking in to the problem more deeply, i could found the solution for the problem. The issue was with the message builder class that i have used and missing a property. We need to use the message builder "org.apache.axis2.format.BinaryBuilder" for this. Apart from that  we need to include the property "ClientApiNonBlocking" in the proxy configuration.

As a sample i have defined a custom content type "chs/binary".

Entries in axis2.xml


<messageFormatters>
        ...    
        <messageFormatter contentType="chs/binary"
        class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>

        ...

</messageFormatters>

    <messageBuilders>

        ...

        <messageBuilder contentType="chs/binary"

        class="org.apache.axis2.format.BinaryBuilder"/>

        ...

   </messageBuilders>

Then we need to have a sample VFS proxy as :

 <proxy xmlns="http://ws.apache.org/ns/synapse"
       name="FileProxy"
       transports="vfs"
       startOnLoad="true"
       trace="disable">
    <description/>
    <target>
       <inSequence>
          <log level="custom">
             <property name="FileProxy" value="Processing file"/>
          </log>
          <property name="OUT_ONLY" value="true"/>
          <property name="ClientApiNonBlocking"
                   value="true"
                  scope="axis2"
                  action="remove"/>
          <send>
             <endpoint name="FileEpr">
                <address uri="vfs:file:////home/shammi/file-out"/>
             </endpoint>
          </send>
       </inSequence>
    </target>
    <parameter name="transport.vfs.Streaming">true </parameter>
    <parameter name="transport.PollInterval">15 </parameter>
    <parameter name="transport.vfs.ActionAfterProcess">MOVE </parameter>
    <parameter name="transport.vfs.FileURI">file:///home/shammi/file-in </parameter>
    <parameter name="transport.vfs.MoveAfterProcess">file:///home/shammi/file-original </parameter>
    <parameter name="transport.vfs.MoveAfterFailure">file:////home/shammi/file-failure </parameter>
    <parameter name="transport.vfs.Locking">enable </parameter>
    <parameter name="transport.vfs.FileNamePattern">.*.zip|.*.test </parameter>
    <parameter name="transport.vfs.ContentType">chs/binary </parameter>
    <parameter name="transport.vfs.ActionAfterFailure">MOVE </parameter>
 </proxy>

    

With this proxy , i could use VFS transport to transfer a file with the size 1 GB with out any issue. I have made the points bold which needs to pay more attention..

Cheers.. Thats it......





[1]http://docs.wso2.org/wiki/pages/viewpage.action?pageId=15471427
[2]http://wso2.org/library/articles/2011/01/wso2-esb-example-file-processing
[3]2012-12-19 11:49:04,797 [-] [Framework Event Dispatcher]  WARN PollTableEntry transport.vfs.FileURI parameter is missing in the proxy service configuration
2012-12-19 11:50:08,896 [-] [vfs-Worker-3] ERROR NativeWorkerPool Uncaught exception
java.lang.OutOfMemoryError: Java heap space
at org.apache.commons.io.output.ByteArrayOutputStream.needNewBuffer(ByteArrayOutputStream.java:124)
at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:155)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1263)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1236)
at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:360)
at org.apache.axis2.format.BinaryBuilder.processDocument(BinaryBuilder.java:72)
at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:558)
at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:312)
at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:158)
at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:107)
at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)


[4]

2012-12-19 13:59:58,033 [-] [Axis2 Task] ERROR VFSTransportSender IO Error while creating response file : file:///home/shammi/wso2/foo/file-out/response.xml
org.apache.axis2.AxisFault: Error serializing binary content of element : {http://ws.apache.org/commons/ns/payload}binary
at org.apache.axis2.format.BinaryFormatter.writeTo(BinaryFormatter.java:66)
at org.apache.synapse.transport.vfs.VFSTransportSender.populateResponseFile(VFSTransportSender.java:235)
at org.apache.synapse.transport.vfs.VFSTransportSender.sendMessage(VFSTransportSender.java:173)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:627)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.commons.vfs2.FileNotFoundException: Could not read from "file:///home/shammi/wso2/foo/file-in/a.zip" because it is a not a file.
at org.apache.commons.vfs2.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1316)
at org.apache.commons.vfs2.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:397)
at org.apache.synapse.transport.vfs.FileObjectDataSource.getInputStream(FileObjectDataSource.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.axis2.format.ManagedDataSourceFactory$DataSourceManager.invoke(ManagedDataSourceFactory.java:91)
at $Proxy15.getInputStream(Unknown Source)
at javax.activation.DataHandler.writeTo(DataHandler.java:290)
at org.apache.axis2.format.BinaryFormatter.writeTo(BinaryFormatter.java:64)
... 7 more
Caused by: java.io.FileNotFoundException: /home/shammi/wso2/foo/file-in/a.zip (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:120)
at org.apache.commons.vfs2.provider.local.LocalFile.doGetInputStream(LocalFile.java:210)
at org.apache.commons.vfs2.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1308)
... 17 more
2012-12-19 14:00:26,458 [-] [Timer-7]  WARN TimeoutHa

Exception during startup: org.wso2.andes.server.Broker$InitException: File null could not be found. Check the file exists and is readable.

Exception during startup: org.wso2.andes.server.Broker$InitException: File null could not be found. Check the file exists and is readable.
org.wso2.andes.server.Broker$InitException: File null could not be found. Check the file exists and is readable.
at org.wso2.andes.server.Broker.getConfigFile(Broker.java:323)
at org.wso2.andes.server.Broker.startupImpl(Broker.java:120)
at org.wso2.andes.server.Broker.startup(Broker.java:102)
at org.wso2.andes.server.Main.startBroker(Main.java:227)
at org.wso2.andes.server.Main.execute(Main.java:220)
at org.wso2.andes.server.Main.(Main.java:63)
at org.wso2.andes.server.Main.main(Main.java:53)
.....

You may get the pointed out exception when installed Message Broker features in WSO2 carbon or in any other wso2 products. The root cause for this problem is missing a system property in the start up

script. 

You can solve this by adding the system property to the wso2server.sh file or the wso2server.bat file of your server. The property is :

-DandesConfig=qpid-config.xml \

When you added this property to the wso2server.sh file , it should look like;

 $JAVACMD \

    -Xbootclasspath/a:"$CARBON_XBOOTCLASSPATH" \

    -Xms256m -Xmx1024m -XX:MaxPermSize=256m \

    -XX:+HeapDumpOnOutOfMemoryError \

    -XX:HeapDumpPath="$CARBON_HOME/repository/logs/heap-dump.hprof" \

    $JAVA_OPTS \

     -DandesConfig=qpid-config.xml \

    -Dcom.sun.management.jmxremote \

    -classpath "$CARBON_CLASSPATH" \

    -Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \

    -Djava.io.tmpdir="$CARBON_HOME/tmp" \

This issue will be fixed with the 2.0.2 release of the wso2 message broker.

Find and Replace Strings in Linux with Grep and Sed

It would be very useful if we can find a string in multiple files and replace it with a new string. It takes a single line of code to do that :

grep -rl matchstring somedir/ | xargs sed -i 's/string1/string2/g'

matchString - the string you want to find
string1 - same string you want to find
string2 - new string that you want to replace with


eg :
I need to find the string "foo" and replace it with "bar"

grep -rl 'foo' ./ | xargs sed -i 's/foo/bar/g'  

 There is a nice post on describing more on this:
http://vasir.net/blog/ubuntu/replace_string_in_multiple_files/

WSO2 ESB Message Stores with WSO2 MB 2.0 -M2

When using JMS Message stores in WSO2 ESB, there is a small difference scenario need to be followed 

 other than in the article written by charith http://wso2.org/library/articles/2011/12/implementing-store-forward-messaging-patterns-wso2esb-part-2

We need to place "andes-client-0.13.wso2v1.jar" , "slf4j.api_1.6.1.jar" and "slf4j.log4j12_1.6.1.jar" libraries in the root lib directory of wso2 ESB before starting the server. 

Note : Offset configuration is not working in the M2 release of MB 2.0. So  you need to change the offset value of ESB instead of MB and the connection String should be like:

connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/carbon?brokerlist='tcp://localhost:5672'

WSO2 Message Broker 2.0 - M2 Released

WSO2 integration development team is pleased to announce that the feature freezed version of WSO2 Message Broker 2.0 is released. In this release we have added Clustering support for Message broker where it can act as a distributed Message broker.

You can find the documentation on different cluster scenarios,setup information in the deployment guide in the doc pack. 

http://people.wso2.com/~charith/

You can find the current issues here

https://wso2.org/jira/browse/CARBON/component/10761

WSO2 MB - Samples : SQS SOAP Client Sample

Many users experiencing problems when testing this sample.



Here i am going to mention few simple steps to execute this sample.

1. Download WSO2MB 1.0.2 pack from http://wso2.org/downloads/message-broker

2. Extract and start the product.

3. Log in to the admin console of MB with credentials admin/admin



4. Navigate to the WSDL2Java tool which can be found in "Tools" left menu



5. Copy and paste the url of sqs wsdl to the provided test box in the "uri" option

6. Press generate button which can be found in the bottom of that page and it will give you the
zip file of generated source pack and save it.

7. Extract the source pack to your project folder. Once you extracted it, you can see a structure like :
.
|-- build.xml
|-- pom.xml
`-- src
`-- com
`-- amazonaws
`-- queue
`-- doc
`-- _2009_02_01
|-- MessageQueueCallbackHandler.java
|-- MessageQueueStub.java
|-- QueueServiceCallbackHandler.java
`-- QueueServiceStub.java


8. Go inside the project folder where you can find the above pom.xml file in the command line. Create the idea project by typing "mvn idea:idea".

9. Open the created project with IDEA

10. Add the SQSClient class to the project.

11. Change the "accessKey" and the "secretAccessKey" to the keys which can be found at "Home > Manage> Message Boxes(SQS)> Access Keys" of wso2mb server.


12.Build the project and execute the sqs client




NOTE:

Possible errors to be occurred when building the project

"cannot find symbol method createOMElement(org.apache.axiom.om.OMDataSource,javax.xml.namespace.QName)"

This is due to version mismatch of the axiom. Valid version of all axiom libraries is the version specified in the pom.xml

eg: 1.2.11.wso2v1

But where there are axiom libraries which is different from the above version, it gives above error.

You need to remove that version mismatching library from your class path .
Eg: axiom-api-1.2.7.jar

With this way you can get rid of this problem.

Allowing a port from firewall in ubuntu

Lets say the port we need to make open from firewall is "2181"

$sudo ufw allow 2181