How to deal with "java.nio.charset.MalformedInputException: Input length = 1" WSO2 ESB

Some times we are receiving unusual characters in our responses from various back ends. Then WSO2 ESB is facing problems in understanding those characters from the response and it tends to throw following exception.

[2015-07-09 12:42:49,651] ERROR - TargetHandler I/O error: Input length = 1
java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:277)
    at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:193)
    at org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:171)
    at org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection$LoggingNHttpMessageParser.parse(LoggingNHttpClientConnection.java:210)
    at org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection$LoggingNHttpMessageParser.parse(LoggingNHttpClientConnection.java:192)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:171)
    at org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection.consumeInput(LoggingNHttpClientConnection.java:106)
    at org.apache.synapse.transport.passthru.ClientIODispatch.onInputReady(ClientIODispatch.java:83)
    at org.apache.synapse.transport.passthru.ClientIODispatch.onInputReady(ClientIODispatch.java:41)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
    at java.lang.Thread.run(Thread.java:745)

Most of the case,  by adding following property to the "passthru-http.properties" file will solve the case.

Eg:  once i was getting a response like following

[2015-07-09 12:47:55,236] DEBUG - wire >> "HTTP/1.1 100 Continue[\r][\n]"
[2015-07-09 12:47:55,237] DEBUG - wire >> "[\r][\n]"
[2015-07-09 12:47:55,274] DEBUG - wire >> "HTTP/1.1 201 Cr[0xe9]e[\r][\n]"
[2015-07-09 12:47:55,274] DEBUG - wire >> "Set-Cookie: JSESSIONID=87101C43FCABBB97D049C0F0C8DD216D; Path=/api/; Secure; HttpOnly[\r][\n]"
[2015-07-09 12:47:55,274] DEBUG - wire >> "Location: https://abc.foo.com/api/office/files/2038[\r][\n]"
[2015-07-09 12:47:55,274] DEBUG - wire >> "Content-Type: application/json[\r][\n]"
[2015-07-09 12:47:55,274] DEBUG - wire >> "Transfer-Encoding: chunked[\r][\n]"
[2015-07-09 12:47:55,275] DEBUG - wire >> "Vary: Accept-Encoding[\r][\n]"
[2015-07-09 12:47:55,275] DEBUG - wire >> "Date: Thu, 09 Jul 2015 16:47:55 GMT[\r][\n]"
[2015-07-09 12:47:55,275] DEBUG - wire >> "Server: qa-byp[\r][\n]"
[2015-07-09 12:47:55,275] DEBUG - wire >> "[\r][\n]"
[2015-07-09 12:47:55,275] DEBUG - wire >> "1f[\r][\n]"
[2015-07-09 12:47:55,275] DEBUG - wire >> "{"person":john}[\r][\n]"
[2015-07-09 12:47:55,275] DEBUG - wire >> "0[\r][\n]"
[2015-07-09 12:47:55,275] DEBUG - wire >> "[\r][\n]"

Then it thew the above exception.

By adding the above entry as follows, i could get rid of that.

http.protocol.element-charset=iso-8859-1

සිතුවිල්ල

එදා සුදු ගවුම ඇඳන් බස් එකෙන් ඇවිත් බහිනකල්
නොඉවසිල්ලෙන් බස් ස්ටැන්ඩ් එකට වෙලා බලන් උන්නු මම...
අද..
පුංචි පුතාගෙ හුරතල් හිනාව බලන්න
ගෙදර යන්න ඔරලොසුවෙ කටුව කැරකෙනකල් බලන් ඉන්නවා..
කාලය... නුඹේ අරුමය..

WSO2 ESB API with JMS Queues - HTTP GET

We are going to discuss on how we can handle HTTP GET methods with JMS Proxy services in WSO2. With this blog we are going to discuss following message flow.

Flow is like follows;

• Client invokes an API defined in ESB
• From the API it sends the message to a JMS queue specifying a "ReplyTo" queue
• There is a JMS proxy consume message from the above queue 
• After consuming message, it will invoke the backend and response will be sent back to JMS "Reply To" queue
• Rest API will consume the message from that "Reply To" queue and send back the response to the client.

In order to have this flow working, you need to setup ActiveMQ with wso2 ESB. You can find the documentation to do it in this documentation page.

Lets see the API

<api xmlns="http://ws.apache.org/ns/synapse" name="WeatherAPI" context="/getQuote">
   <resource methods="POST GET" uri-template="/details?*">
      <inSequence>
         <property name="transport.jms.ContentTypeProperty" value="Content-Type" scope="axis2"></property>
         <log>
            <property name="httpMethod" expression="get-property('axis2','HTTP_METHOD')"></property>
         </log>
         <property name="HTTP_METHOD" expression="get-property('axis2','HTTP_METHOD')" scope="transport" type="STRING"></property>
         <send>
            <endpoint>
               <address uri="jms:/SMSStore?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&java.naming.provider.url=tcp://localhost:61616&transport.jms.DestinationType=queue&transport.jms.ContentTypeProperty=Content-Type&transport.jms.ReplyDestination=SMSReceiveNotificationStore"></address>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <property name="TRANSPORT_HEADERS" scope="axis2" action="remove"></property>
         <send></send>
      </outSequence>
   </resource>
</api>

In the above API , there are few things to be focused on.

• You can see that we are setting the HTTP_METHOD to a property extracted from the incoming message as follows.  By default, for HTTP POST method, API invocation works perfectly with out setting this property at the end. But If we are doing a HTTP GET , we need to obtain the HTTP method at the consumer side of the queue to invoke the back end with a http GET. So we are setting it here. 

<property name="HTTP_METHOD" expression="get-property('axis2','HTTP_METHOD')" scope="transport" type="STRING"></property>

• In the endpoint of the send mediator we are setting up following information which you should have 
• transport.jms.ContentTypeProperty=Content-Type  - We are saying that we are passing the content type in JMS transport
• ransport.jms.ReplyDestination=SMSReceiveNotificationStore - We are saying that ESB is expecting a response to this queue for this request 

Then in the JMS proxy which we consume the message, it will look like follows.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="SMSForwardProxy"
       transports="jms"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target faultSequence="fault">
      <inSequence>
         <property name="HTTP_METHOD" expression="$trp:HTTP_METHOD" scope="axis2"/>
         <send>
            <endpoint>
               <address uri="http://localhost:8080/foo">
                  <suspendOnFailure>
                     <errorCodes>-1</errorCodes>
                     <progressionFactor>1.0</progressionFactor>
                  </suspendOnFailure>
                  <markForSuspension>
                     <errorCodes>-1</errorCodes>
                  </markForSuspension>
               </address>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <parameter name="transport.jms.ContentType">
      <rules>
         <jmsProperty>contentType</jmsProperty>
         <default>text/xml</default>
      </rules>
   </parameter>
   <parameter name="transport.jms.ConnectionFactory">myQueueConnectionFactory</parameter>
   <parameter name="transport.jms.DestinationType">queue</parameter>
   <parameter name="transport.jms.Destination">SMSStore</parameter>
   <description/>
</proxy>

In the above proxy configuration, you can see following line

 <property name="HTTP_METHOD" expression="$trp:HTTP_METHOD" scope="axis2"/>

In that , what we do is , we read the HTTP_METHOD we already set in previous API before sending the message to JMS Queue and now we are reading it from transport level property.  Then we are setting it to "HTTP_METHOD" property with the "axis2" scope.

With that it will be able to figure out the HTTP method to be used on invoking the actual endpoint.


After getting a response, this Proxy will send the reply to the "SMSReceiveNotificationStore" queue. Then it will consumed from the previous API and send back the response to the client.

This will be working perfectly for HTTP POST requests. If it is a HTTP GET Request , you need to do some additional changes from the client side and the server side.

Client Side Changes

• We need to pass the content type from the client side as follows. You can set it as a header in SOAP UI

       contentType:  application/x-www-form-urlencoded

Server Side Changes

• You need to change the builder and formatter for above content type in the axis2.xml file located in WSO2ESB/repository/conf/axis2 directory as follows. 

.....

<messageFormatter contentType="application/x-www-form-urlencoded"
                        class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>

.......

<messageBuilder contentType="application/x-www-form-urlencoded"
                        class="org.wso2.carbon.relay.BinaryRelayBuilder"/>

......

If you does not set above , there will be message building and formatting errors.


With this way you ll be able to get this working for GET http methods.

This is tested with WSO2 ESB 4.8.1 with it's latest patches and Apache Activemq 5.10.0

How to generate a custom Error Message with Custom HTTP Status Code for unavailable Resources in WSO2 ESB

WSO2 ESB 4.8.1  does not throw any exception or error message when an API defined is access with incorrect HTTP method and it will just respond with 202.  In this blog post , i am explaining on how we can get a custom HTTP status code for the above.


In order to get a custom error message , you need to add following sequence to ESB which is not there by default.

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="_resource_mismatch_handler_">
   <payloadFactory media-type="xml">
      <format>
         <tp:fault xmlns:tp="http://test.com">
            <tp:code>405</tp:code>
            <tp:type>Status report</tp:type>
            <tp:message>Method not allowed</tp:message>
            <tp:description>The requested HTTP method for resource (/$1) is not allowed.</tp:description>
         </tp:fault>
      </format>
      <args>
         <arg xmlns:ns="http://org.apache.synapse/xsd"
              xmlns:ns3="http://org.apache.synapse/xsd"
              evaluator="xml"
              expression="$axis2:REST_URL_POSTFIX"/>
      </args>
   </payloadFactory>
   <property name="NO_ENTITY_BODY" scope="axis2" action="remove"/>
   <property name="HTTP_SC" value="405" scope="axis2"/>
   <respond/>
   <drop/>
</sequence>

In ESB documentation [1] , it has explained that in order to handle non-matching resources, it is needed to define this sequence _resource_mismatch_handler_


[1] https://docs.wso2.com/display/ESB481/Configuring+Specific+Use+Cases#ConfiguringSpecificUseCases-Handlingnon-matchingresources

How to generate a custom Error Message with Custom HTTP Status Code for unavailable Resources in WSO2 API Manager

We are going to explain on how we can generate a custom HTTP Status code for a request which is addressed to a un-matching resource of an API.

Problem :

When an API exposed with resource "GET" , if the client invoke the API with "POST","PUT" or any other which is not "GET", By default API manager returns following.

{  
   "fault":{  
      "code":"900906",
      "type":"Status report",
      "message":"Runtime Error",
      "description":"No matching resource found in the API for the given request"
   }
}

In the RAW level you ll see it as follows

HTTP/1.1 403 Forbidden
Access-Control-Allow-Headers: authorization,Access-Control-Allow-Origin,Content-Type
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE,OPTIONS
Content-Type: application/xml; charset=UTF-8
Date: Mon, 29 Jun 2015 14:46:29 GMT
Server: WSO2-PassThrough-HTTP
Transfer-Encoding: chunked
Connection: Keep-Alive

<ams:fault xmlns:ams="http://wso2.org/apimanager/security">
   <ams:code>900906</ams:code>
   <ams:message>No matching resource found in the API for the given request</ams:message>
   <ams:description>Access failure for API: /sss, version: v1 with key: 4a33dc81be68d1b7a5b48aeffebe7e</ams:description>
</ams:fault>

Expected Solution :

We need to change this HTTP Response code 405 [1] with a custom error message.



Solution :

We need to create a sequence which builds the custom error message and the error code and deploy it in API manager's default sequences folder.

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="converter">
    <payloadFactory media-type="xml">
        <format>
            <am:fault xmlns:am="http://wso2.org/apimanager">
                <am:message>Resource not found</am:message>
                <am:description>Wrong http method</am:description>
            </am:fault>
        </format>
    </payloadFactory>
    <property name="RESPONSE" value="true"/>
    <header name="To" action="remove"/>
    <property name="HTTP_SC" value="405" scope="axis2"/>
    <property name="messageType" value="application/xml" scope="axis2"/>
    <send/>
</sequence>

You can save this as converter.xml in wso2am-1.8.0/repository/deployment/server/synapse-configs/default/sequences folder.

Then we need to invoke this sequence in _auth_failure_handler_.xml which is located in the above sequences folder. In order to do that , we need to change it as follows.

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="_auth_failure_handler_">
   <property name="error_message_type" value="application/xml"/>
    <filter source="get-property('ERROR_CODE')" regex="900906">
      <then>
          <sequence key="converter"/>
          <drop/>
      </then>
      <else>
      </else>
    </filter>
    <sequence key="_build_"/>
</sequence>

Once you done the above changes, save them. Then you can test your scenario. If you are successful with this , you ll be able see following response

HTTP/1.1 405 Method Not Allowed
Host: 10.210.1.202:8243
Content-Type: application/xml
Date: Mon, 29 Jun 2015 14:59:12 GMT
Server: WSO2-PassThrough-HTTP
Transfer-Encoding: chunked
Connection: Keep-Alive

<am:fault xmlns:am="http://wso2.org/apimanager">
   <am:message>Resource not found</am:message>
   <am:description>Wrong http method</am:description>
</am:fault>

Explanation : 

By default, when we invoke an non-existing resource it will send the default 403 error code with the message "No matching resource found in the API for the given request". If you check the log of the WSO2 AM, you can see that it has thrown following exception in the backend.

[2015-06-29 10:59:12,103] ERROR - APIAuthenticationHandler API authentication failure
org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Access failure for API: /sss, version: v1 with key: 4a33dc81be68d1b7a5b48aeffebe7e
    at org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate(OAuthAuthenticator.java:212)
    at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest(APIAuthenticationHandler.java:94)
    at org.apache.synapse.rest.API.process(API.java:284)
    at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:83)
    at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:64)
    at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:220)
    at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
    at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
    at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:344)
    at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:385)
    at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:183)
    at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

When it throws above exception, the flow will hit the  _auth_failure_handler_.xml
sequence. So what we have done in this sequence, with using the filter mediator, we have filtered the error code "900906" and for that error code, we invoke our custom sequence and drop the message then.

In the custom sequence , we have used the payload factory mediator to create the payload and added required properties to make it as response. You can find the information further on each of those properties from [2][3][4]

Then after invoking the custom sequence, it will invoke the "_build_" sequence in the same folder which invoke the message builders to build the message.

I have used resources [4] on creating this blog post.


[1] http://www.checkupdown.com/status/E405.html#
[2] https://docs.wso2.com/display/ESB481/Generic+Properties#GenericProperties-RESPONSE
[3] https://docs.wso2.com/display/ESB481/Generic+Properties#GenericProperties-messageType
[4] http://sanjeewamalalgoda.blogspot.com/2015/04/how-to-generate-custom-error-message.html

How to add a thread sleep to a Proxy Service

Here i am going to provide you a example on how we can create a mock service with WSO2 ESB and adding a sleep to that service.

In order to do that we need to use ;

1. Payload Factory mediator to create the mock response
2. script mediator to do a thread sleep

Here is the simple mock service proxy with a thread sleep.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="MyMockProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log>
            <property name="===Before sleep===" value="===Before sleep==="/>
         </log>
         <script language="js">java.lang.Thread.sleep(75000);</script>
         <log>
            <property name="===After sleep===" value="===After sleep==="/>
         </log>
         <payloadFactory media-type="xml">
            <format>
               <Response xmlns="">
                  <status>OK</status>
                  <code>1</code>
               </Response>
            </format>
            <args/>
         </payloadFactory>
         <header name="To" action="remove"/>
         <property name="RESPONSE" value="true" scope="default" type="STRING"/>
         <property name="NO_ENTITY_BODY" scope="axis2" action="remove"/>
         <send/>
      </inSequence>
   </target>
   <description/>
</proxy>

I have used the blog of miyuru [1] to create this.

[1] http://miyurudw.blogspot.com/2012/08/how-to-create-simple-mock-services.html

WSO2 IS User Store as ReadOnly/ReadWrite LDAP secondary user store

In most of the testing scenarios, we need to connect our products in to a secondary user store which is ReadOnly or ReadWrite Ldap User stores.

This is a simple way to get it done with WSO2 Identity Server.

Not as other WSO2 products, IS ships LDAP User store as it's primary user store. So if we need to point any of the other products in to a LDAP secondary user store, we can easily use WSO2 IS for that.



Case 01: Pointing WSO2 AM to a ReadOnlyLDAP Secondary user store

• Download, Extract, Start WSO2 IS
• Download, Extract WSO2 AM
• If we are running both products in the same machine, we need to change the offset of the AM
• Open the carbon.xml file located in "wso2am-1.9.0/repository/conf" folder and change the Offset value to "1". (By default it is "0")
• Start AM
• Browse url https://localhost:9444/carbon/
• Login with credentials admin/admin
• From the left menu , click on "Configure"

• Click on "User Store Management"
• Then click on "Add Secondary User Store" button 
• From the drop down at the top, select "ReadOnlyLdapUserStoreManager" as the user store manager class.
• Then provide parameters as follow

• Domain Name : Any Name (wso2.com)
• Connection Name : uid=admin,ou=system
• Connection URL : ldap://localhost:10389
• Connection Password : admin
• User search base : ou=Users,dc=wso2,dc=org
• User Object Class : (objectClass=person)
• Username Attribute : uid
• User search filter : (&(objectClass=person)(uid=?))
• Then click on Add. 
• After few seconds, it will be displayed in the user Store list 
• You can find these configurations in user-mgt.xml file located in  "wso2am-1.9.0/repository/conf" folder. But you need to focus on the parameter "User search base".  By default it is given as "ou=system". But with that you ll not be able to view the users of the secondary user store. Here i have added the correct parameter value " ou=Users,dc=wso2,dc=org"

Case 02: Pointing WSO2 AM to a ReadWriteLDAP Secondary user store

Please follow the documentation https://docs.wso2.com/display/AM190/Configuring+Secondary+User+Stores