Skip to main content

Processing large payloads with the esb script mediator iteratively


Overview

WSO2 ESB uses Rhino engine to execute JavaScripts. Rhino engine converts the script to a method inside a Java class. Therefore, when processing large JSON data volumes, the code length must be less than 65536 characters, since the Script mediator converts the payload into a Java object. However, you can use the following alternative options to process large JSON data volumes.

The script mediator which is used in ESB is powered by the Rhino engine. Therefore, when processing large JSON data volumes, the code length must be less than 65536 characters which is a limitation in the script mediator being used in the esb versions less than 5.0.0. In ESB 5.0.0 there is a higher capability to process larger payloads using script mediator. In order to process such large payloads we can follow the below two approaches.

1. Replace the javascript tranformation logic using java code by writing a custom mediator. [1]
2. Break down the large payload and execute them as sections using the script mediator.

This blog post will be demonstrating how to carry out the use case #2 using wso2 esb.

Let's get started

The approach we would be using is to iterate over the large response and pass each section through the script mediator. After which we will be aggregating the tranformed sections to display the modified and transformed response.

1) Let's create a mock service which returns the response mentioned in this text file. I have used the online tool http://www.mocky.io/ to generate this response. The invocation URL for my mock response is http://www.mocky.io/v2/57a881a3110000b6171d4546

2) Then we need to create the In sequence. In the ESB in the Service bus section , select the Sequences option and add a new sequence. This is the source view of the in sequence we will be using which simply has a endpoint invocation. This is the in sequence configuration and you can also download it from HERE

    
<sequence name="sendSequence" xmlns="http://ws.apache.org/ns/synapse">
    <log level="custom">
        <property name="test" value="Execution Started"/>
    </log>
    <send>
        <endpoint>
            <http method="GET" uri-template="http://www.mocky.io/v2/57a881a3110000b6171d4546"/>
        </endpoint>
    </send>
</sequence>


3) Next we need to create the out sequence which does the needed transformation. I have listed what happens within the sequence to get a better understanding.

  • I have used the response generated from the above invocation here which has an array of json objects. Since the objects inside the array are repititive we can split it using the iterate mediator. (In this sequence the splitting is done by the ARR_ITEM expression)
  • After splitting it using the iterate mediator we can manipulate each JSON object by passing it through the script mediator.
  • At the end we are placing an aggregate mediator which will aggregate all the modified JSON elements which went through the iterate mediator and display the response.
  • We will be storing the aggregated response inside the rootElement property as json objects. Follow is the sequence which executes this logic. This is the outsequence of our API.
Simillar to above you need to add another sequence which we would be using as the out sequence. This is the sequence source view corresponding to the out sequence described above. You can also download the sequence HERE

       
<sequence xmlns="http://ws.apache.org/ns/synapse" name="iterOutTest">
   <property name="rootElement" scope="default">
      <jsonObject xmlns=""/>
   </property>
   <iterate xmlns:ns="http://org.apache.synapse/xsd"
            continueParent="true"
            expression="//ARR_ITEM">
      <target>
         <sequence>
            <log level="custom">
               <property name="iterator" value="in the iterate"/>
            </log>
            <payloadFactory media-type="json">
               <format>
                $1
            </format>
               <args>
                  <arg evaluator="json" expression="$."/>
               </args>
            </payloadFactory>
            <script language="js">
               mc.setProperty('CONTENT_TYPE', 'application/json');
               var payLoad = mc.getPayloadJSON();
               var catalogItem = payLoad.ARR_ITEM;
               catalogItem.CT.CT_ITEM.ED = "456"; 
               catalogItem.DISCRIPTIONS = "This is the modified description";                
               mc.setPayloadJSON(payLoad);</script>
            <log level="full"/>
            <aggregate>
      <completeCondition>
         <messageCount min="-1" max="-1"/>
      </completeCondition>
      <onComplete xmlns:ns="http://org.apache.synapse/xsd"
                  expression="//jsonObject/ARR_ITEM"
                  enclosingElementProperty="rootElement">
         <log level="full">
            <property name="MSG" value="Aggregated Msg"/>
         </log>
         <respond/>
      </onComplete>
   </aggregate>
         </sequence>
      </target>
   </iterate>
</sequence>

4) After we have created the sequences we need to then add it to the API. Refer the below screenshot. We need to add the above created in and out sequences to the API respectively


Here you can find the xml configuration corresponding to the above API. Also you can download the API from HERE.


<api xmlns="http://ws.apache.org/ns/synapse" name="IterateAggregateAPI" context="/iterate">
   <resource methods="POST" uri-template="/*" inSequence="sendSequence" outSequence="iterateOUT"/>
</api>

5) Using the soap UI or any other rest client you can invoke the API and observe how the response payload is iterated and processed to display the aggregated and tranformed response.


I hope that following this tutorial you were able to understand how large payloads could be split using common elements to process using the script mediator.

References:

[1] https://docs.wso2.com/display/ESB490/Creating+Custom+Mediators
[2] https://docs.wso2.com/display/ESB490/Iterate+Mediator
[3] https://docs.wso2.com/display/ESB490/Aggregate+Mediator
[4] https://docs.wso2.com/display/ESB490/Creating+APIs
[5] https://docs.wso2.com/display/ESB490/Mediation+Sequences


Comments

Popular posts from this blog

WSO2 Cloud

Introduction WSO2 Cloud consists of three commercial cloud offerings namely the API cloud, Integration Cloud and the Identity Cloud. WSO2 Integration Cloud WSO2 Integration Cloud is a public hosted version of the powerful, 100% open-source WSO2 Enterprise Integrator. It allows you to host your cloud-to-cloud, cloud-to-on-premise integrations and API backends on a scalable cloud platform WSO2 API Cloud WSO2 API Cloud is an enterprise-ready, self-service platform that enables you to expose business capabilities as managed APIs and leverage off-the-shelf potential for advertising and selling business APIs. The API Cloud supports all aspects of an API’s lifecycle, from its creation to publication and retirement. The API Cloud is based on WSO2 API Manager , WSO2’s complete solution for designing and publishing APIs and managing a developer community. [2] WSO2 Identity Cloud WSO2 Identity Cloud leverages the functionality of WSO2 Identity Server’s identity and ac...

Exposing a SOAP service as a REST API

In this post i will be explaining how we can transform a SOAP based backend to receive requests in a restful manner through the WSO2 API Cloud. Steps. First log into the WSO2 Cloud and navigate to the API Cloud. In the API cloud select the option to add a new API. We will be creating an API to demonstrate an invocation to the backend soap service ws.cdyne.com/phoneverify/phoneverify.asmx?wsdl Give a name, context and version to the API and add a resource name with a GET Method. The resource name can be anything which you like since we will invoke the actual service usiing a custom sequence. Mention the URI template as indicated in the below screenshot. Next go to the implement tab. And select the endpoint type as HTTP/SOAP Endpoint and specify the endpoint as http://ws.cdyne.com/phoneverify/phoneverify.asmx. There is an important step we need to do here. We need to set the SOAP version for this request. In order to do that we need to select the advanced option ...

Adding error sequence to a custom sequence

In my previous posi i explained how to call an endpoint using the call mediator. Usually when errors are thrown from the call mediator we can handle them using a custom sequence. When call mediators fails the execution stops hence handling errors is very essential. For this post I will be modifying the same sequence which i created in the blog post [1] The change which we need to do is we need to modify the sequence definition to map with the error sequence as shown below. You can find the modified sequence here . <sequence xmlns="http://ws.apache.org/ns/synapse" name="countrySequenceWithError" onError="countryErrorSequence"> <property name="uri.var.countryCode" expression="$url:countryCode"/> <property name="REST_URL_POSTFIX" scope="axis2" action="remove"/>     <call blocking="true">         <endpoint>             <http method="get" uri-temp...