Month: July 2016

WSO2 Data Mapper Common Type Operators

WSO2 Data Mapper Common Type Operators

INTRODUCTION

WSO2 data mapper is a data mapping solution that convert and transform one format of data to a different format. It provides a WSO2 Developer Studio based tool to create the graphical mapping configuration and generate the files needed to execute the mapping configuration by the WSO2 Data mapper engine.

If you first came to here look for the two previous posts which will give you the basic understanding of WSO2 Data Mapper and how to integrate it with WSO2 ESB.

  1. How to Use WSO2 Data Mapping Mediator in ESB (Updated for-ESB-5.0.0-Beta2)
  2. Understanding WSO2 Data Mapper 5.0.0

OVERVIEW

This post is one of many to introduce data mapper operations available and how to use it in mapping configuration. There are mainly six categories in the operator pallet as shown in the following diagram.

diagramEditor

They are,

  • Links
  • Common
  • Arithmetic
  • Conditional
  • Boolean
  • Type Conversion
  • String

In this post we will discuss about Common Operators, given in data mapper.

Constant Constant- define string, number or boolean constant values.

This operator is basically to define values we need to use in mapping. When you drag and drop the constant operator from the pallet to editor area you will get it as following diagram.

constantdefault

As you can see it only has a output connector. And that value can be a Number, String or Boolean. To configure it right click on constant operator and click the configure constant operator menu action. It will pop up a dialog box to configure the operator as follows.

configuremenuconstant.png

So you can define the value you need in here.

CustomFunction Custom Function – define custom function to use in mapping

Custom function is for more advanced users who need more functionality other than offered in the data mapper. And if you know Java Script well you could do many things by using custom functions with global variable operator. But here I will only describe the basic features and how to use this operator to define our own functionality.

When you drag and drop custom function operator for editor you will get the following default operator and the configuration.

defaultcustomfunctionconfiguratoncustomfunction

You should first give a name to your function in the Function Name text box. You can not use same name to two functions in the same mapping.

Then define the number of parameters/inputs you need in the Number of Inputs text box.

Then you need to define the parameters inside the function definition () according to the number of inputs you specified above. Use those parameters or any Global Variable value inside the custom function to do the functionality and return the value.

Properties Properties – use product specific runtime variables

This operator is discussed in great detail in the following post by Eranda.

GlobalVariable Global Variable – instantiate global variable to access from any where

Global variable operator is given to instantiate a variable and use in any where in the mapping. When you drag and drop the operator to editor you will see following default operator and the configuration.

GVoperatorGVconfiguration

This is basically a JavaScript variable defined in the outer scope so that any one can see and access it. You could use this variable in any custom function you define in the mapping. In the configuration you can change the name and default value of it.

Compare Compare – compare two inputs in the mapping

This operator can be used to compare any two values in the mapping. There are 8 different compare operations you could do. They are,

  • equal value (two inputs are equal or not)
  • equal value type (two inputs are equal in both value and type)
  • not equal value (two inputs are not equal or not)
  • not equal value type (two inputs are not equal in both value and type)
  • greater than
  • greater than or equal
  • lower than
  • lower than or equal

compareOperatorcompareconfiguration

 

 

Advertisements
Understanding WSO2 Data Mapper 5.0.0

Understanding WSO2 Data Mapper 5.0.0

INTRODUCTION

WSO2 data mapper is a data mapping solution that convert and transform one format of data to a different format. It provides a WSO2 Developer Studio based tool to create the graphical mapping configuration and generate the files needed to execute the mapping configuration by the WSO2 Data mapper engine.

This is the second of WSO2 Data Mapper posts which will explain about WSO2 Data Mapper from the user perspective. First post was to describe the end to end work flow on how to create, deploy and test data mapper configuration.(How to Use WSO2 Data Mapping Mediator in ESB)

COMPONENTS OF WSO2 DATA MAPPER

Before we going into look the operations lets look how things happen under the hood. WSO2 Data Mapper comes as two components ideally. They are Data Mapper Engine and Data Mapper Tooling component.

Data Mapper Tooling component

Data Mapper Tooling component is the interface to user to create configuration files needed for engine to execute the mapping. There are three configuration files needed by the Data Mapper engine. They are,

  1. Input schema file
  2. Output schema file
  3. Mapping configuration file

Input schema and output schema defines the input/output format of input message and required message. It is basically  a custom defined json schema. It will be generated by the Data Mapper tool for you when loading the input and output files.

inputType

As above image shows user can load input/output message format in several different ways. They are,

  • XML – Sample XML file
  • JSON- Sample JSON file
  • CSV  – Sample CSV file with column names as the first record
  • XSD  – XSD file which is actually schema for XML files
  • JSONSCHEMA – WSO2 data mapper json schema
  • CONNECTOR

I guess only CONNECTOR option is the confusing part for you. It is to use data mapper with connectors straightly. WSO2 connectors will contain json schemas for each operations which will define the message formats both it will respond and expect. So when user integrate connectors in a project this connector option will search through the workspace and find the available connectors. Then user can select the respective connector and the operation so that related json schema will be loaded for data mapper by the tool.

Mapping configuration file is simply a Java Script file generated looking at the diagram user will draw in the data mapper diagram editor by connecting input elements to output elements. Every operation user defines in diagram will convert to Java Script operation.

So these three files will be generated by the Data Mapper Tool and saved in a Registry Resource project to deploy in a wso2 server.

regresource.png

“.datamapper” and “.datamapper_diagram” files contain meta data related to Data Mapper Diagram. They will be ignored when trying to deploy in a server to use by data mapper engine. Only two schema files and .dmc (data mapper configuration) file will be deploy.

Need an Intermediate Component to Use With WSO2 Products

WSO2 Data Mapper is an independent component. It is not depending on any other WSO2 product. But other products can use data mapper to achieve/offer data mapping capability. For that we need a intermediate component. Data Mapper Mediator is this intermediate component which  will give the data mapping capability into WSO2 ESB.

DMMediator.png

Data Mapper Mediator will find the configuration files from the registry and configure data mapper engine with the input message type(XML/JSON/CSV) and output message type(XML/JSON/CSV) . Then it will take request message from ESB message flow and use configured data mapper engine to execute the transformation and output message will be added to ESB message flow.

Can Use Product Specific Runtime Variables

Data Mapper engine also given a way to use run time product specific variables in the mapping. The way it works is, intermediate component should construct a map object containing run time product specific variables and send it to  data mapper engine. So when the mapping happens in the data mapper engine, these variables will be available. For Eg: Data Mapper Mediator provides esb axis2/transport/synapse/axis2client/operation/.. properties like this. In the Data Mapper diagram user can use the Property operator and define the scope and the property name and use it in the mapping. Data Mapper Mediator will identify the required properties to execute the mapping and populate a map with the required properties and will send it to engine. We will look it with more detail when we discuss about Property Operator.

Data Mapper Engine

Data Mapper engine need basically following information to be configured,

  • Input Message Type
  • Output Message Type
  • Input schema
  • Output schema
  • Mapping configuration

At the runtime it will get the input message and runtime variable map object and output the transformed message. To execute the mapping configuration data mapper engine use java scripting api. So if your runtime is JAVA 7 rhino JS engine will be used and if your runtime is JAVA 8 nashorn JS engine will be used.

There are several limitations in the rhino engine that will directly affect data mapper engine when JAVA 7 is used. There are several functions that will not support in rhino,

for eg: String object functions like startsWith() endsWith()

So user may need to aware of that when using custom functions and operators, rhino may have limitations executing those.

OPERATIONS IN WSO2 DATA MAPPER

Let’s look into the operations we could use to do the mapping. Following diagram shows the WSO2 Data Mapping Diagram Editor.

diagramEditor.png

As you can see in the left side we have the operations pallet. These operations can be drag and drop to the Editor area.

There are six categories listed down

  • Links
  • Common
  • Arithmetic
  • Conditional
  • Boolean
  • Type Conversion
  • String

LINKS

DataMapperLink.gif Data Mapping Link- map elements with other operators and elements.

COMMON

Constant Constant- define string, number or boolean constant values.

CustomFunction Custom Function – define custom function to use in mapping

Properties Properties – use product specific runtime variables

GlobalVariable Global Variable – instantiate global variable to access from any where

Compare Compare – compare two inputs in the mapping

ARITHMETIC

Add Add – add two numbers

Subtract Subtract – subtract two or more numbers

Multiply Multiply – multiply two or more numbers

Divide Divide –  divide two numbers

Celi Ceiling – get the ceiling value of a number (closest larger integer value)

Floor Floor – get the floor value of a number (closest lower integer value)

Round Round – get the nearest integer value

SetPrecision Set Precision – Format a number into a specified length

AbsoluteValue Absolute Value – get the absolute value of a rational number

Min Min – get the minimum number from given inputs

Max Max – get the maximum number from given inputs

CONDITIONAL

IfElse IfElse – use a condition and select one input from two

BOOLEAN

AND AND – boolean AND operation on inputs

OR OR – boolean OR operation on inputs

NOT NOT – boolean NOT operation

TYPE CONVERSION

StringToNumber StringToNumber – convert string value to number (“0” -> 0)

StringToBoolean StringToBoolean – convert string value to boolean(“true” -> true)

ToString ToString – convert number or boolean to string

STRING

Concat Concat – concat two or more strings

Split Split – split a string by a matching string value

UpperCase Uppercase – convert a string to uppercase letters

LowerCase Lowercase – convert a string to lowercase letters

StringLength String Length – get the length of the string

StartsWith StratsWith – check whether string starts with a specific value (JAVA7)

EndsWith EndsWith – check whether string ends with a specific value (JAVA7)

Substring Substring – extract a part of the string value

Trim Trim – remove white spaces from beginning and end of a string

Replace Replace – replace the first occurrence of a target string with other

Match Match – check whether the input match with a (JS)regular expression

We will discuss more on some of the operations in the next post.

 

How to Use WSO2 Data Mapping Mediator in ESB (Updated for-ESB-5.0.0-Beta2)

How to Use WSO2 Data Mapping Mediator in ESB (Updated for-ESB-5.0.0-Beta2)

INTRODUCTION

WSO2 data mapper is a data mapping solution that convert and transform one format of data to a different format. It provides a WSO2 Developer Studio based tool to create the graphical mapping configuration and generate the files needed to execute the mapping configuration by the WSO2 Data mapper engine as shown in the below diagrams. As the this is really useful feature in message mediation, WSO2 ESB comes with a data mapper mediator which can be integrate into a mediation sequence since WSO2 ESB 5.0.0.

PREREQUISITES

First we need to have the data mapper mediator supported ESB distribution. WSO2 ESB 5.0.0 is the distribution packed with data mapper mediator. You can download beta2 pack from here.

And also we need to have a data mapping editor supported Developer Studio ESB Tool. You can follow this article to install the WSO2 Developer Studio ESB Tool 5.0.0.

For this post I will use 5.0.0-beta2 packs of wso2 esb runtime and tooling, which are the latest packs. So the operations mentioned here will be only available with 5.0.0 beta2 or onwards.

We will use the Postman as out client to send the requests to esb and receive responses.

OVERVIEW

Since our main focus is to check the functionality of the Data Mapper we will use a very simple configuration in the esb with a data mapping mediator and a respond mediator. We will check following message conversions,

  • XML->XML/JSON/CSV
  • JSON->XML/JSON/CSV
  • CSV->XML/JSON/CSV

To understand the data mapper further and to examine the operators provided in the data mapper, which we can use to map elements from input to output look in to Understanding WSO2 Data Mapper post.

CREATE ESB CONFIGURATION PROJECT

First this we would need to do is create esb configuration to contain the data mapper mediator. ESB Tooling now provides a new option to create a ESB Solution Project so that you can define the all different projects you need for the project in the wizard. Right click on the Project Explorer area and go to new-> ESB Solution Project

ESBSolutionProject

You will get the following wizard page and give project name and unpick the Connector Exporter Project since we are not going to need connector in our configuration.

ESBSolutionProjectWizard

And you will get the following project files created in your workspace.

ProjectExplorer

Now what we will do is create a new REST Api project in ESB and configure it to listen for POST requests on url “/convertMenu”.

RESTAPIfoodMenuAPI

Then create the API resource with data mapper mediator and listening to POST request.

createAPIAndDataMapperapiPropertiesTable

To configure data mapping mediator double click on the mediator and you will get a dialog box asking to create a registry resource project as below.

createMappingFiles

This name would be the prefix for the configuration files we will deploy in esb server related to data mapper. Since we created a ESB Solution Project we were directly pointed that project in here otherwise go to Create new project link and create a new Registry Resource project.

Then click OK after giving the prefix and pointing the Registry resource project to save it.  Following diagram editor will appeared in new perspective named “WSO2 Data Mapper Graphical”. You can switch this perspective as you wish by selecting in top tool bar tags or Window->perspective->Open Perspective->Other

dataMappingEditor

In the left hand side you can see the operation pallet and in the middle input output boxes. We need to provide the input and output message formats to begin the mapping. So right click on the top title bar of the input box and select command Load Input.

loadInputinputType

As you can see in the above image, there are several options for you to load the input and output message formats to data mapper.

You can select the resource type as XML, JSON, CSV ,XSD ,JSONSCHEMA or CONNECTOR.

If you selects XML, you can load a sample xml message you expect and WSO2 Data Mapper Editor will generate the JSON schema to represent the xml according to the WSO2 Data Mapper Schema specification.

Same as for XML, if you select  JSON/CSV as resource type you can load a sample JSON/CSV message. For CSV you need to provide the column names as the first record.

If you have a xsd schema which defines you xml message format, you can select the resource type as XSD and load the xsd file.

And also if you have the JSON schema for your message according to the WSO2 Data Mapper schema specification you can load it by selecting resource type as JSON schema.

Connector type will come if you are going to map message which is an output of a connector, in the input box you can select connector type and it will list down available connectors and then select the operation in-front of data mapper. This is not the time to explain the connector integration with data mapper. We will discuss it later in a different post. 🙂

For this moment we will load from a sample XML file.

Input xml sample for food menu.

<?xml version="1.0" encoding="UTF-8"?>
<breakfast_menu>
<food>
<name>Belgian Waffles</name>
<price>$5.95</price>
<description>Two of our famous Belgian Waffles with plenty of real maple syrup</description>
<calories>650</calories>
<orgin>Belgian</orgin>
<veg>true</veg>
</food>
<food>
<name>Strawberry Belgian Waffles</name>
<price>$7.95</price>
<description>Light Belgian waffles covered with strawberries and whipped cream</description>
<calories>900</calories>
<orgin>Belgian</orgin>
<veg>true</veg>
</food>
<food>
<name>Berry-Berry Belgian Waffles</name>
<price>$8.95</price>
<description>Light Belgian waffles covered with an assortment of fresh berries and whipped cream</description>
<calories>900</calories>
<orgin>Belgian</orgin>
<veg>true</veg>
</food>
<food>
<name>French Toast</name>
<price>$4.50</price>
<description>Thick slices made from our homemade sourdough bread</description>
<calories>600</calories>
<orgin>French</orgin>
<veg>true</veg>
</food>
<food>
<name>Homestyle Breakfast</name>
<price>$6.95</price>
<description>Two eggs, bacon or sausage, toast, and our ever-popular hash browns</description>
<calories>950</calories>
<orgin>French</orgin>
<veg>false</veg>
</food>
</breakfast_menu>

Output xml sample for food menu.

<?xml version="1.0" encoding="UTF-8"?>
<menu>
<item>
<name>Belgian Waffles</name>
<price>$5.95</price>
<calories>650</calories>
<orgin>Belgian</orgin>
<veg>true</veg>
<description>Two of our famous Belgian Waffles with plenty of real maple syrup</description>
</item>
<item>
<name>Strawberry Belgian Waffles</name>
<price>$7.95</price>
<calories>900</calories>
<orgin>Belgian</orgin>
<veg>true</veg>
<description>Light Belgian waffles covered with strawberries and whipped cream</description>
</item>
<item>
<name>Berry-Berry Belgian Waffles</name>
<price>$8.95</price>
<calories>900</calories>
<orgin>Belgian</orgin>
<veg>true</veg>
<description>Light Belgian waffles covered with an assortment of fresh berries and whipped cream</description>
</item>
<item>
<name>French Toast</name>
<price>$4.50</price>
<calories>600</calories>
<orgin>French</orgin>
<veg>true</veg>
<description>Thick slices made from our homemade sourdough bread</description>
</item>
<item>
<name>Homestyle Breakfast</name>
<price>$6.95</price>
<calories>950</calories>
<orgin>French</orgin>
<veg>false</veg>
<description>Two eggs, bacon or sausage, toast, and our ever-popular hash browns</description>
</item>
</menu>

Load above files as input and outputs and look the input output boxes with sample message. Is the element types are correctly identified or not. (Arrays, Objects and Primitive values). Following signs will help you to identify it correctly.

  • {}  – represent object elements
  • []   – represent array elements
  • <> – primitive field values
  • A   – xml attribute values

Now use operators and do the mapping as you desire. You can only connect primitive data values such as Strings, numbers, boolean and etc. Array and object values can not be mapped.

mapping

What I have done here is name is mapped via uppercase operator and calories undergoes and mathematical calculation to the output.

output calories =Round( (calories*1.13) + 6.75)

Now save the all configurations. We can now deploy the created REST API in the ESB server and test our mapping.

To deploy our configuration we need to include these in a CAPP project. When we created the ESB Solution Project we created the CAPP project too. So open the CAPP project and double click on the pom file. You will get the following page to select project files into CAPP.

CAPPProject.png

Here you do not see the registry resource files. We need to refresh the page to see it. Use refresh button in the top right conner to load newly added registry files.

refreshBUtton

Then you will see the all artifacts in the workspace. Select REST API file and the three registry resource files containing mapping configuration , input schema and output schema.

cappProjectselection

 

  • Mapping Configuration : This file contain the script file used to execute the mapping
  • Input Schema: JSON schema which represent input message format
  • Output Schema: JSON schema which represent the output message format

And also see the data mapper mediator configuration in the REST API.

DMMconfig.png

 

  • Input Type : Expected input message type (xml/json/csv)
  • Output Type: Target Output message type (xml/json/csv)

If your mapping failed at the runtime the first thing you need check whether the mediator configuration. Is the input type and output type are set correctly.

Now start WSO2 ESB server and connect it to ESB tool as a remote server or start ESB server from inside the ESB tool. And add out CAPP to the server and deploy it.

addwso2servercappAdding

Then goto management console from https://localhost:9443/carbon

See the deployed API and get the url it is listening.

APIDeployed.png

Start the POST Man and send the request for the above url.

postMan.png

You will get the response with accordingly to post man when you send the message.

To see the XML -> JSON conversion  just go to REST API  configuration and data mapper properties. Select Output Type as JSON . Deploy the configuration and send the request. You will get the JSON message.

Do the same for see the XML->CSV.

But this is not guaranteed every occasion. If t you have defined complex XML output message with name spaces and attributes JSON message or CSV will not built as you will imagine. In this case output message format is fully compatible to represent as JSON and CSV.

Now you have the basic end to end knowledge to develop a ESB configuration with a data mapper mediator. So you can try this and report the bugs and suggest improvements from the following link as JIRAs.