Tag: data mapper

Understanding WSO2 Data Mapper 5.0.0

Understanding WSO2 Data Mapper 5.0.0

INTRODUCTION

WSO2 data mapper is a data mapping solution that convert and transform one format of data to a different format. It provides a WSO2 Developer Studio based tool to create the graphical mapping configuration and generate the files needed to execute the mapping configuration by the WSO2 Data mapper engine.

This is the second of WSO2 Data Mapper posts which will explain about WSO2 Data Mapper from the user perspective. First post was to describe the end to end work flow on how to create, deploy and test data mapper configuration.(How to Use WSO2 Data Mapping Mediator in ESB)

COMPONENTS OF WSO2 DATA MAPPER

Before we going into look the operations lets look how things happen under the hood. WSO2 Data Mapper comes as two components ideally. They are Data Mapper Engine and Data Mapper Tooling component.

Data Mapper Tooling component

Data Mapper Tooling component is the interface to user to create configuration files needed for engine to execute the mapping. There are three configuration files needed by the Data Mapper engine. They are,

  1. Input schema file
  2. Output schema file
  3. Mapping configuration file

Input schema and output schema defines the input/output format of input message and required message. It is basically  a custom defined json schema. It will be generated by the Data Mapper tool for you when loading the input and output files.

inputType

As above image shows user can load input/output message format in several different ways. They are,

  • XML – Sample XML file
  • JSON- Sample JSON file
  • CSV  – Sample CSV file with column names as the first record
  • XSD  – XSD file which is actually schema for XML files
  • JSONSCHEMA – WSO2 data mapper json schema
  • CONNECTOR

I guess only CONNECTOR option is the confusing part for you. It is to use data mapper with connectors straightly. WSO2 connectors will contain json schemas for each operations which will define the message formats both it will respond and expect. So when user integrate connectors in a project this connector option will search through the workspace and find the available connectors. Then user can select the respective connector and the operation so that related json schema will be loaded for data mapper by the tool.

Mapping configuration file is simply a Java Script file generated looking at the diagram user will draw in the data mapper diagram editor by connecting input elements to output elements. Every operation user defines in diagram will convert to Java Script operation.

So these three files will be generated by the Data Mapper Tool and saved in a Registry Resource project to deploy in a wso2 server.

regresource.png

“.datamapper” and “.datamapper_diagram” files contain meta data related to Data Mapper Diagram. They will be ignored when trying to deploy in a server to use by data mapper engine. Only two schema files and .dmc (data mapper configuration) file will be deploy.

Need an Intermediate Component to Use With WSO2 Products

WSO2 Data Mapper is an independent component. It is not depending on any other WSO2 product. But other products can use data mapper to achieve/offer data mapping capability. For that we need a intermediate component. Data Mapper Mediator is this intermediate component which  will give the data mapping capability into WSO2 ESB.

DMMediator.png

Data Mapper Mediator will find the configuration files from the registry and configure data mapper engine with the input message type(XML/JSON/CSV) and output message type(XML/JSON/CSV) . Then it will take request message from ESB message flow and use configured data mapper engine to execute the transformation and output message will be added to ESB message flow.

Can Use Product Specific Runtime Variables

Data Mapper engine also given a way to use run time product specific variables in the mapping. The way it works is, intermediate component should construct a map object containing run time product specific variables and send it to  data mapper engine. So when the mapping happens in the data mapper engine, these variables will be available. For Eg: Data Mapper Mediator provides esb axis2/transport/synapse/axis2client/operation/.. properties like this. In the Data Mapper diagram user can use the Property operator and define the scope and the property name and use it in the mapping. Data Mapper Mediator will identify the required properties to execute the mapping and populate a map with the required properties and will send it to engine. We will look it with more detail when we discuss about Property Operator.

Data Mapper Engine

Data Mapper engine need basically following information to be configured,

  • Input Message Type
  • Output Message Type
  • Input schema
  • Output schema
  • Mapping configuration

At the runtime it will get the input message and runtime variable map object and output the transformed message. To execute the mapping configuration data mapper engine use java scripting api. So if your runtime is JAVA 7 rhino JS engine will be used and if your runtime is JAVA 8 nashorn JS engine will be used.

There are several limitations in the rhino engine that will directly affect data mapper engine when JAVA 7 is used. There are several functions that will not support in rhino,

for eg: String object functions like startsWith() endsWith()

So user may need to aware of that when using custom functions and operators, rhino may have limitations executing those.

OPERATIONS IN WSO2 DATA MAPPER

Let’s look into the operations we could use to do the mapping. Following diagram shows the WSO2 Data Mapping Diagram Editor.

diagramEditor.png

As you can see in the left side we have the operations pallet. These operations can be drag and drop to the Editor area.

There are six categories listed down

  • Links
  • Common
  • Arithmetic
  • Conditional
  • Boolean
  • Type Conversion
  • String

LINKS

DataMapperLink.gif Data Mapping Link- map elements with other operators and elements.

COMMON

Constant Constant- define string, number or boolean constant values.

CustomFunction Custom Function – define custom function to use in mapping

Properties Properties – use product specific runtime variables

GlobalVariable Global Variable – instantiate global variable to access from any where

Compare Compare – compare two inputs in the mapping

ARITHMETIC

Add Add – add two numbers

Subtract Subtract – subtract two or more numbers

Multiply Multiply – multiply two or more numbers

Divide Divide –  divide two numbers

Celi Ceiling – get the ceiling value of a number (closest larger integer value)

Floor Floor – get the floor value of a number (closest lower integer value)

Round Round – get the nearest integer value

SetPrecision Set Precision – Format a number into a specified length

AbsoluteValue Absolute Value – get the absolute value of a rational number

Min Min – get the minimum number from given inputs

Max Max – get the maximum number from given inputs

CONDITIONAL

IfElse IfElse – use a condition and select one input from two

BOOLEAN

AND AND – boolean AND operation on inputs

OR OR – boolean OR operation on inputs

NOT NOT – boolean NOT operation

TYPE CONVERSION

StringToNumber StringToNumber – convert string value to number (“0” -> 0)

StringToBoolean StringToBoolean – convert string value to boolean(“true” -> true)

ToString ToString – convert number or boolean to string

STRING

Concat Concat – concat two or more strings

Split Split – split a string by a matching string value

UpperCase Uppercase – convert a string to uppercase letters

LowerCase Lowercase – convert a string to lowercase letters

StringLength String Length – get the length of the string

StartsWith StratsWith – check whether string starts with a specific value (JAVA7)

EndsWith EndsWith – check whether string ends with a specific value (JAVA7)

Substring Substring – extract a part of the string value

Trim Trim – remove white spaces from beginning and end of a string

Replace Replace – replace the first occurrence of a target string with other

Match Match – check whether the input match with a (JS)regular expression

We will discuss more on some of the operations in the next post.

 

Advertisements
How to Use WSO2 Data Mapping Mediator in ESB

How to Use WSO2 Data Mapping Mediator in ESB

Please find the updated post from here

https://nuwanpallewela.wordpress.com/2016/07/16/how-to-use-wso2-data-mapping-mediator-in-esb-updated/

INTRODUCTION

WSO2 data mapper is a data mapping solution that convert and transform one format of data to a different format. It provides a WSO2 Developer Studio based tool to create the graphical mapping configuration and generate the files needed to execute the mapping configuration by the WSO2 Data mapper engine as shown in the below diagrams. As the this is really useful feature in message mediation, WSO2 ESB comes with a data mapper mediator which can be integrate into a mediation sequence since WSO2 ESB 5.0.0.

PREREQUISITES

First we need to have the data mapper mediator supported ESB distribution. WSO2 ESB 5.0.0 is the distribution packed with data mapper mediator. You can download alpha pack from here.

And also we need to have a data mapping editor supported Developer Studio ESB Tool. You can follow this article to install the WSO2 Developer Studio ESB Tool 5.0.0.

OVERVIEW

This post will focus on how to create a configuration with Data Mapper mediator. So I will chose a very simple ESB configuration with only a data mapper mediator and respond mediator to check the converted message.

 

employeeToEngineerUserCase.png

Our Employee message will be a xml message and we will send a JSON response to the client. We will use the Postman as out client to send the requests.

Creating Data Mapping Configuration

Lets create a ESB Configuration project named “DataMapperEmployeeToEngineerConfig” from the WSO2 Developer Studio ESB Tool and create a Rest API as follows.

RestAPiWizard.png

Then we can add the Data Mapper Mediator in to the API resource and a Respond mediator to send the converted message back to client.

apiconfig.png

Then double click on the data mapper mediator to configure the mapping. It will open a dialog box asking to create a Registry Resource project. This is to save and deploy the required configuration and schema files in the ESB Server to read at the runtime when message conversion happens. So name the configuration file as “EmployeeToEngineerMapping” and registry resource project as “EmployeeToEngineerRegistryResource”.registryresourcefile.pngIt will open the Data Mapper Diagram Editor where we should do the mapping.

diagrameditor.png

But first we need to complete the API configuration. So go back to ESB Diagram editor with the API Resource and click on the Data Mapper mediator. You will see the configurations of the Data Mapper mediator on the property table.

data mapper configuration.png

As you can see there are five configurations listed. They are

  • Input Type : Expected input message type (xml/json/csv)
  • Output Type: Target Output message type (xml/json/csv)
  • Configuration : This file contain the script file used to execute the mapping
  • Input Schema: JSON schema which represent input message format
  • Output Schema: JSON schema which represent the output message format

As we are going to convert xml message to json message change the output type to json.

propertytable.png

And then click on the API resource and  go to the property table. And change the configurations to following.

  • Url Style : URL_MAPPING
  • URL-Mapping : /employeetoengineer
  • Methods : POST

apiconfigchanges.png

Save the API configuration. Now lets move to data mapper editor to create the mapping from employee message to engineer.

First we need to load the input and output message formats into editor. For the right click on the input box title bar and select load input schema command.

loadschemafromfile.png

Then following dialog box appears and you will have multiple ways to load the input message format.

loadinputclose.png

You can select the resource type as XML, JSON, XSD or JSONSCHEMA.

If you selects XML, you can load a sample xml message you expect and WSO2 Data Mapper Editor will generate the JSON schema to represent the xml accoring to the WSO2 Data Mapper Schema specification.

Same as for XML, if you select  JSON as resource type you can load a sample JSON message.

If you have a xsd schema which defines you xml message format, you can select the resource type as XSD and load the xsd file.

And also if you have the JSON schema for your message according to the WSO2 Data Mapper schema specification you can load it by selecting resource type as JSON schema.

Lets use a sample xml message to load input format to the data mapper editor. Use following xml as the sample input.

<?xml version='1.0' encoding='utf-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body><ns1:employees xmlns:ns1="http://wso2.employee.info" xmlns:ns2="http://wso2.employee.address">
<ns1:employee>
<ns1:firstname>Mike</ns1:firstname>
<ns1:lastname>Jhonson</ns1:lastname>
<ns2:addresses>
<ns2:address location="home" >
<ns2:city postalcode="30000">KS</ns2:city>
<ns2:road>main rd</ns2:road>
</ns2:address>
<ns2:address location="office" >
<ns2:city postalcode="10003">NY</ns2:city>
<ns2:road>cross street</ns2:road>
</ns2:address>
</ns2:addresses>
</ns1:employee>
<ns1:employee>
<ns1:firstname>Patric</ns1:firstname>
<ns1:lastname>Jane</ns1:lastname>
<ns2:addresses>
<ns2:address location="home" >
<ns2:city postalcode="60000" >Melborne</ns2:city>
<ns2:road>park street</ns2:road>
</ns2:address>
<ns2:address location="office" >
<ns2:city postalcode="10003">NY</ns2:city>
<ns2:road>cross street</ns2:road>
</ns2:address>
</ns2:addresses>
</ns1:employee>
<ns1:employee>
<ns1:firstname>Thelesa</ns1:firstname>
<ns1:lastname>Lisbon</ns1:lastname>
<ns2:addresses>
<ns2:address location="home" >
<ns2:city postalcode="60000">Madrid</ns2:city>
<ns2:road>Palace street</ns2:road>
</ns2:address>
<ns2:address location="office" >
<ns2:city postalcode="10003">NY</ns2:city>
<ns2:road>cross street</ns2:road>
</ns2:address>
</ns2:addresses>
</ns1:employee>
</ns1:employees>
</soapenv:Body>
</soapenv:Envelope>

And use following JSON message as the sample output and load output message format to data mapper editor.

{
"engineers":{
"engineerList":[
{
"fullname":"Mike Jhonson",
"addresses":{
"address":[
{
"location":"home",
"city":{
"postalcode":30000,
"text":"AA"
},
"road":"BB"
},
{
"location":"office",
"city":{
"postalcode":10003,
"text":"BB"
},
"road":"CC"
}
]
}
},
{
"fullname":"Mike Jhonson",
"addresses":{
"address":[
{
"location":"home",
"city":{
"postalcode":30000,
"text":"AA"
},
"road":"BB"
},
{
"location":"office",
"city":{
"postalcode":10003,
"text":"BB"
},
"road":"CC"
}
]
}
}
]
}
}

Now the editor will have the input and output message formats as bellow.

operations.png

Now we can specify how the output message should build by using the input message. We have several operations given in the operations pallet in the left side.

Click on the operator and it will come to the editor. Then construct the mapping as the following diagram.

mappingdiagram.png

You can only connect primitive data values such as Strings, numbers, boolean and etc. Array and object values can not be mapped.

  • {}  – represent object elements
  • []   – represent array elements
  • <> – primitive field values
  • A   – xml attribute values

After loading the message formats you have to observe whether the formats are correctly identified by the data mapper diagram editor.

Now save the all configurations. We can now deploy the created REST API in the ESB server and test our mapping.

Deploy the REST API in ESB Server

We can create a CAPP including the WSO2 synapse configurations and the registry resource files as below.

DMCAPP.png

Start ESB server and add it to Developer Studio ESB Tool servers and deploy.

deployment.png

Then go to API’s list from the ESB management console and confirm the REST API invocation URL.

deployedAPIs.png

Invoke Created REST API

Open Postman and create the client message as follows.

postmanxmlmessage.png

Send the message from Postman and you will receive the respond message but in the xml format. This is since though we converted the message to type json in the axis2 level we need to change the property “messageType” to “application/json” too.

messagetypeproperty.png

So add a property mediator and configure the following parameters.

  • Property Name : messageType
  • Value : application/json
  • Property Scope : axis2

Then redeploy the CAPP and send the employee message again from the Postman. Then you will receive the expected json message as follows.

jsonresponse.png

{
"engineerList": [
{
"addresses": {
"address": [
{
"city": {
"postalcode": 30000,
"text": "KS"
},
"location": "home",
"road": "main rd"
},
{
"city": {
"postalcode": 10003,
"text": "NY"
},
"location": "office",
"road": "cross street"
}
]
},
"fullname": "Mike Jhonson"
},
{
"addresses": {
"address": [
{
"city": {
"postalcode": 60000,
"text": "MELBORNE"
},
"location": "home",
"road": "park street"
},
{
"city": {
"postalcode": 10003,
"text": "NY"
},
"location": "office",
"road": "cross street"
}
]
},
"fullname": "Patric Jane"
},
{
"addresses": {
"address": [
{
"city": {
"postalcode": 60000,
"text": "MADRID"
},
"location": "home",
"road": "palace street"
},
{
"city": {
"postalcode": 10003,
"text": "NY"
},
"location": "office",
"road": "cross street"
}
]
},
"fullname": "Thelesa Lisbon"
}
]
}

Now you have the basic end to end knowledge to develop a ESB configuration with a data mapper mediator. So you can try this and report the bugs and suggest improvements from the following link as JIRAs.

I will discuss more advanced discussion about how things work inside and mapping configuration is generated from the diagram which will help to understand the behavior of the data mapper and the limitations.