Understanding WSO2 Data Mapper 5.0.0

Understanding WSO2 Data Mapper 5.0.0

INTRODUCTION

WSO2 data mapper is a data mapping solution that convert and transform one format of data to a different format. It provides a WSO2 Developer Studio based tool to create the graphical mapping configuration and generate the files needed to execute the mapping configuration by the WSO2 Data mapper engine.

This is the second of WSO2 Data Mapper posts which will explain about WSO2 Data Mapper from the user perspective. First post was to describe the end to end work flow on how to create, deploy and test data mapper configuration.(How to Use WSO2 Data Mapping Mediator in ESB)

COMPONENTS OF WSO2 DATA MAPPER

Before we going into look the operations lets look how things happen under the hood. WSO2 Data Mapper comes as two components ideally. They are Data Mapper Engine and Data Mapper Tooling component.

Data Mapper Tooling component

Data Mapper Tooling component is the interface to user to create configuration files needed for engine to execute the mapping. There are three configuration files needed by the Data Mapper engine. They are,

  1. Input schema file
  2. Output schema file
  3. Mapping configuration file

Input schema and output schema defines the input/output format of input message and required message. It is basically  a custom defined json schema. It will be generated by the Data Mapper tool for you when loading the input and output files.

inputType

As above image shows user can load input/output message format in several different ways. They are,

  • XML – Sample XML file
  • JSON- Sample JSON file
  • CSV  – Sample CSV file with column names as the first record
  • XSD  – XSD file which is actually schema for XML files
  • JSONSCHEMA – WSO2 data mapper json schema
  • CONNECTOR

I guess only CONNECTOR option is the confusing part for you. It is to use data mapper with connectors straightly. WSO2 connectors will contain json schemas for each operations which will define the message formats both it will respond and expect. So when user integrate connectors in a project this connector option will search through the workspace and find the available connectors. Then user can select the respective connector and the operation so that related json schema will be loaded for data mapper by the tool.

Mapping configuration file is simply a Java Script file generated looking at the diagram user will draw in the data mapper diagram editor by connecting input elements to output elements. Every operation user defines in diagram will convert to Java Script operation.

So these three files will be generated by the Data Mapper Tool and saved in a Registry Resource project to deploy in a wso2 server.

regresource.png

“.datamapper” and “.datamapper_diagram” files contain meta data related to Data Mapper Diagram. They will be ignored when trying to deploy in a server to use by data mapper engine. Only two schema files and .dmc (data mapper configuration) file will be deploy.

Need an Intermediate Component to Use With WSO2 Products

WSO2 Data Mapper is an independent component. It is not depending on any other WSO2 product. But other products can use data mapper to achieve/offer data mapping capability. For that we need a intermediate component. Data Mapper Mediator is this intermediate component which  will give the data mapping capability into WSO2 ESB.

DMMediator.png

Data Mapper Mediator will find the configuration files from the registry and configure data mapper engine with the input message type(XML/JSON/CSV) and output message type(XML/JSON/CSV) . Then it will take request message from ESB message flow and use configured data mapper engine to execute the transformation and output message will be added to ESB message flow.

Can Use Product Specific Runtime Variables

Data Mapper engine also given a way to use run time product specific variables in the mapping. The way it works is, intermediate component should construct a map object containing run time product specific variables and send it to  data mapper engine. So when the mapping happens in the data mapper engine, these variables will be available. For Eg: Data Mapper Mediator provides esb axis2/transport/synapse/axis2client/operation/.. properties like this. In the Data Mapper diagram user can use the Property operator and define the scope and the property name and use it in the mapping. Data Mapper Mediator will identify the required properties to execute the mapping and populate a map with the required properties and will send it to engine. We will look it with more detail when we discuss about Property Operator.

Data Mapper Engine

Data Mapper engine need basically following information to be configured,

  • Input Message Type
  • Output Message Type
  • Input schema
  • Output schema
  • Mapping configuration

At the runtime it will get the input message and runtime variable map object and output the transformed message. To execute the mapping configuration data mapper engine use java scripting api. So if your runtime is JAVA 7 rhino JS engine will be used and if your runtime is JAVA 8 nashorn JS engine will be used.

There are several limitations in the rhino engine that will directly affect data mapper engine when JAVA 7 is used. There are several functions that will not support in rhino,

for eg: String object functions like startsWith() endsWith()

So user may need to aware of that when using custom functions and operators, rhino may have limitations executing those.

OPERATIONS IN WSO2 DATA MAPPER

Let’s look into the operations we could use to do the mapping. Following diagram shows the WSO2 Data Mapping Diagram Editor.

diagramEditor.png

As you can see in the left side we have the operations pallet. These operations can be drag and drop to the Editor area.

There are six categories listed down

  • Links
  • Common
  • Arithmetic
  • Conditional
  • Boolean
  • Type Conversion
  • String

LINKS

DataMapperLink.gif Data Mapping Link- map elements with other operators and elements.

COMMON

Constant Constant- define string, number or boolean constant values.

CustomFunction Custom Function – define custom function to use in mapping

Properties Properties – use product specific runtime variables

GlobalVariable Global Variable – instantiate global variable to access from any where

Compare Compare – compare two inputs in the mapping

ARITHMETIC

Add Add – add two numbers

Subtract Subtract – subtract two or more numbers

Multiply Multiply – multiply two or more numbers

Divide Divide –  divide two numbers

Celi Ceiling – get the ceiling value of a number (closest larger integer value)

Floor Floor – get the floor value of a number (closest lower integer value)

Round Round – get the nearest integer value

SetPrecision Set Precision – Format a number into a specified length

AbsoluteValue Absolute Value – get the absolute value of a rational number

Min Min – get the minimum number from given inputs

Max Max – get the maximum number from given inputs

CONDITIONAL

IfElse IfElse – use a condition and select one input from two

BOOLEAN

AND AND – boolean AND operation on inputs

OR OR – boolean OR operation on inputs

NOT NOT – boolean NOT operation

TYPE CONVERSION

StringToNumber StringToNumber – convert string value to number (“0” -> 0)

StringToBoolean StringToBoolean – convert string value to boolean(“true” -> true)

ToString ToString – convert number or boolean to string

STRING

Concat Concat – concat two or more strings

Split Split – split a string by a matching string value

UpperCase Uppercase – convert a string to uppercase letters

LowerCase Lowercase – convert a string to lowercase letters

StringLength String Length – get the length of the string

StartsWith StratsWith – check whether string starts with a specific value (JAVA7)

EndsWith EndsWith – check whether string ends with a specific value (JAVA7)

Substring Substring – extract a part of the string value

Trim Trim – remove white spaces from beginning and end of a string

Replace Replace – replace the first occurrence of a target string with other

Match Match – check whether the input match with a (JS)regular expression

We will discuss more on some of the operations in the next post.

 

Advertisements

6 thoughts on “Understanding WSO2 Data Mapper 5.0.0

  1. How can map from a csv file that uses ; as field delimiter? Data mapper csv input seems to only work with files that use , as the delimiter. 😦

    Like

    1. I’m sorry to say that is it limited for delimiter , . I will add a JIRA on data mapper as a improvement to specify the delimiter also in csv mappings. Thank you very much for your feedback.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s