INTRODUCTION
WSO2 data mapper is a data mapping solution that convert and transform one format of data to a different format. It provides a WSO2 Developer Studio based tool to create the graphical mapping configuration and generate the files needed to execute the mapping configuration by the WSO2 Data mapper engine.
This is the second of WSO2 Data Mapper posts which will explain about WSO2 Data Mapper from the user perspective. First post was to describe the end to end work flow on how to create, deploy and test data mapper configuration.(How to Use WSO2 Data Mapping Mediator in ESB)
COMPONENTS OF WSO2 DATA MAPPER
Before we going into look the operations lets look how things happen under the hood. WSO2 Data Mapper comes as two components ideally. They are Data Mapper Engine and Data Mapper Tooling component.
Data Mapper Tooling component
Data Mapper Tooling component is the interface to user to create configuration files needed for engine to execute the mapping. There are three configuration files needed by the Data Mapper engine. They are,
- Input schema file
- Output schema file
- Mapping configuration file
Input schema and output schema defines the input/output format of input message and required message. It is basically a custom defined json schema. It will be generated by the Data Mapper tool for you when loading the input and output files.
As above image shows user can load input/output message format in several different ways. They are,
- XML – Sample XML file
- JSON- Sample JSON file
- CSV – Sample CSV file with column names as the first record
- XSD – XSD file which is actually schema for XML files
- JSONSCHEMA – WSO2 data mapper json schema
- CONNECTOR
I guess only CONNECTOR option is the confusing part for you. It is to use data mapper with connectors straightly. WSO2 connectors will contain json schemas for each operations which will define the message formats both it will respond and expect. So when user integrate connectors in a project this connector option will search through the workspace and find the available connectors. Then user can select the respective connector and the operation so that related json schema will be loaded for data mapper by the tool.
Mapping configuration file is simply a Java Script file generated looking at the diagram user will draw in the data mapper diagram editor by connecting input elements to output elements. Every operation user defines in diagram will convert to Java Script operation.
So these three files will be generated by the Data Mapper Tool and saved in a Registry Resource project to deploy in a wso2 server.
“.datamapper” and “.datamapper_diagram” files contain meta data related to Data Mapper Diagram. They will be ignored when trying to deploy in a server to use by data mapper engine. Only two schema files and .dmc (data mapper configuration) file will be deploy.
Need an Intermediate Component to Use With WSO2 Products
WSO2 Data Mapper is an independent component. It is not depending on any other WSO2 product. But other products can use data mapper to achieve/offer data mapping capability. For that we need a intermediate component. Data Mapper Mediator is this intermediate component which will give the data mapping capability into WSO2 ESB.
Data Mapper Mediator will find the configuration files from the registry and configure data mapper engine with the input message type(XML/JSON/CSV) and output message type(XML/JSON/CSV) . Then it will take request message from ESB message flow and use configured data mapper engine to execute the transformation and output message will be added to ESB message flow.
Can Use Product Specific Runtime Variables
Data Mapper engine also given a way to use run time product specific variables in the mapping. The way it works is, intermediate component should construct a map object containing run time product specific variables and send it to data mapper engine. So when the mapping happens in the data mapper engine, these variables will be available. For Eg: Data Mapper Mediator provides esb axis2/transport/synapse/axis2client/operation/.. properties like this. In the Data Mapper diagram user can use the Property operator and define the scope and the property name and use it in the mapping. Data Mapper Mediator will identify the required properties to execute the mapping and populate a map with the required properties and will send it to engine. We will look it with more detail when we discuss about Property Operator.
Data Mapper Engine
Data Mapper engine need basically following information to be configured,
- Input Message Type
- Output Message Type
- Input schema
- Output schema
- Mapping configuration
At the runtime it will get the input message and runtime variable map object and output the transformed message. To execute the mapping configuration data mapper engine use java scripting api. So if your runtime is JAVA 7 rhino JS engine will be used and if your runtime is JAVA 8 nashorn JS engine will be used.
There are several limitations in the rhino engine that will directly affect data mapper engine when JAVA 7 is used. There are several functions that will not support in rhino,
for eg: String object functions like startsWith() endsWith()
So user may need to aware of that when using custom functions and operators, rhino may have limitations executing those.
OPERATIONS IN WSO2 DATA MAPPER
Let’s look into the operations we could use to do the mapping. Following diagram shows the WSO2 Data Mapping Diagram Editor.
As you can see in the left side we have the operations pallet. These operations can be drag and drop to the Editor area.
There are six categories listed down
- Links
- Common
- Arithmetic
- Conditional
- Boolean
- Type Conversion
- String
LINKS
Data Mapping Link- map elements with other operators and elements.
COMMON
Constant- define string, number or boolean constant values.
Custom Function – define custom function to use in mapping
Properties – use product specific runtime variables
Global Variable – instantiate global variable to access from any where
Compare – compare two inputs in the mapping
ARITHMETIC
Add – add two numbers
Subtract – subtract two or more numbers
Multiply – multiply two or more numbers
Divide – divide two numbers
Ceiling – get the ceiling value of a number (closest larger integer value)
Floor – get the floor value of a number (closest lower integer value)
Round – get the nearest integer value
Set Precision – Format a number into a specified length
Absolute Value – get the absolute value of a rational number
Min – get the minimum number from given inputs
Max – get the maximum number from given inputs
CONDITIONAL
IfElse – use a condition and select one input from two
BOOLEAN
AND – boolean AND operation on inputs
OR – boolean OR operation on inputs
NOT – boolean NOT operation
TYPE CONVERSION
StringToNumber – convert string value to number (“0” -> 0)
StringToBoolean – convert string value to boolean(“true” -> true)
ToString – convert number or boolean to string
STRING
Concat – concat two or more strings
Split – split a string by a matching string value
Uppercase – convert a string to uppercase letters
Lowercase – convert a string to lowercase letters
String Length – get the length of the string
StratsWith – check whether string starts with a specific value (JAVA7)
EndsWith – check whether string ends with a specific value (JAVA7)
Substring – extract a part of the string value
Trim – remove white spaces from beginning and end of a string
Replace – replace the first occurrence of a target string with other
Match – check whether the input match with a (JS)regular expression
We will discuss more on some of the operations in the next post.
How can map from a csv file that uses ; as field delimiter? Data mapper csv input seems to only work with files that use , as the delimiter. 😦
LikeLike
I’m sorry to say that is it limited for delimiter , . I will add a JIRA on data mapper as a improvement to specify the delimiter also in csv mappings. Thank you very much for your feedback.
LikeLike
Hello, look at http://stackoverflow.com/questions/40178395/data-mapping-xml-rpc-to-json
Is there an option to mapping fields based in indexed array?
This is because of this response in the example:
https://download.automation.odin.com/pa/6.0/doc/portal/6.0/ba/index.htm?fileName=75131.htm
LikeLike
Sure. Answered the question there. Please have a look.
Thanks
LikeLike
Thank you, it was a really good work around
LikeLike
Hello, it is me again, I am facing a problem with escaping characters. I didn’t found anything that can help me.
See: http://stackoverflow.com/questions/42306137/wso2-esb-datamapper-escaping-underscore
TLDR: I am getting error in tag with _ in message result. What can I do to fix it?
Thank you.
LikeLike