Skip to end of metadata
Go to start of metadata


Contents
Diyotta Modern Data Integration Suite
Integration
Opening Integration
Integration Interface Components
Working with Integration
Working with Data Point
Creating Data Point
Editing Data Point
Copying Data Point
Deleting Data Point
Unlocking Data Point
Closing Data Point
Testing Data Point
Showing Data Point Dependencies
Working with Data Object
Creating Data Object
Importing Data Object
Creating Transient Data Object
Editing Data Object
Copying Data Object
Refreshing Data Object
Deleting Data Object
Unlocking Data Object
Creating/Modifying Data Object as Transient
Copy as Other Data Object
Viewing DDL
Closing Data Object
Working with Data Flow
Creating Data Flow
Creating Source
Creating Transformation as Target
Editing Data Flow
Refreshing Data Flow
Deleting Data Flow
Copying Data Flow
Validating Data Flow
Executing Data Flow
Unlocking Data Flow
Showing Data Flow Dependencies
Arranging Data Flow
Closing Data Flow
Working with Transformations
Editing Transformations
Editing Expression
Editing Rollup
Editing Joiner
Editing Filter
Editing Temp Stage
Editing Splitter
Editing VennGroup
Copying and Pasting Transformations
Deleting Transformations
Creating Transformation as Transient
Creating Transformation as Tstage
Working with Job Flow
Creating Job Flow
Deleting Job Flow
Editing Job Flow
Copying Job Flow
Arranging Job Flow
Working with Data SubFlow
Creating Input in Data SubFlow
Editing Output Target
Editing Data SubFlow
Refreshing Data SubFlow
Deleting Data SubFlow
Copying Data SubFlow
Validating Data SubFlow
Unlocking Data SubFlow
Closing Data SubFlow
Showing Data SubFlow Dependencies
Arranging Data SubFlow
Working with Sequence
Creating Sequence
Editing Sequence
Deleting Sequence
Working with Expression
Creating Expression
Editing Expression
Deleting Expression
Working with User Defined Function
Creating User Defined Functions
Editing UDF
Deleting UDF
Parent Object
Export Object
Import Object
Working with SQL Editor
Admin
Opening Admin
Admin Interface Components
Working with Admin
Admin Configurations
Repository Details
LDAP Configuration
License Details
Enable Data Points
Server Settings
Data Type Mapping
Data Extract Properties
Data Load Properties
Runtime Properties
Application Settings
Admin View
Viewing Server Log
Viewing Active Users
Viewing Object Locks
Organizations
Working with Organizations
Creating Organization
Creating Project
Editing Project
Deleting Project
Creating User
Editing User
Deleting User
Creating User Groups
Editing User Groups
Deleting User Group
Creating Role
Editing Role
Deleting Role
Creating Agent
Editing Agent
Deleting Agent
Creating Agent Groups
Editing Agent Groups
Monitor
Opening Monitor
Monitor Interface Components
Working with Monitor
Monitoring Activities
Getting Log
Lineage
Opening Lineage
Scheduler
Opening Scheduler
Scheduler Interface
Working with Scheduler
Creating Task Group
Deleting Task Group
Creating Command Task
Creating File Watcher Task
Deleting File Watcher Task
Deleting Calendar
Creating Email Action
Creating File Event
Deleting File Event
Inserting Into Scheduler
View Activity
Task Actions
Appendix
Expression Editor
Netezza Params
Data Extraction and Data Load Properties
Glossary

Diyotta Modern Data Integration Suite

Welcome to Diyotta Modern Data Integration (MDI) Suite help.
The MDI Suite introduces a robust architecture to efficiently distribute the processing workload across different processing platforms and linearly scale across different source systems, irrespective of their location. It provides easy portability to different execution engines on demand, and implements the five principles of modern data integration to support the data modernization process.
At the core of this data management platform is the concept of a "Controller" and "Agent" that manage data processing where it resides by generating the optimized instructions which can be natively executed on the processing platform. It manages the entire end-to-end process including sourcing, ingestion, integration, and provisioning while maintaining a central but lightweight metadata repository to track the entire execution and orchestration process.
The MDI Suite consists of browser-based modules where interaction with the system takes place.
The MDI Suite supports a role-based access model such as the developer, operator, analyst etc. It provides an interactive design and data view capability supporting a multi-team developer environment through the following modules:

 
        Integration - The Integration module offers a collaborative multiuser environment to develop and implement data integration processes in an ELT approach, with a rich set of functions in a seamless fashion as all the native platform supported functions are in-built.
        Admin - The Admin module is exclusively offered for management and administrative purposes to control projects, users, hosts, and add flexibility to monitor user work and server logs. The Admin provides role-based security to manage projects in a multi-team environment.
        Monitor - The Monitor module offers a UI to monitor the execution of jobs and operational statistics and gives the ability to manage and debug execution of jobs and streams.
        Lineage - A complete view of data lineage from data origination, to each level of its transformation, to a target can be viewed through this module to provide a level of enterprise data management and object reuse that may not otherwise be available.
        Scheduler - The Scheduler executes planned tasks or designs, manages notifications, and builds dependencies, which can be automated and scheduled based on jobs linked to file events, calendar events, etc.

Integration

The Integration module of the Modern Data Integration (MDI) Suite offers a collaborative multiuser environment to develop and implement data integration processes in an ELT approach, with a rich set of functions in a seamless fashion as all the native platform supported functions are in-built.
It is in Integration module that all the data integration rules are defined, metadata is defined for participating data objects, data points, data movements and transformations. The Integration tool provides various debugging options during development.
The Integration module has components that can help you build Data Points, Data Objects, and Data Flows using which you can specify how to move and transform the data between source and target databases. You can use the Integration module to create source definitions, transformations, and target definitions to create the mappings.
The MDI Suite natively handles connections to the following databases:

Database

 

 

Teradata

Oracle

Netezza

Hadoop

BigInsights

Exadata

GreenPlum

Hawq

Splice Machine

Flat file

DB2

MSSQL

Cobol

Salesforce

JSON

XSD

PostgreSQL

Spark

JMS

Sybase

SAS

Facebook/Twitter

AWS

Kafka

 
 

Related Topics

Opening Integration
Integration Interface Components
Working with Integration
 
 

Opening Integration

Log in to the MDI Suite using your Login Id and password. If you have sufficient privileges (provided by the Admin), then you will be directed to the Integration module home page, by default.
If you are logged into some other module such as Admin, Lineage, Monitor or Scheduler, then-
1.      Click the icon ( ) at the top left corner of the page.
2.      The icon will expand to display all the tiles of the available modules.

3.      Click Integration to login to the Integration module.
 

Related Topics

Integration
Integration Interface Components
Working with Integration
 
 

Integration Interface Components

The Integration home page looks as shown in the screen shot given below.

The home page of the Integration module displays the Dashboard of the system. It displays the number of Data Points, Data Objects, Data Flows and Job Flows used in the project last opened. The home page also displays a graphical presentation about the status of the project.
Object Explorer - The Object explorer on the left displays the icons of the objects used in a project such as Data Point, Data Object, Data Flow, Job Flow, etc.
Canvas Area - The Canvas is the main working area where you can create/view/modify Data Points, Sequences, UDFs, Data Objects, Job Flows, Expressions, Transformations.
Side bar - The side bar on the right displays options which can be performed on an object such as Save, Delete, Refresh, Validate etc.
Quick Access Tool bar - The Quick Access tool bar on the extreme right bottom provides quick links or shortcuts to create new objects.
 

Related Topics

Integration
Working with Integration
 
 

Working with Integration

The Integration module is a workplace for developers to build complex ELT packages encompassing the following basic tasks:

  • Create/Reuse/Manage Data Points
  • Import/Create/Modify Data Objects
  • Add Data Objects to Data Flows
  • Create/Modify Data Flows including setting up various transformations to manipulate data
  • Create Job Flows, set dependencies and orchestrate Data Flows
  • Assign Data Points and mappings in Jobs
  • Execute Jobs
  • Create reusable components to maximize productivity and reduce maintenance
  • Export and import various objects to streamline development across projects

 
These points also describe the development workflow of designing packages within the MDI Integration module. In addition, the Integration module offers help and tips on using the supported processing platform extraction and load utilities; while seamlessly building custom code to accommodate data movement between various source and target platforms.
 

Related Topics

Integration
Working with Data Point
Working with Data Object
 
 
 

Working with Data Point

Data Points are used to establish a relationship with the native or external database. These Data Points are essential during run-time. For example, when you create a job to pull or push data within the database, you need to have the connectivity to the database for executing the job successfully.
        Click the Data Point icon ( ) in the Object Explorer in the left pane of the home page to work with it. You can create a new Data Point in the Canvas area.
The Data Point home page displays a list of all the existing Data Points. The general details such as the type of database, whether it is a private Data Point, the associated Agent, group name, last edited by and last edited on are displayed on the screen.

  Creating Data Point

Data Point is a connection built to access and communicate with the database. This is required during run-time.
1.      You can create a new Data Point through any of the following ways:
        Object Explorer: Click Home in the Object explorer pane on the left of the dashboard. On the side menu bar click
        Side Menu bar: Click the icon in the side menu bar.  
        Quick Access tool bar: Under the Quick Links icon ( ) at the bottom of the side menu bar, click .
2.      A New Data Point window opens prompting you to select the database type for which the Data Point is being created. Select a database type from the list. You will be directed to the General tab page, by default.

General Tab

The following details need to be provided in the General tab:

Field

Description

Name

Specifies the name of the Data Point being created. You can modify the name if required, using the arrow ( ) beside the field.

Description

Specifies a brief description (optional) of the Data Point being created.

Private Data Point

Indicates if the Data Point being created is private or accessible by all users. Select to make it a private Data Point.

Owner

Specifies the owner of the Data Point

Group Name

Specifies the name of the group or database type to which the Data Point is associated.

Parameter Name

Specifies the name of the parameter associated to the Data Point. This is read-only. However, this name changes automatically when the name of the Data Point is changed.

Data Point Type

Specifies the database type associated to the Data Point being created. For example, Netezza, Teradata, Oracle, Hive.

Last Updated By

Specifies the unique user name and the timestamp when the Data Point was last updated.

Properties Tab

Provide the details in the Properties tab according to the description given below:

Field

Description

Host

Specifies the host address.

Port

Specifies the port number that is associated to the selected host.

App User

Specifies the name of the user who will be using the Data Point.

Password

Password associated to the App user ID.

Database Version

Specifies the version of the database to which the Data Point is established from the drop-down list.

Agent

By default, the Agent assigned to that project will be displayed. To change the Agent, click the Change hyperlink, and select the required agent from the list of agent names displayed.
If there are no Agents assigned to your project then by default, the Agent name is blank with Add hyperlink.


Databases Tab

In the Databases tab, use the Click Here button to create new databases.
The following details need to be provided in the Databases tab:

Field

Description

Name

Specifies the name of the database being created.

Database

Specifies the name of the physical database.

Transform DB

Select this option to indicate if the transformation happens in the selected database.
You can select only one database as Tform database.

3.      Use the Add Database icon ( ) to include databases. The system however, allows you to use only one database during runtime.
4.      Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected database, respectively and Paste icon ( ) to paste the copied database. The system allows you to use only unique names for the database; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
5.      Use the up arrow ( ) or down arrow ( ) to arrange the order of the database sequence in the list.
6.      Use the Delete icon ( ) to delete the selected database.
7.      Use the Search icon ( ) to search for a specific database from the list.
8.      Click Save ( )in the side menu bar to save the changes made to the created Data Point or click Revert ( )to discard the changes.
9.      Once the details are set for the Data Point, click Test Data Point.
10.   Set the Data Extract and Load properties.
 

Editing Data Point

Use this option to edit the details of the selected Data Point.
1.      Click the Data Point icon ( ) available in the Object explorer in the left pane of the dashboard.
2.      From the list of Data Points, click to open the Data Point that needs to be edited. The MDI Suite displays the details of the selected Data Point to be edited.
Edit or modify a Data Point based on the description of the fields given below.

General Tab

Field

Description

Name

Displays the name of the Data Point. You can modify the name, if required.

Description

Type a brief description (optional) of the Data Point being edited.

Private Data Point

Indicates if the Data Point being created is private or accessible by all users.

Group Name

Displays the name of the group to which the Data Point is associated.

Parameter Name

Displays the name of the parameter associated to the Data Point. This is read-only. However, this name changes automatically when the name of the Data Point is changed.

Data Point Type

Displays the database to which the Data Point is established. For example, Netezza, Teradata, Oracle, Hive.

Last Updated By and Date

Specifies the unique user and the timestamp when the Data Point was last updated.

Properties Tab

Field

Description

Host

Displays the host address.

Port

Displays the port number that is associated to the selected host.

App User

Displays the name of the user who will be using the Data Point.

Password

The password that is associated to the App user ID.

Database Version

Displays the version of the database to which the Data Point is established.

Agent

Displays the agent associated to the Data Point. You can change the agent associated to the Data Point using the Change hyperlink. Click the hyperlink, and select the required agent from the list of agent names displayed.

Databases Tab

Field

Description

Name

Displays the name of the database being created.

Database

Displays the name of the physical database.

Transform DB

Indicates if the transformation happens in the selected database.

3.      Use the Add icon ( ) to include databases. The system however, allows you to use only one database.
4.      Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected database, respectively and Paste icon ( ) to paste the copied database. The system allows you to use only unique names for the database; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
5.      Use the up arrow ( ) or down arrow ( ) to arrange the order of the database sequence in the list.
6.      Use the Delete icon ( ) to delete the selected database.
7.      Use the Search icon ( ) to search for a specific database from the list.
8.      Click Save ( )in the side menu bar to save the changes made to the created Data Point or click Revert ( )to discard the changes.   
 

Copying Data Point

To create a duplicate of an existing Data Point established with the database:
1.      Click the Data Point icon ( ) on the Object explorer in the left pane.
2.      From the list of Data Points, select the Data Point that needs to be copied. Now, click the Create Copy icon ( )available in the side menu bar.
Or
Click to open the Data Point that needs to be copied. The MDI Suite displays the details of the selected Data Point to be copied. Click the Create Copy option from the side menu bar.
3.      A duplicate Data Point is created with sequence number added as a suffix. The number increments by one every time a copy is made of the same Data Point.
4.      Click the Save button to save the changes made to the created Data Point or click Revert to discard the changes.
 

Deleting Data Point

To delete a Data Point, follow the given steps:
1.      Click the Data Point icon ( ) on the Object explorer in the left pane. A list of all the available Data Points is displayed.
2.      From the list of Data Points, select the Data Point that needs to be deleted.Or
Click to open the Data Point that needs to be deleted. The MDI Suite displays the details of the selected Data Point.
3.      Click the Delete option from the side menu bar. The system strikes out the selected Data Point to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The stricken out Data Points are deleted from the list. Click Revert to undo deleting.
 

Unlocking Data Point

When you are updating a Data Point, the MDI Suite automatically locks the Data Point on your user ID. To unlock the Data Point to enable other users to update it, use the following steps:
1.      Click the Data Point icon ( ) available on the Object explorer in the left pane. A list of all the available Data Points is displayed.
2.      From the list of Data Points, select the Data Point that needs to be unlocked. Now, select the Unlock option from the side menu bar.  
3.      A message is displayed confirming that the selected Data Point is unlocked successfully.
4.      Click the Close button to exit the window.
 

Closing Data Point

You can close the Data Point you are currently working on or close all the open Data Points through this option.
        On the side menu bar, click the Close icon ( ). The current Data Point will be closed; the page again displays the list of available Data Points.
        On the side menu bar, click the tab Opened Objects.

A list of all the open Data Points is displayed.
        Click Close All to close all the open Data Points.
        Click Close Others to close all the Data Points other than the current one on the screen.

Testing Data Point

Once you create a Data Point, you need to test it whether the connectivity to the database is successful. The Data Point can be native or external.
To test whether a Data Point is established successfully:  
1.      Click the Data Point icon ( ) in the Object explorer in the left pane. A list of all the available Data Points is displayed.
2.      From the list select the Data Point that needs to be tested.
3.      On the side menu bar, click the option Test Data Point ( ). A confirmation message stating that the Data Point is established successfully is displayed.
4.      Click the Close button to exit the screen.
5.      You can also directly test a Data Point at the time of creating a Data Point under the Properties tab.
 

Showing Data Point Dependencies

You can use this option to view dependencies of the selected Data Point.
1.      Click the Data Point icon ( ) on the Object explorer in the left pane. A list of all the available Data Points is displayed.
2.      From the list of Data Points, select the Data Point whose dependencies needs to be viewed. On the side menu bar, click ( )
Or
From the list of Data Points, click to open the Data Point whose dependencies needs to be viewed. Now, click Dependencies on the side menu bar.
The dependencies associated with the selected Data Point are displayed.
The following details are available in the window:

Field/Button

Description

Dependent

Displays the name of the dependent object.

Object Type

Displays the type of dependent object.

Project

Displays the name of the project to which the selected Data Point is associated.

Last Updated By

Specifies the user who last updated it.

Last Updated Date

Displays the date on which the dependent object was updated.

3.      Click the dependent object to view the details. You will be redirected to the object's page in the canvas area where the design and the properties are defined.  
4.      Click the Cancel button to exit the screen.
 

Related Topics

Integration
Working with Integration
Creating Data Point
 

Working with Data Object

Data Objects are metadata definitions of source and target database tables that are used in designs to transfer data from one place to another. The MDI Suite provides the options of either creating or importing a Data Object. Data Objects can be created for any database supported by MDI.
The Data Object home page displays a list of all the existing Data Objects. The following general details are displayed on the screen:

Field Name

Description

Data Object Name

Specifies the name of the Data Object.

Type

Specifies the type of database the selected Data Object is associated to.

Data Point

Specifies the name of the Data Point that the selected Data Object is associated to.

Scope

Specifies the scope of the selected Data Object. For example, Project.

Group Name

Specifies the name of the group to which the selected Data Object is associated.

Last Edited By

Specifies the name of the user who last updated the selected Data Object.

Last Edited On

Specifies date and time when the selected Data Object was last updated.

 

Creating Data Object

Data Objects are metadata definitions of source and target database tables that are used in designs to transfer data from one place to another.
1.      You can create a new Data Object through any of the following ways:
        Object Explorer: In the Object explorer, click Data Object icon ( ). A list of all the Data Objects available is displayed. In the right menu bar, click ( ).
        Quick Access tool bar: Click the Quick Links icon ( ) at the bottom of the side menu bar. Click the ( ) icon.
2.      A New Data Object window opens prompting you to select the Data Point type for which the Data Object is being created. Select a Data Point type from the list. You will be directed to the General tab page, by default.
3.      Select the Data Point from the list for which the Data Object needs to be created. The system displays the General tab by default.

General

The following details need to be provided in the General tab:

Field/Button

Description

Name

Specifies the name of the Data Object being edited. Click the icon ( ) to view the Data Object Group Name wizard. You can edit the following details in this wizard:
        Name - The name of the Data Object being modified. When you edit this field, the MDI Suite performs a check to identify if another Data Object is available with the same name. If the name is already in use, the system displays a warning message to opt for a different name.
        Group Name - The name of the group to which the selected Data Object belongs. You can associate the Data Object to different group using the drop-down list containing the names of different group.

Description

Describes the Data Object being created. This is optional.

Group Name

Specifies the name of the group to which the selected Data Object belongs to. You can associate the Data Object to different group using the Data Object Group Name wizard described above.

Parameter Name

Specifies the name of the parameter associated to the Data Object being created.

Data Point

Specifies the name of the Data Point associated to the Data Object being created.

Data Point Database

Specifies the name of the database associated to the Data Point.

Transient Object

Specifies if the created Data Object is a Transient object.

Database Type

Specifies the type of database the selected Data Object is associated with. For example, Netezza, Oracle, Teradata, Hive.

Last Updated By

Specifies the unique user who last updated the Data Object and displays the date and time when the selected data object was last updated.

Attributes

The following details need to be provided in the Attributes tab:
If the Data Object does not have any existing attributes, click the Click Here button to add new attributes.

Field/Button

Description

Attribute Name

Specifies the name of the attribute. Click the Add Attribute icon ( ) to add an attribute into the existing list of attributes.
You can edit the name of the attributes by clicking the name once.

Data Type

Specifies the type of data associated to the attribute. For example, varchar, bigint, boolean, byte, byteint, char, etc.

Prec

Specifies the maximum number of characters that the selected attribute can accommodate.

Scale

Specifies the number of characters that the selected attribute can accommodate after the decimal.

Not Null

Specifies if the selected attribute is not null. Select the check box to indicate it is a null value.

Key Type

Specifies the key type of the selected attribute. The options are:
        Not a key - Select this option to indicate the attribute is not a key.
        Primary Key - Select this option to indicate the attribute is the primary key.
        Foreign Key - Select this option to indicate the attribute is a foreign key.

Dist

Indicates if the distribution of the attributes is based on the selected attribute.
Note: This column is available only for Native Netezza Data Objects.

Org

Indicates if the organization is based on the selected attribute. You can select up to four columns based on which the table is organized.
Note: This column is available only for Native Netezza Data Objects.

UPI

Indicates if the selected attribute is unique primary index.
Note: This column is available only for Native Teradata Data Objects.

NUPI

Indicates if the selected attribute is non-unique primary index.
Note: This column is available only for Native Teradata Data Objects.

USI

Indicates if the selected attribute is unique secondary index.
Note: This column is available only for Native Teradata Data Objects.

NUSI

Indicates if the selected attribute is non-unique secondary index.
Note: This column is available only for Native Teradata Data Objects.

DDL Tab

This tab displays the Data Distribution Language (DDL) of the Data Object being created. Note: DDL is read-only. You cannot modify the script here.

View Data

This tab displays the data available for the Data Object being created. The View Data tab will expand to show three options:

Name

Description

Refresh

Click this option to refresh the data for any latest updates.

Export

Click this option to export the data from the open Data Object to an excel sheet.

View SQL

Click this option to view the SQL query associated with the selected Data Object. You will be redirected to the SQL Editor page.

4.      Click the Save button to save the changes made to the created Data Object or click Revert to discard the changes.
 

  Importing Data Object

Data Objects can be created or imported from sources. Since tables can be large, it is advantageous to import them whenever possible, thereby negating the need to map attributes manually.
1.      You can import a new Data Object through any of the following ways:

  • Side Menu bar: Click the Data Object icon ( ) or the Data Point icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar, click ( ) icon.
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar. Click Import Data Object.

2.      A list of all the available Data Points is displayed in a new window. Expand the Data Point and select the database to be imported.
This launches a wizard which guides you through importing the Data Object. The window displays the Source Database connection details where the fields have the following description.

Field

Description

Data Point Name

Displays the name of the Data Point associated with the Data Object.

Data Point Type

The database type associated to the Data Point. For example, Netezza, Teradata, Oracle, Hive.

Host

Specifies the host address.

Port

Specifies the port number that is associated to the selected host.

Database

Specifies the name of the database being imported.

Data Object Filter

  • mark is displayed to import the Data Objects with the certain set of filters. For example,
            A* will retrieve all the Data Objects name existing in the connected database that starts with the alphabet A.
            *A will retrieve all the data objects name existing in the connected database that ends with the alphabet A.
            A will retrieve the only Data Object name existing in the connected database.

Database Version

Specifies the version of the database.

Agent

Specifies the agent associated to the Data Object.

Ignore Constraints

Select the option "Ignore constraints", if the constraints should be ignored. Constraints are the rules enforced on data columns/ tables. These are used to prevent invalid data entering into the database. For example, if you want to allow nulls in a column in a target but it is constrained in the source, check that box.

Test Data Point

Click this button to test the selected Data Point before proceeding. The MDI Suite displays a confirmation message that the Data Point is valid.

Back

Click this button to go back to the previous window.

Next

Click this button to go to the next window.

Cancel

Click this button to exit the window without importing the Data Object.

The database connection details will vary depending on the type of database. For example, a Hadoop connection would have other fields such as Yarn Resource Manager Port, Job Tracker Port and Zookeeper Client Port.
3.      Click Next and a new window is launched displaying all the available Data Objects. Select the Data Object(s) that you want to import from the list. You can import one or multiple Data Objects.
The window displays the following fields

Fields

Description

Data Objects

Specifies the name of the available Data Objects

Options

Specifies the action to perform on the Data Object.
1        Import - This option is selected by default. Imports the selected Data Object.
2        Replace - Imports the Data Object as it is available in the database.
3        Rename - Renames the Data Object being imported

Data View

Click the icon to see the data.

Difference

Displays the differences (if any) in the original database and the new database.

4.      Now, click the Import button to successfully import the Data Object or click Cancel to exit the window without importing the Data Object. The MDI Suite will display a message confirming that the Data Object has been imported successfully. Click Close to exit the window.
 

  Creating Transient Data Object

Transient Data Objects represent temporary tables created in a Native database that are used as part of a design and are instantiated at run time. They are created the same way as any other Data Object, however have slightly different properties. The purpose of a Transient table is to persist temporary data that may need to be used across designs and runs. The Transient Data Object can be used as a source or target in a design. It will persist for as long as there is no change to the metadata.
To create a Transient Data Object or temporary Data Object:
1.      Click the Data Object icon ( ) in the Object explorer in the left pane of the dashboard.
2.      On the side menu bar, click the icon ( ). This launches the New Transient Data Object window and displays all the available Data Points.
3.      Select the Data Point from the list for which the Transient Data Object needs to be created. The system displays the General tab by default.
Set all the fields according to the description given below:

General

The following details need to be provided in the General tab:

Field/Button

Description

Name

Specifies the name of the Data Object being edited. Click the icon ( ) to view the Data Object Group Name wizard. You can edit the following details in this wizard:
        Name - The name of the Data Object being modified. When you edit this field, the MDI Suite performs a check to identify if another Data Object is available with the same name. If the name is already in use, the system displays a warning message to opt for a different name.
        Group Name - The name of the group to which the selected Data Object belongs to. You can associate the Data Object to different groups using the drop-down list containing the names of different groups.

Description

Specifies the description (optional) for the Transient Data Object being created.

Scope

Select any of the following option to specify the scope of the Data Object:
        Project - Select this option to indicate that the scope of the Data Object is for project.
        Job Flow - Select this option to indicate that the scope of the Data Object is for job flow.

Group Name

Specifies the name of the group to which the selected Data Object belongs to. You can associate the Data Object to different group using the Data Object Group Name wizard described above.

Parameter Name

Specifies the name of the parameter associated to the Data Object being created.

Data Point

Specifies the name of the Data Point associated to the Data Object being created.

Data Point Database

Specifies the name of the database associated to the Data Point.

Transient Object

Specifies if the created Data Object is a Transient object.

Database Type

Specifies the type of database the selected Data Object is associated with. For example, Netezza, Oracle, Teradata, Hive.

Last Updated By and Date

Specifies the unique user who last updated the Data Object and displays the date and time when the selected data object was last updated.


Attributes

The following details need to be provided in the Attributes tab:
Note: If the data object does not have any existing attributes, click the Click Here button to add new attributes.

Field/Button

Description

Attribute Name

Specifies the name of the attribute. Click the Add Attribute icon ( ) to add an attribute into the existing list of attributes.
You can edit the name of the attributes by clicking the name once.

Data Type

Specifies the type of data associated to the attribute. For example, varchar, bigint, boolean, byte, byteint, char, etc.

Prec

Specifies the maximum number of characters that the selected attribute can accommodate.

Scale

Specifies the number of characters that the selected attribute can accommodate after the decimal.

Not Null

Specifies if the selected attribute is not null. Select the check box to indicate it is a null value.

Key Type

Specifies the key type of the selected attribute. The options are:
        Not a key - Select this option to indicate the attribute is not a key.
        Primary Key - Select this option to indicate the attribute is the primary key.
        Foreign Key - Select this option to indicate the attribute is a foreign key.

Dist

Indicates if the distribution of the attributes is based on the selected attribute.
Note: This column is available only for Native Netezza Data Objects.

Org

Indicates if the organization is based on the selected attribute. You can select up to four columns based on which the table is organized.
Note: This column is available only for Native Netezza Data Objects.

UPI

Indicates if the selected attribute is unique primary index.
Note: This column is available only for Native Teradata Data Objects.

NUPI

Indicates if the selected attribute is non-unique primary index.
Note: This column is available only for Native Teradata Data Objects.

USI

Indicates if the selected attribute is unique secondary index.
Note: This column is available only for Native Teradata Data Objects.

NUSI

Indicates if the selected attribute is non-unique secondary index.
Note: This column is available only for Native Teradata Data Objects.

DDL Tab

This tab displays the Data Distribution Language (DDL) of the Data Object being created. Note: DDL is read-only. You cannot modify the script here.

View Data

This tab displays the data available for the Transient Data Object being created. The View Data tab will expand to show three options:

Name

Description

Refresh

Refreshes the data for any latest updates.

Export

Exports the data from the open Data Object to an Excel sheet.

View SQL

Displays SQL query associated with the selected Data Object. You will be redirected to the SQL Editor page.

4.      Click the Save button to save the created Transient Data Object or click Revert to discard the changes.
 

Editing Data Object

To edit a Data Object, follow the steps given below:
1.      Click the Data Object icon ( ) available in the Object explorer in the left pane of the dashboard. A list of all available Data Objects appears.
2.      From the list of Data Objects, click to open the Data Object that needs to be edited. The MDI Suite displays the details of the selected Data Object to be edited.
By default, you will be redirected to the page displaying the properties of the Data object to be edited.



General

The following details need to be provided in the General tab:

Field/Button

Description

Name

Specifies the name of the Data Object being edited. Click the icon ( ) to view the Data Object Group Name wizard. You can edit the following details in this wizard:
        Name - The name of the Data Object being modified. When you edit this field, the MDI Suite performs a check to identify if another Data Object is available with the same name. If the name is already in use, the system displays a warning message to opt for a different name.
        Group Name - The name of the group to which the selected Data Object belongs. You can associate the Data Object to different group using the drop-down list containing the names of different group.

Group Name

Specifies the name of the group to which the selected Data Object belongs to. You can associate the Data Object to different group using the Data Object Group Name wizard described above.

Parameter Name

Specifies the name of the parameter associated to the Data Object being created.

Data Point

Specifies the name of the Data Point associated to the Data Object being created.

Transient Object

Specifies if the created Data Object is a Transient object.

Database Type

Specifies the type of database the selected data object is associated with. For example, Netezza, Oracle, Teradata, Hive.

Last Updated By

Specifies the unique user who last updated the Data Object and the date and time when the selected data object was last updated.







Attributes

The following details need to be provided in the Attributes tab:
Note: If the Data Object does not have any existing attributes, click the Click Here button to add new attributes.

Field/Button

Description

Attribute Name

Specifies the name of the attribute. Click the Add Attribute icon ( ) to add an attribute into the existing list of attributes.
You can edit the name of the attributes by clicking the name once.

Data Type

Specifies the type of data associated to the attribute. For example, varchar, bigint, boolean, byte, byteint, char, etc.

Prec

Specifies the maximum number of characters that the selected attribute can accommodate.

Scale

Specifies the number of characters that the selected attribute can accommodate after the decimal.

Not Null

Specifies if the selected attribute is not null. Select the check box to indicate it is a null value.

Key Type

Specifies the key type of the selected attribute. The options are:
        Not a key - Select this option to indicate the attribute is not a key.
        Primary Key - Select this option to indicate the attribute is the primary key.
        Foreign Key - Select this option to indicate the attribute is a foreign key.

Dist

Indicates if the distribution of the attributes is based on the selected attribute.
 
Note: This column is available only for Native Netezza Data Objects.

Org

Indicates if the organization is based on the selected attribute. You can select up to four columns based on which the table is organized.
 
Note: This column is available only for Native Netezza Data Objects.

UPI

Indicates if the selected attribute is unique primary index.
 
Note: This column is available only for Native Teradata Data Objects.

NUPI

Indicates if the selected attribute is non-unique primary index.
 
Note: This column is available only for Native Teradata Data Objects.

USI

Indicates if the selected attribute is unique secondary index.
 
Note: This column is available only for Native Teradata Data Objects.

NUSI

Indicates if the selected attribute is non-unique secondary index.
 
Note: This column is available only for Native Teradata Data Objects.

DDL Tab

This tab displays the data distribution language (DDL) of the Data Object being created. Note: DDL is read-only. You cannot modify the script here.

View Data

This tab displays the data available for the Data Object being created.
The View Data tab will expand to show three options:

Name

Description

Refresh

Click this option to refresh the data for any latest updates.

Export

Click this option to export the data from the open Data Object to an excel sheet.

View SQL

Click this option to view the SQL query associated with the selected Data Object.

The following items are available in the tool bar available in the View Data tab:

Menu

Description

New Editor

Click this menu item to open a new window for writing custom SQL query. However, in order to create any custom query, the database, for which the SQL query needs to written, must be selected from SQL Editor window displayed by the system.

Run SQL

Click this menu item to execute the SQL query.

View Data Objects

Click this option to view the SQL query of the Data Objects associate with the selected Data Point.

Export

Click this option to export the data from the open Data Object to an excel sheet.

3.      Click the Save button to save the changes made to the Data Object or click Revert to discard the changes.
 

Copying Data Object

To create a duplicate Data Object:
1.      Click the Data Object icon ( ) on the Object explorer in the left pane. A list of all the available Data Objects is displayed.
2.      From the list of Data Objects, select the Data Object that needs to be copied. Now, select the Create Copy option ( ) from the side menu bar.
Or
Click to open the Data Object that needs to be copied. The MDI Suite displays the details of the selected Data Object to be copied. Select the Create Copy option ( )from the side menu bar.
3.      A duplicate Data Object is created with sequence number added as a suffix. The number increments by one every time a copy is made of the same Data object.
4.      Click the Save button to save the changes made to the created Data Point or click Revert to discard the changes.

Refreshing Data Object

To refresh the properties of a Data Object which is being used by a different user:
1.      Click the Data Object icon ( ) available in the Object explorer in the left pane of the dashboard.
2.      Select one or more Data Object that needs to be refreshed from the list.
3.      Click the Refresh icon ( )on the side menu bar. The MDI Suite refreshes the selected Data Object(s) to reflect the latest updates made to it.
 

Deleting Data Object

To delete a Data Object, follow the given steps:
1.      Click the Data Object icon ( ) in the Object explorer in the left pane. A list of all the available Data Objects is displayed.
2.      From the list of Data Objects, select the Data Object that needs to be deleted.Or
Click to open the Data Object that needs to be deleted. The MDI Suite displays the details of the selected Data Object.
3.      Select Delete ( ) from the side menu bar. The system strikes out the selected Data Object to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The stricken out Data Objects are deleted from the list. Click Revert to undo deleting.
 

Unlocking Data Object

When you are updating a Data Object, the MDI Suite automatically locks the Data Object on your user ID. To unlock the Data Object to enable other users to update it, use the following steps:
1.      Click the Data Object icon ( ) available in the Object explorer in the left pane. A list of all the available Data Objects is displayed.
2.      From the list of Data Objects, select the Data Object that needs to be unlocked. Now, select the Unlock icon ( ) on the side menu bar.
3.      A message is displayed confirming that the selected Data Object is unlocked successfully.
4.      Click the Close button to exit the window.
 

Creating/Modifying Data Object as Transient

You can use the following steps to create a Transient Data Object or temporary Data Object from an existing Data Object:
1.      Click the Data Object icon ( ) in the Object explorer in the left pane of the dashboard. A list of all the Data Objects is displayed.
2.      Select the Data Object which you want to convert to Transient Data Object from the list.
Or
Click to open the Data Object. On the side menu bar click the icon ( ) .
A duplicate Data Object is created but this is a Transient Data Object. The Transient Data Object has a name with a sequence number as suffix. The number increments by one every time a copy is made of the same Data Object.
3.      Click the Save button to save the changes made to the created Data Object or click Revert to discard the changes.
 

Copy as Other Data Object

You can create another Data Object of another database type from an existing Data Object. For example, you want to change a Netezza Data Object to a Teradata Data Object, use this option Copy As Other.
1.      Click the Data Object icon ( ) available in the Object explorer in the left pane of the dashboard. A list of all the available Data Objects is displayed.
2.      Select the Data Object from the list.  
Or
Click the Data Object to open the details. From the side menu bar, click . A new window Copy as Other is launched.
3.      From the list of Data Points, select the Data Point for which the Data Object needs to be copied. A duplicate Data Object is created with sequence number added as a suffix (if the Data Object is of the same database type) The number increments by one every time a copy is made of the same Data object of the same database type.
If the new copy created is of a different database type, then the name and details will remain the same and the Data Object will appear on the list with the same name but with a different symbol of the database type.
4.      Click the Save button to save the changes made to the created Data Object or click Revert to discard the changes.
 

Viewing DDL

You can use the following steps to view DDL of an existing Data Object:
1.      Click the Data Object icon ( ) in the Object explorer in the left pane of the dashboard. A list of all the available Data Objects is displayed.
2.      Select one or more Data Objects from the list.
3.      On the side menu bar, click the icon ( ). The MDI Suite displays the DDL script in a new window.
4.      Alternatively, click to open a Data Object. Click the DDL tab on the left. An example is shown below:

5.      Click Close to exit.

 

Closing Data Object

You can close the Data Object you are currently working on or close all the open Data Objects through this option.
        On the side menu bar, click the close icon ( ). The current Data Object will be closed; the page again displays the list of available Data Points.
 
        On the side menu bar, click the tab Opened Objects.

A list of all the open objects is displayed.
        Click Close All to close all the open Data Objects.
        Click Close Others to close all the objects other than the current one on the screen.

 

Showing Data Object Dependencies

To see the dependencies of a Data Object:
1.      Click the Data Object icon ( ) available in the Object explorer in the left pane of the dashboard.
2.      From the list of Data Objects, select the Data Object whose dependencies needs to be viewed. On the side menu bar, click ( )
Or
From the list of Data Objects, click to open the Data Object whose dependencies needs to be viewed. Now, click Dependencies on the side menu bar.
The dependencies associated with the selected Data Object are displayed.
The following details are available in the window:

Field/Button

Description

Dependent

Displays the name of the dependant object.

Object Type

Displays the type of dependent object.

Project

Displays the name of the project to which the selected Data Object is associated.

Last Updated By

Specifies the user who last updated the object.

Last Updated Date

Displays the date on which the dependent object was updated.

3.      Click the dependent object to view the details. You will be redirected to the object's page in the canvas area where the design and the properties are defined.  
4.      Click the Cancel button to exit the screen.
 

Working with Data Flow

A Data Flow is an ETL/ELT mapping that represents the movement of data between source(s) and target(s). These source and target definitions are linked by transformation objects, which define the rules that are applied to the data as it moves between the source(s) and target(s).
The flow of data, transformations and rules governing the transformations are defined in Data Flow work area or the canvas area. When you save the mapping that is defined in the work area, the MDI Suite automatically validates the mapping and the appropriate messages are displayed.
A Data Flow consists of the following components:
        Source – Defines the source Data Object or file being mapped.
        Target – Defines the target Data Object or file to which the source is being mapped.
        Transformation – Defines the functions that must be performed on the source data before writing the data into the target. You can use different transformations to perform different functions. Not all designs have transforms, for example when simply moving data as is between database platforms.
The Data Flow home page displays a list of all the existing Data Flows. The following general details are displayed on the screen:

Menu

Description

Data Flow Name

Specifies the name of the Data Flow.

Type

Specifies the database type.

Layer

Specifies the Layer where the Data Flow resides.

Native Data Point

Specifies the Native Data Point associated.

Last Edited By

Specifies the name of the user who last updated the selected Data Object.

Last Edited On

Specifies date and time when the selected Data Object was last updated.

 

Creating Data Flow

The very basic Data Flow design involves moving data from one table to another. The workflow to accomplish this is to define the source, define the target and link them creating a pipeline. As Data Flow designs grow in complexity, they include various transformations and can involve multiple pipelines and transformations.
1.      You can create a new Data Flow through any of the following ways:
        Object Explorer: In the Object explorer, click Home or click the Data Flow icon ( ). On the side menu bar at the right, click the New Data Flow icon ( )
        Quick Access tool bar: Click the Quick Links icon ( ) at the bottom of the side menu bar. Click the New Data Flow icon ( )
 
2.      A New Data Flow window opens prompting you to select the native Data Point using which the Data Flow will be created. Select the Data Point from the list. You will be directed to the Canvas area.
3.      The Canvas area is divided into two panes. The upper area is where you create the Data Flow and the lower pane displays the properties and the script of the Data Flow currently being created.
4.      Notice that the Object explorer on the left has one more tool bar which displays all the available transformations which you can use in your Data Flow. Click the small arrow on the tool bar to expand it.

The Log pane at the bottom displays the following properties:

General Tab

Fields

Description

Name

Specifies a name for the Data Flow being created. The field accepts only alpha, numeric, or alphanumeric characters.

Description

Specifies a description (optional) for the Data Flow being created.

Database Type

Specifies the database to which the Data Flow is associated.

Native Data Point

The name of the Data Point for which the Data Flow is created.

Last Updated By

Specifies the unique user who last updated the data flow and the date and time when the selected data flow was last updated.

Properties Tab

Fields

Description

Logging Level

Specifies the Logging Level. The options are:
        INFO
        ERROR
        WARN
        DEBUG
        TRACE

TForm Cleanup

Specifies if TForm cleanup is required. The options are:
        Yes
        No

JDBC Transaction Control

Indicates if the selected Data Flow has JDBC transaction control. The options are:
        Yes
        No

Parameters Tab

        If no parameters exist then click to add parameters.
        If you want to add parameters, click the Add Parameter icon ( ) present on the top tool bar.
The following details are available under this tab:

Field

Description

Parameter Name

Specifies the type the system-defined name of the parameter with suffix of $

Data Type

Specifies the data type such as varchar, bigint, etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Default Value

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Validity Indicator

The red dot indicates that the parameter is not valid. Once you validate the expression, the red dot disappears indicating that the expression is now valid.
- Invalid

 
        You can edit the name of any parameter by simply clicking on it.
        Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected parameter, respectively and Paste icon ( ) to paste the copied parameter. The system allows you to use only unique names for the parameter; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
        Use the up arrow ( ) or down arrow ( ) to arrange the order of the parameter sequence in the list.
        Use the Delete icon ( ) to delete the selected parameter.
        Use the Search icon ( ) to search for a specific parameter from the list.
SQL Parameters Tab
The following details are available in the SQL Parameters tab:

Field

Description

Param Name

Specifies the name of the parameter. Click the name once to modify it.

Data Type

Specifies the type of data associated to the parameter. For example,  varchar, bigint, boolean, etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Context

Specifies the context.

SQL

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Default

Specifies the default value.

Validity Indicator

The red dot indicates that the parameter is not valid. Once you validate the expression, the red dot disappears indicating that the expression is now valid.
- Invalid

Load Order Tab

You can use this tab to create user defined parameters.

Field

Description

Change Batch To

Specifies the batch. The options are:
        Parallel
        Sequential
Click the Apply Batch icon ( ) to apply the selected batch. You can use the up arrow icon ( ) or down arrow icon ( ) to arrange the load order sequence in the list.

Pipeline Order

Specifies the load order of the pipeline in Data Flow. You can modify the order using the drop-down list.

Target Order

Specifies the order of the load in the target database. You can modify the order using the drop-down list.

Target Name

Specifies the name of the target Data Objects into which the data must be loaded in order.

Runtime Properties Tab

Specify the Runtime properties of the database under this tab.

Script Tab

This tab displays the SQL query that is generated automatically by the MDI Suite for the selected Data Flow.
 
5.      Click the Save button on the side menu bar to save all the properties to the Data Flow being created.
 



 

Creating Source

A Data Flow is an ETL/ELT mapping that represents the movement of data between source(s) and target(s). These source and target definitions are linked by transformation objects, which define the rules that are applied to the data as it moves between the source(s) and target(s).
To create a Source, follow the steps given.
1.      Drag the Source icon ( ) from the Transformation tool bar on to the Canvas area.
2.      A new window opens prompting you to select the Data Point already created.
3.      Select the Data Point and the database; the source of the data and click Ok.
 
4.      The log pane displays the details of the Source Data Object.
5.      Click Save to create the Source Data Object of a Data Flow.
 

  Creating Transformation as Target

You can use the following steps to create a Transient Transformation as Target Transformation:
1.      Select the Source Transformation in the Canvas area. Now, on the side menu bar, select Create as Target.
2.      From the Create Data Object window, select the Data Point for which the target transformation needs to be created. The MDI Suite places the Target Data Object in the Canvas area linking it to the Source Data Object.
3.      Click the Save button to save the changes or click Revert to discard the changes.
 

Editing Data Flow

You can edit an already existing Data Flow to suit your requirements.
1.      Click the Data Flow icon ( ) in the Object explorer in the left pane of the dashboard. A list of Data Flows appears on the screen.
2.      Select the Data Flows that needs to be edited from the list.
The Data Flow design appears in the canvas area and you can select any object such as the Data Point or the Data Object to display its properties in the Log pane at the bottom of the page. You can change the details of these objects in the log pane.
Change or edit the Data Flow fields based on the description given below.

General Tab

Fields

Description

Name

Specifies a name for the Data Flow being created. The field accepts only alpha, numeric, or alphanumeric characters.

Description

Specifies a description (optional) for the Data Flow being created.

Database Type

Specifies the database to which the Data Flow is associated.

Native Data Point

The name of the Data Point for which the Data Flow is created.

Last Updated By

Specifies the unique user who last updated the data flow and the date and time when the selected data flow was last updated.

Properties Tab

Fields

Description

Logging Level

Specifies the Logging Level. The options are:
        INFO
        ERROR
        WARN
        DEBUG
        TRACE

TForm Cleanup

Specifies if TForm cleanup is required. The options are:
        Yes
        No

JDBC Transaction Control

Indicates if the selected Data Flow has JDBC transaction control. The options are:
        Yes
        No

Parameters Tab

        If no parameters exist then click to add parameters.
        If you want to add parameters, click the Add Parameter icon ( ) present on the top tool bar.
The following details are available under this tab:

Field

Description

Parameter Name

Specifies the type the system-defined name of the parameter with suffix of $

Data Type

Specifies the data type such as varchar, bigint, etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Default Value

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Validity Indicator

The red dot indicates that the parameter is not valid. Once you validate the expression, the red dot disappears indicating that the expression is now valid.
- Invalid

 
3.      You can edit the name of any parameter by simply clicking on it.
4.      Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected parameter, respectively and Paste icon ( ) to paste the copied parameter. The system allows you to use only unique names for the parameter; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
5.      Use the up arrow ( ) or down arrow ( ) to arrange the order of the parameter sequence in the list.
6.      Use the Delete icon ( ) to delete the selected parameter.
7.      Use the Search icon ( ) to search for a specific parameter from the list.
8.      Click Save to save the changes made to the created Data Flow or click Revert to discard the changes.
SQL Parameters Tab
The following details are available in the SQL Parameters tab:

Field

Description

Param Name

Specifies the name of the parameter. Click the name once to modify it.

Data Type

Specifies the type of data associated to the parameter. For example,  varchar, bigint, boolean, etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Context

Specifies the context.

SQL

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Default

Specifies the default value.

Validity Indicator

The red dot indicates that the parameter is not valid. Once you validate the expression, the red dot disappears indicating that the expression is now valid.
- Invalid

Load Order Tab

You can use this tab to create user defined parameters.

Field

Description

Change Batch To

Specifies the batch. The options are:
        Parallel
        Sequential
Click the Apply Batch icon ( ) to apply the selected batch. You can use the up arrow icon ( ) or down arrow icon ( ) to arrange the load order sequence in the list.

Pipeline Order

Specifies the load order of the pipeline in Data Flow. You can modify the order using the drop-down list.

Target Order

Specifies the order of the load in the target database. You can modify the order using the drop-down list.

Target Name

Specifies the name of the target Data Objects into which the data must be loaded in order.

Runtime Properties Tab

Specify the Runtime properties of the database under this tab.

Script Tab

This tab displays the SQL query that is generated automatically by the MDI Suite for the selected Data Flow.
Click the Save button on the menu bar to save all the properties to the Data Flow being created or click Revert to undo the changes.
 

Refreshing Data Flow

You can use the following steps to refresh the properties of a Data Flow:
1.      Click the Data Flow icon ( ) in the Object explorer in the left pane of the dashboard. A list of Data Flows appears on the screen.
2.      Select the Data Flow that needs to be refreshed from the list.
Or
Click the Data Flow to open the details.
3.      Now, click the icon ( ) on the side menu bar. The MDI Suite refreshes the selected Data Flow.
 

Deleting Data Flow

To delete a Data Flow, follow the given steps:
1.      Click the Data Flow icon ( ) in the Object explorer in the left pane. A list of all the available Data Points is displayed.
2.      From the list of Data Flows, select one or more Data Flows that need to be deleted.Or
Click to open the Data Flow that needs to be deleted. The MDI Suite displays the details of the selected Data Flow.
3.      From the side menu bar, click the icon ( ). The system strikes out the selected Data FLow to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The stricken out Data Flows are deleted from the list. Click Revert to undo deleting.
 

Copying Data Flow

To create a duplicate of a Data Flow established with the database:
1.      Click the Data Flow icon ( ) on the Object explorer in the left pane. A list of all the Data Flows appears on the screen.
2.      From the list of Data Flows, select the Data Flow that needs to be copied. Now, click the Create Copy ( )option on the side menu bar.  
Or
Click to open the Data Flow that needs to be copied. The MDI Suite displays the details of the selected Data Flow to be copied. Select the Create Copy option from side menu bar.
3.      A duplicate Data Flow is created with a sequence number added as a suffix. The number increments by one every time a copy is made of the same Data Flow.
4.      Click the Save button to save the changes made to the created Data Flow or click Revert to discard the changes.
 

Validating Data Flow

To validate a Data Flow:
1.      Click the Data Flow icon ( ) on the Object explorer in the left pane. A list of all the Data Flows appears on the screen.
2.      From the list of Data Flows, open the Data Flow that needs to be validated. The MDI Suite displays the details of the selected Data Flow to be validated. Select the Validate ( ) option from the side menu bar.
3.      Click the Save button to save the changes made to the created Data Flow or click Revert to discard the changes.

Executing Data Flow

To execute a Data Flow:
1.      Click the Data Flow icon ( ) on the Object explorer in the left pane. A list of all the Data Flows appears on the screen.
2.      From the list of Data Flows, open the Data Flow that needs to be executed. The MDI Suite displays the details of the selected Data Flow to be executed. Select the Run icon ( ) on the side menu bar.
3.      A Confirmatory message stating that the selected data flow is executed successfully is displayed in the bottom pane below the canvas area.

4.      To check the execution status, click the link (View Execution Status). You will be directed to the monitor module and the detailed execution status can be seen.
5.      Click the Save button to save the changes made to the created Data Flow or click Revert to discard the changes.
 

Unlocking Data Flow

When you are updating a Data Flow, the MDI Suite automatically locks the Data Flow on your user ID. To unlock the Data Flow to enable other users to update it, use the following steps:
1.      Click the Data Flow icon ( ) on the Object explorer in the left pane. A list of all the available Data Flows is displayed.
2.      From the list of Data Flows, select the Data Flow that needs to be unlocked. Now, select the Unlock option ( ) from side menu bar at the right.
 
3.      A message is displayed confirming that the selected Data Flow is unlocked successfully.
4.      Click the Close button to exit the window.

 

Showing Data Flow Dependencies

You can use this option to view the dependent objects to the selected Data Flow.
1.      Click the Data Flow icon ( ) on the Object explorer in the left pane. A list of all the available Data Flows is displayed.
2.      From the list of Data Flows, select to open the Data Flow for which the dependencies needs to be viewed. On the side menu bar, click
The following details are available in the window:

Field/Button

Description

Dependent

Displays the name of the dependant object.

Object Type

Displays the type of dependent object.

Project

Displays the name of the project to which the selected data flow is associated.

Last Updated By

Specifies the unique identifier of the user who last updated.

Last Updated Date

Displays the date on which the dependent object was updated.

3.      Select the dependent object and click Open to open the dependent or click Cancel to exit the screen.
 

Arranging Data Flow

While you are creating a Data Flow, it may not look quite neat and organized. To arrange the design:
        Click the Arrange option ( ) available in the side menu bar. The MDI Suite arranges all the Data Object instances and Transformations used in the Data Flow in a neat, organized manner.
 


Closing Data Flow

You can close the Data Flow you are currently working on or close all the open Data Flows through this option.
        On the side menu bar, click the close icon ( ). The current Data Flow will be closed; the page again displays the list of available Data Flows.
 
        On the side menu bar, click the tab Opened Objects.

A list of all the open objects is displayed.
        Click Close All to close all the open Data Flows.
        Click Close Others to close all the objects other than the current one on the screen.
 

Working with Transformations

A transformation is a data manipulation instruction that changes something about the data as it moves from source to target. Transformations become part of the design pipeline and are typically used to do things like join, split, filter, rearrange, compare, calculate, rollup, or track the history of data changes, as it moves from source to target.
Transformations define the functions that must be performed on the source data before writing the data into the target. You can use different transformations for various types of data manipulation. The purpose of these objects is to essentially write the SQL code required for data transformation. Multiple transformations can be used in a single design to incrementally manipulate data, the individual steps can be staged as needed.
 

Icon

Transform

Description

Source

This is a dummy object instance created based on the user requirement, the attributes of different data types. When similar Design logic is required, Source can be transformed along with the Design.

Target

This is a dummy object instance created based on the user requirement. The attributes available in the Out transformation need not be the same as in Source. However, it is expected to have the same data type of the attributes that are required to be moved from Source to Target. In this case, the data movement is managed by mapping the Source and Target attributes manually.

Expression

Performs calculations such as addition, subtraction, multiplication, division, and uppercase and lowercase conversions. The Expression transformation can only be linked to one source and one target.

Joiner

Combines data from two source transformations based on the common relationship
between one or more attributes. It also allows you to join data from the same source
transformation. The output of the Joiner is a single result set which is inserted
into the target transformation.

Rollup

Performs calculations such as average and sum on the selected attributes through grouping.

Filter

Queries a set of data from a source transformation that meets the specified filter condition. The data can be filtered based on one or more condition. The data that meets the specified filter conditions are allowed to be inserted into the target transformation. The data that does not meet the filter conditions are dropped.

Temp Stage

Acts as temporary tables for the intermediate transformations in a mapping instance. Will be rebuilt for every run – data in a TStage table will not be persisted.

Splitter

Splits the source data into one or more groups in the data flow. You can also use this transformation to create identical copies of the source.

VennGroup

Combines data from multiple sources and produce a single output. All the sources and the result set must have matching attributes. The data type, precision, and scale must be identical across the attributes.

Data SubFlow

These are reusable transformation designs which you can simply drag to the canvas area in our design.

 
To create or insert a Transformation in any Data Flow:
1.      Click the Data Flow icon ( ) on the Object explorer in the left pane of the dashboard.
2.      Click a Data Flow to open it. The MDI Suite displays the details of the selected Data Flow in the canvas area.
3.      Drag and drop the transformation you require from the transformation tool bar displayed on the left side of the work area.

Editing Transformations

Once you bring in the transformation in the canvas area, you need to edit the particular transformation to suit your requirements. Almost all transform objects will be modified to introduce the appropriate criteria to the "WHERE" clause of the data pipeline. Click the links given below to edit the respective transformations.
        Source
        Target
        Expression
        Joiner
        Rollup
        Filter
        Temp Stage
        Splitter
        VennGroup

Editing Input and Output in a Data SubFlow

        Editing Input
        Editing Output
 

Editing Expression

You can condition an expression to suit your requirements in the canvas area. To edit an expression:
        Click the Expression Transformation icon ( ) to be edited in the work area. The details of the Expression transformation can be seen in the lower pane at the bottom.

General Tab

Edit or modify the Expression transformation based on the description of the fields given below.

Field/Button

Description

Name

Specifies the name of the Temp Stage transformation. You can edit the name of the transformation, if required.

Description

Specifies the information related to the selected transformation.

Persist Data

Specifies if the data persists.

Last Updated By

Specifies the user who last updated the selected transformation and when it was last updated.

Attributes Tab
  • Click the Add Attribute ( ) icon. Edit the attributes based on the description given below.

    Field/Button

    Description

    Attribute Name

    Specifies the Attribute name. Click the system-inserted column name and type a name for the attribute to be derived.

    Data Type

    Specifies the data type for the attribute and are available under the drop-down list.

    Prec

    Specifies the maximum number of characters in a string or digits in a number to be accommodated for the selected attribute using the drop-down list.

    Scale

    Specifies the maximum number of digits to be accommodated after the decimal point in a number using the drop-down list.

    Order

    Specifies if the sorting of the result-set is based on the selected attribute. The attributes are sorted in ascending order by default.

    Expression Editor ( )

    Click the Expression Editor icon ( ) to open the Expression Editor window to create a condition for the attributes to be derived. For details on creating the expression, see the Expression Editor section.

    Validity Indicator

    The red dot indicates that the attribute is not yet validated successfully.
    - Invalid

    1.      You can edit the name of any attribute by simply clicking it.
    2.      Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected attribute, respectively and Paste icon ( ) to paste the copied attribute.                          Note: The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
    3.      Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
    4.      Use the Delete icon ( ) to delete the selected attribute.
    5.      Use the Search icon ( ) to search for a specific attribute from the list.
    Script Tab
    This tab displays the SQL query that is generated automatically by the MDI Suite for the selected transformation. The Script will be displayed only when all the attributes are validated.
    Data Tab
    This tab displays the data used in the selected transformation.

    Button

    Value

    Refresh

    Refreshes the data in the selected object.

    Refresh All

    Refreshes all the data of the objects.

    Export

    Exports the data into an .xlsx format.

    As soon as you click Data, the following message is displayed.

    Click Run Status if you want to see the status of your data being fetched. The screen would then look like this-

    The description of each of the columns is given below:

    Field/Button

    Value

    Stop

    Stops the data extract.

    Abort

    Aborts the data extract process.

    Name

    Specifies the name of the data objects available in the selected data flow.

    Status

    Indicates the status of the extract.

    Extract

    Specifies the number of extracts.

    Process

    Specifies the number of processes.

    Load

    Specifies the number of rows loaded.

    Bad

    Specifies the number of bad rows.

    Run By

    Specifies the name of the user who executed the data flow.

    Start Date

    Specifies the date and time when the execution began.

    End Date

    Specifies the date and time when the execution completed.

    (Get Log)

    Displays the log of the Data activity in a new window.

    6.      Click the Save button to save the changes made to the created Expression transformation.
     

    Editing Rollup

    You can edit the Rollup transformation as described below.
    1.      Click the Rollup Transformation ( ) to be edited in the work area. The MDI Suite displays the details of the selected transformation in the lower pane.
    General Tab
    The following details are available in the General tab:

    Field/Button

    Description

    Name

    Specifies the name of the Temp Stage transformation. You can edit the name of the transformation, if required.

    Description

    Specifies the information related to the selected transformation.

    Persist Data

    Specifies if the data persists.

    Last Updated By

    Specifies the user who last updated the selected transformation and when it was last updated.

    Group Attributes Tab
    2.      Click the Click Here button to add a group attribute. The MDI Suite displays Select Attributes window.
    3.      Select the attributes to be rolled up from the respective source objects and click the OK button to save the changes. The MDI Suite displays the selected attributes.

    Field/Button

    Description

    Alias Name

    Specifies the name of the attribute based on which the data must be grouped. You can modify the name of the attribute, if required.

    Select Column

    Specifies the names of the selected attributes.

    Data Type

    Specifies the data types of the respective attributes available in the tab.

    Prec

    Specifies the maximum number of characters in a string or digits in a number to be accommodated for the selected attribute.

    Scale

    Specifies the maximum number of digits to be accommodated after the decimal point in a number.

    Dist

    Specifies if the distribution of the attributes is based on the selected attribute. This column is available only for Native Netezza data objects.

    Org

    Specifies if the organization is based on the selected attribute. You can select up to four columns based on which the table is organized. This column is available only for Native Netezza data objects.

    Order

    Specifies if the sorting of the result-set is based on the selected attribute. The attributes are sorted in ascending order by default. This column is available only for Native Netezza data objects.

    UPI

    Specifies if the selected attribute is unique primary index. This column is available only for Native Teradata data objects.

    NUPI

    Specifies if the selected attribute is non-unique primary index. This column is available only for Native Teradata data objects.

    USI

    Specifies if the selected attribute is unique secondary index. This column is available only for Native Teradata data objects.

    NUSI

    Specifies if the selected attribute is non-unique secondary index. This column is available only for Native Teradata data objects.

    Part

    Specifies if the partition of the attributes is based on the selected attribute. This column is available only for Native Hive data objects.

            You can edit the name of any attribute by simply clicking it.
            Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected attribute, respectively and Paste icon ( ) to paste the copied attribute.                          The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
            Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
            Use the Delete icon ( ) to delete the selected attribute.
            Use the Search icon ( ) to search for a specific attribute from the list.
    Attributes Tab
    4.      Click the Add Attribute ( ) icon. The MDI Suite inserts a row into the Attributes tab.

    Field/Button

    Description

    Attribute Name

    Click the system-inserted column name and type a name for the attribute to be derived.

    Data Type

    Specifies the data type for the attribute using the drop-down list.

    Prec

    Specifies the maximum number of characters in a string or digits in a number to be accommodated for the selected attribute using the drop-down list.

    Scale

    Specifies the maximum number of digits to be accommodated after the decimal point in a number using the drop-down list.

    Order

    Specifies if the sorting of the result-set is based on the selected attribute. The attributes are sorted in ascending order by default.

    Rollup Expression

    Click the Expression Editor icon ( ) to open the Expression Editor window to create a rollup condition for the attributes to be derived. For details on creating the expression, see the Expression Editor section.

    Validity Indicator

    The red dot indicates that the attribute is not yet validated successfully.
    - Invalid

            You can edit the name of any attribute by simply clicking it.
            You can unlink the attributes using the icon ( ).
            You can automatically map the attributes using the Auto Map icon ( ).
            Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected attribute, respectively and Paste icon ( ) to paste the copied attribute.                          The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
            Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
            Use the Delete icon ( ) to delete the selected attribute.
            Use the Search icon ( ) to search for a specific attribute from the list.
    Script Tab
    This tab displays the SQL query that is generated automatically by the MDI Suite for the selected transformation.
    Data Tab
    This tab displays the data available in the selected transformation.
    Data Tab

    Button

    Value

    Refresh

    Click this button to refresh the data in the selected object.

    Refresh All

    Click this button to refresh all the objects.

    Export

    Click this button to export the data into an .xlsx format.

    Run Status
    The following details are available in this window:

    Field/Button

    Value

    Stop

    Click this button to stop the data extract.

    Abort

    Click this button to abort the data extract process.

    Name

    Specifies the name of the Data Objects available in the selected Data Flow.

    Status

    Specifies the status of the extract.

    Extract

    Specifies the number of extracts.

    Process

    Specifies the number of processes.

    Load

    Specifies the number of rows loaded.

    Bad

    Specifies the number of bad rows.

    Run By

    Specifies the name of the user who executed the Data Flow.

    Start Date

    Specifies the date and time when the execution began.

    End Date

    Specifies the date and time when the execution completed.

    (Get Log)

    Specifies the log associated to the Data Flow execution.

     
    5.      Click the Save button to save the changes made to the created Expression transformation or click Undo to exit the window without saving the changes.
     

    Editing Joiner

    A Joiner transformation is used to combine data from two sources based on the common relationship between one or more attributes.  
    To edit a Joiner:
    1.      Click the Joiner Transformation icon ( ) to be edited in the work area. The MDI Suite displays the details of the selected transformation in the bottom pane.
    General Tab
    Edit the fields according to the description given below:

    Field/Button

    Description

    Name

    Specifies the name of the Temp Stage transformation. You can edit the name of the transformation, if required.

    Description

    Specifies the information related to the selected transformation.

    Persist Data

    Specifies if the data persists.

    Last Updated By

    Specifies the user who last updated the selected transformation and when it was last updated.

    Properties Tab
    Provide all the necessary details under this tab according to the description of the fields given below:

    Field/Button

    Description

    Primary Object

    Specifies the Source that must be considered as primary from the drop-down list. The primary source must be considered as the LEFT source against which the secondary source is compared.

    Joiner Type

    Indicates the type of join to be applied between the primary and secondary sources from the drop-down list. The options are:
            INNER JOIN
            LEFT OUTER JOIN
            FULL OUTER JOIN
            RIGHT OUTER JOIN

    Secondary Object

    Specifies the source to be considered as secondary from the drop-down list. This source must be compared against the primary for querying data between them based on the relationship between certain attributes. Note: The secondary source automatically changes when you change the source in the Primary Object field.

    Joiner Condition

    Click the Expression Editor icon ( ) to open the Expression Editor window to create a joiner condition for the selected sources. For details on creating the expression, see the Expression Editor section.

    Validity Indicator

    The red dot indicates that the attribute is not yet validated successfully.
    - Invalid

    Where Clause

    Click the Expression Editor icon ( ) to open the Expression Editor window to provide a where clause to exclude certain data from the result set. For details on creating the expression, see the Expression Editor section.

    Attributes Tab
    2.      Click the Add Attribute ( ) icon. Edit the attributes based on the description given below.

    Field/Button

    Description

    #

    Specifies the sequence numbering.

    Attribute Name

    Specifies the Attribute name. Click the system-inserted column name and type a name for the attribute to be derived.

    Data Type

    Specifies the data type for the attribute and are available under the drop-down list.

    Prec

    Specifies the maximum number of characters in a string or digits in a number to be accommodated for the selected attribute using the drop-down list.

    Scale

    Specifies the maximum number of digits to be accommodated after the decimal point in a number using the drop-down list.

    Order

    Specifies if the sorting of the result-set is based on the selected attribute. The attributes are sorted in ascending order by default.

    Expression Editor ( )

    Click the Expression Editor icon ( ) to open the Expression Editor window to create a condition for the attributes to be derived. For details on creating the expression, see the Expression Editor section.

    Validity Indicator

    The red dot indicates that the attribute is not yet validated successfully.
    - Invalid

            You can edit the name of any attribute by simply clicking it.
            Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected attribute, respectively and Paste icon ( ) to paste the copied attribute.                          Note: The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
            Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
            Use the Delete icon ( ) to delete the selected attribute.
            Use the Search icon ( ) to search for a specific attribute from the list.
    Script Tab
    This tab displays the SQL query that is generated automatically by the Diyotta DI Suite for the selected transformation.
    Data Tab
    This tab displays the data available in the selected transformation.
    Data Tab

    Button

    Value

    Refresh

    Click this button to refresh the data in the selected object.

    Refresh All

    Click this button to refresh all the objects.

    Export

    Click this button to export the data into an .xlsx format.

    Run Status
    The following details are available in this window:

    Field/Button

    Value

    Stop

    Click this button to stop the data extract.

    Abort

    Click this button to abort the data extract process.

    Name

    Specifies the name of the Data Objects available in the selected Data Flow.

    Status

    Specifies the status of the extract.

    Extract

    Specifies the number of extracts.

    Process

    Specifies the number of processes.

    Load

    Specifies the number of rows loaded.

    Bad

    Specifies the number of bad rows.

    Run By

    Specifies the name of the user who executed the Data Flow.

    Start Date

    Specifies the date and time when the execution began.

    End Date

    Specifies the date and time when the execution completed.

    (Get Log)

    Specifies the log associated to the Data Flow execution.

    3.      Click the Save button to save the changes.
     



    Editing Filter

    You can edit the Filter transformation using the following method:
    1.      Click the Filter Transformation ( ) to be edited in the work area. The MDI Suite displays the details of the selected transformation in the bottom pane.
    General Tab
    The following details are available in the General tab:

    Field/Button

    Description

    Name

    Specifies the name of the Temp Stage transformation. You can edit the name of the transformation, if required.

    Description

    Specifies the information related to the selected transformation.

    Persist Data

    Specifies if the data persists.

    Last Updated By

    Specifies the user who last updated the selected transformation and when it was last updated.

    Properties Tab
    The following details are available in the Properties tab:

    Field/Button

    Description

    Filter Condition

    Click the Expression Editor icon ( ) to open the Expression Editor window to create a filter condition for the selected source. For details on creating the expression, see the Expression Editor section.

    Validity Indicator

    The red dot indicates that the attribute is not yet validated successfully.
    - Invalid

    Script Tab
    This tab displays the SQL query that is generated automatically by the MDI Suite for the selected transformation.
    Data Tab
    This tab displays the data available in the selected transformation.
    Data Tab

    Button

    Value

    Refresh

    Click this button to refresh the data in the selected object.

    Refresh All

    Click this button to refresh all the objects.

    Export

    Click this button to export the data into an .xlsx format.

    Run Status
    The following details are available in this window:

    Field/Button

    Value

    Stop

    Click this button to stop the data extract.

    Abort

    Click this button to abort the data extract process.

    Name

    Specifies the name of the Data Objects available in the selected Data Flow.

    Status

    Specifies the status of the extract.

    Extract

    Specifies the number of extracts.

    Process

    Specifies the number of processes.

    Load

    Specifies the number of rows loaded.

    Bad

    Specifies the number of bad rows.

    Run By

    Specifies the name of the user who executed the Data Flow.

    Start Date

    Specifies the date and time when the execution began.

    End Date

    Specifies the date and time when the execution completed.

    (Get Log)

    Specifies the log associated to the Data Flow execution.

    2.      Click the Save button to save the changes.
     

    Editing Temp Stage

    You can edit the Temp Stage transformation using the following method:
    1.      Click the Temp Stage transformation to be edited in the work area. The MDI Suite displays the details of the selected transformation in the bottom pane.
    General Tab
    The following details are available in the General tab:

    Field/Button

    Description

    Name

    Specifies the name of the Temp Stage transformation. You can edit the name of the transformation, if required.

    Description

    Specifies the information related to the selected transformation.

    Persist Data

    Specifies if the data persists.

    Last Updated By

    Specifies the user who last updated the selected transformation and when it was last updated.

    Attributes Tab
    2.      Click the Add Attribute ( ) icon. Diyotta DI Suite inserts a row into the Attributes tab.

    Field/Button

    Description

    Attribute Name

    Click the system-inserted column name and type a name for the attribute to be derived.

    Data Type

    Define the data type for the attribute using the drop-down list.

    Prec

    Specify the maximum number of characters in a string or digits in a number to be accommodated for the selected attribute using the drop-down list.

    Scale

    Specify the maximum number of digits to be accommodated after the decimal point in a number using the drop-down list.

    Order

    Indicates if the sorting of the result-set is based on the selected attribute. The attributes are sorted in ascending order by default.

    Expression

    Click the Expression Editor icon ( ) to open the Expression Editor window to create a condition for the attributes. For details on creating the expression, see the Expression Editor section.

    Validity Indicator

    The red dot indicates that the attribute is not yet validated successfully.
    - Invalid

            You can edit the name of any attribute by simply clicking it.
            Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected attribute, respectively and Paste icon ( ) to paste the copied attribute.                          The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
            Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
            Use the Delete icon ( ) to delete the selected attribute.
            Use the Search icon ( ) to search for a specific attribute from the list.
    Script Tab
    This tab displays the SQL query that is generated automatically by the MDI Suite for the selected transformation.
    Data Tab
    This tab displays the data available in the selected transformation.
    Data Tab

    Button

    Value

    Refresh

    Click this button to refresh the data in the selected object.

    Refresh All

    Click this button to refresh all the objects.

    Export

    Click this button to export the data into an .xlsx format.

    Run Status
    The following details are available in this window:

    Field/Button

    Value

    Stop

    Click this button to stop the data extract.

    Abort

    Click this button to abort the data extract process.

    Name

    Specifies the name of the Data Objects available in the selected Data Flow.

    Status

    Specifies the status of the extract.

    Extract

    Specifies the number of extracts.

    Process

    Specifies the number of processes.

    Load

    Specifies the number of rows loaded.

    Bad

    Specifies the number of bad rows.

    Run By

    Specifies the name of the user who executed the Data Flow.

    Start Date

    Specifies the date and time when the execution began.

    End Date

    Specifies the date and time when the execution completed.

    (Get Log)

    Specifies the log associated to the Data Flow execution.

    3.      Click the Save button to save the changes.
     

    Editing Splitter

    You can edit the Splitter transformation in the following way:
    1.      Click the Splitter transformation ( ) to be edited in the work area. The MDI Suite displays the details of the selected transformation in the bottom pane.
    General Tab
    The following details are available in the General tab:

    Field/Button

    Description

    Name

    Specifies the name of the Temp Stage transformation. You can edit the name of the transformation, if required.

    Description

    Specifies the information related to the selected transformation.

    Persist Data

    Specifies if the data persists.

    Last Updated By

    Specifies the user who last updated the selected transformation and when it was last updated.

    Conditions Tab
    Click the Click Here button to add a Splitter condition. The MDI Suite inserts a row into the Conditions tab.

    Field/Button

    Description

    Condition Name

    Click the system-inserted column name and type a name for the attribute based on which the dataobject must be split into two output connections.

    Splitter Condition

    Click the Expression Editor icon ( ) to open the Expression Editor window to create a splitter condition. For details on creating the expression, see the Expression Editor section.

    Validity Indicator

    The red dot indicates that the attribute is not yet validated successfully.
    - Invalid

            You can edit the name of any attribute by simply clicking it.
            Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected attribute, respectively and Paste icon ( ) to paste the copied attribute. Note: The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
            Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
            Use the Delete icon ( ) to delete the selected attribute.
            Use the Search icon ( ) to search for a specific attribute from the list.
    Script Tab
    This tab displays the SQL query that is generated automatically by the MDI Suite for the selected transformation.
    Data Tab
    This tab displays the data available in the selected transformation.
    Data Tab

    Button

    Value

    Refresh

    Click this button to refresh the data in the selected object.

    Refresh All

    Click this button to refresh all the objects.

    Export

    Click this button to export the data into an .xlsx format.

    Run Status
    The following details are available in this window:

    Field/Button

    Value

    Stop

    Click this button to stop the data extract.

    Abort

    Click this button to abort the data extract process.

    Name

    Specifies the name of the Data Objects available in the selected Data Flow.

    Status

    Specifies the status of the extract.

    Extract

    Specifies the number of extracts.

    Process

    Specifies the number of processes.

    Load

    Specifies the number of rows loaded.

    Bad

    Specifies the number of bad rows.

    Run By

    Specifies the name of the user who executed the Data Flow.

    Start Date

    Specifies the date and time when the execution began.

    End Date

    Specifies the date and time when the execution completed.

    (Get Log)

    Specifies the log associated to the Data Flow execution.

    2.      Click the Save button to save the changes.
     

    Editing VennGroup

    You can edit the VennGroup transformation in the following way:
    1.      Click the VennGroup transformation ( ) to be edited in the work area. The MDI Suite displays the details of the selected transformation in the bottom pane.
    General Tab
    The following details are available in the General tab:

    Field/Button

    Description

    Name

    Specifies the name of the Temp Stage transformation. You can edit the name of the transformation, if required.

    Description

    Specifies the information related to the selected transformation.

    Persist Data

    Specifies if the data persists.

    Last Updated By and Last Updated Date

    Specifies the user who last updated the selected transformation and when it was last updated.

    Properties Tab
    The following details are available in the Properties tab:

    Field/Button

    Description

    Primary Object <Source Transformation Name>

    Specifies the name of the primary source data object.

    Venn Type

    Select the type of transformation from the drop-down list. The valid values are:
            Union
            Union All
            Minus
            Minus All

    Secondary Object <Secondary Source Transformation Name>

    Specifies the name of the secondary source data object.

    Note: Use the up icon ( ) or down icon ( ) to arrange the order of object sequence in the list.
    Attributes Tab
    2.      Click the Click Here button to add a group attribute. The MDI Suite displays the Select Attributes window.
    3.      Select the group attribute from the list and click the OK button to save the changes. Click Cancel to exit the window without saving the changes.

    Field/Button

    Description

    Alias Name

    Click the system-inserted column name and type a name for the attribute to be derived.

    (Object Name) (Primary)

    Specifies the name of the attributes available in the primary object.

    (Secondary Object Name)

    Specifies the name of the attributes available in the secondary object.

    Validity Indicator

    The red dot indicates that the attribute is not yet validated successfully.
    - Invalid

            Click the Add Attribute ( ) icon. MDI Suite inserts a row into the Attributes tab.
            You can edit the name of any attribute by simply clicking it.
            Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected attribute, respectively and Paste icon ( ) to paste the copied attribute. Note: The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
            Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
            Use the Delete icon ( ) to delete the selected attribute.
            Use the Search icon ( ) to search for a specific attribute from the list.
    4.      Click the OK button to save the changes.
     
     
     

      Copying and Pasting Transformations

    To copy and paste a transformation of any type:
    1.      Select one or more transformations to be duplicated.
    2.      From the side menu bar, click the icon ( ). The system creates a duplicate of the selected Transformation(s).
    3.      To use the transforms in other Data Flows, use the paste transform option.
    4.      From the side menu bar, click the icon ( ). The system pastes the selected transformation in the canvas area with a sequence number added as a suffix to the name.
    5.      Click the Save button to save the changes or click Revert to undo the changes.
     

    Deleting Transformations

    You can use the following steps to delete a transformation of any type:
    1.      Select one or more transformations that needs to be deleted from the Data Flow.
    2.      From side menu bar, click . This deletes the selected Transformation.
    3.      Click the Save button to save the changes or click Revert to undo the changes.
    Note: To delete the link between two transforms, click on the link to select it and then click the icon ( ).
     

    Creating Transformation as Transient

    You can use the following steps to create a transient transformation or temporary transformation from an existing transformation:
    1.      Select the Transformation from the Canvas area. Now, from the Transform menu available in the menu bar, select Create as Transient.
    2.      From the Create Data Object window, select the Data Point for which the transient transformation needs to be created. The MDI Suite places the Transient Transformation in the Canvas area Linking it to the parent transformation.
    3.      Click the Save button to save the changes made to the created transformation or click Revert to discard the changes.
     

    Creating Transformation as Tstage

    You can use the following steps to create a Transient Transformation as Tstage transformation:
    1.      Select the Source Data Object from the Canvas area. Now, from the Transform menu available in the menu bar, select Create As Tstage.
    2.      The MDI Suite places the Tstage transformation in the work area linking it to the parent transformation.
    3.      Click the Save button to save the changes made to the created Transformation or click Revert to discard the changes.
     

    Working with Job Flow

    Once a Data Flow is created, a Job Flow needs to be created to execute the Data Flow as a job. Job Flows instantiate Data Flows and execute the data movement. A mapping within the Job Flow represents the "path" for movement of data between source and target databases. When a Data Flow is added to a Job Flow, it essentially becomes a job and must be mapped. More than one job can run in a Job Flow; this is the concept of grouping.
    The following types of jobs can be created:

    Icon

    Job

    Description

    Command Job

    This job is used to execute any external scripts such as shell or batch scripts

    Design Job

    This job is used to execute the mapping instances, that is, Design

    DBCommand Job

    This job is used to execute the DBCommand jobs

    DFSCommand Job

    This job is used to execute the HDFS Command jobs

    Loop

    Looper is used to loop through the jobs placed between the Loop Start job and Loop End job depending on the Looper conditions provided.

    Email Notification

    Email Notification is used to publish a mail to a person or a group when the job being executed undergoes a status change.

    File Transfer Job

    The File Transfer job is usually used to move files from source location to target location if they are associated to different agents.

    Creating Job Flow

    A job is created for a mapping instance. When you execute a job, the MDI Suite moves the data from a source to destination after performing the required transformations, modifications and filtering of the data as defined within the mapping instance.
    1.      You can create a new Job Flow through any of the following ways:
  • Home: In the Object explorer, click Home. On the side menu bar click ( ) icon.
  • Object Explorer: Click the Job Flow icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar click ( ).
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and click ( ).

2.      A New Job Flow window opens prompting you to select the layer where the Job Flow will be created. Select the Layer from the list. You will be directed to the Canvas area.
3.      In the bottom pane of the page, enter the details of the Job Flow being created according to the description given below.

General Tab

Fields

Description

Name

Specifies a name for the Job Flow being created. The field accepts only numeric and alphanumeric characters.

Description

Specifies a description (optional) for the Job Flow being created.

Last Updated By

Specifies the unique user who last updated the Job flow and the date and time when the selected Job flow was last updated.

Properties Tab

Fields

Description

Param File Path

Specifies the reference path of the parameter file. Click the name once to modify it.

Param File Name

Specifies the name associated to the selected Job Flow parameter. Based on reference name, you can use the Job Flow parameters in other transformations and expressions.

Logging Level

Select the type of log from the drop-down list. The options are:
        INFO
        ERROR
        WARN
        DEBUG
        TRACE

TForm Cleanup

Select if TForm cleanup is required:
        Yes
        No

Include Properties

Specifies if the properties must be included.

Show Param Template

Click to see the parameter template

Parameters Tab

Fields

Description

Param Name

Specifies the reference name of the parameter. Click the name to modify it.

Data Type

Specifies the type of data associated to the parameter. For example, char, varchar etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Default

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Validity Indicator

The red dot indicates that the attribute is not yet validated successfully.
- Invalid

SQL Parameters

Fields

Description

Param Name

Specifies the reference name of the parameter. Click the name to modify it.

Data Type

Specifies the type of data associated to the parameter. For example, char, varchar etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Data Point

Displays the Data Point once the database is selected.

Database

Specifies the database

Dependent Job Name

Specifies the dependent job status

Job Order

Specifies the job order

SQL

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Default

Specifies the default value.

Validity Indicator

The red dot indicates that the attribute is not yet validated successfully.
- Invalid

4.      Click Save to create a new Job Flow.

Deleting Job Flow

To delete a Job Flow, follow the given steps:
1.      Click the Job Flow icon ( ) on the Object explorer in the left pane. A list of all the available Job Flows is displayed.
2.      From the list of Job Flows, select the Job Flow that needs to be deleted.Or
Click to open the Job Flow that needs to be deleted. The MDI Suite displays the details of the selected Job Flow.
3.      Click the Delete option ( ) from the side menu bar. The system strikes out the selected Job Flow to indicate it as deleted.

4.      Click the Save button to save the changes. The stricken out Job Flows are deleted from the list. Click Revert to undo deleting.
 

Editing Job Flow

Follow the given steps to edit a Job Flow.  
1.      Click the Job Flow icon ( ) in the Object explorer in the left pane of the dashboard. A list of all the Job Flows appear on the screen.
2.      Click to open the Job Flow such that all the details are displayed on the screen.
3.      Edit the details of the Job Flow according to the description given below.

General Tab

Fields

Description

Name

Specifies a name for the Job Flow being created. The field accepts only  numeric and alphanumeric characters.

Description

Specifies a description (optional) for the Job Flow being created.

Last Updated By

Specifies the unique user who last updated the Job flow and the date and time when the selected Job flow was last updated.

Properties Tab

Fields

Description

Param File Path

Specifies the reference path of the parameter file. Click the name once to modify it.

Param File Name

Specifies the name associated to the selected Job Flow parameter. Based on reference name, you can use the Job Flow parameters in other transformations and expressions.

Logging Level

Select the type of log from the drop-down list. The options are:
        INFO
        ERROR
        WARN
        DEBUG
        TRACE

TForm Cleanup

Select if TForm cleanup is required:
        Yes
        No

Include Properties

Specifies if the properties must be included.

Show Param Template

Click to see the parameter template

Parameters Tab

Fields

Description

Param Name

Specifies the reference name of the parameter. Click the name to modify it.

Data Type

Specifies the type of data associated to the parameter. For example, char, varchar etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Default

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Validity Indicator

The red dot indicates that the attribute is not yet validated successfully.
- Invalid

SQL Parameters

Fields

Description

Param Name

Specifies the reference name of the parameter. Click the name to modify it.

Data Type

Specifies the type of data associated to the parameter. For example, char, varchar etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Data Point

Displays the Data Point once the database is selected.

Database

Specifies the database.

Dependent Job Name

Specifies the dependent job status.

Job Order

Specifies the job order.

SQL

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Default

Specifies the default value.

Validity Indicator

The red dot indicates that the attribute is not yet validated successfully.
- Invalid

4.      Click Save to create a new Job Flow.
 

Copying Job Flow

To create a duplicate of a Data Flow established with the database:
 
1.      Click the Job Flow icon ( ) in the Object explorer in the left pane of the dashboard. A list of all the Job Flows appear on the screen.
2.      From the list of Job Flows, select the Job Flow that needs to be copied. Now, click the Create Copy ( )option on the side menu bar.  
Or
Click to open the Job Flow that needs to be copied. The MDI Suite displays the details of the selected Job Flow to be copied. Select the Create Copy option from side menu bar.
3.      A duplicate Job Flow is created with a sequence number added as a suffix. The number increments by one every time a copy is made of the same Job Flow.
4.      Click the Save button to save the changes made to the created Job Flow or click Revert to discard the changes.
 

Arranging Job Flow

While you are creating a Job Flow, it may not look quite neat and organized. To arrange the design:
        Click the Arrange option ( ) available in the side menu bar. The MDI Suite arranges all the Data Flow instances and Transformations used in the Job Flow in a neat, organized manner.
 
 
 
 

Working with Data SubFlow

A Data Sub Flow is basically a reusable Data Flow which you may require frequently while creating Data Flows. A Data SubFlow consists of a combination of required Transforms and an input and output instead of a source or target Data Objects.
A mapping must contain the following components:
        Input – Defines the input data object or file being mapped.
        Output – Defines the output data object or file to which the input is being mapped.
        Transformation – Defines the functions that must be performed on the input data. You can use different transformations to perform different functions.
 
The following diagram illustrates a Data SubFlow concept:
 

 








Creating Data SubFlow

1.      You can create a Data SubFlow through either of the following ways:
        Object explorer: Click MORE icon ( ) available in the Object explorer. It expands to display the available options. From the list, select Data SubFlow. A list of all the existing Data SubFlows is displayed.

        Click on the side menu bar. The MDI Suite displays the New Data SubFlow window.
        Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and then click on .
2.      A new window opens prompting you to choose the database type to create a Data SubFlow.
3.      In the Log pane at the bottom of the Canvas area, enter the details according to the description given below.

General Tab

Fields

Description

Name

Specifies a name for the Data SubFlow being created. The field accepts only numeric and alphanumeric characters.

Description

Specifies a description for the Data SubFlow being created.

Reusable Dataflow

Indicates if the Data SubFlow being created is reusable.

Database Type

Specifies the database to which the Data SubFlow is associated.

Last Edited By and Last Edited On

The unique user who last updated the Data SubFlow and the date and time when the selected Data SubFlow was last updated.



Properties Tab

Fields

Description

Logging Level

Select the level form the drop-down menu. The options are:
        INFO
        ERROR
        WARN
        DEBUG
        TRACE

TForm Cleanup

Select from the drop-down list if TForm cleanup is required. The options are:
        Yes
        No

JDBC Transaction Control

Indicate if the selected data SubFlow has JDBC transaction control. The options are:
        Yes
        No

Parameters Tab

Click the Click Here button to create a new parameter if there are no parameters.
The following details are available in this tab:

Field

Description

Parameter Name

Specifies the type the system-defined name of the parameter with suffix of $.

Data Type

Specifies the data type such as varchar, bigint, etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Default Value

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Validity Indicator

The red dot indicates that the parameter is not valid. Once you validate the expression, the red dot disappears indicating that the expression is now valid.
- Invalid

4.      Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected parameter, respectively and Paste icon ( ) to paste the copied parameter. The system allows you to use only unique names for the parameter; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
5.      Use the up arrow ( ) or down arrow ( ) to arrange the order of the parameter sequence in the list.
6.      Use the Delete icon ( ) to delete the selected parameter.
7.      Use the Search icon ( ) to search for a specific parameter from the list.
8.      Click Save to create a Data SubFlow.
 

Creating Input in Data SubFlow

This is a dummy object instance created based on the user requirement. When similar Data Flow logic is required repeatedly, a Data SubFlow is created which can be reused. Just as in a Data Flow a Source is required, while creating a Data SubFlow an Input instance is required, which acts as a dummy Source.
To create an Input:
1.      Click MORE icon ( ) available in the Object explorer. It expands to display the available options. From the list, select Data SubFlow. The MDI Suite displays the New Data SubFlow window.
2.      Select the database type to create a new Data SubFlow.
3.      Drag the Input symbol ( ) on the tool bar to the canvas area.
4.      In the lower pane enter the details of the Input Source according to the description given below.

General Tab

The following details are available in the General tab:

Field/Button

Description

Name

Specifies the name of the Source transformation. You can edit the name of the transformation, if required.

Description

A brief description related to the selected transformation.

Last Updated By

Displays the user who last updated the selected transformation and the date when the selected transformation was last updated.

Attributes Tab

        Click the Add Attribute ( ) icon.

Field/Button

Description

Attribute Name

Specifies the name of the attribute. Click the Add Attribute icon ( ) to add an attribute into the existing list of attributes.
You can edit the name of the attributes by clicking the name once.

Data Type

Specifies the type of data associated to the attribute. For example, varchar, bigint, boolean, byte, byteint, char, etc.

Prec

Specifies the maximum number of characters that the selected attribute can accommodate.

Scale

Specifies the number of characters that the selected attribute can accommodate after the decimal.

5.      Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected parameter, respectively and Paste icon ( ) to paste the copied attribute. The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
6.      Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
7.      Use the Delete icon ( ) to delete the selected attribute.
8.      Use the Search icon ( ) to search for a specific attribute from the list.
9.      Click Save to save the changes made to the created Data Flow or click Revert to discard the changes.

Editing Output Target

You can edit the Output/Target transformation using the following method:
1.      Click MORE icon ( ) available in the Object explorer. It expands to display the available options. From the list, select Data SubFlow. The MDI Suite displays the New Data SubFlow window.
2.      Select the database type to create a new Data SubFlow.
3.      Double click the Output symbol ( ) on the tool bar.
4.      In the lower pane enter the details of the Output Target according to the description given below.



General Tab

The following details are available in the General tab:

Field/Button

Description

Name

Displays the name of the Target transformation. You can edit the name of the transformation, if required.

Description

Type the business information related to the selected transformation.

Persist Data

Select the check box to indicate if the data should persist.

Last Updated By

Displays the unique identity of the user who last updated the selected transformation and the date when it was last updated.

Attributes Tab

        Click the Add Attribute ( ) icon.

Field/Button

Description

Attribute Name

Specifies the name of the attribute.
You can edit the name of the attributes by clicking the name once.

Data Type

Specifies the type of data associated to the attribute. For example, varchar, bigint, boolean, byte, byteint, char, etc.

Prec

Specifies the maximum number of characters that the selected attribute can accommodate.

Scale

Specifies the number of characters that the selected attribute can accommodate after the decimal.

5.      Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected parameter, respectively and Paste icon ( ) to paste the copied attribute. The system allows you to use only unique names for the attribute; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
6.      Use the up arrow ( ) or down arrow ( ) to arrange the order of the attribute sequence in the list.
7.      Use the Delete icon ( ) to delete the selected attribute.
8.      Use the Search icon ( ) to search for a specific attribute from the list.

Output Mapper Tab

Displays the target attributes that must be mapped to the input attributes.
 
9.      You can automatically unlink the input and target attributes using the Unlink Target Attribute icon ( ).
10.   You can automatically map the input source and output target attributes using the Auto Map icon ( ).
This tab has the following details:

Field/Button

Description

Attribute Name

Specifies the name of the attribute.
You can edit the name of the attributes by clicking the name once.

Data Type

Specifies the type of data associated to the attribute. For example, varchar, bigint, boolean, byte, byteint, char, etc.

Input Attribute

Select the input attributes to be mapped to the target attribute.

Input Data Type

Displays the data type of the input attribute.

Validity Indicator

The red dot indicates that the attribute mapping is not valid. Once you map the attribute, the red dot disappears indicating that the attributes are now mapped successfully.

  • Invalid

11.   Now, click the Save button to save the changes or click the Close button to exit the window without saving the changes.
 

Editing Data SubFlow

You can edit a Data SubFlow through either of the following ways:
        Object explorer: Click MORE icon ( ) available in the Object explorer. It expands to display the available options. From the list, select Data SubFlow. A list of all the existing Data SubFlows is displayed.

1.      Click to open the Data SubFlow you want to edit.
2.      In the Log pane at the bottom of the Canvas area, modify/edit the details according to the description given below.

General Tab

Fields

Description

Name

Specifies a name for the Data SubFlow being created. The field accepts only numeric and alphanumeric characters.

Description

Specifies a description for the Data SubFlow being created.

Reusable Dataflow

Indicates if the Data SubFlow being created is reusable.

Database Type

Specifies the database to which the Data SubFlow is associated.

Last Edited By and Last Edited On

The unique user who last updated the Data SubFlow and the date and time when the selected Data SubFlow was last updated.

Properties Tab

Fields

Description

Logging Level

Select the level form the drop-down menu. The options are:
        INFO
        ERROR
        WARN
        DEBUG
        TRACE

TForm Cleanup

Select from the drop-down list if TForm cleanup is required. The options are:
        Yes
        No

JDBC Transaction Control

Indicate if the selected data SubFlow has JDBC transaction control. The options are:
        Yes
        No





Parameters Tab

Click the Click Here button to create a new parameter if there are no parameters.
The following details are available in this tab:

Field

Description

Parameter Name

Specifies the type the system-defined name of the parameter with suffix of $.

Data Type

Specifies the data type such as varchar, bigint, etc.

Prec

Specifies the maximum number of characters that the selected parameter can accommodate.

Scale

Specifies the number of characters that the selected parameter can accommodate after the decimal.

Default Value

Specifies the expression that is associated to the selected Data Flow. Click the Expression Editor icon ( ) to open the Expression Editor window. For details, see Expression Editor.

Validity Indicator

The red dot indicates that the parameter is not valid. Once you validate the expression, the red dot disappears indicating that the expression is now valid.
- Invalid

3.      Use the Cut icon ( ) or the Copy icon ( ) to cut or copy the selected parameter, respectively and Paste icon ( ) to paste the copied parameter. The system allows you to use only unique names for the parameter; so, if the copy-paste function is used, the name will need to be adjusted to be unique.
4.      Use the up arrow ( ) or down arrow ( ) to arrange the order of the parameter sequence in the list.
5.      Use the Delete icon ( ) to delete the selected parameter.
6.      Use the Search icon ( ) to search for a specific parameter from the list.
7.      Click Save to save the changes or click Revert to undo the changes.
 
 
 


Refreshing Data SubFlow

You can use the following steps to refresh the properties of a Data SubFlow:
1.      Click MORE icon ( ) available in the Object explorer. It expands to display the available options.
2.      Select Data SubFlow. A list of all the Data SubFlows is displayed.
3.      Select the Data SubFlow that needs to be refreshed from the list.
Or
Click the Data SubFlow to open the details.
4.      Now, click the Refresh icon ( )available in the side menu bar. The selected Data SubFlow is refreshed.
 

Deleting Data SubFlow

Follow the given steps to delete a Data SubFlow permanently from a project:
1.      Click the MORE icon ( ) in the Object explorer to view a list of other options. Select Data SubFlow. A list of all the Data Sub Flows is displayed.
2.      Select the Data SubFlow that needs to be deleted and from the side menu bar click . Or
3.      Click to open the Data SubFlow that needs to be deleted. The details of the selected Data SubFlow appear on the screen.
4.      Click the delete icon ( ) on the side menu bar. The deleted Data SubFlow is stricken out.

5.      Click Save and you will see that the deleted Data SubFlow is deleted from the list or click Revert to undo deleting.
 
 

Copying Data SubFlow

You can use this option to copy the details of the data SubFlow.
1.      Click MORE icon ( ) available in the Object explorer. It expands to display the available options.
2.      Select Data SubFlow. A list of all the available Data SubFlows is displayed.
3.      From the list of Data SubFlows, select the Data SubFlow that needs to be copied. Now, click the Create Copy ( )icon on the side menu bar.  
Or
Click to open the Data SubFlow that needs to be copied. The MDI Suite displays the details of the selected Data SubFlow to be copied. Select the Create Copy option from side menu bar.
4.      A duplicate Data SubFlow is created with a sequence number added as a suffix. The number increments by one every time a copy is made of the same Data SubFlow.
5.      Click the Save button to save the changes made to the created Data Flow or click Revert to discard the changes.
 

Validating Data SubFlow

You can use the following step to validate the Data SubFlow:
1.      Click the MORE icon ( ) in the Object explorer to view a list of other options. Select Data SubFlow. A list of all the Data Sub Flows is displayed.
2.      Click to open the Data SubFlow that needs to validated.
3.      Select the Validate ( ) option from the side menu bar.
4.      Click the Save button to save the changes made to the created Data Flow or click Revert to discard the changes.
  

Unlocking Data SubFlow

When you are updating a Data SubFlow, the MDI Suite automatically locks the Data SubFlow on your user ID. To unlock the Data SubFlow to enable other users to update it, use the following steps:
1.      Click the MORE icon ( ) in the Object explorer to view a list of other options. Select Data SubFlow. A list of all the Data Sub Flows is displayed.
2.      Click to open the Data SubFlow that needs to unlocked.
3.      From the list of Data SubFlows, select the Data SubFlow that needs to be unlocked. Now, select the Unlock option ( ) from side menu bar at the right.
4.      A message is displayed confirming that the selected Data SubFlow is unlocked successfully.
5.      Click the Close button to exit the window.
 

Closing Data SubFlow

You can close the Data SubFlow you are currently working on or close all the open Data SubFlows through this option.
        On the side menu bar, click the close icon ( ). The current Data SubFlow will be closed; the page again displays the list of available Data SubFlows.
 
        On the side menu bar, click the tab Opened Objects.

A list of all the open objects is displayed.
        Click Close All to close all the open Data SubFlows.
        Click Close Others to close all the objects other than the current one on the screen.
 

Showing Data SubFlow Dependencies

You can use this option to view the dependent objects of the selected Data SubFlow.
1.      Click the MORE icon ( ) in the Object explorer to view a list of other options. Select Data SubFlow. A list of all the Data Sub Flows is displayed.
2.      From the list of Data Flows, select the Data Flow for which the dependencies need to be viewed. On the menu bar, beside Filter, click the three expansion dots. The menu will expand to show the available options. Select Dependencies from the drop-down list.
3.      Click to open the Data SubFlow. The details are displayed. On the side menu bar, click
The following details are available in the window:

Field/Button

Description

Dependent

Displays the name of the dependant object.

Object Type

Displays the type of dependent object.

Project

Displays the name of the project to which the selected data SubFlow is associated.

Last Updated By

Specifies the unique identifier of the user who last updated.

Last Updated Date

Displays the date on which the dependent object was updated.

4.      Click Open to open and view the details the dependency or click Cancel to exit the screen.
 

Arranging Data SubFlow

While you are creating a Data Flow, it may not look quite neat and organized. To arrange the design:
        Click the Arrange option ( ) available in the side menu bar. The MDI Suite arranges all the Data Object instances and Transformations used in the Data SubFlow in a neat, organized manner.
 

Working with Sequence

The Sequence option available in the MDI Suite allows you to create parameters. These parameters are appropriately used to create unique values for primary keys used in tables.
You can use the following steps to work with Sequence:
1.      Click the MORE icon ( ) available in the Object explorer to view the list of other options.
2.      From the list, select Sequence. The MDI Suite displays the Sequence window.

 

Creating Sequence

You can use the following steps to create a sequence:
1.      Click the MORE icon ( ) available in the dashboard to view the list of other options.
2.      From the MORE list, select Sequence. The MDI Suite displays the Sequence window.
3.      Click on the side menu bar.
4.      A new window opens. Select the database type for which the sequence needs to be created.

General Tab

The following details need to be provided in the General tab:

Field

Description

Name

Type a name for the sequence being created.

Description

Type the description for the sequence being created.

Group Name

Specifies the group to which the sequence being created is associated.

Database Type

Specifies the database type to which the sequence being created is associated.

Last Updated By

The name of the user who last updated the selected sequence and the date and time when it was updated.

Properties Tab

The following details are available in the Properties tab:

Field

Description

Name

The name of the sequence.

Parameter Name

The name of the parameter associated to the database.

5.      Click the Save button in the menu bar to save the details.

 

Editing Sequence

You can use this option to edit the details of the sequence..
1.      Click the MORE icon ( ) available in the dashboard to view the list of other options.
2.      From the list, select Sequence. The MDI Suite displays the Sequence window.
3.      Click to open the Sequence to be edited. The MDI Suite displays the details of the selected sequence.

General Tab

The following details need to be provided in the General tab:

Field

Description

Name

Type a name for the sequence being created.

Description

Type the description for the sequence being created.

Group Name

Specifies the group to which the sequence being created is associated.

Database Type

Specifies the database type to which the sequence being created is associated.

Last Updated By

The name of the user who last updated the selected sequence and the date and time when it was updated.

Properties Tab

The following details are available in the Properties tab:

Field

Description

Name

The name of the sequence.

Parameter Name

The name of the parameter associated to the database.

4.      Click the Save button on the menu bar to save the details.

Deleting Sequence

You can delete an existing sequence from the project using the following steps:
 
1.      Click the MORE icon ( ) available in the Object explorer to view the list of other options.
2.      From the list, select Sequence. The MDI Suite displays the Sequence window with the list of available sequences.
3.      Select the sequence to be deleted from the list.
Or
Click to open the sequence to view the details.
4.      From the side bar menu, select Delete. The MDI Suite marks the selected sequence as deleted.
5.      Click the Save button to delete the sequence.

 

Working with Expression

An expression is a predefined formula which takes one or more arguments as input, and processes the arguments to return an output.
You can use the following steps to work with Expressions:
1.      Click the MORE icon ( ) available in the Object explorer to view the list of other options.
2.      From the list, select Expression. The MDI Suite displays the Expression window.

Creating Expression

You can create expressions using the following steps:
1.      Click the MORE icon ( ) available in the Object explorer to view the list of other options.
2.      From the list, select Expression. The MDI Suite displays the Expression window.
3.      Click on the side menu bar. New Expression window is displayed by the MDI Suite. Select the Database Type for which the expression is being created.

General Tab

The following details need to be provided in the General tab:

Field

Description

Name

The name of the expression. You can modify the name, if required.

Description

Type a description for the expression being created. This field accepts only up to 500 characters.

Group Type

The name of the group type of the expression being created.

Database Type

The type of database for which the expression is being created.

Last Updated By and Date

The name of the user who last updated the selected expression and the date and time when it was updated.

Input Attributes Tab

The following details need to be provided in the Input Attributes tab:

Field

Description

Attribute Name

The name of the attribute. Click the Add Attribute icon ( ) to add an attribute into the existing list of attributes.
You can edit the name of the attributes by clicking the name once.

Data Type

The type of data associated to the attribute.

Prec

The maximum number of characters that the selected attribute can accommodate.

Scale

The number of characters that the selected attribute can accommodate after the decimal.

4.      Use the Cut icon ( ) or Copy icon ( ) to copy the selected expression and Paste icon ( ) to paste the copied expression. The system allows you to use only unique names for the expression.
5.      Use the upper arrow icon ( ) or lower arrow icon ( ) to arrange the order of expression sequence in the list.
6.      Use the Delete icon ( ) to delete the selected expression.
7.      Use the Search icon ( ) to search for a specific expression from the list.

Properties Tab

The following details need to be provided in the Properties tab:

Field

Description

Data Type

The type of data associated to the attribute.

Prec

The maximum number of characters that the selected attribute can accommodate.

Expression

Click the Expression Editor icon ( ) to open the Expression Editor window to create a condition for the output attributes to be derived. For details on creating the expression, see the Expression Editor section.

8.      Click the Save button to save the changes.
 

Editing Expression

You can update or modify the details of an existing function using the following steps:
1.      Click the MORE icon ( ) available in the Object explorer to view the list of other options.
2.      From the list, select Expression. The MDI Suite displays the Expression window.
3.      Click to open the Expression to be edited. The MDI Suite displays the details of the expression.


General Tab

The following details need to be provided in the General tab:

Field

Description

Name

The name of the expression. You can modify the name, if required.

Description

Type a description for the expression being created. This field accepts only up to 500 characters.

Group Type

The name of the group type of the expression being created.

Database Type

The type of database for which the expression is being created.

Last Updated By and Date

The name of the user who last updated the selected expression and the date and time when it was updated.

Input Attributes Tab

The following details need to be provided in the Input Attributes tab:

Field

Description

Attribute Name

The name of the attribute. Click the Add Attribute icon ( ) to add an attribute into the existing list of attributes.
You can edit the name of the attributes by clicking the name once.

Data Type

The type of data associated to the attribute.

Prec

The maximum number of characters that the selected attribute can accommodate.

Scale

The number of characters that the selected attribute can accommodate after the decimal.

4.      Use the Cut icon ( ) or Copy icon ( ) to copy the selected expression and Paste icon ( ) to paste the copied expression. The system allows you to use only unique names for the expression.
5.      Use the upper arrow icon ( ) or lower arrow icon ( ) to arrange the order of expression sequence in the list.
6.      Use the Delete icon ( ) to delete the selected expression.
7.      Use the Search icon ( ) to search for a specific expression from the list.


Properties Tab

The following details need to be provided in the Properties tab:

Field

Description

Data Type

The type of data associated to the attribute.

Prec

The maximum number of characters that the selected attribute can accommodate.

Expression

Click the Expression Editor icon ( ) to open the Expression Editor window to create a condition for the output attributes to be derived. For details on creating the expression, see the Expression Editor section.

8.      Click the Save button to save the changes.

Deleting Expression

You can delete an existing expression from the project using the following steps:
 
1.      Click the MORE icon ( ) available in the Object explorer to view the list of other options.
2.      From the list, select Expression. The MDI Suite displays the Expression window with the list of available expressions.
3.      Select the expression to be deleted from the list.
Or
Click to open the expression to view the details.
4.      From the side bar menu, select Delete. The MDI Suite marks the selected expression as deleted.
5.      Click the Save button to delete the expression.

Working with User Defined Function

A User Defined Function (UDF) is defined as programmed routine that has certain parameters set by the user. A UDF takes zero or more input parameters and returns either a scalar value or a table.
You can use the following steps to work with UDFs:
1.      Click the MORE icon ( ) available in the Object explorer to view the list of other options.
2.      From the list, select UDF. The MDI Suite displays the UDF window.

Creating User Defined Functions

You can create user defined functions (UDF) using the following steps:
1.      Click the MORE icon ( ) available in the Object explorer to view the list of other options.
2.      From the list, select UDF. The MDI Suite displays the UDF window.
3.      Click on the side menu bar. A new window is opened. Select the Database type for which the UDF is being created.

General Tab

The following details need to be provided in this tab:

Field

Description

Name

Type the name of the UDF. This field accepts letters, numerals and all special characters.

Description

Type a description for he UDF being created.

Group Name

The name of the group for which the UDF is being created.

Database Type

The type of database for which the UDF is being created.

Last Updated By

The name of the user who last updated the selected UDF and the date and time when it was updated.

Properties Tab

The following details need to be provided in the Properties tab:

Field

Description

Class Name

The class name of the UDF being modified.

Return Type

The data type which the UDF function returns. The return data type can be changed using the drop-down menu.

Syntax

Displays the syntax of the UDF being created.

UDF Jar File Name

The name of jar file being created.

4.      Click the Save button in the tool bar to save the details.
 
 
 
 

Editing UDF

You can update or modify the details of an existing UDF using the following steps:
1.      Click the MORE ellipsis ( ) available in the Object explorer to view the list of other options and select UDF.
2.      Click to open the UDF to be edited. The MDI Suite displays the details of the UDF.

General Tab

The following details need to be provided in this tab:

Field

Description

Name

Type the name of the UDF. This field accepts letters, numerals and all special characters.

Description

Type a description for he UDF being edited.

Group Name

The name of the group for which the UDF is being edited.

Database Type

The type of database for which the UDF is being edited.

Last Updated By

The name of the user who last updated the selected UDF and the date and time when it was updated.

Properties Tab

The following details need to be provided in the Properties tab:

Field

Description

Class Name

The class name of the UDF being modified.

Return Type

The data type which the UDF function returns. The return data type can be changed using the drop-down menu.

Syntax

Displays the syntax of the UDF being modified.

UDF Jar File Name

The name of jar file being modified.

3.      Click the Save button in the menu bar to save the details.



 

Deleting UDF

You can delete an existing UDF from the repository using the following steps:
1.      Select the UDF to be deleted from the list.
Or
Click to open the UDF to be deleted from the list.
2.      From the side menu bar, select Delete. The MDI Suite marks the selected UDF as deleted.
3.      Click the Save button to save the changes.
 

Parent Object

This option gives you the ease of opening the object contained within the Data Object, Data Flow etc.
 
1.      Click to open a Data Object, Data Flow or a Job Flow. The details appear in the Canvas area.
2.      Now, the Data Object contains a Data Point; a Data Flow contains objects like the Source, Target; Job Flow contains Data Flows.
3.      In case of Data Object, select the Data Point and click the ( ) option on the side menu bar. The details of the Data Point are displayed.
4.      Similarly, in case of Data Flow, select the source object and click ( ). The details of the Source Data Object are displayed.
You can edit/modify these objects and click Save or click Revert to undo the changes.
 

Export Object

You can export any object related to a project; may it be a Data Point, Data Object, Data Flow from the Integration module or a Calendar, Task or File event from the Scheduler module and save it locally.
The MDI Suite exports the object in a file format with the extension .json.


To export any object from the MDI Suite, follow the steps given below:
1.      In the Integration module, click the Data Point icon ( ), the Data Object icon ( ) or the Data Flow icon ( )-depends on what you want to export, in the Object explorer in the left pane. A list of all the available Data Points/Data Objects/Data Flows is displayed.
2.      On the side menu bar, in the Integration module, click ( )
3.      If you want to export a Task, a Calendar or any other object from the Scheduler module, click the Tasks, Calendars, Email Actions, or the File events icon in the left pane.
4.      On the side menu bar, select the Export option. The MDI Suite displays the Export Object window. The following details are displayed in this window:

Field/Button

Description

Export Object Names

Displays the name of the object being exported.

Type

Displays the type of the object being exported. For example, Data Point, Data Object.

Save

Saves the selected object in the .json format.

Cancel

Discards exporting and exits the window.

6.      Click Save and the Save As dialog box opens for you to save the file at the desired location.
 

Import Object

You can import an object such as a Data Point, Data Object or Data Flow saved locally into the MDI Suite in the same format as it was exported. (In this case, a .json format only)
To import any object from the MDI Suite, follow the steps given below:
1.      Click the Data Point icon ( ) or the Data Object icon ( ) depends on what you want to import, in the Object explorer in the left pane. A list of all the available Data Points/Data Objects is displayed.
2.      On the side menu bar, click ( ) icon.
This launches the Import Object dialog box. Browse to the location to select the object to be imported. Once you select the object to be imported, a preview of the object can be seen.
3.      Click Next. This launches another window which displays the details of the object. Under Name- the name of the project to which it originally belonged to is displayed. Under Mapped to- the name of the project in which it is imported is displayed. For example in this case, the object originally belonged to the project test_project and now is mapped or imported to the project TEST_NEW. You can choose another project by clicking the tab- Mapped to.

4.      Click Next to import all the associated objects. The import wizard will guide you in importing the Data Points and objects associated with the object to be imported. You can edit the name of the Data Point or choose not to import. You can set the preferences by clicking the Options. The available options are-
        Import- Select this option to import the object.
        Rename - Select this option to rename the object name.
        Ignore - Select this option to ignore the object. You cannot ignore an important Data Point or Data Object associated with the object.
5.      Click Finish once you finish importing all objects. You can see the imported object now in the list of Data Objects if you have imported a Data Object or you can see the imported Data Point in the list of available Data Points.
Note: Click the option Import with Default to import the object with default settings.
Importing a Data Object is different from importing any other object in .json format.
 





Working with SQL Editor

SQL Editor available in MDI Suite enables you to write new SQL queries for the selected database or edit an existing query.
        Click the SQL Editor icon ( ) available in the left pane of the dashboard. The MDI Suite displays the SQL Editor window by default where you can write new queries.
The following menu options are available in this window:

Field

Description

New Editor

Click this button on the menu bar to create SQL queries for the selected database.
From the SQL Editor window, select the type of database for which the SQL query needs to be written.

Run SQL

After the query is developed, click this button to execute it.

View Data Object

Click this button to view the Data Objects associated to the selected database.

Export

Click this button to export the data available within the selected Data Object into .xlsx document.

 

Admin

The Admin module is exclusively offered for management and administrative purposes to control projects, users, hosts, and add flexibility to monitor user work and server logs. The Admin provides role-based security to manage projects in a multi-team environment.
In addition, the Admin module is used to enable connectors to integrate source and target systems.
All maintenance tasks such as licensing, client module based maintenance such defining security roles, user access, project and application parameters, as well as server tasks such as agent setup and managing external resources, all happens in the Admin module. This is also where users can be monitored, and objects can be unlocked in the case of a multi-developer environment
 
 

Opening Admin

Log in to the MDI Suite using your login Id and password. If you have sufficient privileges, you will be directed to the Admin module home page, by default.
If you are logged into some other module such as Integration, Lineage, Monitor or Scheduler, then-
1. Click the icon ( ) at the top left corner of the page.
2. The icon will expand to display all the tiles of the available modules.

3. Click Admin to login to the Admin module.
 


 

Admin Interface Components

Once you login, the Admin home page will be displayed. The Admin home page has the following interface components:

 
The home page of the Admin module summarizes the overall status graphically.
Object Explorer - Set on the extreme left, this explorer is used to set up various configurations.
Dashboard- The dashboard is the workspace which is used to enter, modify, delete the details pertaining to a project, user, role, Agent etc.
Menu Bar - Displays different menus based on the work area. For example, Edit, Filter, Save and Revert.
 

Working with Admin

You can use the Admin tool to perform the following activities:
        Change password
        Add, modify, or delete projects
        Add, modify, or delete users
        Add, modify, or delete frictionless agents
        Add, modify, or delete roles
        Apply for server license
        Modify the settings of Diyotta server and applications such as Monitor, Scheduler, AppLogs, Netezza Specific Constants, Teradata Specific Constants, Oracle Specific Constants, DB2 Specific Constants, Hive Specific Constants, and MySQL Specific Constants
You can use Admin to create the following:
        Projects
        Users
        Roles
        Agents

 

Admin Configurations

Repository Details

You can find the Repository Details here:

Fields

Description

Repository Name

Specifies the Repository name

Description

Specifies a description (optional) of the Repository

Enable LDAP Authentication

Indicates whether LDAP Authentication is enabled

Enable Multi Organization

Indicates whether Multi Organization is enabled

Release Version

Specifies the Release version

Repository Version

Specifies the Repository version

Last Updated By

Specifies the unique user who last updated the Repository details.

 
Click Save to save the changes or click Revert to undo the changes.

  

LDAP Configuration

Lightweight Directory Access Protocol (LDAP) is a software protocol used for enabling any user to locate organizations, individuals, and other resources such as files and devices in a network, whether on the public internet or on a corporate intranet.
You can use this option to modify the MDI Suite LDAP Configuration.
1.      Click to expand the Configurations ( ) option in the Object explorer in the left pane of the dashboard.
2.      Click LDAP Configuration on the list.

Settings Tab

The following details are available on this page:

Field

Description

Server Type

Specifies the server type. The options are:
        Active Directory
        LDAP Server
By default, Active Directory is selected.

Host

Specifies the host number to access the selected server.

Port

Specifies the port number that is associated to the selected host.

Authentication Type

Specifies the Authentication type. The options are:
        Simple
        SSL
Default value selected is SSL.

User Name

Specifies the name of the user who will be using the selected server.

Password

Password that must be associated to the user name.

Domain Name

Specifies the domain name to access the server.

Base DN

Specifies the Base DN to access the server.

Properties Tab

Field

Description

Authentication Attribute

Specifies the required authentication attribute. In server, it searches the specific attribute based on the set criteria.

User Filter

Type the user details to search for in the object class. For example, objectClass=user.

User Name Attribute

Type the name of the user.

User Email Attribute

Type the e-mail ID of the user.

User Telephone Attribute

Type the telephone number of the user.

3.      Click Test Configuration to check if the connection is established.
4.      Click Save to save the connection details.
 

License Details

To view the details of license:
1.      Click to expand the Configurations ( ) option in the Object explorer in the left pane of the dashboard.
2.      Click Licence Details on the list.
The following details are available in this page:

Field

Description

Licensed to

Specifies the name of the customer to which the license is granted.

Date of Expiry

Specifies the date when the license is expected to expire.

Environment

Specifies the name of the environment for which the license is granted. For example, Development, Testing, Production.

Server Host Name

Specifies the name of the server in which the setup is established.

Server MAC Address

Specifies the hardware address of the server.

Native Target Database

Specifies the type of the target database supported by the MDI Suite. For example, Netezza, Teradata, Oracle, Flat file, Hive.

Enterprise License

Specifies if the license is granted to an enterprise.

3.      You can select a new licence key by clicking the menu option on the menu bar. You can browse to the location where your licence key file is placed.
4.      Click Save to save the changes.

Enable Data Points

To enable all Data Points which the users can use in their projects:
1.      Click to expand the Configurations ( ) option in the Object explorer in the left pane of the dashboard.
2.      Click Data Points. A list of all the available Database is displayed.

3.      Select the databases by checking the respective boxes for which the Data Point needs to be enabled.
4.      Click Save on the menu bar or click Revert to undo the changes made to page.

Server Settings

You can use this option to modify the Diyotta server settings:
1.      Click to expand the Configurations ( ) option in the Object explorer in the left pane of the dashboard.
2.      Click on Server Settings from the available options.

Server Paths

This section provides details of the locations where the server home and logs are saved. You can edit the values of the properties. The following details are available in this window:

Field

Description

diserver.home

Displays the location of the Diyotta server.

diserver.serverlogs

Displays the location where the DI server logs are saved.

Server Ports

This section provides details of the ports that are used for different Diyotta servers. You can edit the values of the properties.

Field

Description

servicemanager.startup_port

Displays the port number that is applicable for starting the service manager.

servicemanager.shutdown_port

Displays the port number that is applicable for shutting down the service manager.

servicemanager.service_port

Displays the port number that is applicable for service.

processengine.startup_port

Displays the port number that is applicable for starting the process engine.

processengine.shutdown_port

Displays the port number that is applicable for shutting down the process engine.

processengine.service_port

Displays the port number that is applicable for process engine.

Server Log Settings

This section provides the details of the server logs settings. You can edit the values of the properties.

Field

Description

diserver.loglevel

Displays the name of the logs that are saved in the selected server. For example, DEBUG.

diserver.logsize

Displays the maximum size of the log that can be saved in this server.

diserver.logbkup_file_count

Displays the maximum number of files that can used to save the back up of the logs.

Repository Connection Details

Field

Description

jdbc.driverClassName

Displays the name of the driver that is used to connect to the database. For example, oracle.jdbc.driver.OracleDriver.

jdbc.url

Displays the unified resource locator used to connect to the Diyotta Suite. For example, jdbc:oracle:thin:@us.idataxpress.com:1523:IDXORADB03

jdbc.username

Displays the name of the repository to which the connection is established.

jdbc.password

Displays the password to be used to connect to the defined repository.

hibernate.dialect

Displays the name of the dialect used in hibernate. This is used to format the SQL statements.

hibernate.show_sql

Specifies if the SQL statements can be displayed to the users. It accepts two values: True and False.

hibernate.c3p0.init_size

Displays the number of connections a pool can try to acquire upon startup. This number must be between minimum and maximum pool size.

hibernate.c3p0.min_size

Displays the minimum number of connections a pool can maintain at any given time.

hibernate.c3p0.max_size

Displays the maximum number of connections a pool can maintain at any given time.

hibernate.c3p0.acquire_increment

Displays how many connections at a time c3p0 can try to acquire when the pool is exhausted.

hibernate.c3p0.idle_test_period

Displays the number of seconds when c3p0 must check the idle connections once.

hibernate.c3p0.max_statements

Displays the maximum number of statements that must be cached for all connections.

hibernate.c3p0.max_conn_statements

Displays the maximum number of statements that c3p0 must cache for a single pooled connection.

hibernate.c3p0.timeout

Displays the number of seconds after which the inactive connection must automatically be timed out. If the value is set to "0', then the connection does not time out.

hibernate.c3p0.helper_threads

Displays the maximum number of operations that can be performed simultaneously.

hibernate.c3p0.test_table

Displays the name of the empty table against which the SQL statements can be used to test the connection.

JMX Settings

This section provides details of the Java Management Extension settings. You can edit the values of the properties.

Field

Description

diserver.jmx_service

Displays if the JMX services used for monitoring and management of networks is on.

servicemanager.jmx_port

Displays the port number that is applicable for JMX.

processengine.jmx_port

Displays the JMX port number that is applicable for starting the process engine.

3.      You can modify the details on this screen and click the Save button. The server must be restarted for the settings to be effective.
 

Data Type Mapping

Data Type Mapping is used to map data types for global use. For example, if data conversions between Oracle and Teradata are known to be a common job design, it may be worthwhile to set up the default data type conversions globally so they don't have to be set on each Data Flow individually. Whenever those data sources are used in a Data Flow, these settings would be applied automatically.
To set the default data type mapping:
1.      In the Object explorer, in the left pane, click to expand the Configurations option and select Data Type Mapping.
2.      Select the fields from the drop-down lists to map the Data Types according to the description given below.

Field

Description

Source Database

Select the source database.

Target Database

Select the target database.

Source Data Type

Displays all the data types available in the source database.

Map to <Target DataType>

Displays the data types available in the target database that are mapped to the corresponding source data types.

3.      Click the target data type you want to edit and select the required data type from the drop-down list.
4.      Click the Save button to save the changes or click the Revert to undo the changes.
 

Data Extract Properties

You can use this option to set the default Data Extract properties of a database.
1.      Click to expand the Configurations ( ) option in the Object explorer in the left pane of the dashboard.
2.      Select Extract Properties from the list. The Extract Properties page opens.
3.      Select the Database from the drop-down list of which you want to set the extract properties of.
4.      Next, select the Property type and then set the values of the property as shown in the screen shot given below. In the image, the properties of Netezza are displayed. For different databases, the properties would be different.

5.      Click Save in the menu bar at the top to save the settings or click Save and set as default to set the properties as the default ones.

Data Load Properties

You can use this option to set the default Data load properties of a database.
1.      Click to expand the Configurations ( ) option in the Object explorer in the left pane of the dashboard.
2.      Select Load Properties from the list. The Load Properties page opens.
3.      Select the Database from the drop-down list of which you want to set the Load properties of.
4.      Next, select the Property type and then set the values of the property as shown in the screen shot given below. In the image, the properties of Hive are displayed. For different databases, the properties would be different.

5.      Click Save in the menu bar at the top to save the settings or click Save and set as default to set the properties as the default ones.
  

Runtime Properties

As with the Data Mapping properties and the Data Load properties, this is a place to customize the behavior of the Hive or Spark processors. There are many options available from operational to formats, to optimizations. However, it is recommended to make changes to these defaults only under the direction of a Hive or Spark Administrator. The logging level and TFORM cleanup options are set to INFO level and ON by default.
To set the Runtime properties:
1.      Click to expand the Configurations ( ) option in the Object explorer in the left pane of the dashboard.
2.      Select Runtime Properties from the list. The Runtime Properties page opens.
3.      Select the Database (Hive or Spark) from the drop-down list of which you want to set the Runtime properties of.
4.      Next, select the Property type and then set the values of the property as shown in the screen shot given below. In the image, the properties of Hive are displayed.

5.      Click Save in the menu bar at the top to save the settings.
6.      If the property type - Optimization is chosen, you can add more properties by clicking( _button in the menu bar. A new window asking for the details opens:

7.      Enter the Property Name and Property Value and click Ok. New Optimization properties will be added.

Application Settings

You can use this option to modify the settings for different applications such as Monitor, Scheduler, AppLogs, Netezza Specific Constants, Teradata Specific Constants, and Oracle Specific Constants.
1.      Click to expand the Configurations ( ) option in the Object explorer in the left pane of the dashboard.
2.      Click Applications Settings on the list. The MDI Suite displays the Server Application Properties Information window.

Scheduler

This section provides details of the settings pertaining to Scheduler. You can edit the values of the properties. The following details are available in this window:

Field

Description

scheduler.sch_status

 

scheduler.queue_limit

Displays the maximum number of queues you want to scheduled.

scheduler.show_days

Displays the maximum number of days the details of the schedules must be displayed.

scheduler.show_runs

Displays the maximum number of schedules that must be displayed with the highest numbered run occupying the last position.

scheduler.chk_status

Specifies if the status of a schedule being run can be checked. The possible values are: Y and N.

scheduler.mail_type

Displays the type of mail that is used send emails. For example, mailx.

scheduler.SMTP_HOST_NAME

Displays the host name of the Simple Mail Transfer Protocol (SMTP) server that is used to trigger mails.

scheduler.SMTP_PORT

Displays the port number of the SMTP sever.

scheduler.SMTP_USR_NAME

Displays the user name that must be used to access the SMTP server.

scheduler.SMTP_PWD

Displays the password that is associated to the user name.




Monitor

Field

Description

monitor.show_days

Displays the maximum number of days the status of jobs must be displayed.

monitor.show_runs

Displays the maximum number of status of jobs must be displayed.

Applogs

Field

Description

ops.OPS_FILE_PREFIX

Displays the characters that must be prefixed to the log file being created.

ops.OPS_FILE_SUFFIX

Displays the characters that must be suffixed to the log file being created.

ops.OPS_BAD_FILE_EXT

Displays the file extension of the bad file being created.

ops.OPS_CTRL_FILE_EXT

Displays the file extension of the control file being created.

ops.OPS_LOG_FILE_EXT

Displays the file extension of the log file being created.

ops.OPS_DATA_FILE_EXT

Displays the file extension of the data file being created.

TFORM Table

Field

Description

ops.OPS_XFORM_TAB_PERSIST

Specifies if the transformation table must be persisting. The valid values are: TRUE and FALSE.

ops.OPS_XFORM_TAB_PREFIX

Displays the characters that must be prefixed to the transformation table being created. For example, TFORM_.

ops.OPS_XFORM_TAB_SUFFIX

Displays the characters that must be suffixed to the transformation table being created. For example, _$$DxAuditId.

ops.TFORM_CLEANUP

Specifies if the TFORM tables must be cleaned up automatically. The valid values are: Y and N.

ops.OPS_LOC_TAB_PREFIX

Displays the characters that must be prefixed to the local table being created. For example, LOCAL_.

ops.OPS_LOCAL_TAB_SUFFIX

Displays the characters that must be suffixed to the local table being created. For example, _$$ProjectId.




Application Path settings

Field

Description

diserver.apphome

Displays the location where the application home is residing in the server.

diserver.opslogs

Displays the location where the operational logs are saved in the server.

diserver.datafiles

Displays the location where the data files are saved in the server.

diserver.sourcefiles

Displays the location where the source files are saved in the server.

diserver.targetfiles

Displays the location where the target files are saved in the server.

diserver.badfiles

Displays the location where the bad files are saved in the server.

diserver.paramfiles

Displays the location where the parameter files are saved in the server.

diserver.ctrlfiles

Displays the location where the control files are saved in the server.

diserver.stagefiles

Displays the location where the stage files are saved in the server.

Hive Settings

Field

Description

hive.SERDE_JAR_LOCATION

Displays the location of in which the jar file is placed.

SQL time out

Field

Description

ops.SQL_TIMEOUT_LIMIT

Displays the time in seconds after which the connection to the SQL server will be removed automatically when there is not activity.

Oracle

Field

Description

ops.OPS_BLK_ORA_FETCH_SIZE

Specifies the number of rows that a bulk loading batch can have.

ops.OPS_ORA_DRIVER

Displays the name of the oracle driver being used. For example, T1.

Netezza

Field

Description

ops.OPS_CLI_NZPWD

Specifies if the command line interface requires Netezza password is required. The valid values are: Y and N.

 

System

Field

Description

ops.OPS_BLK_DATA_LOGGING_BATCH

Specifies the number of rows that a bulk loading batch can have.

ops.OPS_FLOW_TYPE_REGULAR

Specifies the type of flow being run. The valid value is R. You cannot edit this value.

ops.OPS_FLOW_TYPE_DEBUG

Specifies the type of flow being run. The valid value is D. You cannot edit this value.

ops.OPS_NZ_SQL_DRIVER

Displays the name of the database connection driver being used by the system. For example, JDBC.

ops.DATEFORMAT

Displays the date format being used by the system. For example, MM/dd/yyyy HH:mm:ss

ops.SQL_LOGGING

Displays the SQL log when set to ON. If it is set to OFF, the system does not display the SQL log.

ops.NZLOAD_PIPE_DELAY

Displays the time delay in seconds from extraction to the load of data.

 
Note:
3.      You can modify the details on this screen and click the Save button to save the changes or click Close to exit the screen without saving the changes.
4.      The server does not need to be restarted for the settings to be effective.
 

Admin View

Viewing Server Log

To view the log generated by the server:
1.      Click to expand the View icon ( ) in the Object explorer in the left pane of the dashboard.
2.      Select Server Log from the available options. The list displays the following fields:

Field

Description

Date/Time stamp

Displays the date and time stamp of the activity.

Log Level

Specifies whether the log level is Debug or Trace.

Class Name

Specifies the class name.

Method Name

Specifies the method name.

Log Message

Displays the log message

3.      Select Log level from the drop-down list as ALL, DEBUG or TRACE to refine your display results.

Viewing Active Users

You can use this option to view the list of users who are currently logged into the MDI Suite.
1.      Click to expand the View icon ( ) in the Object explorer in the left pane of the dashboard.
2.      Select Active Users from the available options. A list of all the active users will be displayed.
3.      The list shows the Login ID, name and Login time of the users.
4.      Click Refresh to get the updated list.

Viewing Object Locks

While working on the projects, the users may lock objects for editing by other users. The Admin can view all the objects that are locked by different users and can unlock them also. You can also view the project and its user.
To see the locked objects and unlock them:
1.      Click to expand the View icon ( ) in the Object explorer in the left pane of the dashboard.
2.      Select Object Locks from the available options. A list of all the objects that are locked or editing will be displayed.

Field

Description

ID

Displays the default ID of the object.

Project Name

Specifies the name of the project to which the object belongs to.

Object Name

Specifies the name of the object.

Object Type

Specifies the Object type. For example, Data Object, Data Flow, Job Flow etc.

Login ID

Displays the Login ID of the user who locked the object.

Name

Displays the name of the user who locked the object

Locked Time

Specifies the time at which the object was locked.

3.      If you wish to unlock any object, select the object and click Unlock in the menu bar at the top. The object will be unlocked.

Organizations

Working with Organizations

The MDI Suite lets you create an organized hierarchical structure for complete privacy and security of the projects of the company. The figure given below describes the hierarchy.

        The Super Admin creates one or more Organizations and assigns Tenant Admin for each Organization.
        The Super Admin can access all the projects of all the organizations.
        The Tenant Admin can not access projects assigned to the other Organizations. For example, the Tenant Admin1 of Organization 1 will not be able to access any projects allotted to Organization 2 or Organization 3.

Creating Organization

To create an Organization:
1.      Click the Organizations icon ( ) in the Object explorer in the left pane of the dashboard.
2.      Click New in the menu bar at the top. A new page opens.
3.      Enter the details according to the description given below:

  Organization Details

Field

Description

Organization Name

Specifies the name of the Organization

Description

Specifies a brief description (optional) of the organization

Last Updated by

Specifies the unique user who last updated the details

4.      Click the Native Targets tab. A list of all the database types is displayed. From the list the database type you want to allot as the Native Targets to the Organization being created.
5.      Click the External Targets tab. A list of all the database types is displayed. From the list the database type you want to allot as the External Targets to the Organization being created.  
6.      Click Save to create an Organization or click Revert to undo the changes.
 

Creating Project

Projects are the main organization mechanism of the metadata repository. All the Data Points, Data Objects, Job Flows and Data Flows belong to a project created and assigned by the Admin.
Projects are defined by an Administrator in the Admin console in one of two ways - Global or Local.

Global Projects

Global projects are by definition "Global" in nature; in other words, they hold objects that are reusable in other projects. These are typically set up as place-holders for generic Data Points or Data Objects that are going to be used repeatedly. They can be sourced into a local project by copying the object from the Global folder into a Local folder, thereby saving the developer time by not having to set up the same type of object for each project they are working on.

Local Projects

Local project folders are set up for one time use as a method of organizing objects related to a specific project. They are typically used for new projects. These folders are usually only applicable to the developers working on a specific project. Therefore, access to them can be restricted by the Administrator. On the other hand, they are flexible and can be used in any way to organize the flow of data in an organization.
It should be noted that once a project is defined as Local (by clicking "OK" to commit the project name) it can't be converted into a Global project. This would have to be done by manually copying objects from a Local folder into a new Global project.
 
To create a project:
1.      Click the Projects icon ( ) in the Object explorer, in the left pane of the dashboard.
2.      Click New in the menu bar at the top. Two options will be displayed- Click New Project to create a new local project or select New Global Project to create a global project.
3.      Enter the details according to the description given below:

Project Details Tab

Field

Description

Project Name

Specifies the name for the project being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Project Code

Specifies the Project Code, typically used to differentiate global objects with local objects having same name.

Project Description

Specifies a description for the project being created. This field accepts only up to 500 characters.

Global

Specifies if the project being created is a global or a local one. If you are creating a new global project, then by default this will be checked and deactive for you to change.

Organization

Specifies the Organization to which this project will belong.

Last Updated By

Specifies the unique user who last updated the project.

Add Layers Tab

Project layers are another method of organizing objects within a project. These layers can be thought of as "sub-folders" and can be named as desired.
4.      Click the Add Layer icon ( ) to create a layer within the project being created. At least one layer is required for any project.
5.      Change the name of the layer simply by clicking on it.
6.      To delete a layer from the list, click the Delete ( )icon.

Assign Agents Tab

An Agent is a light Java program that helps move data between servers in a frictionless manner.
7.      To assign Agents to the project, click the Add Agent icon ( ) at the top. A new window Assign Agents opens.
8.      Click to select one or more agents to be assigned to the project and click Ok.
9.      Click the Save button to create the project or click Revert to undo creating the project.
 

Editing Project

You can update or modify an existing project using the following steps:
1.      Click the Projects icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the projects is displayed.
2.      Click the arrow beside Edit in the menu bar at the top. Click Open to open a project to edit. The details of the project are displayed.
3.      Modify/edit the details of the project according to the description given below:

Project Details Tab

Field

Description

Project Name

Specifies the name for the project being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Project Code

Specifies the Project Code, typically used to differentiate global objects with local objects having same name.

Project Description

Specifies a description for the project being created. This field accepts only up to 500 characters.

Global

Specifies if the project being created is a global or a local one. If you are creating a new global project, then by default this will be checked and deactive for you to change.

Organization

Specifies the Organization to which this project will belong.

Last Updated By

Specifies the unique user who last updated the project.

Add Layers Tab

4.      Click the Add Layer icon ( ) to create more layers within the project. At least one layer is required for any project.
5.      Change the name of the layer simply by clicking on it.
6.      To delete a layer from the list, click the Delete ( )icon.

Assign Agents Tab

7.      To assign agents to the project, click the Add Agent icon ( ) at the top. A new window Assign Agents opens.
8.      Click to select one or more agents to be assigned to the project and click Ok.
9.      Click the Save button to save the project or click Revert to undo the changes to the project.
 

Deleting Project

To delete an existing project from the repository:
1.      Click the Projects icon ( ) in the Object explorer, in the left pane of the dashboard.
2.      From the menu bar, click to open the Edit menu. Select Delete from the options. A message box opens confirming whether you want to delete the selected projects. It will also show you the number of dependent objects of the project.
3.      Click Ok to confirm and delete the projects. Remember, once you click Ok the projects would be deleted permanently.
 

Creating User

An Admin can create user accounts and provide access to one or more projects available in the repository using the following steps:
1.      Click the Users icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing users is displayed.
2.      Click New in the menu bar to create a new user.
Enter the information in the fields according to the description given below:

User Details Tab

Field

Description

Login ID

Specifies the unique identifier of the user to login to the MDI Suite.

User Name

Specifies the name of the user for whom the access permission is being created. This field accepts letters and numerals with the first character must be a letter. Among the list of special characters, this field accepts only underscore (_).

Password

Password for logging into the MDI Suite for the user being created.

Email

Specifies the e-mail ID of the user being created. This is required to send any communications to the user about the actions that are performed on the account details or if the user forgets his password.

Phone

Specifies the primary contact number of the user.

Description

Specifies a description for the user account being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last updated the user account details.

 

Organizations Tab

If you want the user to be in some organization created earlier, click this tab and assign the user to an Organization otherwise the user would, by default, be added to the Default Organization.
To assign the user to any other Organization-
3.      Click the Add Organization icon ( ) at the top. A new window opens with a list of all the available Organizations created earlier.
4.      Select the Organization to which this User will belong to. The name of the Organization will be displayed.
5.      Select the option- Tenant Admin Privilege if you want the user to be the Admin of that particular Organization.

6.      Click the Delete icon ( ) to delete any Organization from the list.

Privileges Tab

7.      Click the Add projects icon ( ) given at the top. This will launch a window displaying all the available projects.
8.      Select one or more projects to be assigned to the user and click OK. The project appears on the page.
9.      Grant Administrator Access- Check this option if the user can be granted Admin access and can log in as an Administrator.

Field

Description

Project

Specifies the project(s) which the user has been assigned.

Roles

Specifies the role of the user.

Layers

Specifies the layer(s) in which the user can work

Actions

Use this option if the privileges need to be changed.

User Groups Tab

The users can be grouped together by clubbing them in a particular group. To assign a user group to a user:
10.   Click the Add User group icon ( ) given at the top.
11.   Select the User group from the list of user groups and click Ok.
12.   Click Save to create the User or click Revert to undo the changes.
 
 

Editing User

You can update or modify the details of an existing user account by following the given steps:
1.      Click the Users icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing users is displayed.
2.      Click the user whose details need to be modified and click from the drop-down list of Edit menu, select Open.
Or
3.      Click to open the details of the user from the list. The details of the user will be displayed.
4.      Change/ Modify the details according to the description of the fields given below:

User Details Tab

Field

Description

Login ID

Specifies the unique identifier of the user to login to the MDI Suite. You cannot edit/modify the Login ID of an existing user.   

User Name

Specifies the name of the user for whom the access permission is being created. This field accepts letters and numerals with the first character must be a letter. Among the list of special characters, this field accepts only underscore (_).

Password

Password for logging into the MDI Suite for the user being created.

Email

Specifies the e-mail ID of the user being created. This is required to send any communications to the user about the actions that are performed on the account details or if the user forgets his password.

Phone

Specifies the primary contact number of the user.

Description

Specifies a description for the user account being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last updated the user account details.

Organizations Tab

You can change the Organization of the User under this tab.
5.      Click the Add Organization icon ( ) at the top. A new window opens with a list of all the available Organizations created earlier.
6.      Select the new Organization to which this User will belong to. The name of the Organization will be displayed.
7.      Select the option- Tenant Admin Privilege if you want the user to be the Admin of that particular Organization.

8.      Click the Delete icon ( ) to delete any Organization from the list.

Privileges Tab

9.      Click the Add projects icon ( ) given at the top. This will launch a window displaying all the available projects.
10.   Select one or more projects to be assigned to the user and click OK. The project appears on the page.
11.   Grant Administrator Access- Check this option if the user can be granted Admin access and can log in as an Administrator.

Field

Description

Project

Specifies the project(s) which the user has been assigned.

Roles

Specifies the role of the user.

Layers

Specifies the layer(s) in which the user can work

Actions

Use this option if the privileges need to be changed.

User Groups Tab

The users can be grouped together by clubbing them in a particular group. To assign a user group to a user:
12.   Click the Add User group icon ( ) given at the top.
13.   Select the User group from the list of user groups and click Ok.
14.   Click Save to create the User or click Revert to undo the changes.
 

Deleting User

To delete an existing user from the repository:
1.      Click the Users icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing users is displayed.
2.      From the menu bar, click to open the Edit menu. Select Delete from the options. A message box opens confirming whether you want to delete the selected User.
3.      Click Ok to confirm and delete the user/users. Remember, once you click Ok the users would be deleted permanently.

Creating User Groups

An Admin can create User Groups and club in all the users based on same privileges:
1.      Click the User Groups icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing user groups is displayed.
2.      Click New in the menu bar to create a new user.
3.      In the User Group page, enter the details according to the description given below:

User Group Details

Field

Description

Organization Name

Specifies the name of the Organization

Description

Specifies a brief description (optional) of the organization

Organization

Assigns the Organization to the User Group. You can change the Organization by clicking Change.  

Last Updated by

Specifies the unique user who last updated the details

Privileges

Field

Description

Grant Administrator Access

Select this option to give the User group the Admin access. In case this is selected, the projects will be disabled as the Super Admin can access all the projects, by default.

Grant Tenant Administrator Access

Select this option to give the User group the Tenant Admin Access.

Project

Specifies the project/projects which to be assigned to the User Group.

Roles

Specifies the role of all the users in the User Group.

Layers

Specifies the layers in which the User Group would work.

Actions

Deletes the project from the list.

Users Tab

4.      Click the Add User icon ( ) a the top. A list of all the users with their login Ids and user names is displayed.
5.      Select all the users you want to be in this User Group and click Ok.
6.      Click Save on the menu bar to save the User Group or click Revert to undo the changes.
 

Editing User Groups

To edit or modify a User Group:
1.      Click the User Groups icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing user groups is displayed.
2.      Click the User Group you want to edit and on the menu bar, click Edit and select Open.
Or
3.      Click to open the select user Group. The details of the user Group are displayed.
4.      Edit the user Group according to the fields and their description given below.

User Group Details

Field

Description

Organization Name

Specifies the name of the Organization

Description

Specifies a brief description (optional) of the organization

Organization

Assigns the Organization to the User Group. You can change the Organization by clicking Change.  

Last Updated by

Specifies the unique user who last updated the details

Privileges

Field

Description

Grant Administrator Access

Select this option to give the User group the Admin access. In case this is selected, the projects will be disabled as the Super Admin can access all the projects, by default.

Grant Tenant Administrator Access

Select this option to give the User group the Tenant Admin Access.

Project

Specifies the project/projects which to be assigned to the User Group.

Roles

Specifies the role of all the users in the User Group.

Layers

Specifies the layers in which the User Group would work.

Actions

Deletes the project from the list.

Users Tab

5.      Click the Add User icon ( ) a the top. A list of all the users with their login Ids and user names is displayed.
6.      Select all the users you want to be in this User Group and click Ok.
7.      Click Save on the menu bar to save the User Group or click Revert to undo the changes.
 

Deleting User Group

To delete an existing User Group from the repository:
1.      Click the Users icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing user groups is displayed.
2.      From the menu bar, click to open the Edit menu. Select Delete from the options. A message box opens confirming whether you want to delete the selected User Group.
3.      Click Ok to confirm and delete the User Group. Remember, once you click Ok the user group would be deleted permanently.

Creating Role

Before adding users, it is a best practice to understand and set up the Roles that will be used to assign users their privileges within the MDI Suite tools.
You can create roles and assign the roles to the users in the project using the following steps:
1.      Click the Roles icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing Roles is displayed.
2.      Click New in the menu bar to create a new user.
3.      Enter the details in the fields according to the description given below.

Role Details Tab

Field

Description

Role Name

Specifies the name of the role being created.

Role Description

Specifies a description for the role being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last updated the role details.

Available Privileges

The list of total number of privileges that can be assigned to the role being created. To assign a privilege to the role, select the respective privilege and click the right-arrow to move it to the Assigned Privileges table.
The following are the available privileges:
        Studio Read
        Studio Write
        Studio Execute
        Monitor Read
        Monitor Execute
        Metaview Read
        Metaview Write

Assigned Privileges

Specifies the list of privileges that are assigned to the role being created. To remove a privilege from the list, select the respective privilege and click the left-arrow to move it to the Available Privileges table.

Assign Privileges Tab

Click the Add Privileges icon ( ) given at the top.

Field

Description

Available Privileges

Displays a list of all the privileges that can be assigned to the role being created. To assign a privilege to the role, select one or more privileges from the list.
The following are the available privileges:
        Integration Write
        Integration Execute
        Monitor Read
        Monitor Execute
        Lineage Read
        Lineage Write
        Scheduler Read
        Scheduler Write
        Scheduler Execute

4.      Click the Save button to create a role or click Revert to undo the changes.
 

Editing Role

You can update or modify the details of an existing role using the following steps:
1.      Click the Roles icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing Roles is displayed.
2.      Select the role to be edited by clicking on it. The details of the role are displayed.  
3.      Enter/ Modify the details in the fields according to the description given below.

Role Details Tab

Field

Description

Role Name

Specifies the name of the role being created.

Role Description

Specifies a description for the role being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last updated the role details.

Available Privileges

The list of total number of privileges that can be assigned to the role being created. To assign a privilege to the role, select the respective privilege and click the right-arrow to move it to the Assigned Privileges table.
The following are the available privileges:
        Studio Read
        Studio Write
        Studio Execute
        Monitor Read
        Monitor Execute
        Metaview Read
        Metaview Write

Assigned Privileges

Specifies the list of privileges that are assigned to the role being created. To remove a privilege from the list, select the respective privilege and click the left-arrow to move it to the Available Privileges table.

Assign Privileges Tab

Click the Add Privileges icon ( ) given at the top.

Field

Description

Available Privileges

Displays a list of all the privileges that can be assigned to the role being created. To assign a privilege to the role, select one or more privileges from the list.
The following are the available privileges:
        Integration Write
        Integration Execute
        Monitor Read
        Monitor Execute
        Lineage Read
        Lineage Write
        Scheduler Read
        Scheduler Write
        Scheduler Execute

4.      Click the Save button to create a role or click Revert to undo the changes.
 

  Deleting Role

You can delete an existing role from the repository using the following steps:
1.      Click the Roles icon ( ) in the Object explorer, in the left pane of the dashboard. A list of all the existing Roles is displayed.
2.      From the menu bar, click to open the Edit menu. Select Delete from the options. A message box opens confirming whether you want to delete the selected Role.
3.      Click Ok to confirm and delete the Role/Roles.  
 
 
 

Creating Agent

An Agent is a light Java program that helps move data between servers in a frictionless manner. These agents can be created in MDI server or external servers based on the requirements.
When the data is handled through the Agents, the server compresses the data which eventually reduces the bandwidth required for loading it from source server to target server.
1.      To create an Agent, click the Agents icon ( ) in the object explorer in the left pane of the dashboard.
2.      On the menu bar, click New.
3.      Enter the details to create an Agent according to the description given below.

Agent Details

Field

Description

Agent Name

Specifies the name of the Agent being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Description

Specifies a description (optional) for the Agent being created. This field accepts only up to 500 characters.

Host

Specifies the host details where the Agent will be located.

Installation Location

Specifies the path to the location where the Agent being created will be installed.

Organization

The Agent is assigned to the Default Organization. Click Change to assign the Agent to a different Organization.

Enable Agent Data Receiver Service

Select the check box to enable the Data Receiver Port field.

Data Receiver Port

Type the Data Receiver Port details which acts like a Target Agent.

Private Agent

Specifies if the Agent being created is assigned as a private Agent. You can change the owner of the Agent by clicking the link Change.

Configurations Tab

Field

Description

Flagent.loglevel

Select the required log level agent from the drop-down list.

Flagent.logsize

Specifies the required log size.

Flagent.logbkup_file_count

Specifies the required file count.

Flagent.logtype

Specifies the required log type.

Ops.BUF_SIZE

Specifies  the required buffer size.

Ops.OPS_NO_OF_THREADS

Specifies the required number of threads.

Ops.QUERY_TIMEOUT

Specifies the required time out.

Ops.OPS_LOGGING_LEVEL

Specifies the required log level from the drop-down list.

Flagent.encryption

Select "Y" to enable the encryption or "N" to disable the encryption.

Flagent.compression

Select "Y" to enable the compression or "N" to disable the compression.

Last Updated By

Specifies the unique user who last updated the agent details.

4.      Click the Assign Projects tab and click the Add Projects icon ( ) to add the projects which would be using the Agent being created.
5.      You can include the Agent being created in an Agent Group. To do this, click the Assign Agent Group tab.
6.      Click the icon ( ) to open the list of Agent Groups. Select the group and click Ok.
7.      Click the Save button to create an Agent click Revert to undo the changes.
 

Editing Agent

You can update or modify the details of an existing Agent using the following steps:
1.       To edit an Agent, click the Agents icon ( ) in the object explorer in the left pane of the dashboard. A list of all the Agents is displayed.
2.      Select an Agent and click the Edit menu on the menu bar. Select Open to display the details of an Agent.
Or
3.      Click to open an Agent and the details are displayed on the screen. Edit/Modify the details according to the description given below.

Agent Details

Field

Description

Agent Name

Specifies the name of the Agent being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Description

Specifies a description (optional) for the Agent being created. This field accepts only up to 500 characters.

Host

Specifies the host details where the Agent will be located.

Installation Location

Specifies the path to the location where the Agent being created will be installed.

Organization

The Agent is assigned to the Default Organization. Click Change to assign the Agent to a different Organization.

Enable Agent Data Receiver Service

Select the check box  to enable the Data Receiver Port field.

Data Receiver Port

Type the Data Receiver Port details which acts like a Target Agent.

Private Agent

Specifies if the Agent being created is assigned as a private Agent. You can change the owner of the Agent by clicking the link Change.

Configurations Tab

Field

Description

Flagent.loglevel

Select the required log level agent from the drop-down list.

Flagent.logsize

Specifies the required log size.

Flagent.logbkup_file_count

Specifies the required file count.

Flagent.logtype

Specifies the required log type.

Ops.BUF_SIZE

Specifies  the required buffer size.

Ops.OPS_NO_OF_THREADS

Specifies the required number of threads.

Ops.QUERY_TIMEOUT

Specifies the required time out.

Ops.OPS_LOGGING_LEVEL

Specifies the required log level from the drop-down list.

Flagent.encryption

Select "Y" to enable the encryption or "N" to disable the encryption.

Flagent.compression

Select "Y" to enable the compression or "N" to disable the compression.

Last Updated By

Specifies the unique user who last updated the agent details.

4.      Click the Assign Projects tab and click the Add Projects icon ( ) to add the projects which would be using the Agent being created.
5.      You can include the Agent being created in an Agent Group. To do this, click the Assign Agent Group tab.
6.      Click the icon ( ) to open the list of Agent Groups. Select the group and click Ok.
7.      Click the Save button to create an Agent click Revert to undo the changes.
 

Deleting Agent

You can delete an existing frictionless agent from the repository.
1.       To delete an Agent, click the Agents icon ( ) in the object explorer in the left pane of the dashboard. A list of all the Agents is displayed.
2.      Select an Agent and click the Edit menu on the menu bar. Select Delete to delete the Agent from the list. The Agent will be deleted.
 

Creating Agent Groups

You can make Agent Groups to club in Agents. To create an Agent group follow the steps given below.
1.      To create an Agent Group, click the Agent Groups icon ( ) in the object explorer in the left pane of the dashboard.
2.      On the menu bar, click New.
3.      Enter the details to create an Agent Group according to the description given below.

Agent Details

Field

Description

Agent Group Name

Specifies the name of the Agent Group being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Description

Specifies a description (optional) for the Agent Group being created. This field accepts only up to 500 characters.

Organization

The Agent is assigned to the Default Organization. Click Change to assign the Agent to a different Organization.

Private Agent

Specifies if the Agent being created is assigned as a private Agent. You can change the owner of the Agent by clicking the link Change.

4.      Click the Assign Agents tab. Click the Add icon ( ) to open the list of available Agents. Select the ones which you want to be in the same group.
5.      Now, on the Assign Projects tab, click the Add icon ( ) to add the projects which would use this Agent Group. The selected projects would be displayed on the screen.
6.      Click Save to create the Agent Group or click Revert to undo the changes.
 

Editing Agent Groups

You can update or modify the details of an existing Agent group using the following steps:
1.       To update or modify an Agent group, click the Agent Groups icon ( ) in the object explorer in the left pane of the dashboard. A list of all the Agent Groups is displayed.
2.      Select an Agent Group and click the Edit menu on the menu bar. Select Open to display the details of an Agent.
Or
3.      Click to open an Agent Group and the details are displayed on the screen. Edit/Modify the details according to the description given below.

Agent Details

Field

Description

Agent Group Name

Specifies the name of the Agent Group being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Description

Specifies a description (optional) for the Agent Group being created. This field accepts only up to 500 characters.

Organization

The Agent is assigned to the Default Organization. Click Change to assign the Agent to a different Organization.

Private Agent

Specifies if the Agent being created is assigned as a private Agent. You can change the owner of the Agent by clicking the link Change.

4.      Click the Assign Agents tab. Click the Add icon ( ) to open the list of available Agents. Select the ones which you want to be in the same group.
5.      Now, on the Assign Projects tab, click the Add icon ( ) to add the projects which would use this Agent Group. The selected projects would be displayed on the screen.
6.      Click Save to create the Agent Group or click Revert to undo the changes.
 

Monitor

Monitor is one of the modules of the MDI Suite. It allows to view or monitor the status of Jobs that are running and to restart, abort, or stop them. The Monitor provides access to Local and Global repositories of information simultaneously making the process of managing Job runs easy.
In the Monitor module, the following activities can be performed:
        View or monitor the status of a Data Flow or a Job Flow
        Refresh the status of Data Flow or Job Flow
        View graphical representation of Data Flows and Job Flows in a bar chart format.
 

Opening Monitor

Log in to the MDI Suite using your Login Id and password. If you have sufficient privileges (provided by the Admin), then you will be directed to the Integration module home page, by default.
If you are logged into some other module such as Admin, Integration, Lineage or Scheduler, then-
1. Click the icon ( ) at the top left corner of the page.
2. The icon will expand to display all the tiles of the available modules.

3. Click Monitor to login to the Monitor module.





Monitor Interface Components

When you are logged into the Monitor module, the home page will be like the one shown below:
 
The home page of the Monitor module displays the Dashboard. It displays the number of Data Flows or Job Flows which have succeeded, failed etc. of the current project. The home page also displays a graphical presentation about the status of the project.
Object Explorer - The Object explorer on the left displays the icons of the options available to view the statuses of the jobs.
Menu Bar - The menu bar at the top displays different menus such as Save, Refresh, Edit, Filter, etc.
Dashboard - This area displays the progress or the status of the selected job that is running as part of the transformation process.
 

Working with Monitor

You can use the Monitor tool to perform the following activities:
        View or monitor the status of a Job Flow or Data Flow
        Refresh the status of Job Flow or Data Flow
        View graphical representation of Data Flows, Job Flows etc. in a bar chart format.
 


 

Monitoring Activities

To check the status or monitor the activities of your Job Flow or Data Flow:
1.      Click the Activity icon ( ) in the Object explorer in the left pane of the dashboard. A list of all the Data Flows and Job Flows of your project appear on the screen.
The details of each object are given as:

Field

Description

Name

Specifies the name of the Data Flow or the Job Flow.

Type

Specifies whether it is Data Flow or Job Flow.

Layer

Specifies the layer to which the project belongs .

Status

Displays the status of the job.

Start Time

Specifies the time when the job started.

End Time

Specifies the end time of the job.

Duration

Specifies the duration of the job.

Run By

Species the user who executed the job.

Progress

Displays the status in a graphical format. The different colors signify different statuses.

The color codes for the Progress are:
o       Green - Indicates the successful/Completed status of the Job.
o       Red - Indicates the failed execution of the Job.
o       Light Blue - Indicates the in-queue/Start/Initialize/on hold/Release status of the Job.
o       Dark Blue - Indicates the active status of the Job.
o       Orange - Indicates the Cancelled status of the Job.
o       Grey - Indicates the waiting on dependency/Deferred/Disabled status of the Job.
o       Pink- Indicates the Timed-out status of the job
o       Maroon - Indicates the Stopped/Aborted status of the job



Exploring the menu

Click any of the below given options at any time during the execution of job on the menu bar at the top. A description of the menu options is given below.  

Icon

Menu

Description

Stop

Stops the execution of selected job. The job can be stopped only if it is active.

Abort

Aborts the running job immediately.

Run

Runs the selected job.

Rerun

Reruns the selected job.

Restart from Failure

Restarts the job from the point where it fails.

Filter

Uses advanced search options.

Refresh

Refreshes the jobs and updates the page to the latest status.

 

  Getting Log

To acquire log details pertaining to a Data Flow or a Job Flow in a selected project:
1.      Click the Activity icon ( ) in the left pane of the dashboard. A list of all the Data Flows / Job Flows is displayed.
2.      Click to select the Job Flow. The Log pane at the bottom will display the details of the run status of the job.  
3.      Click the Statistics tab to view the statistics of the Data Objects.
4.      Click the Log tab to get the log of the job.
5.      Select the Log Level from the drop-down list. The options are: ALL, ERROR, INFO and WARN.
6.      This tab has the following details:

Field

Description

Date

Displays the date and time of the execution of the job.

Log Level

Displays the level at which you want to view the log details. For example, INFO, ERROR etc.

Unit Name

Displays the object which was executed.

Controller/Agent

Specifies whether it is Controller/Agent.

Log Message

Displays the complete details of the log.

7.      Click Save to save the log details in a text format locally.
8.      Click the History tab to view the run status history of the selected job.
 

Lineage

Lineage is one of the modules of the MDI Suite. Lineage allows you to view the details of the metadata pertaining to a selected Data Object or attributes associated to the Data Object.
This module also allows you to view the different transformations that the Data Object underwent both in the forward and backward lineage.
 

Opening Lineage

Log in to the MDI Suite using your Login Id and password. If you have sufficient privileges (provided by the Admin), then you will be directed to the Integration module home page, by default.
If you are logged into some other module such as Integration, Admin, Monitor or Scheduler, then-
1.      Click the icon ( ) at the top left corner of the page.
2.      The icon will expand to display all the tiles of the available modules.

3.      Click Lineage to login to the Lineage module.
 

Scheduler

Scheduler is one of the modules of the MDI Suite. It is used to schedule work flows, invoke jobs, setup event handlers like scheduling tasks, email alerts, file tasks etc. It is used to automate and manage Job Flow execution based on a calendar. This tool allows to schedule a task to run continuously, re-run the same task at a given time or interval, or manually initiate the execution of the task.
The following tasks are carried out in Scheduler:
        Create a task to be scheduled
        Group multiple tasks together to create a group task
        Define calendar for a group task
        Define dependencies between one task and another
        Define file event to watch for a file arrival at a specific location on server
        Define email notification rules to send failure or success notification of task
        Insert/Remove tasks to/from the schedule
        Pause/Resume tasks or entire scheduler for any maintenance
        Automate task executions
        Trigger task executions based on-
o       Calendar
o       File watcher
        Send notifications on failure or success of tasks
        Set dependencies between scheduled objects

 

Opening Scheduler

Log in to the MDI Suite using your Login Id and password. If you have sufficient privileges (provided by the Admin), then you will be directed to the Integration module home page, by default.
If you are logged into some other module such as Integration, Admin, Lineage, Monitor then:
1. Click the icon ( ) at the top left corner of the page.
2. The icon will expand to display all the tiles of the available modules.

3. Click Scheduler to login to the Scheduler module.
 

Scheduler Interface

When you open Scheduler, the home page will look typically like this:

Project Explorer - This explorer on the extreme left is used to create different Tasks, Calendars, Email actions and File Events associated with the selected project.
Canvas - The details of the scheduled tasks can be viewed here.
Side Menu Bar - The menu bar displays different menus to execute actions.
 




Working with Scheduler

You can use the Scheduler tool to perform the following activities:
        Create Task Group
        Create Command Task
        Create File Watcher Task
        Insert Into Scheduler
        Delete Task Group
        Delete Command Task
        Delete File Watcher Task
        Create Calendar
        Delete Calendar
        Create Email Action
        Delete Email Action
        Create File Event
        Delete File Event

 

Creating Task Group

Task group is a logical collection of tasks that should be logically grouped and executed together. For example, putting a File Watcher and Email Event together to be used for a specific Job.
To schedule a set of tasks or multiple tasks that are related, they can be grouped under the same Task Group. The Task group is particularly helpful when there are Job Flows in batches and must be executed simultaneously. This reduces the tedious work of scheduling each task one by one for execution. To expedite the process of adding multiple related tasks to the Scheduler, first create a Task Group.
1.      You can use any of the following methods to add/create a Task Group:

  • Object Explorer: In the Object explorer, click Home. On the side menu bar click ( ).
  • Menu bar: Click the Tasks icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar click ( ).
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and click .

2.      Enter the details in the fields according to the description given below:

Field

Description

Group Name

Specifies a name for the task being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Description

Specifies a description (optional) for the Task Group being created. This field accepts only up to 500 characters.

3.      Click Save to save the Task Group.

Deleting Task Group

To delete a Task Group, follow the given steps:
1.      Click the Tasks icon ( ) in the Object explorer in the left pane. A list of all the available Tasks and Tasks Groups is displayed.
2.      From the list of Tasks and Tasks Groups, select the Task Group that needs to be deleted.Or
Click to open the Task Group that needs to be deleted. The MDI Suite displays the details of the selected Task Group.
3.      Click ( ) from the right menu bar. The system strikes out the selected Task Group to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The stricken out Task Groups are deleted from the list. Click Revert to undo deleting.

 

 

Creating Command Task

A Command task is created to schedule and run any MDI Suite CLI command or any valid external commands.
1.      You can use any of the following methods to add/create a Command Task:

  • Object Explorer: In the Object explorer, click Home. On the side menu bar click ( ).
  • Menu bar: Click the Tasks icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar click ( ).
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and click .

2.      Provide the following details in the New Command Task page according to the description of the fields given below:

General Tab

The following details are available under this tab:

Field

Description

Name

Specifies a name for the Command Task being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Group Name

Specifies the Task Group from the drop-down list to which the Command Task being created must belong.

Description

Gives a description for the Command Task being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last modified/updated the selected Command Task.

Command Tab

The following details are available in this tab:

Field

Description

Command Type

Returns a list of Command types. The options are:
        Execute - Select this option to schedule and run streams
        Export - Select this option to take backup of Project source JSON files
        Import - Select this option to restore any Project source JSON files
        Refresh - Select this option to refresh Project Metadata for Metaview
        Cleanup - Select this option to clean logs, temporary tables etc.
        External  - Select this option to schedule any external shell command

Command

Displays the MDI Suite command that must be executed based on the option selected in the Command Type field.
This field is enabled to allow you to type the external command only when you select the External option in the Command Type field.

Parameters

Specifies the parameters for the selected command type in this field. Click the Select Params icon ( ) next to the Command Type drop-down list to open the Select Params for cmd <Command Type> window and specify the required details.
Note: This window is available only for the command types Execute, Export and Import.

The Command Task created should be linked to the Data Flow which is being scheduled. Click the arrow ( ) next to the Command Type field. This launches a window asking for the parameters to link to the Data Flow. Select the Data Flow from the list appearing in the new window and click OK.

Calendar Tab

Click the Calendar tab and schedule the task using the fields available under the Calendar tab. The Calendar gives various options of setting up the schedule of task to be carried out. The descriptions of each of the field is given in the table below.

Field

Description

Calendar

By default, ADHOC_CALENDAR is created. If you wish to select an existing calendar, click Change. For details on creating the calendar, see the Creating Calendar page.

Frequency

Specifies the number of times the command task being created must be run. The valid values are:
        Once
        Daily
        Weekly
        Monthly

Start Date

Specifies the date when the command task being created must be started.

End Date

Specifies the date when the command task being created must be completed.

Start Time

Specifies the time when the command task being created must be started.

End Time

Select the time when the command task being created must be completed.

Re-run after Minutes

Specifies the time interval, the command task must be executed. Selecting the checkbox, enables this option and the time in minutes is specified in the field next to it.
This field is enabled only when the frequency is set to Daily, Weekly, or Monthly.

Weeks

Specifies the day of the week when the command task being created must be run.
This field is enabled only when the frequency is set to Weekly.

Months

Specifies the month when the command task being created must be run.
This field is enabled only when the frequency is set to Monthly.

Days

Specifies the day when the command task being created must be run on the particular month.
This field is enabled only when the frequency is set to Monthly.

Repeat Count

Displays the number of time the selected command task is executed.
This field is enabled only when the frequency is set to Daily, Weekly, or Monthly.

On Failure

Specifies the action to be performed when the Command Task fails. The options are:
        Do Not Schedule Next Run
        Schedule Next Run

Time Out

A specific time can be set here such that if there is a problem and the task is not carried out properly and it goes beyond that time then a message is displayed. By default, it is set to -1.

Task Dependencies Tab

        Click the Add Dependencies icon ( ) to include tasks to which the command task being created is dependent on. The MDI Suite displays the Select Task Dependencies window. Note: You can open the Select Task Dependencies window only when some tasks are already available.
        Select the tasks from the list to which the Command Task being created is dependent on.
        Click the OK button to add them or click the Cancel button to exit the screen without adding any dependent task.
The following details are available in this tab:

Field

Description

Dependent Task Name

Displays the name of the dependent task.

Status Check Condition

Specifies the condition from the drop-down list to set the dependency between the tasks. The options are:
        Succeeded
        Failed
        Timed Out
        Cancelled
        Active
        Completed

All Dependencies should meet

Specifies that all dependencies between the tasks should meet.

At least One Dependency should meet

Specifies that at least one dependency between the tasks should meet.

Notifications Tab

        Click the Add Emails icon ( ) to select the email actions that must be executed. The MDI Suite displays the Add Email window. Note: You can open the Add Email window only when some Email Actions are already available.
        Select the Email Actions from the list.
        Click the OK button to add them or click the Cancel button to exit the screen without adding any email action.
The following fields and buttons are available in this tab:

Field

Description

Email Action Name

Displays the name of the email action that is attached to the Command Task.

On Status Change Condition

Specifies the condition from the drop-down list to indicate when the selected Email Action must be executed. The options are:
        Succeeded
        Failed
        Timed Out
        Cancelled
        Active
        Completed

Run Book Tab

Type the operations guidelines with priority and steps to be followed for escalation.

Run Status Tab

Note: This tab is available only when the Command Task instance is opened from the Activity canvas.
The following details are available in this tab:

Field

Description

Run Status

Displays the status of the selected command task. The valid values are:
        Scheduled
        Waiting For Dependency
        In Queue
        Active
        Succeeded
        Failed
        Timed Out
        On Hold
        Cancelled

Run Occurrence

Specifies the position in which the selected task is scheduled to be executed if it is required to be run more than once.

Run Log

Displays the list of activities performed on the selected command task.

Start Time

Specifies the date and time when the task has began to run.

Updated By

Displays the unique identity of the user who last updated the task.

  Deleting Command Task

To delete a Command Task, follow the given steps:
1.      Click the Tasks icon ( ) in the Object explorer in the left pane. A list of all the available Command Tasks is displayed.
2.      From the list of Command Tasks, select the Command Task that needs to be deleted.Or
Click to open the Command Task that needs to be deleted. The MDI Suite displays the details of the selected Command Task.
3.      Click the system strikes out the selected Command Task to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The stricken out Command Tasks are deleted from the list. Click Revert to undo deleting.
 

Creating File Watcher Task

File Watcher task is created to check the arrival of files. It checks if a specific file or a file matching a wild card string has arrived. These files are used in conjunction with file events which can be used to trigger a Job. The File Event specifies the path of the file where it is located.
Note: You can create a File Watcher Task only when at least one File Event is available. To work with the File Watcher Task, first a File Event must be created. So, create a File Event first.
1.      You can use any of the following methods to add/create a File Watcher Task:

  • Object Explorer: In the Object explorer, click Home. On the side menu bar click ( )
  • Menu bar: Click the Tasks icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar click ( ).  
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and click ( ).

2.      Provide the following details in the New File Watcher Task page according to the description of the fields given below.

General Tab

Field

Description

Name

Specifies a name for the file watcher task being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Group Name

Specifies the Task Group to which the file watcher task being created must belong.

Description

Specifies a description for the file watcher task being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last modified the selected file watcher task.

File Event Tab

The following details are available in this tab:

Field

Description

Dependent File

Select the File Event from the drop-down list which needs to be executed.

File Directory

Displays the name of the directory in which the expected file must be received.

File Mask

Displays the name of the file that is expected to be masked in the defined directory.

Wait time after file arrival

Displays the time (in minutes) that the event must wait before sending an alert about the arrival of the expected file.

Archive Indicator

Indicates if the received file must be archived.

Archive Directory

Displays the name of the directory in which the received file must be archived.

Calendar Tab

Click the Calendar tab and schedule the task using the fields available under the Calendar tab. The Calendar gives various options of setting up the schedule of task to be carried out. The descriptions of each of the field is given in the table below.

Field

Description

Calendar

By default, ADHOC_CALENDAR is created. If you wish to select an existing calendar, click Change. For details on creating the calendar, see the Creating Calendar page.

Frequency

Specifies the number of times the command task being created must be run. The valid values are:
        Once
        Daily
        Weekly
        Monthly

Start Date

Specifies the date when the command task being created must be started.

End Date

Specifies the date when the command task being created must be completed.

Start Time

Specifies the time when the command task being created must be started.

End Time

Select the time when the command task being created must be completed.

Re-run after Minutes

Specifies the time interval, the command task must be executed. Selecting the check box, enables this option and the time in minutes is specified in the field next to it.
This field is enabled only when the frequency is set to Daily, Weekly, or Monthly.

Weeks

Specifies the day of the week when the command task being created must be run.
This field is enabled only when the frequency is set to Weekly.

Months

Specifies the month when the command task being created must be run.
This field is enabled only when the frequency is set to Monthly.

Days

Specifies the day when the command task being created must be run on the particular month.
This field is enabled only when the frequency is set to Monthly.

Repeat Count

Displays the number of time the selected command task is executed.
This field is enabled only when the frequency is set to Daily, Weekly, or Monthly.

On Failure

Specifies the action to be performed when the Command Task fails. The options are:
        Do Not Schedule Next Run
        Schedule Next Run

Time Out

A specific time can be set here such that if there is a problem and the task is not carried out properly and it goes beyond that time then a message is displayed. By default, it is set to -1.

Notifications Tab

1.      Click the Add Emails icon ( ) to select the email actions that must be executed. The MDI Suite displays the Add Email window.
Note: You can open the Add Email window only when some Email Actions are already available.
2.      Select the Email Actions from the list.
3.      Click the OK button to add them or click the Cancel button to exit the screen without adding any email action.
The following fields and buttons are available in this tab:

Field

Description

Email Action Name

Displays the name of the email action that is attached to the Command Task.

On Status Change Condition

Specifies the condition from the drop-down list to indicate when the selected Email Action must be executed. The options are:
        Succeeded
        Failed
        Timed Out
        Cancelled
        Active
        Completed

Run Book Tab

Type the operations guidelines with priority and steps to be followed for escalation.

Run Status Tab

Note: This tab is available only when the File Watcher Task instance is opened from the Activity canvas.
The following details are available in this tab:

Field

Description

Run Status

Displays the status of the selected file watcher task. The valid values are:
        Scheduled
        Waiting For Dependency
        In Queue
        Active
        Succeeded
        Failed
        Timed Out
        On Hold
        Cancelled

Run Occurrence

Specifies the position in which the selected task is scheduled to be executed if it is required to be run more than once.

Start Time

Specifies the date and time when the task has began to run.

Updated By

Displays the unique identity of the user who last updated the task.

  Deleting File Watcher Task

To delete a File Watcher Task, follow the given steps:
1.      Click the Tasks icon ( ) in the Object explorer in the left pane. A list of all the available File Watcher Tasks is displayed.
2.      From the list of File Watcher Tasks, select the File Watcher Task that needs to be deleted.Or
Click to open the File Watcher Task that needs to be deleted. The MDI Suite displays the details of the selected File Watcher Task.
3.      Click the Delete option on the side menu bar. The system strikes out the selected File Watcher Task to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The system deletes the File Watcher Tasks from the list. Click Revert to undo deleting.
 

Creating Calendar

A task is scheduled to run based on a Calendar. A Calendar is added to a schedule with details about the date and time a task should run. A calendar can be set up for the task in two ways-
        Create a Calendar while creating the Command Task/File Watcher Task
        Create a Calendar first and then fetch it in the Command Task dialog box.
1.      You can use any of the following methods to add/create a Calendar:

  • Object Explorer: In the Object explorer, click Home. On the side menu bar click
  • Menu bar: Click the Calendar icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar click .
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and click .

Enter the details of the fields according to the description given below.

General Tab

Field

Description

Name

Specifies a name for the calendar event being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Description

Specifies a description for the calendar event being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last modified the selected calendar event.

Properties Tab

The following details are available in this tab:

Field

Description

Frequency

Specifies the number of times the command task being created must be run. The valid values are:
        Once
        Daily
        Weekly
        Monthly

Start Date

Specifies the date when the command task being created must be started.

End Date

Specifies the date when the command task being created must be completed.

Start Time

Specifies the time when the command task being created must be started.

End Time

Select the time when the command task being created must be completed.

Re-run after Minutes

Specifies the time interval, the command task must be executed. Selecting the checkbox, enables this option and the time in minutes is specified in the field next to it.
This field is enabled only when the frequency is set to Daily, Weekly, or Monthly.

Weeks

Specifies the day of the week when the command task being created must be run.
This field is enabled only when the frequency is set to Weekly.

Months

Specifies the month when the command task being created must be run.
This field is enabled only when the frequency is set to Monthly.

Days

Specifies the day when the command task being created must be run on the particular month.
This field is enabled only when the frequency is set to Monthly.

Repeat Count

Displays the number of time the selected command task is executed.
This field is enabled only when the frequency is set to Daily, Weekly, or Monthly.

On Failure

Specifies the action to be performed when the Command Task fails. The options are:
        Do Not Schedule Next Run
        Schedule Next Run

Time Out

A specific time can be set here such that if there is a problem and the task is not carried out properly and it goes beyond that time then a message is displayed. By default, it is set to -1.

Cancel

Click this button to exit the screen without creating the calendar.

2.      Click Save to save the Calendar.

Deleting Calendar

To delete a Calendar, follow the given steps:
1.      Click the Tasks icon ( ) in the Object explorer in the left pane. A list of all the available Calendars is displayed.
2.      From the list of Calendars, select the Calendar that needs to be deleted.Or
Click to open the Calendar that needs to be deleted. The MDI Suite displays the details of the selected Calendar.
3.      Click the Delete option on the side menu bar. The system strikes out the selected Calendar to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The system deletes the Calendars from the list. Click Revert to undo deleting.

Creating Email Action

The Email action, if added to any task, sends a mail to a person or a group, when the task being executed undergoes a status change. Provide an email id and the notification will reach the person whenever a task succeeds or fails, depending on the status of the task.
1.      To link an email id to the task, add an Email action. This can be done through:

  • Object Explorer: In the Object explorer, click Home. On the side menu bar click
  • Menu bar: Click the Email Actions icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar click .
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and click .


 
2.      Enter the details of the fields according to the description given below.

General Tab

Field

Description

Name

Specifies a name for the calendar event being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Description

Specifies a description for the calendar event being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last modified the selected calendar event.

Properties Tab

The following details are available in this tab:

Field

Description

To

Specifies the email address of the person or group to which the email event being created must be sent.

CC

Specifies the email address of the person or group to which the email event being created must be copied.

Subject

Specifies the subject line to be carried when the email is sent.

Message

Specifies the message to be sent to the selected person or group.

Attach Log

Check this box if the log of the scheduled task is required.

3.      Click Save to save the Email Action.

Deleting Email Action

To delete an Email Action, follow the given steps:
1.      Click the Email Actions icon ( ) in the Object explorer in the left pane. A list of all the available Email Actions is displayed.
2.      From the list of Email Actions, select the one that needs to be deleted.Or
Click to open the Email Action that needs to be deleted. The MDI Suite displays the details of the selected Email Actions.
3.      Click the Delete option on the side menu bar. The system strikes out the selected Email Action to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The Email Actions are deleted from the list. Click Revert to undo deleting.
 

Creating File Event

A File event is typically used to give the path of the file which is to be received. The File Event specifies the directory and the path where the file is placed. The File Watcher Task typically looks out for the file through the File Event created.
1.      You can use any of the following methods to add a file event:

  • Object Explorer: In the Object explorer, click Home. On the side menu bar click .
  • Menu bar: Click the File Events icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar click .
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and click .

2.      Provide the details in the fields according to the description given below:

General Tab

Field

Description

Name

Specifies a name for the file event being created. This field accepts letters and numerals with the first character being a letter. Among the list of special characters, this field accepts only underscore (_).

Description

Specifies a description for the file event being created. This field accepts only up to 500 characters.

Last Updated By

Specifies the unique user who last modified the selected file event.


Properties Tab

The following details are available in this tab:

Field

Description

File Directory

Specifies the name of the directory in which the expected file must be received.

File Mask

Specifies the name of the file that is expected to be masked in the defined directory.

Wait time after file arrival

Specifies the time (in minutes) that the event must wait before sending an alert about the arrival of the expected file.

Archive Indicator

Select this check box to indicate that the received file must be archived.

Archive Directory

Specifies the name of the directory in which the received file must be archived.

3.      Click Save to save the File Event

Deleting File Event

To delete a File Event, follow the given steps:
1.      Click the File Events icon ( ) in the Object explorer in the left pane. A list of all the available File Events is displayed.
2.      From the list of File Events, select the one that needs to be deleted.
Or
Click to open the File Event that needs to be deleted. The MDI Suite displays the details of the selected File Event.  
3.      Click the Delete option on the side menu bar. The system strikes out the selected File Event to indicate it as deleted as shown in the screen shot given below.

4.      Click the Save button to save the changes. The File Event is deleted from the list. Click Revert to undo deleting.
 

 

Inserting Into Scheduler

To activate a scheduled Job, it must be inserted into the Scheduler. The Job will run on schedule after insertion.
1.      Click the Tasks icon ( ) in the Object explorer in the left pane of the dashboard. A list of all the Tasks and Task Groups appear on the screen with the details.
2.      Select the Command Task or the File Watcher Task or the Task Group which must be scheduled.
3.      Click the Schedule icon( )on the side menu bar. The MDI Suite displays a Scheduler Message confirming that the selected Task Group, Command Task, or File Watcher Task is inserted into the Scheduler.
 
Note: You cannot schedule an empty Task Group. It should contain at least one task.
 

View Activity

You can view the activities related to all the tasks of your project.

  • Object Explorer: In the Object explorer, click Home. On the side menu bar click .
  • Menu bar: Click the View Activity icon ( ) in the Object explorer in the left pane of the dashboard. On the side menu bar click .
  • Quick Access tool bar: Click the Quick Links icon ( ) below the side menu bar and click .

 
The details of the scheduled Tasks are as described below:

Field

Description

Name

Specifies the Name of the Task scheduled for run.

Occurrence

Specifies the number of times the Task has been executed.

Status

Specifies the Status of the Task. The options are:
        Succeeded
        Failed
        Active
        Aborted
        Waiting on Dependency
        In Queue
        Initialize
        Stopped
        Scheduled
        Cancelled
        Time Out

Scheduled

Specifies the date and time of the scheduled task.

Start Time

Specifies the actual start date and time of the scheduled task.

End Time

Specifies the actual end date and time of the scheduled task.

Duration

Specifies the time taken by the task .

Updated By

Indicates the user who last updated the status.

Task Actions

You can Cancel, Stop, Abort, Force Start or Force Complete any Scheduled Task. To select any of the mentioned actions, follow the given steps:
1.      On the Object explorer, in the left pane click View Activity ( ). A list of all the scheduled Tasks will be displayed along with the details and the status.
2.      Select the task which you need to perform an action on.
3.      On the side menu bar at the top, click Action. A list of all Actions appear in the bar.
4.      Select the action based on the description given below.

Icons

Action

Description

Cancel

Cancels the scheduled task. A task can be cancelled only if it is inserted into schedule, Waiting on Dependency or In Queue.

Stop

Stops the execution of the task.

Abort

Aborts a task.

Force Start

Force starts a task. A task can be Force Started only if it is scheduled or Waiting for Dependency.

Force Complete

Force completes a Task. A Task can be Force complete only if is in Failed, Timed Out, Aborted, Stopped or Cancelled state.

Pause

Pauses any scheduled task.

Play

Resumes or starts the paused scheduled task.

Remove

Removes the task from the Schedule.

Override

Overrides the dependency of the task on another task/file event.

 

Appendix

Expression Editor

You can use this window to create conditions for the sources associated to the transformations. The following details are available in the Expression Editor window:

Attributes Tab

This tab displays all the attributes that are available in the source that is associated to the selected transformation.
You can double-click or drag-and-drop the attributes from the Attributes tab into Formula Editor and create the filter condition using the logical operators available on top of the editor.

Parameters Tab

The following details are available in this tab:

Params

Description

<Canvas Name>Parameters

Displays the list of user defined parameters.

System Parameters

Displays the list of system parameters.
        $$RepositoryName - This parameter identifies the name of the repository that you are currently working on.
        $$ProjectID – This parameter identifies the unique identifier of the project you are working on.
        $$ProjectName – This parameter identifies the name of the project you are working on.
        $$LayerName – This parameter identifies the name of the layer within the project you are working on.
        $$DesignName – This parameter identifies the name of the Design.
        $$UserName – This parameter identifies the name of the user currently logged into the project.
        $$CurrentDate – This parameter is used to find the current system date.
        $$CurrentTimeStamp – This parameter is used to find the current timestamp.
        $$AuditID – This parameter identifies the details of a run using the audit ID that is assigned to the run. These details are used for auditing purposes.
        $$StreamName – This parameter identifies the name of the Stream that you are working on.
        $$SrcFileDir - This parameter identifies the name of the Source directory where the files are extracted from.
        $$TgtFileDir - This parameter identifies the name of the Target directory into which the files need to be loaded.
        $$ParamFileDir - This parameter identifies the name of the directory in which the parameter files are available.
        $$ServerRootDir - This parameter identifies the name of the server root directory.
        $$ServerLogDir - This parameter identifies the name of the server log directory.
        $$AppRootDir - This parameter identifies the name of the application root directory.
        $$AppLogDir - This parameter identifies the name of the application log directory.
        $$BadFileDir - This parameter identifies the name of the directory in which the bad files are stored.
        $$CtrlFileDir - This parameter identifies the name of the directory in which the control files are stored.
        $$SourceName – This parameter identifies the name of the Source database from where the data is extracted.
        $$TargetName – This parameter identifies the name of the Target database where the data needs to be loaded.
        $$JobStartTime – This parameter identifies the time when a specified job was started.
        $$JobEndTime - This parameter identifies the time when a specified job was completed.
        $$DateFileDir - This parameter identifies the name of the directory in which the date files are stored.
        $$JobName - This parameter identifies the name of the job.

 


Functions Tab

The following details are available in this page:

Field/Button

Description

Functions

See MDI Suite SQL Functions for more details.

DI_GENERATE_STATS

This function collects statistics on all tables available in the database. The statistics can be each column's proportion of duplicate values, and the maximum and minimum values.
Syntax:
DI_GENERATE_STATS()

DI_GENERATE_STATS_TBL

This function collects statistics on the selected tables within the database. The statistics can be each column's proportion of duplicate values, and the maximum and minimum values.
Syntax:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="68feb07b-0a61-4647-8ce0-dabb784bfa42"><ac:plain-text-body><![CDATA[DI_GENERATE_STATS_TBL(TableName1[,TableName2,TableName3…])

]]></ac:plain-text-body></ac:structured-macro>

DI_GROOM

This function reclaims disk space for deleted or outdated rows on all tables in database.
Syntax:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="bcba0c48-9d58-4d2d-9196-98db71708b17"><ac:plain-text-body><![CDATA[DI_GROOM ([[mode-choice], [reclaim-choice]]) Where [mode-choice]:= RECORDS READY

RECORDS ALL

PAGES ALL

PAGES START

VERSIONS [reclaim-choice]:= RECLAIM BACKUPSET { NONE

DEFAULT}

]]></ac:plain-text-body></ac:structured-macro>

DI_GROOM_TBL

This function reclaims disk space for deleted or outdated rows on selected tables in database.
Syntax:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="25071637-0961-4a7a-acaf-e86eee96d15e"><ac:plain-text-body><![CDATA[DI_GROOM_TBL (TableName,[[mode-choice], [reclaim-choice]]) Where [mode-choice]:= RECORDS READY

RECORDS ALL

PAGES ALL

PAGES START

VERSIONS [reclaim-choice]:= RECLAIM BACKUPSET { NONE

DEFAULT}

]]></ac:plain-text-body></ac:structured-macro>

DI_TRUNCATE

This function truncates all the tables in the database.
Syntax:
DI_TRUNCATE()

DI_TRUNCATE_TBL

This function truncates the selected tables in the database.
Syntax:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="2753a8db-cb51-4240-b854-16a2d0a10f73"><ac:plain-text-body><![CDATA[DI_TRUNCATE_TBL(TableName1[,TableName2,TableName3…])

]]></ac:plain-text-body></ac:structured-macro>

DI_CF

This function executes the custom SQL. Note: The custom SQL must not begin with SELECT.
Syntax:
DI_CF(Custom SQL)

DI_NZMIGRATE

This function moves or copies one or more databases or tables from one Netezza Performance Server (NPS) to another within the same server or between different servers. SUNIXUSER, SHOST, THOST, SDB, TDB are mandatory fields. All other fields are optional.

Netezza Params

The following table lists the Netezza command line parameters that can be used for migration:

Command Line Parameters

Description

Default Value

Sample Value

Mandatory/Optional

shost

Source Host

No default value. Parameter is mandatory, If not specified script will exit with error message

 

Mandatory

thost

Target Host

No default value. Parameter is mandatory, If not specified script will exit with error message

 

Mandatory

 tdb

Target Database

No default value. Parameter is mandatory, If not specified script will exit with error message

 

Mandatory

suser

Source DB User

If not specified it uses default NZ_USER

 

Optional

spassword

Source DB password

If not specified it uses default NZ_PASSWORD

 

Optional

tuser

Target DB User

User is optional

 

Optional

tpassword

Target DB password

If not specified it uses default NZ_PASSWORD

 

Optional

tables

The table(s) within the database to be migrated

If none are specified, then all tables in the source database will be migrated.]

 

Optional

sport

Source Netezza Port

If not specified it uses default NZ_DBMS_PORT

 

Optional

tport

Target  Netezza Port

If not specified it uses default NZ_DBMS_PORT

 

Optional

cksum

Perform the cksum of source table and target table and if not matching will print an error message

If not specified, default will be "false"

 

Optional

createcargettable

Creates Target table if this option is specified as "yes".

If not specified, default will be "no"

 

Optional

truncatetargettable

Truncates Target table if this option is specified as "yes".

If not specified, default will be "no"

 

Optional

genStats

Generates Statstics of migrated tables

If not specified, default will be "none"

 

Optional

format

Data Format <ascii

binary>

Default is ASCII

 

Optional

threads

The number of threads to be used to process one table

By default

  1.      1  thread will be used for small tables ( < 1M rows )
    <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="4645ec7c-b826-4d64-a9cf-4ca28414ea4b"><ac:plain-text-body><![CDATA[#      4  threads will be used for larger tables

threads "n"   [n should be a number] *Note using more threads to process a small table will negativelt affect the performance snce creating threads will also has a overhead.

 

]]></ac:plain-text-body></ac:structured-macro>

to

Email address to be sent a copy of the output from this script

By deafult, no email will be sent

 

Optional

sTable

source Table

This option allows you to migrate a table into a table with a different name.     When using this option, you may only process one table at a time via this script.            This option cannot be used with -CreateTargetTable and -TruncateTargetTable options

 

Optional

tTable

target Table

 

 

Optional

sbin

source bin/executables directory                        # Used when migrating from a 3,0 system

 

 

Optional

tbin

target bin/executables directory                        # Used when migrating from a 3,0 system

 

 

Optional

sourcewhereclause

sourcewhereclause will be applied to the ata from source table

 

 

Optional

stableTXid

stable transaction ID/number

Default will not check for stable transaction ID

 

 

 

Data Extraction and Data Load Properties

The following table provides you the properties pertaining to Data extraction and Data load:

Property Name

Data Extraction Properties

Data Load Properties

 

 

 

Flatfile as source

 

Load Type: NZLoad
File Location: SrcFileDir
File Name :
File Data Reference:Direct
File Type: Delimiter
Column Delimiter:


Escape Characters:  Empty
Text Qualifier: Empty
Date Style:YMD
Y2Base: Empty
Date Delimiter: -
Time Style: 12Hour
Time Delimiter: ' : '
Decimal Delimiter: Empty
Skip Rows: 0
Maximum Rows: Load All Rows
Maximum Errors:Ignore Errors
Null Value: NULL
Boolean Style:1_0
Fill Record: OFF
Truncate String: OFF
CR in String:OFF
Encoding:Internal
Socket Buffer Size: 16 MB
Ctrl Chars: OFF
Login Timeout:  120

Hive as Source

Extraction type: SQOOP Export

Load Type: SQOOP EXPORT
Column Delimiter: Empty
Warehouse: Empty
Row Delimiter: \n
Split By: Empty

MSSQL as Source

Extraction Type: BCP
File Location: $$StgFileDir
Extraction Mode: File
Column Delimiter: ','
Packetsize:65535

Load Type: Nzload
File Type: Delimiter
Escape Character:  Empty
Text Qualifier: Empty
Date Style: YMD
Y2Base: Empty
Date Delimiter: '-'
Time Style: 24 HOURS
Time Delimiter: ':'
Decimal Delimiter: Empty
Skip Rows: 0
Max Rows: Load All Rows
Maximum Errors: Ignore Errors
Null Value: NULL
Boolean Style: 1_0
Fill Record: OFF
Truncate String: OFF
CR in String: OFF
Encoding: Internal
Socket Buffer size: 16 MB
Ctrl Chars:  OFF.
Login Timeout: 120

Netezza as Source

Extraction type: External Table
File Location:    $$StgFileDir
Extraction Mode: File

Load Type: Nzload
File Type:Dellimiter
Column Delimiter: '

'
Escape Character: Empty
Text Qualifier: Empty
Date Style: YMD
Ybase: Empty
Date Delimiter: '-'
Time Style: 24 HOUR
Time Delimiter: ':'
Decimal Delimiter: Empty
Skip Rows: 0
Maximum Rows: Load All Rows
Maximum Errors: Ignore Errors
Null Value: NULL
Boolean Style: 1_0
Fill Record: OFF
Truncate String: OFF
CR in String: OFF
Encoding: internal
Socket Buffer Size: 16MB
Ctrl Chars:OFF
Login Timeout: 120

Teradata as Source

Extraction type: Fast Export
File Location: $$StgFileDir
SESSIONS: 5
TENACITY: 5
SLEEP: 10
MODE: RECORD
FORMAT: DELIM
LOG TABLE: LT_$$AuditId_$$UnitId_JobId
LOG FILE: instNm_$$AuditId.log
Column Delimiter:


DATE STYLE: YMD
DATE DELIMITER: -

Load Type: NZ Load
File Type: Delimiter
Column Delimiter: '

'
Escape Character: Empty
Text Qualifier: Empty
Date Style: YMD
Y2 Base: Empty
Date Delimiter: '-'
Time Style: 12Hour
Decimal Delimiter:Empty
Skip Rows: 0
Maximum Rows: Load All Rows
Maximum Errors : Ignore Errors
Null Value:NULL
Boolean Style: 1_0
Fill Record: OFF
Truncate String : OFF
CR in String: OFF
Encoding: internal
Socket Buffer Size: 16MB
Ctrl Chars: OFF
Login Timeout: 120

Oracle as  Source

Extraction Type: JDBC
File Location: $$StgtFileDir
Extraction Mode: File

Load Type:Nzload
File Type: Delimiter
Column Delimiter:'

'
Escape Characters: Empty
Text Qualifier:Empty
Date Style: YMD
Y2 Base: Empty
Date Delimiter:'-'
Time Style:24 Hour
Time Delimiter: ':'
Decimal Delimiter:Empty
Skip Rows: 0
Maximum Rows:Load All Rows
Maximum Errors:Ignore Errors
Null Value: Null
Boolean Style:1_0
Fill Record:OFF
Truncate String:OFF
CR in String:OFF
Encoding:internal
Socket Buffer Size: 16MB
Ctrl Chars: OFF
Login Timeout: 120

DB2 as  Source

Extraction Type: JDBC
File Location   :$$StgtFileDir
Extraction Mode:File

Load Type:Nzload
File Type: Delimiter
Column Delimiter:'

'
Escape Characters: Empty
Text Qualifier:Empty
Date Style: YMD
Y2 Base: Empty
Date Delimiter:'-'
Time Style:24 Hour
Time Delimiter: ':'
Decimal Delimiter:Empty
Skip Rows: 0
Maximum Rows:Load All Rows
Maximum Errors:Ignore Errors
Null Value: Null
Boolean Style:1_0
Fill Record:OFF
Truncate String:OFF
CR in String:OFF
Encoding:internal
Socket Buffer Size: 16MB
Ctrl Chars: OFF
Login Timeout: 120

Flatfile as Target

File Location: $$TgtFileDir
File Name: Empty
File Type: Dilimiter
Column Delimiter: '

' ,  ','
Escape Character: Empty
Text Qualifier: Empty
Date Style: YMD
Date Delimiter:'-'
Time Style:24 Hour
Time Delimiter: ':'
Decimal Delimiter:Empty
Null Value: Null
Boolean Style:1_0
Encoding:internal
Socket Buffer Size: 16MB

 

Terdata Design

 

 

Netezza as a source

Extraction Type:External Table
File Location:$$StgtFileDir
Column Delimiter:'

'
Date Style:YMD
Date Delimiter:'-'
Time Style:12 Hour
Time Delimiter: ':'
Decimal Delimiter:Empty
Skip Rows:0
Maximum Rows:Load Rows
Maximum Erros:Ignore Errors
Null Value:NULL
Encoding:latin9
Ctrl Chars:OFF
Boolean Style:1_0
Fill Record: OFF
Truncate String:OFF
CR in String :OFF
Y2Base:Empty
Escape Character:Empty

Load Type:Fast Load
Sessions:5
Tenacity:5
Sleep :10
Error Tables:ET1_$$AuditId_$$UnitId_$$JobId ET2_$$AuditId_$$UnitId_$$JobId
Mode:Record
Format:Vartext
CheckPoint:50

Flatfile as Source

 

Load Type: Fload
File Location:$$SrcFileDir
File Name:Empty
Sessions:5
Tenacity:5
Sleep:10
ErrorTables:ET1_$$AuditId_$$UnitId_$$JobId ET2_$$AuditId_$$UnitId_$$JobId
Mode:Record
Format:Vartext
Checkpoint:50
Date Style:YMD
DateDelimiter:'-'
Time Style:12Hour
Time Delimiter: ':'
Column Delimiter: '

'

Flatfile as Target

Extraction Type: Fast Export
File Location:$$TgtFileDir
Sessions:5
Tenacity:5
Sleep:10
Mode:Record
Format:Delim
File Name:Empty
Logtable:Empty
Logfile:Empty
Column Delimiter:'

'
Date Style YMD
Date Delimiter:'-'
Time Style:12 Hour
Time Delimiter: ':'
Null Value:Empty
TPT Type:Export

 

Hive as Source

Extraction Type: SQOOP EXPORT

Load Type:Sqoop Export
Column Delimiter:Empty
Warehouse Location:Empty
Row Delimiter:'\n'
Split By:Empty

Oracle as Source

Extraction type:JDBC
File location:$$StgFileDir
Extraction Mode:File
Date Style: YMD
Date Delimiter :'-'
Time Style: 12 Hour
Time Delimiter: ' : '
Text Qualifier: Empty
Column Delimiter:'

'

Load Type:Fast load
Sessions: 5
Tenacity: 5
Sleep:10
Error Tables:Empty
Mode:Record
Format:Vartext
Checkpoint:50
Error Table:Empty

Teradata Source

Extraction Type :Fast Export
File Location:$$StgFileDir
Sessions:5
Tenacity:5
Sleep:10
Format:Delim
Log Table:Empty
Log File:Empty
Column Delimiter: '

'
Date Style:YMD
Date Delimiter: '-'
Time Style: 12 hour
Time Delimiter: ' : '
Null Value: Empty

Load Type:FastLoad
Sessions:5
Tenacity:5
Sleep:10
Error Tables: Empty
Mode:Record
Format:VarText
Checkpoint: 50

MSSQL as Source

Extraction Type:BCP
File Location: $$StgFileDir
Extraction Mode: File
Date Style: YMD
Date Delimiter : '-'
Time Style : 24 HOURS
Time Delimiter : ':'
Text Qualifier : Empty
Column Delimiter: ','
Packetsize:65535

Load Type:Fast load
File Location: $$StgFileDir
File Name:Empty
Sessions:5
Tenacity:5
Sleep:10
ErrorTables:Empty
Mode:Record
Format:Vartext
Checkpoint:50

DB2 as  Source

Extraction Type:JDBC
File Location: $$StgFileDir
Extraction Mode: File
Date Style: YMD
Date Delimiter : '-'
Time Style: 24 HOURS
Time Delimiter: ':'
Text Qualifier: Empty
Column Delimiter: '

'

Load Type:Fast load
Sessions:5
Tenacity:5
ErrorLimit:100
Sleep:10
Mode:Record
Format:Vartext
OutFile:Empty
Error table1: Empty
Error table2: Empty
Checkpoint:50
LOGTABLE:Empty
ROUTE MESSAGES:Empty
INFILE:100
WORKTABLES:Empty
ERRORTABLES:Empty

Hive Design

 

 

Netezza Source

Extraction type: SQOOP

Load Type : SQOOP
MySql Delimiters : OFF/ON
Column Delimiter: Empty
Row Delimiter: \n
Split By : Empty
Verbose:OFF/ON
Connection Param File :Empty
Append:OFF/ON
File Type : Empty
Boundary Query : Empty
Direct_Mode :OFF/ON
Direct Mode Splitsize :Empty
Inline LOB Limit:Empty
Map Tasks:Empty
Where Condition :Empty
Compression:OFF/ON
Compression Codec:Empty
Null if String:Empty
Null if NonString:Empty
Incremental :Empty
Enclose By :Empty
Escape By :Empty
Optional Enclosed By:Empty
Hive Overide :OFF/ON
CREATE TABLE: OFF/ON
Drop Import Delimits: OFF/ON
Delimit Replacements : Empty
Partition Key: Empty
Partition Values : Empty

Flatfile as Source

 

Load Type : HiveLoad
File Location : $$SrcFileDir
File Name :Empty
File Type :Delimiter
Column Delimiter:


Load From:Local
Text Qualifier: Empty

Flatfile as Target

Extract Type: External SQL
File Location:$$TgtFileDir
File Name:instNM.dat
File Type:Delimiter
Column Delimiter :'

' or ','

 

MSSQL as Source

Extraction type :BCP
File Location :$$tgFileDir
Extraction Mode :File
Date Style: YMD
Date Delimiter:'-'
Time Style:24 Hour
Time Delimiter: ':'
Text Qualifier:Empty
Column Delimiter: '

' or ','
Packetsize:65535

Load Type: HiveLoad
File Location : $$SrcFileDir
File Type :Delimiter
File Transfor Type : hadoop/scp
HDFS Tempory Directory : /tmp

Teradataas Source

Extraction Type :SQOOP

Load Type: SQOOP
MySql Delimiters: OFF/ON.
Column Delimiter:Empty
Row Delimiter:\n
Split By : Empty
Verbose:OFF/ON
Connection Param File: Empty
Append :OFF/ON
File Type : Empty
Boundary Query : Empty
Direct Mode:OFF/ON
Direct Mode Splitsize:Empty
Inline LOB Limit :Empty
Map Tasks :Empty
Where condition: Empty
Compression:OFF/ON
Compression codec:OFF/ON
Null If String:Empty
Null If NonString:Empty
Incremental :Empty
Enclose By :Empty
Escape By:Empty
Optional Enclosed By:Empty
Create Table :OFF/ON
Hive OverRide :OFF/ON
Drop Import Delimits :OFF/ON
Delimit Replacements:Empty
Partition Key :Empty
Partition Value: Empty

Oracle as Source

Extraction type: JDBC
File Location:$$StgFileDir
Extraction Mode:File
Date Style: YMD
Date Delimiter:'-'
Time Style:24 Hour
Time Delimiter: ':'
Text Qualifier:Empty
Column Delimiter: '

' or ','

Load Type :Hive Load
File Type :Delimiter
File Transfor Type : hadoop/scp
HDFS Tempory Directory : /tmp

Hive as Source

Extraction type : External SQl
File Location: $$StgFileDir
HDFS Temp Unload :/tmp
Temp File Name : Empty
Extraction Mode:File

Load Type :Hive Load
HDFS Temp Directory: /tmp
Column Delimiter : '

'
Row Delimiter: \n

Hive as Target

Extraction Type :External SQL
File Location:$$TgtFileDir
Temp File Name:Empty
Extraction Mode :File

Load Type:Hive Load
File Type: Delimiter
Column Delimiter:'

'

DB2 as Source

Extraction type: JDBC
File Location:$$StgFileDir
Extraction Mode:File
Date Style: YMD
Date Delimiter:'-'
Time Style:24 Hour
Time Delimiter: ':'
Text Qualifier:Empty
Column Delimiter: '

' or ','

Load Type:Hive Load
File Type: Delimiter
File Transfor Type : hadoop/scp
HDFS Temp Directory: /tmp
Column Delimiter:'

'

 
 
 

Glossary

A
Attribute: The property of a selected dataobject, map or flow.
C
Character: Type of character such as ASCII, CHR, CHRCODE.
Conversion: Specifies the conversion format such as TO_BIGINT, TO_DATE, TO_DECIMAL
D
Data Type: The type of the data associated to the attribute.
DataCleansing: Specifies the data condition such as BETWEEN, GREATEST, IN.
DataObject: DataObject is a standard object in which you can store more than one piece of data at a time. However, you need to make sure the individual pieces must be of different data format. In case you attempt to store a data of a specific format that is already in use, the system stores the latest data and discards the old data.
Design: The wiork area where mapping between the source and target dataobjects are performed.
Diyotta Data Integration Suite: Diyotta DI Suite is a metadata driven DI suite created for exploiting the complete potential of DW Appliances (MPP) such as Netezza, Oracle, Teradata, Flatfile, and Hive using in-the-box database operations. The intuitive GUI based design makes the data integration process simple.
E
Export: Sending a copy of metadata from one database to another database in the .xml file format.
Expression: An equation used to format the data from one standard to another standard.
F
Filter: Used to filter the data available in the source table.
Financial: Specifies the financial condition such as multiplication, additon, and subtraction.
Flat File: A database type Flat File.
I
Import: Taking a copy of metadata from different database into another database in the .xml file format.
L
Layer: The different layers available within a project.
M
Minus: Used to compare data in two source tables, remove all the data common to both the tables, and save only the remaining data from the first table into the target table.
MJoiner: Used to join data in more than two tables and save them in a single target table.
N
Netezza: A database type Netezza
Numerical: Specifies the numerical condition such as CONV, CUME, EXP.
O
Object: Object canvas is the work area on which dataobjects are extracted from a source database, transformed, and moved to the target database.
Oracle: A database type Oracle.
P
Parameter: Identifies a characteristic, a feature, a measurable factor that can help in defining a particular system.
PostgreSQL: A database type PostgreSQL.
Prec: The maximum number of characters the selected attribute can accommodate.
Primary Object: The first level source table from which the data is extracted for transformation.
R
Repository: The location where the project level details are saved.
Revert: Restores the selected item to the last saved state.
Rollup: Used to perform calculation such as aggregation, and averaging and save them in a target table.
S
Scale: The number of characters the selected attribute can accommodate after the decimal.
Scientific: Specifies the scientific cnditions such as COS, COSH, SIN.
Secondary Object: The second level source table whose attributs are compared against the attributes available in the primary object.
SJoiner: Used to join data in two tables and save them in a single target table.
Special: Specifies special condition such as ABORT, DECODE, ERROR.
Splitter: Used to split the data in the source table and save them in multiple target tables.
Stream: The work area where the flow of data between the source and target database is defined.
T
Target: Defines the target table or file to which the source is being mapped.
Teradata: A database type Teradata.
Test: Specifies test conditions such as ISNULL, IS_DATE, IS_NUMBER.
Transformation: Defines the functions that must be performed on the source data before writing the data in the target. You can use different expressions to perform different functions.
TStage: Used to create temporary tables for the intermediate tables available in a mapping instance.
U
Union: Used to combine the data in different tables and save them in a single target table.
V
Validate: Checking if the specified condition is valid or not.
Version: The number of times the selected expression is modified.

  • No labels