Talend Open Studio Cookbook

Manually checking and setting contexts

Understanding tContextLoad

Understanding implicit context loading

Manipulating contexts in Talend Open Studio

Introduction

Appendix B. Management of Contexts

Appendix A. Common Type Conversions

Stopping memory errors in Talend

Printing tMap variables

Adding job name project name and other job specific information

Capturing tMap rejects

Some complex date formats

Dragging component globalMap variables

Reloads going missing at each row global variable

Finding a new context variable

Finding the code routine

My tab is missing

Introduction

Chapter 12. Common Mistakes and Other Useful Hints and Tips

Executing non-Talend objects and operating system commands

Passing parameters to a child job

Returning codes from a child job without tDie

Capturing and acting on different return codes

Managing job dependencies

Adding command-line context parameters

Using a different context

Creating compiled executables

Introduction

Chapter 11. Deploying and Scheduling Talend Code

Killing a job from within tJavaRow

Testing logic – the most-used pattern

Creating test data using Excel

Creating random test data using lookups

Creating complex test data using tRowGenerator tFlowToIterate tMap and sequences

Creating simple test data using tRowGenerator

Dumping the console output to a file from within a job

Printing out the context

Using tJava to display status messages and variables

Using tJavaRow to display row information

Using tLogRow to show data in a row

Using the Java debugger to debug Talend jobs

Using the Talend debug mode – row-by-row execution

Locating execution errors from the console output

Find the location of compilation errors using the Problems tab

Introduction

Chapter 10. Debugging Logging and Testing

Ensuring lossless queues using sessions

Reading and writing to a queue

Calling a RESTful web service

Calling a SOAP web service

Writing complex XML

Reading complex hierarchical XML

Using tXMLMap to create an XML document

Using tXMLMap to read XML

Introduction

Chapter 9. Working with XML Queues and Web Services

Creating and writing files depending on the input data

Processing control/validation files

Processing multiple files at once

Capturing file information

Moving copying renaming and deleting files and folders

Adding a header and trailer to a file

Using the information in the header and trailer

Reading headers and trailers with no identifiers

Reading headers and trailers using tMap

Storing intermediate data in the memory using tHashMap

Using temporary files

Reading rows using a regular expression

Appending records to a file

Introduction

Chapter 8. Managing Files

Rewritable lookups using an in-process database

Managing surrogate keys for parent and child tables

Database and table management

Capturing individual rejects and errors

Selecting different fields and keys for insert update and delete

Passing a session to a child job

Managing database sessions

Printing your output query

Writing to a database table

Printing your input query

Using context and globalMap variables in SQL queries

Reading from database tables

Importing the table schemas

Setting up a database connection

Introduction

Chapter 7. Working with Databases

Setting the context file location in the operating system

Turning implicit context loading on and off in a job

Using implicit context loading to load contexts

Using tContextLoad to load contexts

Adding contexts to a context group

Adding a context group to your job

Creating a context group

Introduction

Chapter 6. Managing Context Variables

Importing JAR files to allow use of external Java classes

Creating custom functions using code routines

Creating pseudo components using tJavaFlex

Adding complex logic into a flow using tJavaRow

Setting the context and globalMap variables using tJava

Performing one-off pieces of logic using tJava

Introduction

Chapter 5. Using Java in Talend

Using reload at each row to process real-time / near real-time data

Hierarchical joins using tMap

Joining data using tMap

Splitting an input row into multiple outputs based on input conditions

Filtering input rows

Using intermediate variables in tMap

Using the ternary operator for conditional logic

Creating tMap expressions

Simple mapping and tMap time savers

Introduction

Chapter 4. Mapping Data

Creating binary error codes to store multiple test results

Creating validation rules for more complex requirements

Checking a column against a lookup

Checking a column against a list of allowed values

Rejecting rows using tMap

Validating against the schema

Gathering all rejects prior to killing a job

Enabling and disabling reject flows

Introduction

Chapter 3. Validating Data

Creating schemas from lists

Dropping schemas to empty components

Cutting and pasting schema information

Creating a generic schema from the existing metadata

Propagating schema changes

Hand-cranking a built-in schema

Introduction

Chapter 2. Metadata and Schemas

Enabling tHashInput and tHashOutput

Installing the software

Before you begin

Chapter 1. Introduction and General Principles

Customer support

Reader feedback

Conventions

Who this book is for

What you need for this book

What this book covers

Preface

Support files eBooks discount offers and more

www.PacktPub.com

About the Reviewers

About the Author

Credits

Talend Open Studio Cookbook

coverpage

coverpage

Talend Open Studio Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files eBooks discount offers and more

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Chapter 1. Introduction and General Principles

Before you begin

Installing the software

Enabling tHashInput and tHashOutput

Chapter 2. Metadata and Schemas

Introduction

Hand-cranking a built-in schema

Propagating schema changes

Creating a generic schema from the existing metadata

Cutting and pasting schema information

Dropping schemas to empty components

Creating schemas from lists

Chapter 3. Validating Data

Introduction

Enabling and disabling reject flows

Gathering all rejects prior to killing a job

Validating against the schema

Rejecting rows using tMap

Checking a column against a list of allowed values

Checking a column against a lookup

Creating validation rules for more complex requirements

Creating binary error codes to store multiple test results

Chapter 4. Mapping Data

Introduction

Simple mapping and tMap time savers

Creating tMap expressions

Using the ternary operator for conditional logic

Using intermediate variables in tMap

Filtering input rows

Splitting an input row into multiple outputs based on input conditions

Joining data using tMap

Hierarchical joins using tMap

Using reload at each row to process real-time / near real-time data

Chapter 5. Using Java in Talend

Introduction

Performing one-off pieces of logic using tJava

Setting the context and globalMap variables using tJava

Adding complex logic into a flow using tJavaRow

Creating pseudo components using tJavaFlex

Creating custom functions using code routines

Importing JAR files to allow use of external Java classes

Chapter 6. Managing Context Variables

Introduction

Creating a context group

Adding a context group to your job

Adding contexts to a context group

Using tContextLoad to load contexts

Using implicit context loading to load contexts

Turning implicit context loading on and off in a job

Setting the context file location in the operating system

Chapter 7. Working with Databases

Introduction

Setting up a database connection

Importing the table schemas

Reading from database tables

Using context and globalMap variables in SQL queries

Printing your input query

Writing to a database table

Printing your output query

Managing database sessions

Passing a session to a child job

Selecting different fields and keys for insert update and delete

Capturing individual rejects and errors

Database and table management

Managing surrogate keys for parent and child tables

Rewritable lookups using an in-process database

Chapter 8. Managing Files

Introduction

Appending records to a file

Reading rows using a regular expression

Using temporary files

Storing intermediate data in the memory using tHashMap

Reading headers and trailers using tMap

Reading headers and trailers with no identifiers

Using the information in the header and trailer

Adding a header and trailer to a file

Moving copying renaming and deleting files and folders

Capturing file information

Processing multiple files at once

Processing control/validation files

Creating and writing files depending on the input data

Chapter 9. Working with XML Queues and Web Services

Introduction

Using tXMLMap to read XML

Using tXMLMap to create an XML document

Reading complex hierarchical XML

Writing complex XML

Calling a SOAP web service

Calling a RESTful web service

Reading and writing to a queue

Ensuring lossless queues using sessions

Chapter 10. Debugging Logging and Testing

Introduction

Find the location of compilation errors using the Problems tab

Locating execution errors from the console output

Using the Talend debug mode – row-by-row execution

Using the Java debugger to debug Talend jobs

Using tLogRow to show data in a row

Using tJavaRow to display row information

Using tJava to display status messages and variables

Printing out the context

Dumping the console output to a file from within a job

Creating simple test data using tRowGenerator

Creating complex test data using tRowGenerator tFlowToIterate tMap and sequences

Creating random test data using lookups

Creating test data using Excel

Testing logic – the most-used pattern

Killing a job from within tJavaRow

Chapter 11. Deploying and Scheduling Talend Code

Introduction

Creating compiled executables

Using a different context

Adding command-line context parameters

Managing job dependencies

Capturing and acting on different return codes

Returning codes from a child job without tDie

Passing parameters to a child job

Executing non-Talend objects and operating system commands

Chapter 12. Common Mistakes and Other Useful Hints and Tips

Introduction

My tab is missing

Finding the code routine

Finding a new context variable

Reloads going missing at each row global variable

Dragging component globalMap variables

Some complex date formats

Capturing tMap rejects

Adding job name project name and other job specific information

Printing tMap variables

Stopping memory errors in Talend

Appendix A. Common Type Conversions

Appendix B. Management of Contexts

Introduction

Manipulating contexts in Talend Open Studio

Understanding implicit context loading

Understanding tContextLoad

Manually checking and setting contexts