Introduction

This simple sample introduce the use of Kettle's Regex step to process a Tomcat's log file. We will parse the log and separated the row based on a regular expression pattern.

You can download the file samples at the end of the article.

Sample Explanation

1. We get the log file's content at [Tomcat Logs] (Text Input) step.
2. Validate the pattern using [Validasi Regex] (Regex Evalution) step. Here we only get rows with date information plus message pattern in the same row. Save the boolean checking value in a field named result.
   
3. Based on result field, we make decision where the row should go next using [Filter Rows] (Filter Rows) step. Data that match with condition 'result = Y' will go ahead to [Tanggal] step and a non-matching one will go to [Others] step.
4. You can see the results by previewing [Others] and [Tanggal] dummy steps.
5. Finally we split the matching incoming content into 3 fields (tanggal, jam and pesan) using javascript in [Hasil Akhir] step (Modified Java Script Value).
6. Done

Result

Drawback

By using Modified Java Script Value, we have the risk of processing bottleneck in this step as scripting can be slower than if we implemented it in a plugin with specific functionality.

Nevertheless, this sample show the flexibility of Kettle if you know how to manipulate any data type using Java language.

Reference

• http://wiki.pentaho.com/display/EAI/Regex+Evaluation
• http://www.regular-expressions.info/tutorial.html