Matt Casters on Data Integration

Today we received our 10000th post on our Pentaho Data Integration (Kettle) forum. It seems only a short while ago that user “rsheldon” posted the very first post on December 12th 2005.

I can still remember myself thinking: “I just open sourced years of work and you ask me an Oracle question on legal stuff? WTF??”

Fortunately, other people followed and although at first there where only a few posts, these days the forum is the doorway to a vibrant community that makes me proud to be a part of. It seems these days we’re getting hundreds of posts each week and I’m extremely pleased that almost all questions get a reply.

Surely, the insane quick pace of Kettle development is to be blamed for the large number of questions, but I for one can only be glad for this. What once was my pet project is now a full community effort. This is what Ohloh has to say about Kettle:

Factoid: “Large, active development team”

Over the past twelve months, 11 developers contributed new code to Kettle.

This is a relatively large team, putting this project among the top 10% of all project teams on Ohloh.

For this measurement, Ohloh considered only recent changes to the code. Over the entire history of the project, 12 developers have contributed.

In fact more developers have contributed, but all changes before 4 months ago are not taken into account. That is because we recently migrated to a different source control server.

Until next time,

Matt

1 Comment »

I have been using (open) SuSE for my Linux “needs” ever since SuSE version 6.x.
For my new laptop (see previous posts on the topic) however I tought I would give Kubuntu a try. I’ve been reading a lot of positive things about it and the Ubuntu attitude towards Microsoft is a bit clearer than Novells.

I have to say that I’m absolutely impressed. The only change I change I made to the distribution was the installation of the NVidia closed source graphics drivers, but to be honest, the open source drivers worked just the same. So that’s pretty impressive to me.

Now the only thing left that bothered me was the limit to the amount of memory I can use. I packed this machine with 4GB of RAM. Vista can access it so why can’t I. Well apparently the Intel Core 2 Duo 7600 is x86-64 so I probably should have picked that DVD download from the Kubuntu site. I’m guessing that’s the mistake I made here. I won’t know until I find the time to re-install the machine. (not the next couple of weeks anyway) I’m going to try the “live DVD” one of these days to see if that gives any clues. I’ll keep you up to date in the comments.

As for the real reason why I like this laptop and Linux: the complete re-build of the Kettle project takes less than 10 seconds on the new box. This could take minutes on my old (2GHz Pentium-M) machine. That and “apt-get”. ‘nough said.

Until next time,

Matt

No Comments yet »

Now that the API conversion is done, we can continue to port steps. In the mean time, you guys can enjoy better performance on the steps that where already ported.

Here is an example: generating 10M rows with 2 integers in it:

For now, the GUI hasn’t changed at all, but you can start to help out with testing etc. This image can be used to get an initial view of the performance of the currently ported steps. In any case, the example above runs “somewhat” faster than the old version, try for yourself.

Until next time,

Matt

P.S. A new download for the step plugin source code is available.

4 Comments »

We’re smack in the middle of porting Kettle to a new API, version 3.0.  So this is what it comes down to again…

Yep, you got it right.  Refactoring is at times a very time-consuming thing to do.

It also is something that brings me back to the start of Kettle.  You see, I have been doing this sort of thing a few times before with the Kettle codebase.  I can for example remember the time when I changed the Java code from using if/then structures to using Exceptions to make error handling more transparent and to clean up code.  There is something about going through hundreds of classes, fixing all occurences of a certain bad pattern, for days on end … it’s an acquired taste I have to say.

Anyway, the 3.0 porting of Spoon is well under way and I hope you can see the first results after the weekend.

Until then,

Matt

1 Comment »

Harris Ward, one of our finest community members has created a Kettle installer you might fancy.

So, download it, try it out and let us know what you think!!

Until next time,
Matt

3 Comments »