Dead wrong

Belgian consultancy company Element 61 has just posted an opinion piece under the disguise of a review on open source ETL.

What a load of utter nonsens.  Try reading this:

Instead of using SQL statements to transform data, an Open Source ETL tool gives the developer a standard set of functions, error handling rules and database connections. The integration of all these different components is done by the Open Source ETL tool provider. The straightforward transformations can be implemented very quickly, without the hassle of writing queries, connecting to data sources or writing your own error handling process. When there are complex transformations to make, Open Source ETL tools will often not offer out-of-the-box solutions.

Well Mr Jan Claes, we’re perfectly capable of handling quite complex transformation with high performance too.  If Kettle isn’t capable of handling your ETL needs, neither is Informatica, DataStage, OWB or BODI.  If you prefer Oracle Warehouse builder because it allows you to squeeze PL/SQL or SQL into your ETL tool, than that’s fine, just don’t use false arguments to dis open source ETL tools.  ETL tools should allow you to write LESS code and make it easier to maintain your transformations, not more.  Being open source has nothing to do with that fact.

Most reputed ETL-vendors provide an extensive amount of connections to all sorts of data sources. This is a problem with Open Source ETL tools: most of them are based on Java architecture and need JDBC to connect to a database. In the basic license, a few connections are available but when there is a need for extra connections, the customer has to pay an extra fee and/or for some platforms (like mainframe sources) nothing might be available.

You have to be kidding, right?  Kettle supports 34 database types + generic ODBC, OCI and JNDI connections out the box for free.  On top of that we connect to legacy systems like SAP/R3 and obviously your mainframe as well if needed (very few people ever do).  The painful truth is that we’re doing better, not worse.

Java & XML knowledge required for complex transformations.

This comment just made the article provably false since you never need any Java or XML knowledge to use Pentaho Data Integration.  (I’m sure the same goes for Talend by the way)

Lack of skills, knowledge & resources.

Pentaho has plenty of partners , even in Belgium. (Cronos for example)  We also have the lead developer of Pentaho Data Integration (me) working in Belgium as well as Davy Nys our Sales Representative.  Professional support, training (on site if needed) is offered as well.

In these turbulent financial times, open source ETL it exactly the answer to constantly shrinking budgets and that is why Pentaho is doing better than ever before despite the credit crunch.

Element 61 in the mean time needs to get hit with a clue stick.  It’s one thing to accept money from the big boys, it’s a completely different thing to spread demonstrable lies.

Until next time,
Matt

10 comments

  • Hi Matt,

    go get ‘em ;)

    Seriously, these people didn’t do their homework. Take for example the connectivity bit…if you take for example, MS SQL Integration services, nobody is dissing them for only offering ODBC and AdoDB right? And to top it off I am pretty sure they do not offer 30 something connectors…more like 3, or 13 if you’re lucky.

    And btw, what over-generalization is this, how this it make sense to lump all open source ETL in one bucket? From my experience there is substantial difference between different tools and suites and in no universe does it make sense to functionally recognize one large body of “open source etl tools”. The only thing the serious FOSS BI tools have in common is a more or less free license and access to the source code.

    Anyway, back to work ;)

  • >> Java & XML knowledge required for complex transformations.
    >
    > This comment just made the article provably false since you never
    > need any Java or XML knowledge to use Pentaho Data Integration.
    > (I’m sure the same goes for Talend by the way)

    I don’t see any XML requirements in Talend. We may save repository items in an XML format, but that’s internal mecanism.

    Concerning complex transformations, the ones you can do in tMap (the all-in-one component) you indeed need basic knowledge in Java syntax. You can also select Perl as generated language, and then you can use Perl for complex transformations.

    The most important point is that here at Talend, we have very soon decided to use standard and open languages at the opposite of a specific and dedicated language. In any company, you have a much higher probability to find Java/Perl skills than Basic universe skills. In addition to this, with Perl/Java you can imagine any transformation, there is no real limitation.

  • I couldn’t agree more Pierrick. Java, Perl, JavaScript are all open and free standards, unlike the scripting languages used in for example Informatica PowerCenter or even PL/SQL.
    It’s this complete absence of vendor lock-in that scares some people to the delight of our customers :-)

  • >>>> Java & XML knowledge required for complex transformations.
    Hugh…If it was the case, implementing ETL would become silly. People would use TSQL, PL/SQL,…+ JAVA/C# if they have to spend the same time.
    We decided to use PDI because it is quite simple to build complex transformation/job without coding.

    >>>> Most reputed ETL-vendors provide an extensive amount of connections to all sorts of data sources.
    Not really the case today.
    Things are going faster in Open source.
    You don’t need to wait 3 years for the next release.

    >>>> most of them are based on Java architecture and need JDBC to connect to a database.
    I like Java and i don’t see why he complains about it.

    >>>> when there is a need for extra connections, the customer has to pay an extra fee and/or for some platforms
    You also pay extra fee to ETL vendor.
    I believe that community make the difference..and because open source tool are open, you can write you own plugin/connector without paying.

    Samatar

  • “Maturity of Open Source ETL / Why do people fail to do their homework?”

    http://rpbouman.blogspot.com/2008/10/maturity-of-open-source-etl-why-do.html

  • Jan Claes

    Matt,

    First of all, don’t attack element61 for this article, but just me.
    I like the Pentaho Data Integration solution (and Talend), and I am absolutely a supporter of open source tools. This was a general article about open source ETL, not an attack. I honestly believe that you can become a major player, and being an alternative for the “big boys”. And yes, you offer more connections and transformations than the average open source ETL tool (although everybody is doing a lot of work on that). And yes, you don’t need knowledge of XML and Java to use your tool (I know, because I installed it on my PC), but you are not the only vendor in this world.
    You have to see this article as a mean to start a discussion, to bring your message (and that of others) over and convince companies that there are other solutions than the “big ones” (and no, I am not paid by them, bit of an unnecessary attack…).
    Personally, If I have to do a POC for an ETL tool, I would invite Pentaho and Talend to see what they can do in a real life environment.

    kind regards

    Jan Claes

  • Jan,

    If you don’t want to be associated with Element 61, don’t post on their website, it’s as simple as that. There are plenty of free blog services out there.
    You know, I think you are completely unbelievable: just because you downloaded PDI and Talend, you think you know how the tools work and what they do? From your comment I’m sure you simply don’t. You admit to that ignorance by inviting (as vaguely as possible) to do a POC. We don’t need that invitation, PDI and Talend have been deployed at very big customers all over the world. You can check pentaho.com for reference cases.

    With tens of thousands of downloads and users out there, all over the world, we already are a big player. Wake up!

    Have a nice day,

    Matt

  • Karl Popper

    True ignorance is not the absence of knowledge, but the refusal to acquire it – Karl Popper

  • Perhaps there name is a double entendre … they feel compelled to make buzz words like “performance management” a foundational element of there web site and use them at least 61 times.

    Great responses – they clearly are dead wrong (if they are considering Kettle and Talend as the open source front runners). My guess these clowns jumped on a the performance management band wagon 18 months ago are are now grasping as companies didn’t all follow. The comparison was probably prepared to help sell one client who mentioned they were considering open source alternatives.

    Keep it up!

  • Patrick

    The original review is ridiculous. It’s obvious that they just downloaded PDI/Talend and played with it for 5 minutes. What’s even more obvious is they have never used a proprietary ETL tool like Ascential or Hyperion.

    If anyone thinks they can get anything done in less than 10 minutes on either of those proprietary tools out of the box, they’re sorely mistaken. The inflexibility of those tools completely goes against the reason we need ETL programs – we need flexibility to do data transformations because everyone’s data is different. And the programming skills required is shockingly high for those tools, especially when you try to do something complicated. The hardest thing in Kettle? JavaScript – which a 8 yr. old knows how to do these days.

    And clearly the review is full of lies. Kettle (don’t know about Talend) has the greatest number of out of the box connectors I’ve ever seen come out of any tool, and they’re all free. I know for a fact that SAP connectors on Ascential cost upwards of $40k USD as a module option.

    Stuff like this infuriates me, but what should really tick anyone off is that there are salespeople out there with talking points like these on sheets of paper that they just rattle off to spread FUD about open source products. You can only trick a customer once. Let’s see if these people will be in business at the end of 2009.