take a look at this comment I put over on Vincent’s blog and tell me what you think about it. I have to make these transformations available regardless, but as I read more comments about these benchmarks, the more I think that having a set of real world transformations to work against might be interested for all the different vendors.

http://it.toolbox.com/blogs/infosphere/was-the-manapps-etl-benchmark-test-flawed-or-baised-28697#2498479

Again, I think we need to define these tests in an open source way and find some DS, INFA, BODI, OWB, Talend, PDI experts to execute the tests.
That sort of thing needs to happen on common grounds so I’m looking for a “neutral” site to host the benchmark suites and results.

What do you think of that idea?

All the best,

Matt

By the way, it would be much more interesting to compare PDI to proprietary solutions (Datastage, Informatica…).

As I usually say, PDI & Talend are not competitors…

Fabrice

Regarding the limitations, that’s when you use random writes, see my blog post:

http://www.daniel-lemire.com/blog/archives/2008/02/02/random-write-performance-in-solid-state-drives/

But if you are reading a CSV file, you are good. Of course, writing out a transformed output randomly on disk will get you in trouble. But thankfully, you are not doing that (I hope you are loading it up into a table or something).

I think it is with the right settings in this case. Personally I think it has more to do with disk latency. However for argument sake it’s besides the point. Like you said, if you would jump to the new types of SSD or very fast disk subsystems, one or even 2 cores would not be enough to process the +200MByte/s of throughput. I’ve seen one particular situation (can’t disclose) where we needed 10 servers with 4 cores to read the data at peak performance (a single fixed width flat file) : 2GByte/s. Of-course it goes beyond saying that 20Gbit/s disk subsystems don’t come cheap and are not always readily available to us for testing

>> Now, the first thread starts and reads the first line. What the heck can the second thread do? It cannot jump and read b1, b2, b3 because it does not know where b1 before the first thread has read a3.

That’s exactly what it does: jump to a neighbor of line 3. The algorithm is not line based anyway, it’s size based.
Positioning is done with a disk seek, not by actually reading the file. In the worst case the disk heads have to reposition, no other time or CPU cycles are wasted on it.
To prevent the disk from going “crazy” as you put it, it’s better to read in larger blocks. That is why the second test has a large NIO buffer size set.

Obviously, looking at it from the position of a single disk spindle is a bit simple since most enterprise data these days would be stored on disk subsystems, RAID-5, mirrored disks, etc.

By the way, it’s a different subject, but you would need very new SSD drives to surpass a recent hard drive performance, especially when it comes to writing. Up until now, SSD has been marketing, nothing more. Check out Linus’ comments on the subject: http://torvalds-family.blogspot.com/2008/10/so-i-got-one-of-new-intel-ssds.html

That is what I meant by CPU-bound. That is, you must throw a second core to keep the I/O system busy. One core is not enough.

Have you tried the same thing with a SSD drive? If I really wanted to load up CSV files fast, I’d store them on SSDs.

As far as parallel CSV reading, that was a bit tricky to set up… The size of the file is measured and split into the number of threads. All threads know at that point in time what part to read since they all know what number they have and how many there are, even across a cluster. The idea is then to read until you find the next newline/CR and continue until you hit the size limit. There are some borderline cases to handle, but that’s the general idea.

I still do not get it. Suppose that I have the following CSV data:

a1, a2, a3

b1, b2, b3

c1, c2, c3

d1, d2, d3

Now, the first thread starts and reads the first line. What the heck can the second thread do? It cannot jump and read b1, b2, b3 because it does not know where b1 before the first thread has read a3.

What I am guessing is this. You start the first thread at a1. Meanwhile, you have the second thread jump forward in the file, at maybe c2… then it starts reading… of course, the first line it reads is wasted, but you keep going.

That’s nice, but you have some overhead. For example, after each line the first thread reads, it must check whether it reached the area read by thread two.

Then, also, you risk having your disk go crazy because you ask for a lot of random access. Of course, this would not be happening with SSD.