Comments on: EC2 : Scaling large files with S3 and Kettle http://www.ibridge.be/?p=113 Venting steam after a long day of writing code... Fri, 10 Sep 2010 23:55:10 +0000 http://wordpress.org/?v=2.5 By: Matt Casters http://www.ibridge.be/?p=113#comment-22181 Matt Casters Sun, 11 May 2008 07:58:10 +0000 http://www.ibridge.be/?p=113#comment-22181 Alex, lazy conversion will indeed speed that up. I wrote another blog entry on the subject: http://www.ibridge.be/?p=63 And another one on performance : http://www.ibridge.be/?p=78 The high CPU usage basically is the "Java" price you pay for Unicode conversions and data type conversions. To get around that we invented the concept of lazy conversion to try and avoid doing unneeded data conversions. If you're loading into a database, it doesn't really matter all that much since that's going to be the bottleneck. On the slowness of the EC2 nodes: nothing we can do about that really. It's a price/performance thing. There are faster nodes to be had, but they go at $US 0.30/hour. Also please remember that the data is coming in over REST (web services). That is bound to be a bit slower than reading from local disks. Again, the point is that it scales. Perhaps if we would add another 10 nodes, it would scale even further. I'm lacking the time to test it out at the moment. Matt Alex, lazy conversion will indeed speed that up.

I wrote another blog entry on the subject: http://www.ibridge.be/?p=63
And another one on performance : http://www.ibridge.be/?p=78

The high CPU usage basically is the “Java” price you pay for Unicode conversions and data type conversions.
To get around that we invented the concept of lazy conversion to try and avoid doing unneeded data conversions.
If you’re loading into a database, it doesn’t really matter all that much since that’s going to be the bottleneck.

On the slowness of the EC2 nodes: nothing we can do about that really. It’s a price/performance thing. There are faster nodes to be had, but they go at $US 0.30/hour. Also please remember that the data is coming in over REST (web services). That is bound to be a bit slower than reading from local disks. Again, the point is that it scales. Perhaps if we would add another 10 nodes, it would scale even further. I’m lacking the time to test it out at the moment.

Matt

]]> By: Alex http://www.ibridge.be/?p=113#comment-22180 Alex Sun, 11 May 2008 07:11:17 +0000 http://www.ibridge.be/?p=113#comment-22180 Interesting stuff, especially the nearly perfect scaling. I"ve been digging into some aspects of Kettle's performance lately. It would seem that data conversion from source input is pretty CPU-hungry, because the per node throughput seems to be: 1 node: 2400/842 = 2.85mb/s 10 nodes: 2400/96 = 25mb/s aggregate (2.5mb /s /node) It seems like a lot of effort to get 25mb/s given that your average SATA disk can get sequential read speeds of 40-50mb/s. I just tested a simple transform against a 2.5GB file with a series of generated UUIDs and random numbers to load the data and then sum up the random numbers. Sure enough, I can barely make the disk sweat, only managing 85k rows/s (4mb/s). The CPU on this box is a bit old though, but I would not have expected this. Is this the primary goal of lazy conversion and parallel loading -- to speed up the CPU bottleneck of conversions? Sorry if i'm going a bit off topic... Interesting stuff, especially the nearly perfect scaling. I”ve been digging into some aspects of Kettle’s performance lately. It would seem that data conversion from source input is pretty CPU-hungry, because the per node throughput seems to be:

1 node: 2400/842 = 2.85mb/s
10 nodes: 2400/96 = 25mb/s aggregate (2.5mb /s /node)

It seems like a lot of effort to get 25mb/s given that your average SATA disk can
get sequential read speeds of 40-50mb/s.

I just tested a simple transform against a 2.5GB file with a series of generated UUIDs and random numbers to load the data and then sum up the random numbers. Sure enough, I can barely make the disk sweat, only managing 85k rows/s (4mb/s). The CPU on this box is a bit old though, but I would not have expected this.

Is this the primary goal of lazy conversion and parallel loading — to speed up the CPU bottleneck of conversions?

Sorry if i’m going a bit off topic…

]]>