Step performance graphs

One of the things I’ve been working on lately in Kettle / Pentaho Data Integration is the transparency of the performance monitoring.

We don’t just need an API to get the step performance data out, but we also need to visualize this data in a simple way, something like this:
performance graph

Graph with moving average

The next steps will be to also allow this data to be spooled off to a database somewhere and to be accessed remotely using Carte.

Until next time,

Matt

9 comments

  • Samatar

    Hé Matt..that’s great !
    I see that Santa clause visited you :-)
    Is this step will be available in 3.1 (as a plugin or in the code base).

    Any news from data quality plugin?

    A+

    Samatar

  • Hi Samatar,

    It’s no step, it will be available directly from the transformation log view in v3.1 as well as the slave server logging view. (remote)

    No news from the data quality plugin, but I have plans of my own ;-)

    Cheers,

    Matt

  • Samatar

    It’s no step, it will be available directly from the transformation log view in v3.1 as well as the slave server logging view. (remote)
    –> Super

    No news from the data quality plugin, but I have plans of my own
    –> with 3.1 of after?

    Take care

  • Great Job Matt !

    It will make us better on profiling our flow.

    Feris

  • Samatar, I want to put the first DQ stuff in 3.1, yes. It will depend on other endeavors that are going on in the background. I’ll know in a few weeks.

    Matt

  • Samatar

    That’s great Matt..
    Adding this feature will bring more help on easily point on bottleneck steps.
    Nice…Really Nice.

    Samatar

  • Great…Matt..i hope will make better…hope i join term.

  • rusang

    Fantabulous (fantastic + fabulous)…

    >> The next steps will be to also allow this data to be spooled off to a database somewhere.

    Will this be available in 3.1 ??

    Can we help ??

    -rusang

  • rusang, 3.1.0-M1 will indeed carry these changes. You can download one of the nightly builds to try:

    ftp://download.pentaho.org/client/pentaho-data-integration/3.1.0

    To write to disk, I’m thinking about making a step available that exposes the performance data from the result object. Pentaho HQ in Orlando is also creating a management console that will create fancy reports and analytics on that data, probably later next month or so.

    Anyone can help!! If you have a good idea, I would advice you to create a JIRA case or send an e-mail to the kettle-developers mailing list (@ google groups).

    Cheers,

    Matt