[MonetDB-users] MonetDB with Pentaho and Kettle
Hi Guys, As of 3.1 Kettle now has a connector built in for Monet so I thought I'd give it another go after not getting too far earlier in the year. Anyway I have it up and running on a Quad Core Xeon box running Centos x86_64 and it seems pretty happy, but when I try to copy sizable amounts of data into it, kettle stops with an error: 2008/10/17 12:29:22 - Table output.0 - Prepared statement : INSERT INTO voc.d_budgets3 (entity_anl_code, trad_code, Cost, Sales, CostMTD, SalesMTD) VALUES ( ?, ?, ?, ?, ?, ?) 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Because of an error, this step can't continue: 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : org.pentaho.di.core.exception.KettleException: 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Error inserting row into table [d_budgets3] with values: [19], [A09], [0], [0], [0], [0] 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Error inserting/updating row 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : current transaction is aborted (please ROLLBACK) But to my untrained eye the data looks fine(coming from a mysql database) and I don't know where to start debugging this or what settings I could change. The monet table is 2 Strings, 2 Int, 2 Big Num and it always inputs a load of data before stopping. What can I look at to resolve the issues? otherwise, rocking database and keep up the good work. Thanks Tom CONFIDENTIALITY NOTICE This e-mail is confidential and may also be privileged. Please notify us immediately if you are not the intended recipient. You should not copy it, forward it or use it for any purpose or disclose the contents to any person. If you have received this e-mail in error, please notify us as soon as possible and destroy your copy. Unless otherwise expressly stated, this e-mail is not intended to constitute a business letter, order form or other offer or invitation to you, nor does this e-mail form the basis of any contract. Contracts may not be concluded by e-mail and any contract or agreement attached is subject to contract and shall not and is not intended to create a legally binding relationship. This e-mail is sent by or on behalf of Encon Limited or the relevant subsidiary undertaking of Encon Limited with which you are dealing. Encon Limited is registered in England and Wales with company number 3411533 registered office at The Wolseley Center, Harrison Way, Leamington Spa, CV31 3HH, United Kingdom. VAT No 362 0233 93. Encon Ireland is registered in Ireland with company number 262605 registered office at Wolseley Ireland Moydrum Road, Athlone, Co Westmeath. Save a Tree ! - Please think before you print.
Sorry Jos, Matt Casters told me exactly the same on the Pentaho forums, I got a little side tracked data loading. Cheers Tom On Fri, 2008-10-17 at 14:59 +0200, Jos van Dongen wrote:
Hi Tom,
this is due to the fact that Kettle 3.1 still issues an 'Insert Into' statement which MonetDB doesn't handle very well. It's a known issue (see http://wiki.pentaho.com/display/EAI/MonetDB ) which should be resolved in the next Kettle release. I'll forward your experiences to the Kettle team so they can take notice of it as well.
Met vriendelijke groet/Kind regards,
Tom Barber wrote:
Hi Guys,
As of 3.1 Kettle now has a connector built in for Monet so I thought I'd give it another go after not getting too far earlier in the year. Anyway I have it up and running on a Quad Core Xeon box running Centos x86_64 and it seems pretty happy, but when I try to copy sizable amounts of data into it, kettle stops with an error:
2008/10/17 12:29:22 - Table output.0 - Prepared statement : INSERT INTO voc.d_budgets3 (entity_anl_code, trad_code, Cost, Sales, CostMTD, SalesMTD) VALUES ( ?, ?, ?, ?, ?, ?) 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Because of an error, this step can't continue: 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : org.pentaho.di.core.exception.KettleException: 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Error inserting row into table [d_budgets3] with values: [19], [A09], [0], [0], [0], [0] 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Error inserting/updating row 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : current transaction is aborted (please ROLLBACK)
But to my untrained eye the data looks fine(coming from a mysql database) and I don't know where to start debugging this or what settings I could change. The monet table is 2 Strings, 2 Int, 2 Big Num and it always inputs a load of data before stopping.
What can I look at to resolve the issues? otherwise, rocking database and keep up the good work.
Thanks Tom CONFIDENTIALITY NOTICE
This e-mail is confidential and may also be privileged. Please notify us immediately if you are not the intended recipient. You should not copy it, forward it or use it for any purpose or disclose the contents to any person. If you have received this e-mail in error, please notify us as soon as possible and destroy your copy.
Unless otherwise expressly stated, this e-mail is not intended to constitute a business letter, order form or other offer or invitation to you, nor does this e-mail form the basis of any contract. Contracts may not be concluded by e-mail and any contract or agreement attached is subject to contract and shall not and is not intended to create a legally binding relationship.
This e-mail is sent by or on behalf of Encon Limited or the relevant subsidiary undertaking of Encon Limited with which you are dealing. Encon Limited is registered in England and Wales with company number 3411533 registered office at The Wolseley Center, Harrison Way, Leamington Spa, CV31 3HH, United Kingdom. VAT No 362 0233 93.
Encon Ireland is registered in Ireland with company number 262605 registered office at Wolseley Ireland Moydrum Road, Athlone, Co Westmeath.
Save a Tree ! - Please think before you print.
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Dear all, Please explain what version of MonetDB you are using. The latest status of MonetDB on our Linux boxes (Current head) is that it can handle without problems a single COPY INTO statement of a table comprised of 13 Billion tuples of 6 columns and 400Million tuples of 500 columns. This is ran over an ascii file and exploits multicore parallelism. Finished in a few hours to consume 2TB. This is part of the SkyServer application we have been porting and which we will release soon. It is a 3TB database covered by a SQL catalog of about 200 pages and for which only an sql-server implemention exist. (http://cas.sdss.org/dr6/en/) Tom Barber wrote:
Sorry Jos,
Matt Casters told me exactly the same on the Pentaho forums, I got a little side tracked data loading.
Cheers
Tom
On Fri, 2008-10-17 at 14:59 +0200, Jos van Dongen wrote:
Hi Tom,
this is due to the fact that Kettle 3.1 still issues an 'Insert Into' statement which MonetDB doesn't handle very well. It's a known issue (see http://wiki.pentaho.com/display/EAI/MonetDB ) which should be resolved in the next Kettle release. I'll forward your experiences to the Kettle team so they can take notice of it as well.
Met vriendelijke groet/Kind regards,
Tom Barber wrote:
Hi Guys,
As of 3.1 Kettle now has a connector built in for Monet so I thought I'd give it another go after not getting too far earlier in the year. Anyway I have it up and running on a Quad Core Xeon box running Centos x86_64 and it seems pretty happy, but when I try to copy sizable amounts of data into it, kettle stops with an error:
2008/10/17 12:29:22 - Table output.0 - Prepared statement : INSERT INTO voc.d_budgets3 (entity_anl_code, trad_code, Cost, Sales, CostMTD, SalesMTD) VALUES ( ?, ?, ?, ?, ?, ?) 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Because of an error, this step can't continue: 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : org.pentaho.di.core.exception.KettleException: 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Error inserting row into table [d_budgets3] with values: [19], [A09], [0], [0], [0], [0] 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Error inserting/updating row 2008/10/17 12:29:27 - Table output.0 - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : current transaction is aborted (please ROLLBACK)
But to my untrained eye the data looks fine(coming from a mysql database) and I don't know where to start debugging this or what settings I could change. The monet table is 2 Strings, 2 Int, 2 Big Num and it always inputs a load of data before stopping.
What can I look at to resolve the issues? otherwise, rocking database and keep up the good work.
Thanks Tom CONFIDENTIALITY NOTICE
This e-mail is confidential and may also be privileged. Please notify us immediately if you are not the intended recipient. You should not copy it, forward it or use it for any purpose or disclose the contents to any person. If you have received this e-mail in error, please notify us as soon as possible and destroy your copy.
Unless otherwise expressly stated, this e-mail is not intended to constitute a business letter, order form or other offer or invitation to you, nor does this e-mail form the basis of any contract. Contracts may not be concluded by e-mail and any contract or agreement attached is subject to contract and shall not and is not intended to create a legally binding relationship.
This e-mail is sent by or on behalf of Encon Limited or the relevant subsidiary undertaking of Encon Limited with which you are dealing. Encon Limited is registered in England and Wales with company number 3411533 registered office at The Wolseley Center, Harrison Way, Leamington Spa, CV31 3HH, United Kingdom. VAT No 362 0233 93.
Encon Ireland is registered in Ireland with company number 262605 registered office at Wolseley Ireland Moydrum Road, Athlone, Co Westmeath.
Save a Tree ! - Please think before you print.
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (3)
-
Jos van Dongen -
Martin Kersten -
Tom Barber