[MonetDB-users] MonetDB XQuery in Production
The product looks very promising on paper, although not that many people are talking about it and I came across MonetDB by accident. Is anyone using MonetDB in production? Is the product stable enough? I do not feel like spending lots of time testing this XQuery server if nobody runs it in production. I loaded several million records and tested sample queries without issues for the most part, but then once the server just vanished (running latest 32bit version on WindowsXP). I restarted and everything is working again, but now I am very scared. Nothing like this ever happened when using MySQL, MS SQL Server or Oracle. Nothing ever just dies without a trace. Thank you. -Max
Max Fedorov wrote:
The product looks very promising on paper, although not that many people are talking about it and I came across MonetDB by accident. Is anyone using MonetDB in production? Is the product stable enough? I do not feel like spending lots of time testing this XQuery server if nobody runs it in production. I loaded several million records and tested sample queries without issues for the most part, but then once the server just vanished (running latest 32bit version on WindowsXP). I restarted and everything is working again, but now I am very scared. Nothing like this ever happened when using MySQL, MS SQL Server or Oracle. Nothing ever just dies without a trace. Max,
Thanks for trying out the XQuery package. I guess you have been using either Oracle or MySQL system to run a realistic XQuery workload. If so, we would like to learn the experiences in relationship to those. You might consider the size of your database in relationship to the platform you are using. And the characteristics of MonetDB as a database engine that squeezes performance out of it. If a process dies without a trace, and deals with these sizes on a small platform, then it is not unlikely that the OS has identified it as a victim to free up resources. The recovery mechanism ensures that the data is not lost. FYI, one of the customers using the XQuery engine on a daily basis is the Dutch Forensics Institute. regards, Martin Kersten
Thank you. -Max
------------------------------------------------------------------------
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com
------------------------------------------------------------------------
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Thanks for response.
Currently, I am using MySQL to store Websphere MQ statistics for our
software product.
Clients begin complaining about reporting performance once the size goes
beyond 3-4mill records :)
I tried BDB XML (latest), but after loading 50K xml records into collection
it takes many hours to start :)
I loaded around 5 million same XML records into MonetDB and still have great
performance.
Thanks.
-Max
On Tue, Apr 14, 2009 at 12:23 PM, Martin Kersten
Max Fedorov wrote:
The product looks very promising on paper, although not that many people are talking about it and I came across MonetDB by accident. Is anyone using MonetDB in production? Is the product stable enough? I do not feel like spending lots of time testing this XQuery server if nobody runs it in production. I loaded several million records and tested sample queries without issues for the most part, but then once the server just vanished (running latest 32bit version on WindowsXP). I restarted and everything is working again, but now I am very scared. Nothing like this ever happened when using MySQL, MS SQL Server or Oracle. Nothing ever just dies without a trace. Max,
Thanks for trying out the XQuery package.
I guess you have been using either Oracle or MySQL system to run a realistic XQuery workload. If so, we would like to learn the experiences in relationship to those.
You might consider the size of your database in relationship to the platform you are using. And the characteristics of MonetDB as a database engine that squeezes performance out of it. If a process dies without a trace, and deals with these sizes on a small platform, then it is not unlikely that the OS has identified it as a victim to free up resources. The recovery mechanism ensures that the data is not lost.
FYI, one of the customers using the XQuery engine on a daily basis is the Dutch Forensics Institute.
regards, Martin Kersten
Thank you. -Max
------------------------------------------------------------------------
------------------------------------------------------------------------------
This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com
------------------------------------------------------------------------
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Max Fedorov Netflexity, Ltd. tel: 267-408-3699 fax:888-201-7922 www.netflexity.com
On Tue, Apr 14, 2009 at 12:45:52PM -0400, Max Fedorov wrote:
Thanks for response. Currently, I am using MySQL to store Websphere MQ statistics for our software product. What XML extension to MySQL are you using?
Niels
Clients begin complaining about reporting performance once the size goes beyond 3-4mill records :) I tried BDB XML (latest), but after loading 50K xml records into collection it takes many hours to start :) I loaded around 5 million same XML records into MonetDB and still have great performance. Thanks. -Max
On Tue, Apr 14, 2009 at 12:23 PM, Martin Kersten <[1]Martin.Kersten@cwi.nl> wrote:
Max Fedorov wrote:
The product looks very promising on paper, although not that many people are talking about it and I came across MonetDB by accident. Is anyone using MonetDB in production? Is the product stable enough? I do not feel like spending lots of time testing this XQuery server if nobody runs it in production. I loaded several million records and tested sample queries without issues for the most part, but then once the server just vanished (running latest 32bit version on WindowsXP). I restarted and everything is working again, but now I am very scared. Nothing like this ever happened when using MySQL, MS SQL Server or Oracle. Nothing ever just dies without a trace.
Max, Thanks for trying out the XQuery package. I guess you have been using either Oracle or MySQL system to run a realistic XQuery workload. If so, we would like to learn the experiences in relationship to those. You might consider the size of your database in relationship to the platform you are using. And the characteristics of MonetDB as a database engine that squeezes performance out of it. If a process dies without a trace, and deals with these sizes on a small platform, then it is not unlikely that the OS has identified it as a victim to free up resources. The recovery mechanism ensures that the data is not lost. FYI, one of the customers using the XQuery engine on a daily basis is the Dutch Forensics Institute. regards, Martin Kersten > > Thank you. > -Max > > > ------------------------------------------------------------ ------------ > > ------------------------------------------------------------ ------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > [2]http://p.sf.net/sfu/www-ibm-com > > > ------------------------------------------------------------ ------------ > > _______________________________________________ > MonetDB-users mailing list > [3]MonetDB-users@lists.sourceforge.net > [4]https://lists.sourceforge.net/lists/listinfo/monetdb-user s ------------------------------------------------------------ ------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! [5]http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list [6]MonetDB-users@lists.sourceforge.net [7]https://lists.sourceforge.net/lists/listinfo/monetdb-user s
-- Max Fedorov Netflexity, Ltd. tel: 267-408-3699 fax:888-201-7922 [8]www.netflexity.com
References
1. mailto:Martin.Kersten@cwi.nl 2. http://p.sf.net/sfu/www-ibm-com 3. mailto:MonetDB-users@lists.sourceforge.net 4. https://lists.sourceforge.net/lists/listinfo/monetdb-users 5. http://p.sf.net/sfu/www-ibm-com 6. mailto:MonetDB-users@lists.sourceforge.net 7. https://lists.sourceforge.net/lists/listinfo/monetdb-users 8. http://www.netflexity.com/
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Max, out of couriosity: does MySQL and/or the XML extension(s) that you are using also provide XPath, XQuery, and/or XQuery Update functionality? And to align use of language/term: What exactly do you refer to as "records" wrt. XML data? Thanks! Stefan On Tue, Apr 14, 2009 at 10:26:14PM +0200, Niels Nes wrote:
On Tue, Apr 14, 2009 at 12:45:52PM -0400, Max Fedorov wrote:
Thanks for response. Currently, I am using MySQL to store Websphere MQ statistics for our software product. What XML extension to MySQL are you using?
Niels
Clients begin complaining about reporting performance once the size goes beyond 3-4mill records :) I tried BDB XML (latest), but after loading 50K xml records into collection it takes many hours to start :) I loaded around 5 million same XML records into MonetDB and still have great performance. Thanks. -Max
On Tue, Apr 14, 2009 at 12:23 PM, Martin Kersten <[1]Martin.Kersten@cwi.nl> wrote:
Max Fedorov wrote:
The product looks very promising on paper, although not that many people are talking about it and I came across MonetDB by accident. Is anyone using MonetDB in production? Is the product stable enough? I do not feel like spending lots of time testing this XQuery server if nobody runs it in production. I loaded several million records and tested sample queries without issues for the most part, but then once the server just vanished (running latest 32bit version on WindowsXP). I restarted and everything is working again, but now I am very scared. Nothing like this ever happened when using MySQL, MS SQL Server or Oracle. Nothing ever just dies without a trace.
Max, Thanks for trying out the XQuery package. I guess you have been using either Oracle or MySQL system to run a realistic XQuery workload. If so, we would like to learn the experiences in relationship to those. You might consider the size of your database in relationship to the platform you are using. And the characteristics of MonetDB as a database engine that squeezes performance out of it. If a process dies without a trace, and deals with these sizes on a small platform, then it is not unlikely that the OS has identified it as a victim to free up resources. The recovery mechanism ensures that the data is not lost. FYI, one of the customers using the XQuery engine on a daily basis is the Dutch Forensics Institute. regards, Martin Kersten > > Thank you. > -Max > > > ------------------------------------------------------------ ------------ > > ------------------------------------------------------------ ------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > [2]http://p.sf.net/sfu/www-ibm-com > > > ------------------------------------------------------------ ------------ > > _______________________________________________ > MonetDB-users mailing list > [3]MonetDB-users@lists.sourceforge.net > [4]https://lists.sourceforge.net/lists/listinfo/monetdb-user s ------------------------------------------------------------ ------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! [5]http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list [6]MonetDB-users@lists.sourceforge.net [7]https://lists.sourceforge.net/lists/listinfo/monetdb-user s
-- Max Fedorov Netflexity, Ltd. tel: 267-408-3699 fax:888-201-7922 [8]www.netflexity.com
References
1. mailto:Martin.Kersten@cwi.nl 2. http://p.sf.net/sfu/www-ibm-com 3. mailto:MonetDB-users@lists.sourceforge.net 4. https://lists.sourceforge.net/lists/listinfo/monetdb-users 5. http://p.sf.net/sfu/www-ibm-com 6. mailto:MonetDB-users@lists.sourceforge.net 7. https://lists.sourceforge.net/lists/listinfo/monetdb-users 8. http://www.netflexity.com/
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Hi, I was storing very little amount of compressed XML in MySQL as BLOBs. I
entirely changed my domain model for MonetDB.
Here is the record I am talking about:
<queueStat>
<id>fb4d6de1-d9ee-4a0f-931f-51a381665ef9</id>
<qmanagerid>7b551e14-ee94-4d0c-8973-689f4d1dcef3</qmanagerid>
<queueid>c45f92c5-dc06-478b-b1ca-a3ae133bd7a6</queueid>
<queueName>c45f92c5-dc06-478b-b1ca-a3ae133bd7a6</queueName>
<fetchStart>1238953466670</fetchStart>
<fetchEnd>1238953466670</fetchEnd>
<messagesIn>825.4472306961399</messagesIn>
<messagesOut>156.09297299884696</messagesOut>
<highDepth>1084.167374456622</highDepth>
</queueStat>
Thanks.
On Tue, Apr 14, 2009 at 4:40 PM, Stefan Manegold
Max,
out of couriosity: does MySQL and/or the XML extension(s) that you are using also provide XPath, XQuery, and/or XQuery Update functionality?
And to align use of language/term: What exactly do you refer to as "records" wrt. XML data?
Thanks!
Stefan
On Tue, Apr 14, 2009 at 10:26:14PM +0200, Niels Nes wrote:
On Tue, Apr 14, 2009 at 12:45:52PM -0400, Max Fedorov wrote:
Thanks for response. Currently, I am using MySQL to store Websphere MQ statistics for our software product. What XML extension to MySQL are you using?
Niels
Clients begin complaining about reporting performance once the size goes beyond 3-4mill records :) I tried BDB XML (latest), but after loading 50K xml records into collection it takes many hours to start :) I loaded around 5 million same XML records into MonetDB and still have great performance. Thanks. -Max
On Tue, Apr 14, 2009 at 12:23 PM, Martin Kersten <[1]Martin.Kersten@cwi.nl> wrote:
Max Fedorov wrote:
The product looks very promising on paper, although not that many people are talking about it and I came across MonetDB by accident. Is anyone using MonetDB in production? Is the product stable enough? I do not feel like spending lots of time testing this XQuery server if nobody runs it in production. I loaded several million records and tested sample queries without issues for the most part, but then once the server just vanished (running latest 32bit version on WindowsXP). I restarted and everything is working again, but now I am very scared. Nothing like this ever happened when using MySQL, MS SQL Server or Oracle. Nothing ever just dies without a trace.
Max, Thanks for trying out the XQuery package. I guess you have been using either Oracle or MySQL system to run a realistic XQuery workload. If so, we would like to learn the experiences in relationship to those. You might consider the size of your database in relationship to the platform you are using. And the characteristics of MonetDB as a database engine that squeezes performance out of it. If a process dies without a trace, and deals with these sizes on a small platform, then it is not unlikely that the OS has identified it as a victim to free up resources. The recovery mechanism ensures that the data is not lost. FYI, one of the customers using the XQuery engine on a daily basis is the Dutch Forensics Institute. regards, Martin Kersten > > Thank you. > -Max > > > ------------------------------------------------------------ ------------ > > ------------------------------------------------------------ ------------------ > This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > [2]http://p.sf.net/sfu/www-ibm-com > > > ------------------------------------------------------------ ------------ > > _______________________________________________ > MonetDB-users mailing list > [3]MonetDB-users@lists.sourceforge.net > [4]https://lists.sourceforge.net/lists/listinfo/monetdb-user s ------------------------------------------------------------ ------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! [5]http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list [6]MonetDB-users@lists.sourceforge.net [7]https://lists.sourceforge.net/lists/listinfo/monetdb-user s
-- Max Fedorov Netflexity, Ltd. tel: 267-408-3699 fax:888-201-7922 [8]www.netflexity.com
References
1. mailto:Martin.Kersten@cwi.nl 2. http://p.sf.net/sfu/www-ibm-com 3. mailto:MonetDB-users@lists.sourceforge.net 4. https://lists.sourceforge.net/lists/listinfo/monetdb-users 5. http://p.sf.net/sfu/www-ibm-com 6. mailto:MonetDB-users@lists.sourceforge.net 7. https://lists.sourceforge.net/lists/listinfo/monetdb-users 8. http://www.netflexity.com/
This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels http://www.cwi.nl/%7Eniels e-mail: Niels.Nes@cwi.nl
------------------------------------------------------------------------------
This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/http://www.cwi.nl/%7Emanegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Max Fedorov Netflexity, Ltd. tel: 267-408-3699 fax:888-201-7922 www.netflexity.com
participants (4)
-
Martin Kersten -
Max Fedorov -
Niels Nes -
Stefan Manegold