[MonetDB-users] mclient - copy into problem - users-list

[MonetDB-users] mclient - copy into problem

older
[MonetDB-users] Running into swap...

dariuszs

30 Jan 2009 30 Jan '09

6:06 p.m.

Hi, Please verify this - when you output a file from the table with 'copy... into...' and terminate that client - the output file is still being created even though the client was terminated. Thanks. Dariusz.

Show replies by date

Martin Kersten

30 Jan 30 Jan

7:22 p.m.

dariuszs wrote:

...

Hi, Please verify this - when you output a file from the table with 'copy... into...' and terminate that client - the output file is still being created even though the client was terminated. Thanks. Dariusz. Of course. mclient and server are two independent processes. And MonetDB does not let you abort a query easily. So the back-end happily continous to finish the query.

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

dariuszs

7:27 p.m.

Hi, Well then how do you stop that process? Terminate server? And if that's a production machine? Log off hundreds of people? Martin Kersten wrote:

...

dariuszs wrote:

...
Hi, Please verify this - when you output a file from the table with 'copy... into...' and terminate that client - the output file is still being created even though the client was terminated. Thanks. Dariusz.

Of course. mclient and server are two independent processes. And MonetDB does not let you abort a query easily. So the back-end happily continous to finish the query.

...
------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Martin Kersten

7:32 p.m.

...

Hi, Well then how do you stop that process? Terminate server? And if that's a production machine? Log off hundreds of people? The server does not see when a client ceases until it requests the next statement. The protocol does not include an out of bound message to warn

dariuszs wrote: the server. Even if it could, the bulk-processing approach leads to at best stopping between two MAL instructions. Such an instruction can take a lot of time. The developments aimed at distribution and automatic horizontal. partitioning are the handles to better control 'early escape'

...

Martin Kersten wrote:

...
dariuszs wrote:

...
Hi, Please verify this - when you output a file from the table with 'copy... into...' and terminate that client - the output file is still being created even though the client was terminated. Thanks. Dariusz.

Of course. mclient and server are two independent processes. And MonetDB does not let you abort a query easily. So the back-end happily continous to finish the query.

...
------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

dariuszs

7:44 p.m.

Hi, So this database was never intended for production use? Dariusz. Martin Kersten wrote:

...

...
Hi, Well then how do you stop that process? Terminate server? And if that's a production machine? Log off hundreds of people? The server does not see when a client ceases until it requests the next statement. The protocol does not include an out of bound message to warn

dariuszs wrote: the server. Even if it could, the bulk-processing approach leads to at best stopping between two MAL instructions. Such an instruction can take a lot of time.

The developments aimed at distribution and automatic horizontal. partitioning are the handles to better control 'early escape'

...
Martin Kersten wrote:

...
dariuszs wrote:

...
Hi, Please verify this - when you output a file from the table with 'copy... into...' and terminate that client - the output file is still being created even though the client was terminated. Thanks. Dariusz.

Of course. mclient and server are two independent processes. And MonetDB does not let you abort a query easily. So the back-end happily continous to finish the query.

...
------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Mark V

11:13 p.m.

On Sat, Jan 31, 2009 at 6:44 AM, dariuszs wrote:

...

Hi, So this database was never intended for production use? Dariusz.

You've designed your application to do/allow fragile/volatile/experimental things on a production server, a server which hundreds of people depend on - and its MonetDB's fault?

...

Martin Kersten wrote:

...
...
Hi, Well then how do you stop that process? Terminate server? And if that's a production machine? Log off hundreds of people? The server does not see when a client ceases until it requests the next statement. The protocol does not include an out of bound message to warn

dariuszs wrote: the server. Even if it could, the bulk-processing approach leads to at best stopping between two MAL instructions. Such an instruction can take a lot of time.

The developments aimed at distribution and automatic horizontal. partitioning are the handles to better control 'early escape'

...
Martin Kersten wrote:

...
dariuszs wrote:

...
Hi, Please verify this - when you output a file from the table with 'copy... into...' and terminate that client - the output file is still being created even though the client was terminated. Thanks. Dariusz.

Of course. mclient and server are two independent processes. And MonetDB does not let you abort a query easily. So the back-end happily continous to finish the query.

...
------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

dariuszs

31 Jan 31 Jan

12:22 a.m.

Hi, I did not say It's anybodys' fault, I'm simply evaluating the software, I think the software is amazing. All I was asking was - what would I do if I was going to use it in production and I had to deal with situation like this, that's all. Thanks for hour help. Dariusz. Mark V wrote:

...

On Sat, Jan 31, 2009 at 6:44 AM, dariuszs wrote:

...
Hi, So this database was never intended for production use? Dariusz.

You've designed your application to do/allow fragile/volatile/experimental things on a production server, a server which hundreds of people depend on - and its MonetDB's fault?

...
Martin Kersten wrote:

...
dariuszs wrote:

...
Hi, Well then how do you stop that process? Terminate server? And if that's a production machine? Log off hundreds of people?

The server does not see when a client ceases until it requests the next statement. The protocol does not include an out of bound message to warn the server. Even if it could, the bulk-processing approach leads to at best stopping between two MAL instructions. Such an instruction can take a lot of time.

The developments aimed at distribution and automatic horizontal. partitioning are the handles to better control 'early escape'

...
Martin Kersten wrote:

...
dariuszs wrote:

...
Hi, Please verify this - when you output a file from the table with 'copy... into...' and terminate that client - the output file is still being created even though the client was terminated. Thanks. Dariusz.

Of course. mclient and server are two independent processes. And MonetDB does not let you abort a query easily. So the back-end happily continous to finish the query.

...
------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Mark V

12:42 a.m.

On Sat, Jan 31, 2009 at 11:22 AM, dariuszs wrote:

...

Hi, I did not say It's anybodys' fault, I'm simply evaluating the software, I think the software is amazing. All I was asking was - what would I do if I was going to use it in production and I had to deal with situation like this, that's all. Thanks for hour help. Dariusz.

Apologies then. I took your question, "So this database was never intended for production use?", to mean that you'd discovered something about MonetDB which meant it is not suitable for production use, and you wanted to know if that was correct. Whereas, to me, it sounded like your app. was still in the development stage. For what it is worth.... I think several MonetDB servers can be run on the same machine using different port numbers. This might allow you to periodically setup a 'temporary' server for long-running/fragile 'stuff'. If things go pear-shaped you can kill it off and leave your production server MonetDB running. If you are in a production environment with very stringent availability constraints, and if your app could conceivably do really wild things, you might like to consider running the temp MonetDB servers on a physically separate machines. Depends what trade-offs you can make. Maybe someone else can make an authoritative comment on any potential pitfalls in either of these architectures. Mark

...

Mark V wrote:

...
On Sat, Jan 31, 2009 at 6:44 AM, dariuszs wrote:

...
Hi, So this database was never intended for production use? Dariusz.

You've designed your application to do/allow fragile/volatile/experimental things on a production server, a server which hundreds of people depend on - and its MonetDB's fault?

...
Martin Kersten wrote:

...
dariuszs wrote:

...
Hi, Well then how do you stop that process? Terminate server? And if that's a production machine? Log off hundreds of people?

The server does not see when a client ceases until it requests the next statement. The protocol does not include an out of bound message to warn the server. Even if it could, the bulk-processing approach leads to at best stopping between two MAL instructions. Such an instruction can take a lot of time.

The developments aimed at distribution and automatic horizontal. partitioning are the handles to better control 'early escape'

...
Martin Kersten wrote:

...
dariuszs wrote:

> > Hi, > Please verify this - when you output a file from the table with > 'copy... into...' and terminate that client - the output file is > still being created even though the client was terminated. Thanks. > Dariusz. > >

Of course. mclient and server are two independent processes. And MonetDB does not let you abort a query easily. So the back-end happily continous to finish the query.

> > > ------------------------------------------------------------------------------ > > This SF.net email is sponsored by: > SourcForge Community > SourceForge wants to tell your story. > http://p.sf.net/sfu/sf-spreadtheword > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > >

------------------------------------------------------------------------------

This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Ross Bates

3:59 a.m.

I think what Dariusz is hitting on is the gray area between MonetDB as a storage engine to wrap an application around and MonetDB as a pure database server dedicated to user queries. His comment about MonetDB not being "production ready" falls into the latter category and didn't appear to be intentionally disparaging. The original email was about how to stop an export process. Here's another related example, if you open things up to users and someone kicks off a query with a cartesian join there is no way to connect to the mserver5 process and kill that individual query without interrupting other queries. Another thing Dariusz inquired about in an earlier email was mserver5 logging. These are all traditional DBA items which are not available in MonetDB. It's really more a perception issue, if you call something a database people are going to have a natural tendency to compare to the other databases they've worked with. Note: this isn't a commentary on what should or shouldn't be included in MonetDB. MonetDB is open source and the beauty is that people can individually or collectively take things wherever they want. Perhaps you could argue that the collective can work......... awww crap, let's just call it a night. ;) -Ross On Fri, Jan 30, 2009 at 6:42 PM, Mark V wrote:

...

On Sat, Jan 31, 2009 at 11:22 AM, dariuszs wrote:

...
Hi, I did not say It's anybodys' fault, I'm simply evaluating the software, I think the software is amazing. All I was asking was - what would I do if I was going to use it in production and I had to deal with situation like this, that's all. Thanks for hour help. Dariusz.

Apologies then. I took your question, "So this database was never intended for production use?", to mean that you'd discovered something about MonetDB which meant it is not suitable for production use, and you wanted to know if that was correct. Whereas, to me, it sounded like your app. was still in the development stage.

For what it is worth.... I think several MonetDB servers can be run on the same machine using different port numbers. This might allow you to periodically setup a 'temporary' server for long-running/fragile 'stuff'. If things go pear-shaped you can kill it off and leave your production server MonetDB running. If you are in a production environment with very stringent availability constraints, and if your app could conceivably do really wild things, you might like to consider running the temp MonetDB servers on a physically separate machines. Depends what trade-offs you can make.

Maybe someone else can make an authoritative comment on any potential pitfalls in either of these architectures.

Mark

...
Mark V wrote:

...
On Sat, Jan 31, 2009 at 6:44 AM, dariuszs wrote:

...
Hi, So this database was never intended for production use? Dariusz.

You've designed your application to do/allow fragile/volatile/experimental things on a production server, a server which hundreds of people depend on - and its MonetDB's fault?

...
Martin Kersten wrote:

...
dariuszs wrote:

...
Hi, Well then how do you stop that process? Terminate server? And if that's a production machine? Log off hundreds of people?

The server does not see when a client ceases until it requests the next statement. The protocol does not include an out of bound message to warn the server. Even if it could, the bulk-processing approach leads to at best

stopping

...
...
...
...
between two MAL instructions. Such an instruction can take a lot of time.

The developments aimed at distribution and automatic horizontal. partitioning are the handles to better control 'early escape'

...
Martin Kersten wrote:

> > dariuszs wrote: > > >> >> Hi, >> Please verify this - when you output a file from the table with >> 'copy... into...' and terminate that client - the output file is >> still being created even though the client was terminated. Thanks. >> Dariusz. >> >> > > Of course. mclient and server are two independent processes. > And MonetDB does not let you abort a query easily. > So the back-end happily continous to finish the query. > > >> >> >>

...
...
...
...
...
>> >> This SF.net email is sponsored by: >> SourcForge Community >> SourceForge wants to tell your story. >> http://p.sf.net/sfu/sf-spreadtheword >> _______________________________________________ >> MonetDB-users mailing list >> MonetDB-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-users >> >> > > >

...
...
...
...
...
> > This SF.net email is sponsored by: > SourcForge Community > SourceForge wants to tell your story. > http://p.sf.net/sfu/sf-spreadtheword > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > >

...
...
...
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Mark V

4:58 a.m.

On Sat, Jan 31, 2009 at 2:59 PM, Ross Bates wrote:

...

I think what Dariusz is hitting on is the gray area between MonetDB as a storage engine to wrap an application around and MonetDB as a pure database server dedicated to user queries. His comment about MonetDB not being "production ready" falls into the latter category and didn't appear to be intentionally disparaging.

The original email was about how to stop an export process. Here's another related example, if you open things up to users and someone kicks off a query with a cartesian join there is no way to connect to the mserver5 process and kill that individual query without interrupting other queries. Another thing Dariusz inquired about in an earlier email was mserver5 logging. These are all traditional DBA items which are not available in MonetDB. It's really more a perception issue, if you call something a database people are going to have a natural tendency to compare to the other databases they've worked with.

Sure, you can find this a little in the MySQL/Oracle/MSSQL forums - '...but I can do this in Access.' And then you realize they think 'the database' is one and the same thing as 'the GUI query builder'. There is also the perennial, and more difficult, issue of use cases. Some DB servers are optimized and targeted at well defined/behaved queries, other use cases have to allow 'what-ever-you-can-throw-at-it' on a shared server. One of the nice things about MonetDB is the performance. Personally I favour keeping that perfromance the expense of bells and whistles. and at the cost of some convenience. It may be useful to collect some tips on 'traps-for-young-players' about what's different about MonetDB cf <whatever> and how to handle that. E.g. The 'anything-goes' use case might involve suggesting running a 'small-test-first' server which users have to pass through without error to get their script to run on the shared mega-production server, etc., etc. Mark

...

Note: this isn't a commentary on what should or shouldn't be included in MonetDB. MonetDB is open source and the beauty is that people can individually or collectively take things wherever they want. Perhaps you could argue that the collective can work......... awww crap, let's just call it a night. ;)

-Ross

On Fri, Jan 30, 2009 at 6:42 PM, Mark V wrote:

...
On Sat, Jan 31, 2009 at 11:22 AM, dariuszs wrote:

...
Hi, I did not say It's anybodys' fault, I'm simply evaluating the software, I think the software is amazing. All I was asking was - what would I do if I was going to use it in production and I had to deal with situation like this, that's all. Thanks for hour help. Dariusz.

Apologies then. I took your question, "So this database was never intended for production use?", to mean that you'd discovered something about MonetDB which meant it is not suitable for production use, and you wanted to know if that was correct. Whereas, to me, it sounded like your app. was still in the development stage.

For what it is worth.... I think several MonetDB servers can be run on the same machine using different port numbers. This might allow you to periodically setup a 'temporary' server for long-running/fragile 'stuff'. If things go pear-shaped you can kill it off and leave your production server MonetDB running. If you are in a production environment with very stringent availability constraints, and if your app could conceivably do really wild things, you might like to consider running the temp MonetDB servers on a physically separate machines. Depends what trade-offs you can make.

Maybe someone else can make an authoritative comment on any potential pitfalls in either of these architectures.

Mark

...
Mark V wrote:

...
On Sat, Jan 31, 2009 at 6:44 AM, dariuszs wrote:

...
Hi, So this database was never intended for production use? Dariusz.

You've designed your application to do/allow fragile/volatile/experimental things on a production server, a server which hundreds of people depend on - and its MonetDB's fault?

...
Martin Kersten wrote:

...
dariuszs wrote:

> > Hi, > Well then how do you stop that process? Terminate server? And if > that's a production machine? Log off hundreds of people? >

The server does not see when a client ceases until it requests the next statement. The protocol does not include an out of bound message to warn the server. Even if it could, the bulk-processing approach leads to at best stopping between two MAL instructions. Such an instruction can take a lot of time.

The developments aimed at distribution and automatic horizontal. partitioning are the handles to better control 'early escape'

> > Martin Kersten wrote: > >> >> dariuszs wrote: >> >> >>> >>> Hi, >>> Please verify this - when you output a file from the table with >>> 'copy... into...' and terminate that client - the output file is >>> still being created even though the client was terminated. Thanks. >>> Dariusz. >>> >>> >> >> Of course. mclient and server are two independent processes. >> And MonetDB does not let you abort a query easily. >> So the back-end happily continous to finish the query. >> >> >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> This SF.net email is sponsored by: >>> SourcForge Community >>> SourceForge wants to tell your story. >>> http://p.sf.net/sfu/sf-spreadtheword >>> _______________________________________________ >>> MonetDB-users mailing list >>> MonetDB-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/monetdb-users >>> >>> >> >> >> >> ------------------------------------------------------------------------------ >> >> This SF.net email is sponsored by: >> SourcForge Community >> SourceForge wants to tell your story. >> http://p.sf.net/sfu/sf-spreadtheword >> _______________________________________________ >> MonetDB-users mailing list >> MonetDB-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-users >> >> >> >> >> > >

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

dariuszs

6:24 p.m.

Hello, I think Mark is correct, what makes this software great it's the performance. If we can control server/client interaction from the application/os level then that's perfectly fine. The speed of MonetDB is unbelievable, it's very simple to use, excellent support. I've dealt with lot's of different concepts using b-trees, hash, bitmaps and nothing is as easy and fast as MonetDB, great design. Thanks for your help. Dariusz. Mark V wrote:

...

On Sat, Jan 31, 2009 at 2:59 PM, Ross Bates wrote:

...
I think what Dariusz is hitting on is the gray area between MonetDB as a storage engine to wrap an application around and MonetDB as a pure database server dedicated to user queries. His comment about MonetDB not being "production ready" falls into the latter category and didn't appear to be intentionally disparaging.

The original email was about how to stop an export process. Here's another related example, if you open things up to users and someone kicks off a query with a cartesian join there is no way to connect to the mserver5 process and kill that individual query without interrupting other queries. Another thing Dariusz inquired about in an earlier email was mserver5 logging. These are all traditional DBA items which are not available in MonetDB. It's really more a perception issue, if you call something a database people are going to have a natural tendency to compare to the other databases they've worked with.

Sure, you can find this a little in the MySQL/Oracle/MSSQL forums - '...but I can do this in Access.' And then you realize they think 'the database' is one and the same thing as 'the GUI query builder'. There is also the perennial, and more difficult, issue of use cases. Some DB servers are optimized and targeted at well defined/behaved queries, other use cases have to allow 'what-ever-you-can-throw-at-it' on a shared server.

One of the nice things about MonetDB is the performance. Personally I favour keeping that perfromance the expense of bells and whistles. and at the cost of some convenience.

It may be useful to collect some tips on 'traps-for-young-players' about what's different about MonetDB cf <whatever> and how to handle that. E.g. The 'anything-goes' use case might involve suggesting running a 'small-test-first' server which users have to pass through without error to get their script to run on the shared mega-production server, etc., etc.

Mark

...
Note: this isn't a commentary on what should or shouldn't be included in MonetDB. MonetDB is open source and the beauty is that people can individually or collectively take things wherever they want. Perhaps you could argue that the collective can work......... awww crap, let's just call it a night. ;)

-Ross

On Fri, Jan 30, 2009 at 6:42 PM, Mark V wrote:

...
On Sat, Jan 31, 2009 at 11:22 AM, dariuszs wrote:

...
Hi, I did not say It's anybodys' fault, I'm simply evaluating the software, I think the software is amazing. All I was asking was - what would I do if I was going to use it in production and I had to deal with situation like this, that's all. Thanks for hour help. Dariusz.

Apologies then. I took your question, "So this database was never intended for production use?", to mean that you'd discovered something about MonetDB which meant it is not suitable for production use, and you wanted to know if that was correct. Whereas, to me, it sounded like your app. was still in the development stage.

For what it is worth.... I think several MonetDB servers can be run on the same machine using different port numbers. This might allow you to periodically setup a 'temporary' server for long-running/fragile 'stuff'. If things go pear-shaped you can kill it off and leave your production server MonetDB running. If you are in a production environment with very stringent availability constraints, and if your app could conceivably do really wild things, you might like to consider running the temp MonetDB servers on a physically separate machines. Depends what trade-offs you can make.

Maybe someone else can make an authoritative comment on any potential pitfalls in either of these architectures.

Mark

...
Mark V wrote:

...
On Sat, Jan 31, 2009 at 6:44 AM, dariuszs wrote:

...
Hi, So this database was never intended for production use? Dariusz.

You've designed your application to do/allow fragile/volatile/experimental things on a production server, a server which hundreds of people depend on - and its MonetDB's fault?

...
Martin Kersten wrote:

> dariuszs wrote: > > >> Hi, >> Well then how do you stop that process? Terminate server? And if >> that's a production machine? Log off hundreds of people? >> >> > The server does not see when a client ceases until it requests the > next statement. The protocol does not include an out of bound message > to warn > the server. > Even if it could, the bulk-processing approach leads to at best > stopping > between two MAL instructions. Such an instruction can take a lot of > time. > > The developments aimed at distribution and automatic horizontal. > partitioning are the handles to better control 'early escape' > > >> Martin Kersten wrote: >> >> >>> dariuszs wrote: >>> >>> >>> >>>> Hi, >>>> Please verify this - when you output a file from the table with >>>> 'copy... into...' and terminate that client - the output file is >>>> still being created even though the client was terminated. Thanks. >>>> Dariusz. >>>> >>>> >>>> >>> Of course. mclient and server are two independent processes. >>> And MonetDB does not let you abort a query easily. >>> So the back-end happily continous to finish the query. >>> >>> >>> >>>> >>>> ------------------------------------------------------------------------------ >>>> >>>> This SF.net email is sponsored by: >>>> SourcForge Community >>>> SourceForge wants to tell your story. >>>> http://p.sf.net/sfu/sf-spreadtheword >>>> _______________________________________________ >>>> MonetDB-users mailing list >>>> MonetDB-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/monetdb-users >>>> >>>> >>>> >>> >>> ------------------------------------------------------------------------------ >>> >>> This SF.net email is sponsored by: >>> SourcForge Community >>> SourceForge wants to tell your story. >>> http://p.sf.net/sfu/sf-spreadtheword >>> _______________________________________________ >>> MonetDB-users mailing list >>> MonetDB-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/monetdb-users >>> >>> >>> >>> >>> >>> >> > >

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Fabian Groffen

8:39 a.m.

On 30-01-2009 21:59:15 -0600, Ross Bates wrote:

...

The original email was about how to stop an export process. Here's another related example, if you open things up to users and someone kicks off a query with a cartesian join there is no way to connect to the mserver5 process and kill that individual query without interrupting other queries. Another thing Dariusz inquired about in an earlier email was mserver5 logging. These are all traditional DBA items which are not available in MonetDB. It's really more a perception issue, if you call something a database people are going to have a natural tendency to compare to the other databases they've worked with.

I think you address two interesting points here. The first being to be able to shoot a client, e.g. think of MySQL's 'KILL <tid>;' syntax (by the way, that most of the time doesn't work when you really need it, for instance when something's locked, but that aside), and the second being a request for logging information. As creator of Merovingian, I had the latter one in mind when I wrote it. Merovingian's log hence shows some information about what it is doing, as well as what mserver5 processes write out, which usually only are errors and warnings. What kind of logging information are you looking for? Please note that Merovingian is not available on the Windows platform. As to the possibility to kill a client, perhaps we can just create a hook at the MAL level. It is worth to open up a feature request for this, I think.

Martin Kersten

9:23 a.m.

...

On 30-01-2009 21:59:15 -0600, Ross Bates wrote:

As to the possibility to kill a client, perhaps we can just create a hook at the MAL level. It is worth to open up a feature request for this, I think. The hooks are available in the kernel to stop a client between individual MAL statements. It requires protocol extensions for out of bound messages, or a secure scheme to open a second connection using the same user credentials to shoot off another. Then a SQL command like "call resetClient()" could be made to do

Fabian Groffen wrote: the trick. This does not solve the point raised, i.e. to stop instructions in the middle of their processing. This is one of the design issues why MonetDB can be fast, i.e. in the inner loop of bulk operations avoid time checking environment status.

Mark V

9:36 a.m.

On Sat, Jan 31, 2009 at 8:23 PM, Martin Kersten wrote:

...

...
On 30-01-2009 21:59:15 -0600, Ross Bates wrote:

As to the possibility to kill a client, perhaps we can just create a hook at the MAL level. It is worth to open up a feature request for this, I think. The hooks are available in the kernel to stop a client between individual MAL statements. It requires protocol extensions for out of bound messages, or a secure scheme to open a second connection using the same user credentials to shoot off another. Then a SQL command like "call resetClient()" could be made to do

Fabian Groffen wrote: the trick.

This does not solve the point raised, i.e. to stop instructions in the middle of their processing. This is one of the design issues why MonetDB can be fast, i.e. in the inner loop of bulk operations avoid time checking environment status.

Please don't compromise on speed. Please. I can architect and write my apps to ensure they are 'well behaved', whatever that means from MonetDB's perspective - I can't make MonetDB faster than it is :) Mark

...

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Fabian Groffen

9:58 a.m.

On 31-01-2009 10:23:42 +0100, Martin Kersten wrote:

...

...
On 30-01-2009 21:59:15 -0600, Ross Bates wrote:

As to the possibility to kill a client, perhaps we can just create a hook at the MAL level. It is worth to open up a feature request for this, I think. The hooks are available in the kernel to stop a client between individual MAL statements. It requires protocol extensions for out of bound messages, or a secure scheme to open a second connection using the same user credentials to shoot off another. Then a SQL command like "call resetClient()" could be made to do

Fabian Groffen wrote: the trick.

If you login as admin, you can simply use "KILL", otherwise you can't. Simple.

...

This does not solve the point raised, i.e. to stop instructions in the middle of their processing. This is one of the design issues why MonetDB can be fast, i.e. in the inner loop of bulk operations avoid time checking environment status.

Yes, but it does allow to stop a client earlier than it is now. E.g. it allows you to shoot a client that you want to get off of your system, such as when you bring it down to maintenance mode. When you get tired of waiting, you just shoot the clients that are still connected/idle.

Martin Kersten

12:56 p.m.

Fabian Groffen wrote:

...

On 31-01-2009 10:23:42 +0100, Martin Kersten wrote:

...
...
On 30-01-2009 21:59:15 -0600, Ross Bates wrote:

As to the possibility to kill a client, perhaps we can just create a hook at the MAL level. It is worth to open up a feature request for this, I think. The hooks are available in the kernel to stop a client between individual MAL statements. It requires protocol extensions for out of bound messages, or a secure scheme to open a second connection using the same user credentials to shoot off another. Then a SQL command like "call resetClient()" could be made to do

Fabian Groffen wrote: the trick.

If you login as admin, you can simply use "KILL", otherwise you can't. Simple. Well, src/modules/mal/client.mx contains hooks to play with: client.suspend, client.resume, client.stop, client.shutdown. Extensive testing has not been performed.

...
This does not solve the point raised, i.e. to stop instructions in the middle of their processing. This is one of the design issues why MonetDB can be fast, i.e. in the inner loop of bulk operations avoid time checking environment status.

Yes, but it does allow to stop a client earlier than it is now. E.g. it allows you to shoot a client that you want to get off of your system, such as when you bring it down to maintenance mode. When you get tired of waiting, you just shoot the clients that are still connected/idle. Idle clients are no issue. Their connections can be killed safely.

Ross Bates

6:29 p.m.

@markv - I completely agree, the speed of MonetDB is what makes it so compelling and I don't think performance should be compromised. As for features, here's another way to frame the discussion. If you were going to promote MonetDB to a colleague what product(s) would you compare it to? Row based, column based, embedded, open source, commercial, etc? Or, if MonetDB didn't exist what would you be using instead? MDB is an excellent product but in my opinion it doesn't receive the full attention it deserves because it's kinda' stuck between application storage engine and database server. Just hoping to spur a community discussion to see what types of things are important to people and see if there is a common thread. Back to the tech discussion.

...

If you login as admin, you can simply use "KILL", otherwise you can't. Simple.

Using the cartesian join example again , the way I understood Martin's email is that killing the client would only kill the thread which launched the command and the mserver5 process would keep running unaware that the client had stopped. Additionally if the client was remote you wouldn't have the ability to kill it directly. I don't think resource optimization wouldn't need to be built into the server, just the ability to look inside and stop individual requests. Which brings up something Fabian mentioned, Merovingian. Perhaps the discussion about logging and client management moves towards mero. It makes sense, mserver5 can continue to do what it does best while admin/operations are handled at a supervisor level. Merovingian already has a base for things like clustering and failover. Security could be built out. Detailed query logging with different debug levels could be included. This gives people the option to turn on/off overhead functions. -Ross On Sat, Jan 31, 2009 at 6:56 AM, Martin Kersten wrote:

...

Fabian Groffen wrote:

...
On 31-01-2009 10:23:42 +0100, Martin Kersten wrote:

...
...
On 30-01-2009 21:59:15 -0600, Ross Bates wrote:

As to the possibility to kill a client, perhaps we can just create a hook at the MAL level. It is worth to open up a feature request for this, I think. The hooks are available in the kernel to stop a client between individual MAL statements. It requires protocol extensions for out of bound messages, or a secure scheme to open a second connection using the same user credentials to shoot off another. Then a SQL command like "call resetClient()" could be made to do

Fabian Groffen wrote: the trick.

If you login as admin, you can simply use "KILL", otherwise you can't. Simple. Well, src/modules/mal/client.mx contains hooks to play with: client.suspend, client.resume, client.stop, client.shutdown. Extensive testing has not been performed.

...
This does not solve the point raised, i.e. to stop instructions in the middle of their processing. This is one of the design issues why MonetDB can be fast, i.e. in the inner loop of bulk operations avoid time checking environment status.

Yes, but it does allow to stop a client earlier than it is now. E.g. it allows you to shoot a client that you want to get off of your system, such as when you bring it down to maintenance mode. When you get tired of waiting, you just shoot the clients that are still connected/idle. Idle clients are no issue. Their connections can be killed safely.

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

5881

Age (days ago)

5882

Last active (days ago)

List overview

Download

16 comments

5 participants

participants (5)

dariuszs
Fabian Groffen
Mark V
Martin Kersten
Ross Bates