Hi Hans, On Wed, Dec 10, 2008 at 04:33:22PM +0100, Hans van Rijswijk (DT) wrote:

Hi Stefan,

Thanks for your reply. My answers to several questions are below,

Regards, Hans.

...

-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: woensdag 10 december 2008 15:54 To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] Again: Mserver fails on Fedora 9 - 32 bit

Hi Hans,

On Wed, Dec 10, 2008 at 03:17:04PM +0100, Hans van Rijswijk (DT) wrote:

...

Hi MonetDB users,

I downloaded Monet Fedora RPMs from:

http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./20

...

08 1209/RPMs/Fedora9.i386.oid32/

I installed these with yum on my Fedora fc9 installation, running within VMWare on my Windows XP 32 bit machine.

When I startup Mserver everything seems to work well. I start with an empty dbfarm.

Then I run the following scenario: - Add 3 XML documents with pf:add-doc() (some of them updatable) - Add 2 more XML documents with pf:add-doc() - Update one of these docs: do insert <something/> into doc("a.xml")/rootelement - Send a sync() command with mclient mil.

What did/does sync() return? (i.e., TRUE or FALSE?)

I was using echo "sync();" | mclient -p 51240 -lmil This returned nothing at all.

Well, sync() does return a value of type bit (see below), but you did not check the actual value it returned. Honestly, I'd have to look into the code to know what the return value means, but that only makes sense in case I knew what it returned in your case ... ======== MonetDB>help("sync"); COMMAND: sync() : bit MODULE: trans COMPILED: by adm on Tue Dec 9 18:15:39 2008 save all persistent BATs ======== MonetDB>print(sync()); [ true ] ========

...

...

- Check the content of the database with pf:documents() (is OK) - Knock down Mserver: kill <pid> - Startup Mserver again (don't ask me why I do it like this)

Well, I'd actually like to know why you do it like this --- while MonetDB should of course survive such "rude" usage (in case the mclient update transaction has successfully finished before Mserver is killed, that is), the question is whether this is (for you!) the most convenient way to use it --- we can only give advise on this in case we know, what you actually want to do, and/or why you do things the (odd?) way you do them ...

The scenario is a simplified part of my installation procedure. The first time I start Mserver I initialize the database: put some docs in. After initialization I shutdown all used services, including Mserver. At the end of the installation procedure the user may opt for starting all services, or choose for starting them later. Starting all services involves starting Mserver.

Ok, so since you want/need to run Mserver as "service", you need to run it as daemon in the background, right?

...

...

After this scenario I cannot gain access to Mserver with mclient: !xchange_challenge: frontend xquery not found

What does Mserver say at restart? Any meassges, warnings, errors? Any missing messages?

Sometimes Mserver complains about: !ERROR: logger: could not use bat (441) for 1000000000_rid_kind !ERROR: logger: could not use bat (412) for 1000000000_rid_nid Etc.. (see my previous mail post on this subject)

Ah, this IS very crucial info --- since you didn't repeat that info, here, I assumed this problems was "solved" by using a clean (non-corrupted) database; apparently not ... "Sometimes" holds for the case during the restart after the update+kill, that then resulted in `mclient -lxquery` not being able to connect? Well, since this is a logger error message, my guess that this is a login/logger related problem was not that bad --- for efficiency, it is very handy to always give this kind of crucial info; in particular, if error messages precede a problem, these error messages should (obviously!??) always be clearly shared ...

...

...

If I wait half an hour before killing Mserver this works fine.

Questions: - Does sync() do what it is supposed to do?

I hope so --- however, your updates are most probably still under the control of the logger; hence, sync() makes no difference, here, I suppose --- strage, though that replaying the logs at server start-up does not work ...

...

- Why is shutdown() not implemented?

Lacking user/access management in the MonetDB kernel/server it was/is considered insecure.

Clear!

...

...

- What is the right way to shutdown Mserver when it is started as a daemon in the background?

For the time bing: kill.

So you did all the above with Mserver started in deamon mode? What is you exact command line call to start Mserver? (In case you use any MIL scripts or modification in the MonetDB.conf file, please say so, too.)

Did/could you also try, what happens in case you start Mserver in interactive mode (both on the virgin DB and for the restart after the updates), and shut it down by saying "quit();" on the MIL console?

I will give this a try

Please keep us posted (in detail) about yur experiences!

...

...

- How do I guarantee the database to remain intact with the abovementioned scenario?

As long as your update transaction has sucessfully finished, MonetDB should take care of this. In case is doesn't, I consider this a bug, and you should file a respective bug report including all the information to reproduce the problem.

I am working on a simple scenario to reproduce this. If it appears to be a MonetDB bug, I will file a bug report.

Great. Thanks! Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

Hi Niels, I don't understand: sync() should be a command to commit all changes and flush all buffers to a permanent storage, i.e. harddisk. I also don't understand that a logger should be responsible for commits. But the question remains: What should a client do between the last update query and kill Mserver? The only thing I can think of right now is: WAIT.....and hope the logger eventually commits my changes. But for how long??? Is that right? I think the following happens in my case: I start Mserver, add some documents and perform an update, then sync() (You say it does nothing useful for me) and then kill Mserver. Since Mserver is not ready yet writing all changes to disk, the database gets corrupted. The next time I start Mserver it cannot load some parameters it should have stored in BAT, and refuses to startup the frontend, so mclient commands fail. Even waiting 30 seconds between sync() and kill Mserver does not help. Is that right?? Regards, Hans.

-----Original Message----- From: Niels Nes [mailto:Niels.Nes@cwi.nl] Sent: woensdag 10 december 2008 17:13 To: monetdb-users@lists.sourceforge.net Cc: Stefan Manegold Subject: Re: [MonetDB-users] Again: Mserver fails on Fedora 9 - 32 bit

On Wed, Dec 10, 2008 at 04:52:27PM +0100, Stefan Manegold wrote:

...

Hi Hans,

On Wed, Dec 10, 2008 at 04:33:22PM +0100, Hans van Rijswijk (DT) wrote:

...

Hi Stefan,

Thanks for your reply. My answers to several questions are below,

sync() is bad (should be removed fast!). It does a global commit, but the bats are under control of the logger, ie the logger should deside when to commit.

...

...

Regards, Hans.

...

-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: woensdag 10 december 2008 15:54 To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] Again: Mserver fails on

Fedora 9 - 32

...

...

bit

Hi Hans,

On Wed, Dec 10, 2008 at 03:17:04PM +0100, Hans van Rijswijk (DT) wrote:

...

Hi MonetDB users,

I downloaded Monet Fedora RPMs from:

http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds.

...

...

/20

...

08 1209/RPMs/Fedora9.i386.oid32/

I installed these with yum on my Fedora fc9 installation, running within VMWare on my Windows XP 32 bit machine.

When I startup Mserver everything seems to work well. I start with an empty dbfarm.

Then I run the following scenario: - Add 3 XML documents with pf:add-doc() (some of them updatable) - Add 2 more XML documents with pf:add-doc() - Update one of these docs: do insert <something/> into doc("a.xml")/rootelement - Send a sync() command with mclient mil.

What did/does sync() return? (i.e., TRUE or FALSE?)

I was using echo "sync();" | mclient -p 51240 -lmil This returned nothing at all.

Well, sync() does return a value of type bit (see below), but you did not check the actual value it returned. Honestly, I'd have to look into the code to know what the return value means, but that only makes sense in case I knew what it returned in your case ... ======== MonetDB>help("sync"); COMMAND: sync() : bit MODULE: trans COMPILED: by adm on Tue Dec 9 18:15:39 2008 save all

...

======== MonetDB>print(sync()); [ true ] ========

...

...

...

- Check the content of the database with pf:documents() (is OK) - Knock down Mserver: kill <pid> - Startup Mserver again (don't ask me why I do it like this)

Well, I'd actually like to know why you do it like this --- while MonetDB should of course survive such "rude" usage (in case the mclient update transaction has successfully finished before Mserver is killed, that is), the question is whether this is (for you!) the most convenient way to use it --- we can only give advise on this in case we know, what you actually want to do, and/or why you do things the (odd?) way you do them ...

The scenario is a simplified part of my installation procedure. The first time I start Mserver I initialize the database: put some docs in. After initialization I shutdown all used services, including Mserver. At the end of the installation procedure the user may opt for starting all services, or choose for starting them later. Starting all services involves starting Mserver.

Ok, so since you want/need to run Mserver as "service", you need to run it as daemon in the background, right?

...

...

...

After this scenario I cannot gain access to Mserver

with mclient:

...

...

!xchange_challenge: frontend xquery not found

What does Mserver say at restart? Any meassges, warnings, errors? Any missing messages?

Sometimes Mserver complains about: !ERROR: logger: could not use bat (441) for 1000000000_rid_kind !ERROR: logger: could not use bat (412) for 1000000000_rid_nid Etc.. (see my previous mail post on this subject)

Ah, this IS very crucial info --- since you didn't repeat that info, here, I assumed this problems was "solved" by using a clean (non-corrupted) database; apparently not ... "Sometimes" holds for the case during the restart after the update+kill, that then resulted in `mclient -lxquery` not being able to connect? Well, since this is a logger error message, my guess that this is a login/logger related problem was not that bad --- for efficiency, it is very handy to always give this kind of crucial info; in

Niels persistent BATs particular,

...

if error messages precede a problem, these error messages should (obviously!??) always be clearly shared ...

...

...

...

If I wait half an hour before killing Mserver this works fine.

Questions: - Does sync() do what it is supposed to do?

I hope so --- however, your updates are most probably still under the control of the logger; hence, sync() makes no

difference, here,

...

I suppose --- strage, though that replaying the logs at server start-up does not work ...

...

- Why is shutdown() not implemented?

Lacking user/access management in the MonetDB kernel/server it was/is considered insecure.

Clear!

...

...

- What is the right way to shutdown Mserver when it is

started as

...

...

a daemon in the background?

For the time bing: kill.

So you did all the above with Mserver started in deamon mode? What is you exact command line call to start Mserver? (In case you use any MIL scripts or modification in the MonetDB.conf file, please say so, too.)

Did/could you also try, what happens in case you start Mserver in interactive mode (both on the virgin DB and for the restart after the updates), and shut it down by saying "quit();" on the MIL console?

I will give this a try

Please keep us posted (in detail) about yur experiences!

...

...

...

- How do I guarantee the database to remain intact with the abovementioned scenario?

As long as your update transaction has sucessfully finished, MonetDB should take care of this. In case is doesn't, I consider this a bug, and you should file a respective bug report including all the information to reproduce the problem.

I am working on a simple scenario to reproduce this. If it appears to be a MonetDB bug, I will file a bug report.

Great. Thanks!

Stefan

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | CWI, P.O.Box | 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

----------------------------------------------------------------------

...

-------- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at

http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmi

...

x.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

--

Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

--------------------------------------------------------------- --------------- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009. visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

On 2008-12-11 18:52, Niels Nes wrote:

On Thu, Dec 11, 2008 at 08:49:38AM +0100, Hans van Rijswijk (DT) wrote:

...

Hi Niels,

I don't understand: sync() should be a command to commit all changes and flush all buffers to a permanent storage, i.e. harddisk. I also don't understand that a logger should be responsible for commits. But the question remains: What should a client do between the last update query and kill Mserver?

The only thing I can think of right now is: WAIT.....and hope the logger eventually commits my changes. But for how long??? Is that right? No, no waiting should be needed. At least it the XQuery update query finishes the updates should be logged. And logged means its on disk in a save way. I think the following happens in my case: I start Mserver, add some documents and perform an update, then sync() (You say it does nothing useful for me) and then kill Mserver. Since Mserver is not ready yet writing all changes to disk, the database gets corrupted. The next time I start Mserver it cannot load some parameters it should have stored in BAT, and refuses to startup the frontend, so mclient commands fail. Even waiting 30 seconds between sync() and kill Mserver does not help. Is that right?? Just try your script without the sync

The only thing is, when the server starts up again, it may need to replay the logs before it starts serving clients. This replaying may take some time and during this time the server is not responsive. You can make this faster by calling pf_logflush(0LL); before shutting down the server.

Niels

...

Regards, Hans.

...

-----Original Message----- From: Niels Nes [mailto:Niels.Nes@cwi.nl] Sent: woensdag 10 december 2008 17:13 To: monetdb-users@lists.sourceforge.net Cc: Stefan Manegold Subject: Re: [MonetDB-users] Again: Mserver fails on Fedora 9 - 32 bit

On Wed, Dec 10, 2008 at 04:52:27PM +0100, Stefan Manegold wrote:

...

Hi Hans,

On Wed, Dec 10, 2008 at 04:33:22PM +0100, Hans van Rijswijk (DT) wrote:

...

Hi Stefan,

Thanks for your reply. My answers to several questions are below, sync() is bad (should be removed fast!). It does a global commit, but the bats are under control of the logger, ie the logger should deside when to commit.

...

...

Regards, Hans.

...

-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: woensdag 10 december 2008 15:54 To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] Again: Mserver fails on Fedora 9 - 32 bit

Hi Hans,

On Wed, Dec 10, 2008 at 03:17:04PM +0100, Hans van Rijswijk (DT) wrote: > Hi MonetDB users, > > I downloaded Monet Fedora RPMs from: > http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds. /20 > 08 > 1209/RPMs/Fedora9.i386.oid32/ > > I installed these with yum on my Fedora fc9 installation, running > within VMWare on my Windows XP 32 bit machine. > > When I startup Mserver everything seems to work well. I start with an > empty dbfarm. > > Then I run the following scenario: > - Add 3 XML documents with pf:add-doc() (some of them updatable) > - Add 2 more XML documents with pf:add-doc() > - Update one of these docs: do insert <something/> into > doc("a.xml")/rootelement > - Send a sync() command with mclient mil. What did/does sync() return? (i.e., TRUE or FALSE?) I was using echo "sync();" | mclient -p 51240 -lmil This returned nothing at all. Well, sync() does return a value of type bit (see below), but you did not check the actual value it returned. Honestly, I'd have to look into the code to know what the return value means, but that only makes sense in case I knew what it returned in your case ... ======== MonetDB>help("sync"); COMMAND: sync() : bit MODULE: trans COMPILED: by adm on Tue Dec 9 18:15:39 2008 save all

...

======== MonetDB>print(sync()); [ true ] ========

...

...

> - Check the content of the database with pf:documents() (is OK) > - Knock down Mserver: kill <pid> > - Startup Mserver again (don't ask me why I do it like this) Well, I'd actually like to know why you do it like this --- while MonetDB should of course survive such "rude" usage (in case the mclient update transaction has successfully finished before Mserver is killed, that is), the question is whether this is (for you!) the most convenient way to use it --- we can only give advise on this in case we know, what you actually want to do, and/or why you do things the (odd?) way you do them ... The scenario is a simplified part of my installation procedure. The first time I start Mserver I initialize the database: put some docs in. After initialization I shutdown all used services, including Mserver. At the end of the installation procedure the user may opt for starting all services, or choose for starting them later. Starting all services involves starting Mserver. Ok, so since you want/need to run Mserver as "service", you need to run it as daemon in the background, right?

...

...

> After this scenario I cannot gain access to Mserver with mclient: > !xchange_challenge: frontend xquery not found What does Mserver say at restart? Any meassges, warnings, errors? Any missing messages? Sometimes Mserver complains about: !ERROR: logger: could not use bat (441) for 1000000000_rid_kind !ERROR: logger: could not use bat (412) for 1000000000_rid_nid Etc.. (see my previous mail post on this subject) Ah, this IS very crucial info --- since you didn't repeat that info, here, I assumed this problems was "solved" by using a clean (non-corrupted) database; apparently not ... "Sometimes" holds for the case during the restart after the update+kill, that then resulted in `mclient -lxquery` not being able to connect? Well, since this is a logger error message, my guess that this is a login/logger related problem was not that bad --- for efficiency, it is very handy to always give this kind of crucial info; in

Niels persistent BATs particular,

...

if error messages precede a problem, these error messages should (obviously!??) always be clearly shared ...

...

...

> If I wait half an hour before killing Mserver this works fine. > > Questions: > - Does sync() do what it is supposed to do? I hope so --- however, your updates are most probably still under the control of the logger; hence, sync() makes no difference, here, I suppose --- strage, though that replaying the logs at server start-up does not work ...

> - Why is shutdown() not implemented? Lacking user/access management in the MonetDB kernel/server it was/is considered insecure. Clear!

...

> - What is the right way to shutdown Mserver when it is started as > a daemon in the background? For the time bing: kill.

So you did all the above with Mserver started in deamon mode? What is you exact command line call to start Mserver? (In case you use any MIL scripts or modification in the MonetDB.conf file, please say so, too.)

Did/could you also try, what happens in case you start Mserver in interactive mode (both on the virgin DB and for the restart after the updates), and shut it down by saying "quit();" on the MIL console? I will give this a try Please keep us posted (in detail) about yur experiences!

...

...

> - How do I guarantee the database to remain intact with the > abovementioned scenario? As long as your update transaction has sucessfully finished, MonetDB should take care of this. In case is doesn't, I consider this a bug, and you should file a respective bug report including all the information to reproduce the problem. I am working on a simple scenario to reproduce this. If it appears to be a MonetDB bug, I will file a bug report. Great. Thanks!

Stefan

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | CWI, P.O.Box | 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

----------------------------------------------------------------------

...

-------- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at

http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmi

...

x.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users --

Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

--------------------------------------------------------------- --------------- SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009. visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Sjoerd Mullender