On Nov 27, 2007 4:05 PM, Martin Kersten wrote:

m h wrote:

...

I tried running the SQLAlchemy ORM testsuite. The first time it ran for a while and then mserver segfaulted. (It also grew to using 700+Megs of memory. Am assumming this is a monetdb issue, since after

not uncommon size. A point of concern is if you use like/ltrim/rtrim.... because there we detected a leak recently. which is fixed

I'm not aware of any trimming. But I'm also seeing memory growth as I run other unit tests. I'll put it on the backburner for now.

The one show below is more serious... and we would love to be able to reproduce it using a small test. If infeasible.. then run the experiment again with 'valgrind' to detect the place where dirty memory is used, or double frees are generated.

...

each test the tables are usually dropped). The server now has issues. It appears to start fine:

# MonetDB server v5.2.3, based on kernel v1.20.1 # Serving database 'demo' # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked # Copyright (c) 1993-2007 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /root/MonetDB/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.20.3 loaded

When I try to connect with the command line client it complains repeatedly:

!ERROR: BUNdelete: access denied to tmp_537, aborting. !ERROR: BUNdelete: access denied to tmp_537, aborting. ....

Have attached a valgrind log using this command. Let me know if you want --leak-check=full or other options. valgrind --log-file=/tmp/monetval.log ./bin/mserver5 --dbinit="include sql;"

On Nov 29, 2007 5:08 PM, m h wrote:

On Nov 27, 2007 4:29 PM, m h wrote:

...

On Nov 27, 2007 4:05 PM, Martin Kersten wrote:

...

m h wrote:

...

I tried running the SQLAlchemy ORM testsuite. The first time it ran for a while and then mserver segfaulted. (It also grew to using 700+Megs of memory. Am assumming this is a monetdb issue, since after

not uncommon size. A point of concern is if you use like/ltrim/rtrim.... because there we detected a leak recently. which is fixed

I'm not aware of any trimming. But I'm also seeing memory growth as I run other unit tests. I'll put it on the backburner for now.

...

The one show below is more serious... and we would love to be able to reproduce it using a small test. If infeasible.. then run the experiment again with 'valgrind' to detect the place where dirty memory is used, or double frees are generated.

...

each test the tables are usually dropped). The server now has issues. It appears to start fine:

# MonetDB server v5.2.3, based on kernel v1.20.1 # Serving database 'demo' # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked # Copyright (c) 1993-2007 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /root/MonetDB/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.20.3 loaded

When I try to connect with the command line client it complains repeatedly:

!ERROR: BUNdelete: access denied to tmp_537, aborting. !ERROR: BUNdelete: access denied to tmp_537, aborting. ....

Have attached a valgrind log using this command. Let me know if you want --leak-check=full or other options.

valgrind --log-file=/tmp/monetval.log ./bin/mserver5 --dbinit="include sql;"

I ran the SQLAlchemy ORM tests again today (I only have 6 issues left with the SQL tests!, mostly unicode and binary data issues). And the server segfaulted again.

What can I do to help this situation? Run with valgrind? Explain howto run the SQLAlchemy test suite?

thanks much,

-matt

Attached is the valgrind output of the ORM crash (took some 15 minutes+ to run on my laptop).

On Nov 30, 2007 12:25 AM, Niels Nes wrote:

On Thu, Nov 29, 2007 at 05:43:44PM -0700, m h wrote:

...

...

What can I do to help this situation? Run with valgrind? Explain howto run the SQLAlchemy test suite? I guess that would help most.

I'm off to bed now, but hopefully this will get you started. *pull the svn of SQLAlchemy and don't install, but just unzip My code is attached to this bug: http://www.sqlalchemy.org/trac/ticket/874 put the monetdb.py in the sqlalchemy/lib/sqlalchemy/databases/ directory. add 'dialect.monetdb', to modules_to_test in sqlalchemy/test/dialect/alltests.py I have a script that runs the test, here it is: #!/bin/sh pushd sqlalchemy PYTHONPATH=./lib:./test; python -tt test/sql/alltests.py --db monetdb --verbose --log-debug=sqlalchemy.engine #PYTHONPATH=./lib:./test; python -t test/orm/alltests.py --db monetdb --verbose --log-debug=sqlalchemy.engine popd #end script Uncomment the test/orm/alltests.py to crash the server. Repeatedly running the normal test will show that there appears to be a memory leak in the server, as the memory grows by some 10-20 Megs after each run.

...

...

thanks much,

-matt

Attached is the valgrind output of the ORM crash (took some 15 minutes+ to run on my laptop). I guess you forgot to attach it as the attachment is missing.

Sorry, here it is -matt

On Nov 30, 2007 12:17 PM, Niels Nes wrote:

On Fri, Nov 30, 2007 at 12:37:28AM -0700, m h wrote:

...

On Nov 30, 2007 12:25 AM, Niels Nes wrote:

...

On Thu, Nov 29, 2007 at 05:43:44PM -0700, m h wrote:

...

...

What can I do to help this situation? Run with valgrind? Explain howto run the SQLAlchemy test suite? I guess that would help most.

I'm off to bed now, but hopefully this will get you started.

*pull the svn of SQLAlchemy and don't install, but just unzip

My code is attached to this bug: http://www.sqlalchemy.org/trac/ticket/874

put the monetdb.py in the sqlalchemy/lib/sqlalchemy/databases/ directory.

add 'dialect.monetdb', to modules_to_test in sqlalchemy/test/dialect/alltests.py

I have a script that runs the test, here it is:

#!/bin/sh

pushd sqlalchemy PYTHONPATH=./lib:./test; python -tt test/sql/alltests.py --db monetdb --verbose --log-debug=sqlalchemy.engine #PYTHONPATH=./lib:./test; python -t test/orm/alltests.py --db monetdb --verbose --log-debug=sqlalchemy.engine popd

#end script

Uncomment the test/orm/alltests.py to crash the server. Repeatedly I my fresh check out of alchemy it isn't commented. I didn't see crashes. I did have restart problems which are now fixed.

I meant uncomment the line in my script above: #PYTHONPATH=./lib:./test; python -t test/orm/alltests.py --db monetdb --verbose --log-debug=sqlalchemy.engine

On Fri, Nov 30, 2007 at 01:08:19PM -0700, m h wrote:

...

...

...

Uncomment the test/orm/alltests.py to crash the server. Repeatedly I my fresh check out of alchemy it isn't commented. I didn't see crashes. I did have restart problems which are now fixed.

Still crashes for me with a build from 2 hours ago. but can you restart without problems after the crash, ie your other problem?

Niels

------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

On Nov 30, 2007 1:09 PM, Niels Nes wrote:

On Fri, Nov 30, 2007 at 01:08:19PM -0700, m h wrote:

...

...

...

...

Uncomment the test/orm/alltests.py to crash the server. Repeatedly I my fresh check out of alchemy it isn't commented. I didn't see crashes. I did have restart problems which are now fixed.

Still crashes for me with a build from 2 hours ago. but can you restart without problems after the crash, ie your other problem?

Yes, restart works now!

On Fri, Nov 30, 2007 at 01:24:44PM -0700, m h wrote:

On Nov 30, 2007 1:22 PM, m h wrote:

...

On Nov 30, 2007 1:09 PM, Niels Nes wrote:

...

On Fri, Nov 30, 2007 at 01:08:19PM -0700, m h wrote:

...

...

...

> Uncomment the test/orm/alltests.py to crash the server. Repeatedly I my fresh check out of alchemy it isn't commented. I didn't see crashes. I did have restart problems which are now fixed.

Still crashes for me with a build from 2 hours ago. but can you restart without problems after the crash, ie your other problem?

Yes, restart works now!

Whoops, spoke too soon. Build from 2 hours ago gives the following error when trying to connect: !ERROR: BUNdelete: access denied to tmp_537, aborting. !ERROR: BUNdelete: access denied to tmp_537, aborting. !ERROR: BUNdelete: access denied to tmp_537, aborting.

I didn't run into this on a Nov27 build indeed did see this too. I'll have another round.

Niels

------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

On Fri, Nov 30, 2007 at 01:30:22PM -0700, m h wrote:

On Nov 30, 2007 1:26 PM, Niels Nes wrote:

...

On Fri, Nov 30, 2007 at 01:24:44PM -0700, m h wrote:

...

On Nov 30, 2007 1:22 PM, m h wrote:

...

On Nov 30, 2007 1:09 PM, Niels Nes wrote:

...

On Fri, Nov 30, 2007 at 01:08:19PM -0700, m h wrote:

...

> > > Uncomment the test/orm/alltests.py to crash the server. Repeatedly > > I my fresh check out of alchemy it isn't commented. I didn't see > > crashes. I did have restart problems which are now fixed.

Still crashes for me with a build from 2 hours ago. but can you restart without problems after the crash, ie your other problem?

Yes, restart works now!

Whoops, spoke too soon. Build from 2 hours ago gives the following error when trying to connect: !ERROR: BUNdelete: access denied to tmp_537, aborting. !ERROR: BUNdelete: access denied to tmp_537, aborting. !ERROR: BUNdelete: access denied to tmp_537, aborting.

I didn't run into this on a Nov27 build indeed did see this too. I'll have another round.

Is there a way to purge the data that's crashing it so that it will start again? Or do I need to rebuild everything? simply remove your dbfarm and log files, usualy in found in $prefix/var

(ie rm -rf $prefix/var/* should do the trick) Niels

------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

On Fri, Nov 30, 2007 at 01:37:02PM -0700, m h wrote:

...

...

...

Is there a way to purge the data that's crashing it so that it will start again? Or do I need to rebuild everything? simply remove your dbfarm and log files, usualy in found in $prefix/var

(ie rm -rf $prefix/var/* should do the trick)

Yep, that did it also checked in a fix this problem. And now I'm looking why the

SELECT bar.foo_a AS bar_foo_a, bar.foo_b AS bar_foo_b, bar.bar_a AS bar_bar_a, bar.bar_b AS bar_bar_b FROM (SELECT foo.a AS foo_a, foo.b AS foo_b, bar.a AS bar_a, bar.b AS bar_b FROM foo JOIN (SELECT foo.a AS a, foo.b AS b FROM foo WHERE foo.b = 'bar') AS bar ON foo.b = 'bar') AS bar ORDER BY bar.foo_a; make use assert. Niels

------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

...

Whoops, spoke too soon. Build from 2 hours ago gives the following error when trying to connect: !ERROR: BUNdelete: access denied to tmp_537, aborting. !ERROR: BUNdelete: access denied to tmp_537, aborting. !ERROR: BUNdelete: access denied to tmp_537, aborting.

I didn't run into this on a Nov27 build

Here's the valgrind log for that....