June 2014 - users-list

MonetDB just reads and reads when restoring a dump
by Bryan Senseman 16 Jun '14

16 Jun '14

All, I know I have asked this before, but I am still plagued by this error. I am restoring a 50GB database from Linux to Windows, the dump file created via mclient is ~53GB. I am reading that file from an SSD and loading the database onto another SSD. This has been running for 2 DAYS! Using almost all available memory 16GB. Using Windows Resoure Monitor, the mserver5.exe process is just reading tail files for hours. The pattern is that the disk usage will grow by some amount say 5gb (I’m assuming the size of the table actually being loaded) and then it will spend 6-12 hours reading what it just wrote. Why? Can this be eliminated? I removed all indexes on the fact tables so it isn’t indexing (although that runs almost instantly anyway). Only about 10% cpu utilization. Thanx, Bryan -- ------------------------------ This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.

2 1

"LIMIT + ORDER BY" TypeException
by George Ramonov 15 Jun '14

15 Jun '14

Hi, I am still learning about monetdb internals, so I hope someone can enlighten me here. I was playing around with monetdb, and run into the following problem: Suppose we have a table CREATE TABLE test.simple_table("id" INTEGER); INSERT INTO test.simple_table VALUES (1),(2),(3),(4),(5); Let's say I want to run a simple query: SELECT COUNT(*) AS val FROM test.simple_table ORDER BY val DESC; +----+ | v | +==+ | 5 | +----+ Runs fine. However, if add a *LIMIT* clause: SELECT COUNT(*) AS val FROM test.simple_table ORDER BY val DESC LIMIT 5; I get the following TypeException:user.s37_7[17]:'algebra.markT' undefined in: _30:any := algebra.markT(_27:oid, _29:oid) program contains errors Now things let's add a primitive *HAVING* clause SELECT COUNT(*) AS val FROM test.simple_table HAVING 1 > 0 ORDER BY val DESC LIMIT 5; +----+ | v | +==+ | 5 | +----+ And things are magically fine again. Can someone explain what's going on? Thanks, George Ramonov UC Berkeley B.S. Electrical Engineering & Computer Science '13 <http://www.linkedin.com/in/georgeramonov>

2 2

odbc driver 11.17.17 (64-bit) memory allocation error on Windows Server 2012
by László Sándor 10 Jun '14

10 Jun '14

Upon some further reflection, the Stata error message still suggest the error happened within MonetDB's ODBC driver. Anybody else using MonetDB with ODBC on Windows? What shall I check? Thanks! Laszlo

1 2

Re: Getting "BATproject: does not match always"
by Joe Wass 10 Jun '14

10 Jun '14

Update to my previous message. Looks like something went wrong with an mmap. 2014-06-09 17:10:58 ERR my_database[32949]: = gdk_posix.c:560: MT_mremap(bat/11/1150.theap,c0000000,538312704,1076625408): p == MAP_FAILED Looks like the server ran out of memory but just kept going with a corrupted table? Should this have been allowed to happen or should the process have crashed? It looks like this bug is related, although it has a different cause: https://www.monetdb.org/bugzilla/show_bug.cgi?id=3450 Joe

1 0

Getting "BATproject: does not match always"
by Joe Wass 10 Jun '14

10 Jun '14

My schema is: create table my_table (col_a date, col_b varchar(1024), col_c varchar(128), col_d varchar(128), col_e varchar(128)); I have approx 5,000,000 entries. The following queries not work: select count(col_b) from my_table; select distinct(col_b) from my_table; select 1 from my_table group by col_b; Giving the error: GDK reported error. BATproject: does not match always They work fine on all other fields. I don't know precisely what triggers this. I've tried to create some small sample values but I can only reproduce this on my large data set. I'm actually trying to do more interesting aggregation, but this is a minimal way to trigger the error. I'm using Jan2014-SP2 on Mac OS X 10.9.2 . Thanks Joe

1 0

Re: MonetDB problem with mserver.exe
by Hannes Mühleisen 10 Jun '14

10 Jun '14

On 06/10/2014 09:04 AM, Kaspar Burger wrote: > You are right, Anthony, when I run the file independent of R, > mserver.exe shuts down immediately. Could you please share your test.bat file? > > *De :*users-list > [mailto:users-list-bounces+kaspar.burger=iukb.ch@monetdb.org] *De la > part de* Anthony Damico > *Envoyé :* mardi 10 juin 2014 08:51 > *À :* Communication channel for MonetDB users > *Objet :* Re: MonetDB problem with mserver.exe > > kaspar has something changed? when you and i screenshared to try to > resolve this issue, mserver.exe was shutting down immediately before R > ever came into the picture? > > what happens when you run this file independent of R? > > > C:\Users\Kaspar Burger\MonetDB\test.bat > > > > On Tue, Jun 10, 2014 at 2:42 AM, Hannes Mühleisen > <Hannes.Muehleisen(a)cwi.nl <mailto:Hannes.Muehleisen@cwi.nl>> wrote: > > This seems to fall into my backyard... > > > On 10.06.2014, at 08:32, Kaspar Burger <kaspar.burger(a)iukb.ch > <mailto:kaspar.burger@iukb.ch>> wrote: > > I am working in R, configuring a MonetDB test database and trying to > > access MonetDB with the following code: > > > > batfile <- "C:/Users/Kaspar Burger/MonetDB/test.bat" > > pid <- monetdb.server.start( batfile ) > > dbname <- "test" > > dbport <- 50000 > > monet.url <- paste0( "monetdb://localhost:" , dbport , "/" , dbname ) > > db <- dbConnect( MonetDB.R() , monet.url , wait = TRUE ) > > > > When I run the “pid <- monetdb.server.start( batfile )” line, the > > mserver5.exe opens briefly but shuts down immediately afterwards (for a > > more extensive description of the R code, see: > > > https://raw.githubusercontent.com/ajdamico/usgsd/master/MonetDB/monetdb%20d…) > > By contrast, running the batch file from C:\Program > > Files\MonetDB\MonetDB5\M5server.bat or from the Start menu works fine. I > > tried when both drivers were installed as well as when only one > driver was > > installed. Could that problem be related to R or to my code? > > Would not be surprised if the space in the batfile path causes the > problem. Try setting up the DB into another directory without a space in > it and try again. > > Best, > > Hannes > > > > > > > > >> > >> > >> On 2014-06-10 00:06, Stefan Manegold wrote: > >>>> On 09/06/14 16:28, Kaspar Burger wrote: > >>>>> > >>>>> Hello, > >>>>> > >>>>> I seem to have difficulties with mserver.exe. I would like to use > >>>>> MonetDB > >>>>> on a Windows 64 bit computer, so... > >>>>> > >>>>> 1) ...I went to > >>>>> https://www.monetdb.org/downloads/Windows/Jan2014-SP2/ > >>>>> > >>>> Most likely, you need only 1 ODBC driver. > >>>> Otherwise causing a clash. > >>> > >>> For the records, as by our own documentation, on a 64-bit Windows > >>> system, > >>> both the 32-bit and the 64-bit ODBC driver *must* be installed. > >>> (As far as I understand, this is required to support both 32-bit and > >>> 64-bit > >>> applications, or simply because Windows requires it.) > >> > >> You need both in order to support both 32 and 64 bit applications. > >> That's the only reason to install both. > >> > >>> Stefan > >>> > >>>>> 2) ...and first installed the MonetDB ODBC driver, 32 bit (to > >>>>> C:\Program > >>>>> Files (x86)\MonetDB\MonetDB ODBC Driver\) > >>>>> > >>>>> 3) ...then I installed the MonetDB ODBC driver, 64 bit (to C:\Program > >>>>> Files\MonetDB\MonetDB ODBC Driver\) > >>>>> > >>>>> 4) ...and finally I installed the MonetDB SQL server and client, 64 > >>>>> bit > >>>>> (to C:\Program Files\MonetDB\MonetDB5\). > >>>>> > >>>>> When I open up mserver.exe, it shuts down immediately. Does anyone > >>>>> know > >>>>> what could have gone wrong? > >> > >> Did you start the server using the start menu? That actually runs a cmd > >> script to set a bunch of environment variables before starting the > >> actual server mserver5.exe. > >> > >>>>> I would appreciate any hint. Thanks in advance. > >>>>> > >>>>> Kaspar Burger > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> users-list mailing list > >>>>> users-list(a)monetdb.org <mailto:users-list@monetdb.org> > >>>>> https://www.monetdb.org/mailman/listinfo/users-list > >>>>> > >>>> > >>>> _______________________________________________ > >>>> users-list mailing list > >>>> users-list(a)monetdb.org <mailto:users-list@monetdb.org> > >>>> https://www.monetdb.org/mailman/listinfo/users-list > >>>> > >>> > >> > >> -- > >> Sjoerd Mullender > >> > >> _______________________________________________ > >> users-list mailing list > >> users-list(a)monetdb.org <mailto:users-list@monetdb.org> > >> https://www.monetdb.org/mailman/listinfo/users-list > >> > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > https://www.monetdb.org/mailman/listinfo/users-list > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > https://www.monetdb.org/mailman/listinfo/users-list > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list >

3 5

MonetDB problem with mserver.exe
by Kaspar Burger 10 Jun '14

10 Jun '14

Hello, I seem to have difficulties with mserver.exe. I would like to use MonetDB on a Windows 64 bit computer, so... 1) ...I went to https://www.monetdb.org/downloads/Windows/Jan2014-SP2/ 2) ...and first installed the MonetDB ODBC driver, 32 bit (to C:\Program Files (x86)\MonetDB\MonetDB ODBC Driver\) 3) ...then I installed the MonetDB ODBC driver, 64 bit (to C:\Program Files\MonetDB\MonetDB ODBC Driver\) 4) ...and finally I installed the MonetDB SQL server and client, 64 bit (to C:\Program Files\MonetDB\MonetDB5\). When I open up mserver.exe, it shuts down immediately. Does anyone know what could have gone wrong? I would appreciate any hint. Thanks in advance. Kaspar Burger

7 8

Re: Could monetdb have a capacity to surpport our case
by Hannes Mühleisen 09 Jun '14

09 Jun '14

Hello, On 06.06.2014, at 12:06, liuguodong <liuguodong(a)jd.com> wrote: > Our case as following: > > 1. There are several big tables(more than billion rows per table, more than 1 TB) ; > 2. Many dimentions in every table( more than 150 dimentions per table), they random do join operation; > 3. Low latency reponse, it is expected to reponse time being less than 1 second. > > Could monetdb have a capacity to surpport our case or any suggestion? This depends first on your queries and second on your hardware. In general, the columnar architecture of MonetDB would be a good match for what you describe (lots of dimensions, random join combinations). I would suggest to simply try your queries with a subset of data on a scaled-down machine (e.g. 100 GB of your data on a 16GB RAM box) and see how it goes. Best, Hannes

1 0

Re: Re: Re: tomograph does not work
by Cheng Xuntao 07 Jun '14

07 Jun '14

Thanks Martin. Xuntao

1 0

MAL scheduling and stethoscope metrics
by Cheng Xuntao 07 Jun '14

07 Jun '14

Hi all, I am new to monetdb. I want to know how Monetdb schedule its MAL statements to threads, or to which CPU socket in a multi-cpu environment. I didn't find any document talking bout this directly. Shall I start from reading the source code? Which part of the source code shall I start with? Any suggestion will do me a great favour! I also have some confusions about the metrics of stethoscope. So I asked these 2 question in stackoverflow. http://stackoverflow.com/questions/24077439/what-does-the-metrics-c-m-r-w-a… http://stackoverflow.com/questions/24076688/how-does-monetdb-schedule-its-m… Any suggestion? Thanks very much! Xuntao

1 0