- users-list - monetdb.org

creating a binary file for "COPY binary ..."
by Lynn Carol Johnson 19 Jan '16

19 Jan '16

Hi all - I'm creating a large db table with several columns. To speed it up, I'm using monetdb's "COPY BINARY" command. I have a question on creating the binary files. What I've done is use java's DataOutputStream to create the binary file - using code that looks as below (sometimes an int, sometimes a real or byte): writerChr = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(chrFile))); writerChr.writeInt(intChrom); When I read this file using a test program and java's DataInputStream, it correctly shows me the integer values I've written. In the "writerChr" case, all values in this particular column are "10". However, when I load this into monetdb the value is way off - it shows a large number. The command I am using to copy this file and one other into a table with 2 "int" columns is: COPY binary into justchrpos from ('/home/lcj34/chrFileC10.bin','/home/lcj34/posFileC10.bin'); It copies in fine, but when I cheek the values I see this: (not 10!) sql>\d justchrpos CREATE TABLE "testbig"."justchrpos" ( "chr" INTEGER, "pos" INTEGER ); sql>select * from justchrpos limit 10; +-----------+-----------+ | chr | pos | +===========+===========+ | 167772160 | 16777216 | | 167772160 | 33554432 | | 167772160 | 50331648 | | 167772160 | 67108864 | | 167772160 | 83886080 | | 167772160 | 100663296 | | 167772160 | 117440512 | | 167772160 | 134217728 | | 167772160 | 150994944 | | 167772160 | 167772160 | +-----------+-----------+ 10 tuples (3.601ms) sql> QUESTION: Is using java's DataOutputStream a valid way to create the binary files for monetdb? If not, what should I be using? As I mentioned above, when I read back the created file using java's DataInputStream, I see the values "10" as I expect. Thanks - Lynn

3 3

Monet way of handling string columns
by bhuvan Mitra 18 Jan '16

18 Jan '16

Hello, I have recently started looking into Monetdb and want to contribute to it. My complete focus is on string columns as numeric data can already be processed with maximum speed. I know it uses BATs to store every single column, but I read some where that it uses dictionary for string columns. How is it actually using dictionary? And are there any other special handling techniques for strings? When a query like 'contains' comes, how will monet handle that? I found there is no inverted index in monet to deal queries like this. (Please tell a rough model of how query execution is done incase of string columns) Hope this forum gives me some insight of what's happening during string column processing. Thanks in advance, -Mitra

1 0

MonetDB-11.21.13 (Jul2015-SP2) can't start in CentOS 6.3
by chenkaijiang＠bigknow.com.cn 18 Jan '16

18 Jan '16

I downloaded the source codes at https://www.monetdb.org/downloads/sources/Jul2015-SP2/MonetDB-11.21… ./configure --prefix=/usr/local/MonetDBmakemake install No error is reported. But can't start:monetdbd create ./ycmonetdbd: error while loading shared libraries: libstream.so.7: cannot open shared object file: No such file or directory It looks like the libstream.so is lib on debian linux.  How can I fix it on CentOS 6.3?

2 1

How to set query timeout in MonetDB Server
by Luciano Sasso 16 Jan '16

16 Jan '16

How to set query timeout in MonetDB Server? -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Announcement: New Jul2015-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 16 Jan '16

16 Jan '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP2 bugfix release Bug Fixes * 2014: 'null' from copy into gets wrong * 3817: opt_pushselect stuck with multi-table UDF * 3835: windows does not release ram after operations * 3836: rand() only gets evaluated once when used as an expression * 3838: Update column with or without parenthesis produce different results * 3840: savepoints may crash the database * 3841: mclient fails with response "Challenge string is not valid" * 3842: SQL execution fails to finish and reports bogus error messages * 3845: Too many VALUES in INSERT freeze mserver5 * 3847: Wrong SQL results for a certain combination of GROUP BY / ORDER BY / LIMIT * 3848: mserver segfault during bulk loading/updating * 3849: HUGEINT incorrect value * 3850: DEL character not escaped * 3851: expression that should evaluate to FALSE evaluates to TRUE in SELECT query * 3852: CASE statement produces GDK error on multithreaded database: BATproject does not match always * 3854: Complex expression with comparison evaluates incorrectly in WHERE clause * 3855: Foreign key referencing table in a different schema - not allowed. * 3857: Large LIMIT in SELECT may abort the query * 3861: Using window functions cause a crash * 3864: Error in bulk import for chinese character * 3871: NOT x LIKE triggers "too many nested operators" * 3872: mserver crashes under specific combination of JOIN and WHERE conditions * 3873: mserver5: gdk_bat.c:1015: setcolprops: Assertion `x != ((void *)0) || col->type == 0' failed. * 3879: Database crashes when querying with several UNION ALLs. * 3887: Querying "sys"."tracelog" causes assertion violation and crash of mserver5 process * 3889: read only does not protect empty tables * 3895: read only does not protect this table

2 1

error opening DB on External drive
by Lynn Carol Johnson 14 Jan '16

14 Jan '16

Hello - I am running monetdb with the July2015 service pack on MAC OSX. I have no problem running and connecting to a dbfarm that lives on my MAC. But I want to test a very large data set so have created a dbfarm on a Samsung T1 SSD external drive that is USB connected to my MAC. I have successfully started monetdbd server and have attached to my dbfarm. I created/released a new database. Yet when I attempt to run mClient it fails. Here is the "get all" command: rs-btlcj34lt:testMapiSock lcj34$ monetdbd get all /Volumes/Samsung_T1/alt-dbfarm property value hostname rs-btlcj34lt.maize.cornell.edu dbfarm /Volumes/Samsung_T1/alt-dbfarm status monetdbd[740] 1.7 (Jul2015-SP1) is serving this dbfarm mserver /Users/lcj34/development/usr/local/monetdb/bin/mserver5 logfile /Volumes/Samsung_T1/alt-dbfarm/merovingian.log pidfile /Volumes/Samsung_T1/alt-dbfarm/merovingian.pid sockdir /tmp port 50000 exittimeout 60 forward proxy discovery true discoveryttl 600 control no passphrase <unknown> mapisock /tmp/.s.monetdb.50000 controlsock /tmp/.s.merovingian.50000 rs-btlcj34lt:testMapiSock lcj34$ Here is the command line error: rs-btlcj34lt:alt-dbfarm lcj34$ mclient -u monetdb -d maizeFullGenomeDB password: monetdbd: internal error while starting mserver, please refer to the logs And here is the error from the log file. 2016-01-14 10:33:26 MSG maizeFullGenomeDB[749]: # Visit http://www.monetdb.org/ for further information 2016-01-14 10:33:26 MSG maizeFullGenomeDB[749]: !IOException:mal_mapi.listen:operation failed: binding to UNIX socket file /Volumes/Samsung_T1/alt-dbfarm/maizeFullGenomeDB/.mapi.sock failed: No such file or directory If I switch back to a dbfarm local to the MAC, it works fine. Any ideas why I can't connect to a dbfarm on the external drive? Is there something else that needs to be setup for the socket creation to work? Thanks - Lynn

3 3

Error on database startup
by Robin Cijvat 13 Jan '16

13 Jan '16

Hello everyone! Every now and then my laptop fails to wake up from standby mode and I have to do a hard reset, while monetdb is running in the background. I have had the following problem multiple times lately when trying to start the database that was stopped during the hard reset: monetdb start tpch starting database 'tpch'... FAILED start: starting 'tpch' failed: database 'tpch' did not initialise the sql scenario This is from the log: 2016-01-13 07:22:38 MSG tpch[7085]: # Start processing logs sql/sql_logs version 52200 2016-01-13 07:22:38 ERR tpch[7085]: !mvc_init: unable to create system tables 2016-01-13 07:22:38 MSG tpch[7085]: !SQLException:SQLinit:Catalogue initialization failed 2016-01-13 07:22:38 MSG tpch[7085]: !ERROR: Could not read the version number from the file 'sql_logs/sql//log'. 2016-01-13 07:22:38 MSG merovingian[7066]: database 'tpch' (7085) has exited with exit status 0 2016-01-13 07:22:38 MSG merovingian[7066]: database 'tpch' has shut down 2016-01-13 07:22:38 ERR control[7066]: (local): failed to fork mserver: database 'tpch' did not initialise the sql scenario Now I don't keep any important information in this database, so for me it is fine to reload everything. However, since this is a recurring problem, I wondered if anyone has any suggestions as to how to fix such a 'broken' database.

1 0

New syntax filter functions
by Roberto Cornacchia 08 Jan '16

08 Jan '16

Hi there, I'm a bit struggling with the syntax for filter function, after it changed from Oct2014 to Jul2015. I have looked at the release notes, but they don't seem to even mention this change of syntax. This used to work in Oct2014: create filter function maxlevenshtein(s1 string, s2 string, k int) external name str."distjoin_levenshtein"; select t.name, f.name from sys.tables t, sys.functions f where t.name maxlevenshtein(2) f.name; In Jul2015, if I look at Niels' comment in https://www.monetdb.org/bugzilla/show_bug.cgi?id=3702 , it should be something similar to select t.name, f.name from sys.tables t, sys.functions f where [t.name] maxlevenshtein [f.name, 2]; But this is still not right. How should it be? And should I change the SQL / MAL / C interface of the function as well? Thanks,

1 2

Monitoring MonetDB performance
by Alexander Gordt 07 Jan '16

07 Jan '16

Hello together, I’ve got a performance issue with my project, where the DB performance is going down for what ever reason. The activity monitor of OSX shows me the following numbers about the mserver5-process: The CPU usage is very low (15-25% of one core!), the IO traffic on my ssd is fair (about 10-40 MB/s), but the speed slows down. I’ve had a look in the merovingian.log, but there’s only the connection shown. For the first 5000 messages, the CPU usage of mserver5 was about 70-110% - why is it going down?… How could I analyze, why the database performance is going down? Is it possible, that there is an issue with the JDBC-driver, that the performance is decreasing? I attache a log file where you can see that the performance is quite good for some time, but then goes down for whatever reason. 18:40:46 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 15.711 seconds - totally inserted 5200 messages 18:40:46 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:41:03 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 17.189 seconds - totally inserted 5400 messages 18:41:03 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:41:27 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 23.834 seconds - totally inserted 5600 messages 18:41:27 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:42:19 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 52.13 seconds - totally inserted 5800 messages 18:42:19 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:43:09 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 49.892 seconds - totally inserted 6000 messages 18:43:09 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:43:59 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 49.786 seconds - totally inserted 6200 messages 18:43:59 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:45:31 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 92.389 seconds - totally inserted 6400 messages 18:45:31 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:47:26 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 114.412 seconds - totally inserted 6600 messages 18:47:26 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:49:24 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 118.324 seconds - totally inserted 6800 messages One message results in 10-50 insert statements in different tables and the 200 messages are processed in one transaction. The absolute performance isn’t nice, but that is another topic – I’ve read about "copy into“ and that kind of stuff – but that gonna be part in a later optimization. I tried to find out, if there is a long running process, but „select * from sys.queue;“ didn’t show me anything of interest – the result shows only this select. Is there a way to gather some statistics about executed queries and their runtime? Or about blocking resources… Thanks in advance for bringing any light in the black box :) Best regards, Alex

1 1

About JDBC Driver
by Luciano Sasso 07 Jan '16

07 Jan '16

Steps for error: 1- In testing with java client and JDBC 2:19, an error occurs when saving decimal field, for example the value 11:23 is recorded as 13. 2- When I run the same test with the 2.8 driver, saves normally and correct. Not only is the 2.19 driver, all above 2.8 is that same mistake. My Java client is Pentaho Data Integration, My MonetDB is JUL2015-SPI I go zip package test, and send fo tests. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1