March 2012 - users-list

[MonetDB-users] Announcement: New Dec2011-SP2 Bugfix release of MonetDB suite
by Fabian Groffen 16 Mar '12

16 Mar '12

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2011-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at http://www.monetdb.org/. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2011-SP2 bugfix release Java Module * Implemented missing Number types support in PreparedStatement.setObject() Bug Fixes * 2915: Query (mclient or jdbc) with 2 outer joins appears to crash connectionor server * 2947: Queries where a "is null" clause is combined with two equals clauses fail to produce the expected result. * 2968: Memory Leaks * 2974: thetajoin produces invalid/inconsistent bat * 2988: Subquery in select part goes wrong * 2998: Equivalent multi-way join queries give different results * 3002: String plus interval result in program errors upon comparison * 3011: count(*) in subquery results in 1 for query without results * 3012: bitwise comparison fails after a cast * 3029: test sql/test/Dependencies/Tests/Dependencies.sql fails after recent changes * 3031: Virtually all SQL tests fail on Windows * 3036: optimizers are not getting set correctly through set optimizer * 3038: ~190 SQL tests fail on Darwin, FreeBSD & Solaris (only) since introduction of "(extended) relational algebra" * 3043: WITH clause is not allowed in derived table * 3045: algebra.project(0@0,bat) fails at propcheck * 3052: Wrong Error Code on Rollback Exception

1 0

[MonetDB-users] rename table?
by Christian Braun 16 Mar '12

16 Mar '12

Hello! Is it possible to rename a table? Regards, Christian.

1 0

[MonetDB-users] Query ignored
by Mints, Alexey 15 Mar '12

15 Mar '12

Hello! I have an issue with MonetDB query. This one select count(*) from catalogedsources c1 left outer join catalogedsources c2 on ( c2.cat_id = c1.cat_id + 2 AND ((c1.decl - c2.decl) * (c1.decl - c2.decl))< 1e-06) where c1.cat_id = 4 AND c1.decl BETWEEN 77.0 AND 83.0 AND c1.ra BETWEEN 42.0 AND 48.0 ; works fine. But if I replace "c2.cat_id = c1.cat_id + 2" with "c2.cat_id = 6" - i receive nothing. Not zero, not empty dataset, but just nothing at all. "EXPLAIN" does not work as well. Also, with more complex join conditions I have no result as well. I am using: Welcome to mclient, the MonetDB/SQL interactive terminal (Dec2011-SP1) Database: MonetDB v11.7.7 (Dec2011-SP1), 'mapi:monetdb://linux-6d2s.site:50000/*****' What can this be? Regards, Alexey

1 0

[MonetDB-users] MonetDB out of memory errors
by Stathis Gkotsis 14 Mar '12

14 Mar '12

Hi all, I am using MonetDB v11.7.7 (Dec2011-SP1) in an application which constantly feeds data in tables (daily tables). The communication with the server is done through the mapi API. This is accomplished through the "COPY INTO" command. From time to time (every 1-2 days) I see errors in the application logs such as: exception at 00000000004C0109: Connect error (Error code -4): monetdbd: an internal error has occurred, refer to the logs for details, please try again later . exception at 00000000004C0109: Connect error (Error code -4): monetdbd: there are no available connections for 'mydatabase' . exception at 00000000004C0109: Connect error (Error code -4): monetdbd: there are no available connections for 'mydatabase' Then looking at dmesg log: [964932.242080] 251025 pages non-shared [964932.242083] Out of memory: kill process 11585 (monetdbd) score 787911 or a child [964932.302785] Killed process 11781 (mserver5) Sometimes, monetdb can recover from this error and then the loading continues normally, some others I have to manually restart monetdb. Is there anything I can do to troubleshoot this issue? Is there a possibility for me to fall back to an earlier version of MonetDB with potentially less features but more stability? Thanks for all the help, Stathis

2 2

[MonetDB-users] rdf support
by Jürgen Jakobitsch 14 Mar '12

14 Mar '12

hi, first of all congrats to this great project! i came to monetDB via a search for "column stores", since i know that one of the big players in the field of linked data and rdf storage switched to usage of a column store in one of their latest releases. i just built from source without any hassle on openSuse 11.4 with support for rdf since this is my primary interest at the moment. i didn't find much of an info regarding monetDB and rdf besides an old nabble-mail and a small-meeting wiki. is there some more info available? any pointer really appreciated. wkr jürgen -- | Jürgen Jakobitsch, | Software Developer | Semantic Web Company GmbH | Mariahilfer Straße 70 / Neubaugasse 1, Top 8 | A - 1070 Wien, Austria | Mob +43 676 62 12 710 | Fax +43.1.402 12 35 - 22 COMPANY INFORMATION | http://www.semantic-web.at/ PERSONAL INFORMATION | web : http://www.turnguard.com | foaf : http://www.turnguard.com/turnguard | skype : jakobitsch-punkt | xmlns:tg = "http://www.turnguard.com/turnguard#"

1 0

[MonetDB-users] Replication/Redundancy
by Joseph Brower 13 Mar '12

13 Mar '12

We're using MonetDB quite heavily and are really enjoying it. We're now wanting to add replication/redundancy to ensure we have as little downtime as possible. We're also thinking of balancing reads if that is possible. I've heard this is possible, but I'm unsure on the best way to approach it. I've not been able to find many docs that talk about this either. Could someone point me in the right direction? Thanks, Joseph Brower

4 10

[MonetDB-users] monetdbd: binding to datagram socket port 50000 failed: no available address
by Charalampos Nikolaou 09 Mar '12

09 Mar '12

When I start monetdbd I get the following error "monetdbd: binding to datagram socket port 50000 failed: no available address" Prior to this error I had properly terminated the server, so there is no server running. How can I overcome that problem? Also, netstat gives the following: $ netstat -p udp | grep 50000 udp4 1562 0 *.50000 *.* Thanks, Babis P.S.: I am on a Mac if that plays any role.

2 2

[MonetDB-users] some questions regarding queries in MonetDB
by Freddy Priyatna 06 Mar '12

06 Mar '12

Dear all, I've been using MonetDB version 5 on Ubuntu machine for evaluating BSBM queries. The dataset and queries can be found here[1]. It seems that there are some queries that are not supported on MonetDB : 1) SELECT p.nr, p.label FROM product p, producttypeproduct ptp WHERE p.nr=ptp.product AND "productType"=35 AND "propertyNum1">500 AND "propertyNum3"<2000 AND 168 IN (SELECT "productFeature" FROM productfeatureproduct WHERE productfeatureproduct.product=p.nr) AND 147 NOT IN (SELECT "productFeature" FROM productfeatureproduct WHERE product=p.nr) >From the error, the subquery seems can't refer to the table alias "p". 2) SELECT distinct p.nr, p.label FROM product p, product po, (Select distinct pfp1.product FROM productfeatureproduct pfp1, (SELECT "productFeature" FROM productfeatureproduct WHERE product=2) pfp2 WHERE pfp2."productFeature"=pfp1."productFeature") pfp WHERE p.nr=pfp.product AND po.nr=2 AND p.nr <> po.nr AND p."propertyNum1" < (po."propertyNum1"+120) AND p."propertyNum1" > (po."propertyNum1"-120) AND p."propertyNum2" < (po."propertyNum2"+170) AND p."propertyNum2" > (po."propertyNum2"-170) This is strange, because if I just comment one of the last 2 lines, the query works. But the last two lines can't be used together. 3) SELECT nr, label FROM product WHERE label like "%countries%"; It seems that MonetDB doesn't support regex. 4) SELECT * FROM (select label from product where nr=2) p left join ((select o.nr as onr, o.price, v.nr as vnr, v.label from offer o, vendor v where 2=o.product AND o.vendor=v.nr AND v.country='GB' AND o."validTo">'2008-07-01') ov right join (select r.nr as rnr, r.title, pn.nr as pnnr, pn.name, r.rating1, r.rating2 from review r, person pn where r.product=2 AND r.person=pn.nr) rpn on (1=1)) on (1=1); It seems that MonetDB can't do join with true condition because when I tried to run a simple query : SELECT * FROM product p RIGHT JOIN offer o ON (1=1) Then MonetDB gives an error. Could anyone please help me how to run the queries above? Many thanks in advance. Freddy [1] http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/ExploreUse…

5 11

[MonetDB-users] duplicate tables in schema
by Bas Kaptijn 05 Mar '12

05 Mar '12

Hi, repeatedly we get in a situation where tables end up duplicate in the sys.tables table: SELECT count(*) FROM tables WHERE name='mytable'; returns: 2 We think this should never be the case. It seems this gives problems with dumping the database and in other situations. This is an older version of MonetDB (Apr2011 SP1). We want to know whether someone knows this was/is a symptom of any known issue Sincerely, Bas

3 2

[MonetDB-users] Slow range join performance / regression with Dec2011-SP1
by Viktor Rosenfeld 02 Mar '12

02 Mar '12

Hi, I have a few queries that join a table with itself using a ranged predicate on the primary key. The simplest query is: SELECT count(*) AS count FROM rank_copy rank1, rank_copy rank2 WHERE rank1.cat_s = true AND rank2.pos_vvfin = true AND rank1.pre < rank2.pre AND rank2.pre < rank1.post As you can see, the range lookup on rank2.pre is restricted on both sides by rank1.pre and rank1.post. The selectivity of the attributes and the complete query is: - row count : 2431307 - query : 95977 (0.039) - cat_s : 148799 (0.061) - pos_vvfin : 71965 (0.030) PostgreSQL is able to evaluate the query in 2 seconds using an index on pre. (This index is automatically generated because pre is the primary key.) MonetDB Dec2011 requires about 35 seconds. The MAL plan is posted below. I've also tried sorting the table on pre, and created an (advisory) index on pre but this makes no difference. +---------------------------------------------------------------------------+ | mal | +===========================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_9:bat[:oid,:bit] := sql.bind(X_2,"sys","rank_copy","pos_vvfin",0); | | X_10 := algebra.uselect(X_9,true:bit); | | X_11 := algebra.markT(X_10,0@0:oid); | | X_12 := bat.reverse(X_11); | | X_4:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_13 := algebra.leftjoin(X_12,X_4); | | X_17:bat[:oid,:bit] := sql.bind(X_2,"sys","rank_copy","cat_s",0); | | X_18 := algebra.uselect(X_17,true:bit); | | X_19 := algebra.markT(X_18,0@0:oid); | | X_20 := bat.reverse(X_19); | | X_23 := algebra.leftjoin(X_20,X_4); | | X_15:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_21 := algebra.leftjoin(X_20,X_15); | | X_24 := algebra.join(X_13,X_23,X_21,false,false); | | X_26 := algebra.markT(X_24,0@0:oid); | | X_27 := bat.reverse(X_26); | | X_28 := algebra.leftjoin(X_27,X_13); | | X_29 := aggr.count(X_28); | | sql.exportValue(1,"sys.rank2","count":str,"wrd",64,0,6,X_29,""); | | end s1_1; | +---------------------------------------------------------------------------+ 22 tuples If I use a slightly more complicated query that has the boolean attributes pos_vvfin and cat_s on another table that is joined by rank, the situation is even worse. SELECT count(*) AS count FROM node_copy node1 JOIN rank_copy rank1 ON (node1.id = rank1.node_ref), node_copy node2 JOIN rank_copy rank2 ON (node2.id = rank2.node_ref) WHERE node1.cat_s = true AND node2.pos_vvfin = true AND rank1.pre < rank2.pre AND rank2.pre < rank1.post PostgreSQL evaluates the query in 9 seconds, again using the primary key indexes on node.id and rank.pre. MonetDB Dec2011 requires 14 minutes. The MAL plan follows: +-----------------------------------------------------------------------------------------------+ | mal | +===============================================================================================+ | function user.s1_1{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_19:bat[:oid,:oid] := sql.bind_idxbat(X_2,"sys","rank_copy","FK_rank_copy_node_ref",0); | | X_13:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","pos_vvfin",0); | | X_14 := algebra.uselect(X_13,true:bit); | | X_15 := algebra.markT(X_14,0@0:oid); | | X_16 := bat.reverse(X_15); | | X_9:bat[:oid,:int] := sql.bind(X_2,"sys","node_copy","id",0); | | X_11 := bat.mirror(X_9); | | X_17 := algebra.leftjoin(X_16,X_11); | | X_18 := bat.reverse(X_17); | | X_37:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","cat_s",0); | | X_39 := algebra.uselect(X_37,true:bit); | | X_40 := algebra.markT(X_39,0@0:oid); | | X_41 := bat.reverse(X_40); | | X_42 := algebra.leftjoin(X_41,X_11); | | X_43 := bat.reverse(X_42); | | X_24 := algebra.join(X_19,X_18); | | X_25 := algebra.markT(X_24,0@0:oid); | | X_26 := bat.reverse(X_25); | | X_4:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_27 := algebra.leftjoin(X_26,X_4); | | X_29:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_30 := algebra.join(X_27,X_4,X_29,false,false); | | X_32 := algebra.markT(X_30,0@0:oid); | | X_33 := bat.reverse(X_32); | | X_44 := bat.reverse(X_30); | | X_45 := algebra.markT(X_44,0@0:oid); | | X_46 := bat.reverse(X_45); | | X_47 := algebra.leftjoin(X_46,X_19); | | X_48 := algebra.join(X_47,X_43); | | X_49 := algebra.markT(X_48,0@0:oid); | | X_50 := bat.reverse(X_49); | | X_51:bat[:oid,:int] := algebra.leftjoinPath(X_50,X_33,X_27); | | X_52 := aggr.count(X_51); | | sql.exportValue(1,"sys.rank2","count","wrd",64,0,6,X_52,""); | | end s1_1; | +-----------------------------------------------------------------------------------------------+ 37 tuples The Dec2011-SP1 release takes even longer, about 18 minutes. The MAL plan is unchanged. The culprit appears to be the ranged lookup because if I replace it with an equi-join on another attribute MonetDB evaluates it in about 100 ms. SELECT count(*) AS count FROM node_copy node1 JOIN rank_copy rank1 ON (node1.id = rank1.node_ref), node_copy node2 JOIN rank_copy rank2 ON (node2.id = rank2.node_ref) WHERE node1.cat_s = true AND node2.pos_vvfin = true AND rank1.pre = rank2.parent I've tried rewriting the query using WITH and had some success. The first query again takes 35 seconds: WITH rank1 AS (SELECT * FROM rank_copy WHERE cat_s = true), rank2 AS (SELECT * FROM rank_copy WHERE pos_vvfin = true) SELECT count(*) FROM rank1, rank2 WHERE rank1.pre < rank2.pre AND rank2.pre < rank1.post The second query also takes 35 seconds when rewritten: WITH rank1 AS (SELECT * FROM rank_copy JOIN node_copy ON (rank_copy.node_ref = node_copy.id) WHERE node_copy.cat_s = true), rank2 AS (SELECT * FROM rank_copy JOIN node_copy ON (rank_copy.node_ref = node_copy.id) WHERE node_copy.pos_vvfin = true) SELECT count(*) FROM rank1, rank2 WHERE rank1.pre < rank2.pre AND rank2.pre < rank1.post MAL plan: +-----------------------------------------------------------------------------------------------+ | mal | +===============================================================================================+ | function user.s1_2{autoCommit=true}():void; | | X_2 := sql.mvc(); | | X_20:bat[:oid,:oid] := sql.bind_idxbat(X_2,"sys","rank_copy","FK_rank_copy_node_ref",0); | | X_13:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","pos_vvfin",0); | | X_14 := algebra.uselect(X_13,true:bit); | | X_15 := algebra.markT(X_14,0@0:oid); | | X_16 := bat.reverse(X_15); | | X_9:bat[:oid,:int] := sql.bind(X_2,"sys","node_copy","id",0); | | X_11 := bat.mirror(X_9); | | X_17 := algebra.leftjoin(X_16,X_11); | | X_18 := bat.reverse(X_17); | | X_32:bat[:oid,:bit] := sql.bind(X_2,"sys","node_copy","cat_s",0); | | X_33 := algebra.uselect(X_32,true:bit); | | X_34 := algebra.markT(X_33,0@0:oid); | | X_35 := bat.reverse(X_34); | | X_36 := algebra.leftjoin(X_35,X_11); | | X_37 := bat.reverse(X_36); | | X_22 := algebra.join(X_20,X_18); | | X_23 := algebra.markT(X_22,0@0:oid); | | X_24 := bat.reverse(X_23); | | X_5:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","pre",0); | | X_25 := algebra.leftjoin(X_24,X_5); | | X_38 := algebra.join(X_20,X_37); | | X_39 := algebra.markT(X_38,0@0:oid); | | X_40 := bat.reverse(X_39); | | X_43 := algebra.leftjoin(X_40,X_5); | | X_27:bat[:oid,:int] := sql.bind(X_2,"sys","rank_copy","post",0); | | X_41 := algebra.leftjoin(X_40,X_27); | | X_44 := algebra.join(X_25,X_43,X_41,false,false); | | X_46 := algebra.markT(X_44,0@0:oid); | | X_47 := bat.reverse(X_46); | | X_48 := algebra.leftjoin(X_47,X_25); | | X_49 := aggr.count(X_48); | | sql.exportValue(1,"sys.rank2","L1","wrd",64,0,6,X_49,""); | | end s1_2; | +-----------------------------------------------------------------------------------------------+ 35 tuples Is there anything I can do to further speed up the execution of these queries? Thanks, Viktor

1 0