December 2008 - users-list

[MonetDB-users] multi-users and memory problems
by Loredana Afanasiev 21 Dec '08

21 Dec '08

Dear developers, I'm currently running the CVS head of Dec 11, 2008. I apologies in advance for missing on details. If needed I'll provide more. Thank you, and lots of wonderful wishes for Xmas and for the New Year!!! (this is the last email you'll get from me this year, I promise :)) l. Observation 1: When trying to run big numbers of clients simultaneously, the server crushes. For me it crushed with 70 and 100 clients fired at the same time. Observation 2: After running one query in a multi-user scenario (N clients/threads simultaneously) the memory footprint of mserver (% of the memory that the server occupies) grew. After repeating the experiment M times the memory grew each time with a constant. Running the same query sequentially the same number of times leaves the footprint of mserver constant. Could it be a faulty memory cleaning? Query: q3.xq let $col := fn:collection("MotiesTweedeKamer") let $years := fn:distinct-values( for $date in $col//hiddendatum return fn:substring(fn:string($date),1,4)) for $y in $years order by $y ascending return <result year="{$y}" count="{ count($col//document[fn:substring(fn:string(.//hiddendatum),1,4) = $y]) }"/> N=50 M=1 $ perl runNclients.pl N=50 $ top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28563 lafanasi 20 0 6266m 5.1g 100m S 0 26.0 10:50.71 Mserver M=2 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28563 lafanasi 20 0 7546m 5.9g 100m S 0 30.1 21:42.67 Mserver ... N=60 M=10 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31182 lafanasi 20 0 17.3g 8.3g 100m S 0 42.3 263:45.37 Mserver ... M=30 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31182 lafanasi 20 0 22.7g 9.6g 100m S 0 48.8 395:08.87 Mserver Sequential run: N=1 M=50 times PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28372 lafanasi 20 0 839m 430m 100m S 1 2.1 6:26.65 Mserver M=100 times PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28372 lafanasi 20 0 839m 430m 100m S 1 2.1 6:26.65 Mserver Observation 3: The times it takes to run 10 Tijah search queries simultaneously is the same as the time it takes to run them sequentially. The percentage of CPU used in both cases by mserver is also similar. Does tijah support multi-users? Query: q2.xq let $opt := <TijahOptions ft-index="polietiekedata" ir-model="NLLR"/> let $c := collection("HAN") let $qid := tijah:query-id($c, "//spreker[about(.,%KEYWORD%)]", $opt) for $res in tijah:nodes($qid) return <pair>{( string($res/@naam), tijah:score($qid, $res))}</pair> 20 users/threads stopMserver,startMserver (to make sure that the server freed the memory) $time Run20Threads(mclient -lx q2.xq) 0.134u 0.118s 0:20.01 1.1% 0+0k 0+8io 0pf+0w sequential run, 20 times stopMserver, startMserver $ time for i in `seq 1 10`; do mclient -lx q2.xq; done 0.060u 0.067s 0:19.74 0.6% 0+0k 0+8io 0pf+0w Observation 4: The mserver memory footprint grows very fast when running Tijah queries. When the footprint reaches 98% the query processing time gets really slow or the server crushes. Query: the same as above Sequential run, M=90 times $stopMserver, startMserver $for i in `seq 1 10`; do mclient -lx q2.xq; done $ top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24527 lafanasi 20 0 26.2g 19g 80m S 66 98.0 4:24.57 Mserver

2 1

[MonetDB-users] mserver memory grows, system thrashes and hangs
by Bostr 18 Dec '08

18 Dec '08

Hello MonetDB Nov2008 release, built with all standard options on RedHat EL 5 (2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with dbfarm on a RAID0 internal 7-disk array. I'm doing bulk load tests (using mclient to run COPY INTO table FROM file), repeatedly loading 500,000 row (68M) CSV files into a fact table, while periodically running a suite of test queries in a separate process to test query times as the fact table grows. The problem is that as the fact table grows to around 80Million records, the database starts to consume all available system memory and swap, the system thrashes and comes to its knees. I also get database corruption causing foreign key violations. (More details below) Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this be caused by operator error? Suggestions? Cheers Bob PS More detailed account of the problem below: Everything starts off fine, files load in 8-10 seconds, queries quite fast. As the table starts to fill, the queries (not unexpectedly) start to slow. But (unexpectedly) the memory consumption of the mserver5 process steadily increases. At some point, around 80Million records, the system performance rapidly starts to degrade. Query times dramatically increase, load time also dramatically increases, and the entire system thrashes and becomes unusable. Running 'top' shows that mserver5 has consumed vurtually all physical memory, and most of the swap too. top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96 Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total, 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal The first time this happened, I stopped the load scripts, restarted the database, and that helped things for a little while, but it wasn't long before the memory was all consumed again, and performance went to the dogs. Then loads started failing with foreign-key violation errors.. Inspection showed a corrupted dimension table that somehow ended up with a bunch of bad values in it.. So I blew away, and recreated the database. This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file - having seen a defect report related to 'COPY INTO ' being buggy and this being the solution. But the same thing happened again.. it ran fine again until I had about 100Million rows, then memory problems/thrashing. Restarted the server again, and shortly after the data loads failed again with foreign-key violations and a corrupted dimension table.. -- View this message in context: http://www.nabble.com/mserver-memory-grows%2C-system-thrashes-and-hangs-tp2… Sent from the monetdb-users mailing list archive at Nabble.com.

4 5

[MonetDB-users] Again: Mserver fails on Fedora 9 - 32 bit
by Hans van Rijswijk (DT) 11 Dec '08

11 Dec '08

Hi MonetDB users, I downloaded Monet Fedora RPMs from: http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./2008 1209/RPMs/Fedora9.i386.oid32/ I installed these with yum on my Fedora fc9 installation, running within VMWare on my Windows XP 32 bit machine. When I startup Mserver everything seems to work well. I start with an empty dbfarm. Then I run the following scenario: - Add 3 XML documents with pf:add-doc() (some of them updatable) - Add 2 more XML documents with pf:add-doc() - Update one of these docs: do insert <something/> into doc("a.xml")/rootelement - Send a sync() command with mclient mil. - Check the content of the database with pf:documents() (is OK) - Knock down Mserver: kill <pid> - Startup Mserver again (don't ask me why I do it like this) After this scenario I cannot gain access to Mserver with mclient: !xchange_challenge: frontend xquery not found If I wait half an hour before killing Mserver this works fine. Questions: - Does sync() do what it is supposed to do? - Why is shutdown() not implemented? - What is the right way to shutdown Mserver when it is started as a daemon in the background? - How do I guarantee the database to remain intact with the abovementioned scenario? Regards, Hans van Rijswijk Netherlands Forensic Institute

4 7

[MonetDB-users] crash on Windows with local install
by Bryan Lewis 11 Dec '08

11 Dec '08

If I install MonetDB in the normal way on a Windows machine, it installs the C runtime library dlls in a location like C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT... However, we want to install MonetDB along with our product, in a single installation procedure that doesn't require the user to have administrative rights. That means we need to install the CRT directory as "private assemblies", copying it to the MonetDB application directories. I've almost gotten this to work by copying the CRT directory to MonetDB5/lib, bin and lib/MonetDB5. I can run MSQLServer.bat and mclient.bat in separate console windows. But when I try to run some SQL commands (my test case is a 'create table' and a 'copy into' from a CSV file), mclient dies with a Windows exception. The event in Event Viewer shows: Faulting application mclient.exe, version 0.0.0.0, faulting module ntdll.dll In theory the private assemblies are all that Windows needs, and in fact the programs run which implies that the libraries are being found. I don't understand why the crash happens later. I'm hoping someone else has tried this. The exception happens with the last official Windows release as well as with our built-from-cvs version. Thanks, Bryan Lewis

3 2

[MonetDB-users] Mserver fails on Fedora 9 - 32 bit
by Hans van Rijswijk (DT) 09 Dec '08

09 Dec '08

Hi MonetDB users, I downloaded Monet Fedora RPMs from: http://monetdb.cwi.nl/testing/projects/monetdb/Stable/.DailyBuilds./2008 1209/RPMs/Fedora9.i386.oid32/ I installed these with yum on my Fedora fc9 installation, running within VMWare on my Windows XP 32 bit machine. I got the following messages when starting Mserver: # MonetDB Server v4.26.1 # based on GDK v1.26.1 # Copyright (c) 1993-July 2008, CWI. All rights reserved. # Copyright (c) August 2008-, MonetDB B.V.. All rights reserved. # Compiled for i686-redhat-linux-gnu/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/pftijah !ERROR: logger: could not use bat (209) for 1000000000_map_pid !ERROR: logger: could not use bat (210) for 1000000000_rid_size !ERROR: logger: could not use bat (442) for 1000000000_rid_level !ERROR: logger: could not use bat (392) for 1000000000_rid_prop !ERROR: logger: could not use bat (441) for 1000000000_rid_kind !ERROR: logger: could not use bat (412) for 1000000000_rid_nid !ERROR: logger: could not use bat (418) for 1000000000_nid_rid !ERROR: logger: could not use bat (211) for 1000000000_qn_histogram !ERROR: logger: could not use bat (451) for 1000000000_qn_prefix_uri_loc !ERROR: logger: could not use bat (452) for 1000000000_qn_uri_loc !ERROR: logger: could not use bat (453) for 1000000000_qn_prefix !ERROR: logger: could not use bat (454) for 1000000000_qn_uri !ERROR: logger: could not use bat (455) for 1000000000_qn_loc !ERROR: logger: could not use bat (456) for 1000000000_prop_text !ERROR: logger: could not use bat (457) for 1000000000_prop_com !ERROR: logger: could not use bat (458) for 1000000000_prop_ins !ERROR: logger: could not use bat (459) for 1000000000_prop_tgt !ERROR: logger: could not use bat (460) for 1000000000_prop_val !ERROR: logger: could not use bat (379) for 1000000000_attr_own !ERROR: logger: could not use bat (406) for 1000000000_attr_qn !ERROR: logger: could not use bat (427) for 1000000000_attr_prop !ERROR: logger: could not use bat (245) for 1000000001_map_pid !ERROR: logger: could not use bat (246) for 1000000001_rid_size !ERROR: logger: could not use bat (423) for 1000000001_rid_level !ERROR: logger: could not use bat (384) for 1000000001_rid_prop !ERROR: logger: could not use bat (436) for 1000000001_rid_kind !ERROR: logger: could not use bat (421) for 1000000001_rid_nid !ERROR: logger: could not use bat (100) for 1000000001_nid_rid !ERROR: logger: could not use bat (247) for 1000000001_qn_histogram !ERROR: logger: could not use bat (391) for 1000000001_qn_prefix_uri_loc !ERROR: logger: could not use bat (471) for 1000000001_qn_uri_loc !ERROR: logger: could not use bat (383) for 1000000001_qn_prefix !ERROR: logger: could not use bat (478) for 1000000001_qn_uri !ERROR: logger: could not use bat (479) for 1000000001_qn_loc !ERROR: logger: could not use bat (480) for 1000000001_prop_text !ERROR: logger: could not use bat (481) for 1000000001_prop_com !ERROR: logger: could not use bat (482) for 1000000001_prop_ins !ERROR: logger: could not use bat (483) for 1000000001_prop_tgt !ERROR: logger: could not use bat (484) for 1000000001_prop_val !ERROR: logger: could not use bat (414) for 1000000001_attr_own !ERROR: logger: could not use bat (103) for 1000000001_attr_qn !ERROR: logger: could not use bat (102) for 1000000001_attr_prop !ERROR: logger: could not use bat (380) for 1000000002_map_pid !ERROR: logger: could not use bat (425) for 1000000002_rid_size !ERROR: logger: could not use bat (438) for 1000000002_rid_level !ERROR: logger: could not use bat (378) for 1000000002_rid_prop !ERROR: logger: could not use bat (279) for 1000000002_rid_kind !ERROR: logger: could not use bat (399) for 1000000002_rid_nid !ERROR: logger: could not use bat (416) for 1000000002_nid_rid !ERROR: logger: could not use bat (405) for 1000000002_qn_histogram !ERROR: logger: could not use bat (431) for 1000000002_qn_prefix_uri_loc !ERROR: logger: could not use bat (443) for 1000000002_qn_uri_loc !ERROR: logger: could not use bat (229) for 1000000002_qn_prefix !ERROR: logger: could not use bat (472) for 1000000002_qn_uri !ERROR: logger: could not use bat (498) for 1000000002_qn_loc !ERROR: logger: could not use bat (499) for 1000000002_prop_text !ERROR: logger: could not use bat (500) for 1000000002_prop_com !ERROR: logger: could not use bat (501) for 1000000002_prop_ins !ERROR: logger: could not use bat (502) for 1000000002_prop_tgt !ERROR: logger: could not use bat (503) for 1000000002_prop_val !ERROR: logger: could not use bat (433) for 1000000002_attr_own !ERROR: logger: could not use bat (327) for 1000000002_attr_qn !ERROR: logger: could not use bat (468) for 1000000002_attr_prop !ERROR: logger: could not use bat (555) for 1000000003_map_pid !ERROR: logger: could not use bat (238) for 1000000003_rid_size !ERROR: logger: could not use bat (388) for 1000000003_rid_level !ERROR: logger: could not use bat (239) for 1000000003_rid_prop !ERROR: logger: could not use bat (386) for 1000000003_rid_kind !ERROR: logger: could not use bat (556) for 1000000003_rid_nid !ERROR: logger: could not use bat (557) for 1000000003_nid_rid !ERROR: logger: could not use bat (206) for 1000000003_qn_histogram !ERROR: logger: could not use bat (487) for 1000000003_qn_prefix_uri_loc !ERROR: logger: could not use bat (512) for 1000000003_qn_uri_loc !ERROR: logger: could not use bat (510) for 1000000003_qn_prefix !ERROR: logger: could not use bat (509) for 1000000003_qn_uri !ERROR: logger: could not use bat (508) for 1000000003_qn_loc !ERROR: logger: could not use bat (507) for 1000000003_prop_text !ERROR: logger: could not use bat (506) for 1000000003_prop_com !ERROR: logger: could not use bat (283) for 1000000003_prop_ins !ERROR: logger: could not use bat (409) for 1000000003_prop_tgt !ERROR: logger: could not use bat (393) for 1000000003_prop_val !ERROR: logger: could not use bat (465) for 1000000003_attr_own !ERROR: logger: could not use bat (204) for 1000000003_attr_qn !ERROR: logger: could not use bat (474) for 1000000003_attr_prop !ERROR: logger: could not use bat (489) for 1000000004_map_pid !ERROR: logger: could not use bat (370) for 1000000004_rid_size !ERROR: logger: could not use bat (428) for 1000000004_rid_level !ERROR: logger: could not use bat (546) for 1000000004_rid_prop !ERROR: logger: could not use bat (419) for 1000000004_rid_kind !ERROR: logger: could not use bat (196) for 1000000004_rid_nid !ERROR: logger: could not use bat (537) for 1000000004_nid_rid !ERROR: logger: could not use bat (470) for 1000000004_qn_histogram !ERROR: logger: could not use bat (511) for 1000000004_qn_prefix_uri_loc !ERROR: logger: could not use bat (396) for 1000000004_qn_uri_loc !ERROR: logger: could not use bat (390) for 1000000004_qn_prefix !ERROR: logger: could not use bat (382) for 1000000004_qn_uri !ERROR: logger: could not use bat (426) for 1000000004_qn_loc !ERROR: logger: could not use bat (387) for 1000000004_prop_text !ERROR: logger: could not use bat (518) for 1000000004_prop_com !ERROR: logger: could not use bat (532) for 1000000004_prop_ins !ERROR: logger: could not use bat (569) for 1000000004_prop_tgt !ERROR: logger: could not use bat (562) for 1000000004_prop_val !ERROR: logger: could not use bat (515) for 1000000004_attr_own !ERROR: logger: could not use bat (539) for 1000000004_attr_qn !ERROR: logger: could not use bat (224) for 1000000004_attr_prop # MonetDB/XQuery module v0.26.1 loaded (default back-end is 'algebra') # XRPC administrative console at http://127.0.0.1:51241/admin This is not all: also mclient could not communicate with Mserver: $ mclient -p51240 -lx !xchange_challenge: frontend xquery not found The same applies when running this distribution on Fedora 10. Did anyone encounter a similar problem; Does anyone know a workaround? Hope to hear from one of you soon.. Regards, Hans van Rijswijk Netherlands Forensic Institute

2 1

[MonetDB-users] Announcement: New feature release of MonetDB suite
by Sjoerd Mullender 08 Dec '08

08 Dec '08

The MonetDB team at CWI is pleased to announce the new "Nov2008" release of the MonetDB suite of programs. This is a feature release, involving, among other things, a change in the database format, so be sure to dump your databases before attempting an upgrade. More information on the new release is available at <https://sourceforge.net/forum/forum.php?forum_id=892621>. The sources and binaries are available at <http://monetdb.cwi.nl/projects/monetdb/Download/>. -- Sjoerd Mullender

1 0

[MonetDB-users] Mclient exit value
by Floris Ouwendijk (DT) 04 Dec '08

04 Dec '08

Dear users, While investigating a problem where a collection-query returns invalid data, I ran into problems with mclient seemingly not generating a proper exit value. If I run: $ mclient -lxq -p49980 -s"element { \"log\" } { pf:collection(\"cons02_log.xml\")//run }" -removed a lot of xml- </stdout> <fault!ERROR: XML Generation: tmp_3265 BAT does not have a 24 head. MAPI = monetdb@botswana.xiraf:49980 QUERY = element { "log" } { pf:collection("cons02_log.xml")//run } ERROR = !ERROR: !ERROR: XML Generation: NULL BAT has a 24 head, but tail is NULL. !ERROR: !ERROR: xquery_print_result_main: operation failed. $ echo $? 0 The above returns a lot of correct XML, then dumps an error to stderr, but returns error code 0. In contrast, the following works as expected: $ mclient -lxq -p49980 -s"eelement { \"log\" } { pf:collection(\"cons02_log.xml\")//run }" >/dev/null MAPI = monetdb@server:49980 QUERY = eelement { "log" } { pf:collection("cons02_log.xml")//run } ERROR = !parse error: syntax error, unexpected {, expecting $end on line 1, column 10 (next token is `{') !parse error: XQuery parsing failed !# halted in parser.tab.c (PFparse), line 5716 $ echo $? 1 Which is the expected result for a flawed query. Do you agree this is a problem that should be reported, or is there a better way of finding out whether there were problems running the query? Kind regards, Floris

2 1

[MonetDB-users] Building the nightly build on Centos 5
by Schwarz, Omri 01 Dec '08

01 Dec '08

Hi, all. I am trying to do a build on Centos 5 and am getting this bizarre issue which a web search shows has propped up with many other autoconf-based builds, but which does not have an easy resolution: /bin/sh ../../libtool --tag=CC --mode=link gcc -DLIBGDK -O2 -std=c99 -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -fu nroll-loops -frerun-cse-after-loop -frerun-loop-opt -ftree-vectorize -D_REENTRANT -o libbat.la -rpath /root/MonetDB/l ib libbat_la-gdk_scanselect.lo libbat_la-gdk_scanselect_defs.lo libbat_la-gdk_batop.lo libbat_la-gdk_search.lo libbat_la -gdk_tm.lo libbat_la-gdk_align.lo libbat_la-gdk_bbp.lo libbat_la-gdk_heap.lo libbat_la-gdk_setop.lo libbat_la-gdk_utils.l o libbat_la-gdk_atoms.lo libbat_la-gdk_qsort.lo libbat_la-gdk_ssort.lo libbat_la-gdk_storage.lo libbat_la-gdk_bat.lo libb at_la-gdk_delta.lo libbat_la-gdk_relop.lo libbat_la-gdk_system.lo libbat_la-gdk_value.lo libbat_la-gdk_rangejoin.lo libba t_la-gdk_posix.lo libbat_la-gdk_logger.lo ../common/libmutils.la ../common/libstream.la -lz -lpthread -ldl gcc -shared .libs/libbat_la-gdk_scanselect.o .libs/libbat_la-gdk_scanselect_defs.o .libs/libbat_la-gdk_batop.o .libs/lib bat_la-gdk_search.o .libs/libbat_la-gdk_tm.o .libs/libbat_la-gdk_align.o .libs/libbat_la-gdk_bbp.o .libs/libbat_la-gdk_he ap.o .libs/libbat_la-gdk_setop.o .libs/libbat_la-gdk_utils.o .libs/libbat_la-gdk_atoms.o .libs/libbat_la-gdk_qsort.o .lib s/libbat_la-gdk_ssort.o .libs/libbat_la-gdk_storage.o .libs/libbat_la-gdk_bat.o .libs/libbat_la-gdk_delta.o .libs/libbat_ la-gdk_relop.o .libs/libbat_la-gdk_system.o .libs/libbat_la-gdk_value.o .libs/libbat_la-gdk_rangejoin.o .libs/libbat_la-g dk_posix.o .libs/libbat_la-gdk_logger.o -Wl,--rpath -Wl,/var/tmp/test/MonetDB/MonetDB/src/common/.libs -Wl,--rpath -Wl,/ root/MonetDB/lib -L/var/tmp/test/MonetDB/MonetDB/src/common/.libs ../common/.libs/libmutils.so ../common/.libs/libstream. so -lz -lpthread -ldl -Wl,-soname -Wl,libbat.so.0 -o .libs/libbat.so.0.0.0 .libs/libbat_la-gdk_scanselect_defs.o: In function `__strcspn_c1': gdk_scanselect_defs.c:(.text+0x0): multiple definition of `__strcspn_c1' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x0): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strcspn_c2': gdk_scanselect_defs.c:(.text+0x34): multiple definition of `__strcspn_c2' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x34): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strcspn_c3': gdk_scanselect_defs.c:(.text+0x78): multiple definition of `__strcspn_c3' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x78): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strspn_c1': gdk_scanselect_defs.c:(.text+0xd4): multiple definition of `__strspn_c1' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0xd4): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strspn_c2': gdk_scanselect_defs.c:(.text+0xf0): multiple definition of `__strspn_c2' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0xf0): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strspn_c3': gdk_scanselect_defs.c:(.text+0x108): multiple definition of `__strspn_c3' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x108): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strpbrk_c2': gdk_scanselect_defs.c:(.text+0x128): multiple definition of `__strpbrk_c2' .libs/libbat_la-gdk_scanselect.o:gdk_scanselect.c:(.text+0x128): first defined here .libs/libbat_la-gdk_scanselect_defs.o: In function `__strpbrk_c3': [... and so ad nauseam, containing redefinitions of amy low level LibC functions like toupper, tolower, fstat, et cetera. ] What do I do to dodge this? Thanks, all Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

[MonetDB-users] Records disappear in prepared statement insterts via JDCB and ODBC if using autocommit or batch commit
by Jens Oehlschlägel 01 Dec '08

01 Dec '08

Dear monetdb community, I am new to monetdb and I am very impressed about its speed and nice features like its csv bulk load command. I have a question on prepared statements: I noticed that when using prepared statements via ODBC or JDBC strange things happened: when inserting 1000 rows, not only insert statements became very slow, also many inserts silently never reached MonetDB, i.e. a random number of records passes and suddenly - observed with ODBC debugging activated - execs are responded with errors. I tried via ODBC (writing from R's RODBC package) and via JDBC (writing from DBVizualizer). I found out that deactivating autocommit will help (verified for ODBC and JDBC), but the problem reappears - again randomly - if the client tool sends batch commits each x records (verified only in DBVizualizer / JDBC, now with explicit errors). Is that a known limitation or a bug or am I doing wrong? Further down you find information for replication. Thanks for any help Kind regards Dr. Jens Oehlschlägel Munich P.S. Details I run MonetDB under WindowsProfessionalXP with 3 GB RAM. I installed: MonetDB5-SQL-Installer-i386-20080624.msi MonetDB-ODBC-Installer-i386-20080812.msi monetdb-1.8-jdbc.jar Then - as recommended - I created CREATE USER "voc" WITH PASSWORD 'voc' NAME 'VOC Explorer' SCHEMA "sys"; CREATE SCHEMA "voc" AUTHORIZATION "voc"; ALTER USER "voc" SET SCHEMA "voc"; and added my test table CREATE TABLE databasetest ( "id" INTEGER , "ubyte1" SMALLINT , "ubyte2" SMALLINT , "ubyte3" SMALLINT , "ubyte4" SMALLINT , "ubyte5" SMALLINT , "ubyte6" SMALLINT , "ubyte7" SMALLINT , "ubyte8" SMALLINT , "ubyte9" SMALLINT , "ubyte10" SMALLINT , "ubyte11" SMALLINT , "ubyte12" SMALLINT , "ubyte13" SMALLINT , "ubyte14" SMALLINT , "ubyte15" SMALLINT , "smallint1" SMALLINT , "smallint2" SMALLINT , "smallint3" SMALLINT , "smallint4" SMALLINT , "smallint5" SMALLINT , "smallint6" SMALLINT , "smallint7" SMALLINT , "smallint8" SMALLINT , "smallint9" SMALLINT , "smallint10" SMALLINT , "smallint11" SMALLINT , "smallint12" SMALLINT , "smallint13" SMALLINT , "smallint14" SMALLINT , "smallint15" SMALLINT , "float1" DOUBLE , "float2" DOUBLE , "float3" DOUBLE , "float4" DOUBLE , "float5" DOUBLE , "float6" DOUBLE , "float7" DOUBLE , "float8" DOUBLE , "float9" DOUBLE , "float10" DOUBLE , "float11" DOUBLE , "float12" DOUBLE , "float13" DOUBLE , "float14" DOUBLE , "float15" DOUBLE , "float16" DOUBLE , "float17" DOUBLE , "float18" DOUBLE , "float19" DOUBLE , "float20" DOUBLE ) ; The test data contains positive and negative numbers, and the following R script (to be executed with R from www.r-project.org) allows to replicate the problem # R-script commands for replication of problem with prepared statement inserts in MonetDB # You find the installer for the R interpreter under http://cran.at.r-project.org (windows version 2.8.0) # (linux and sourcecode versions also available) # just install and start R # then install the RODBC package (version 1.2-4) install.packages("RODBC") # then paste the following lines # activate ODBC package library(RODBC) # only if we deactivate autocommit the prepared statement inserts work correctly (and much faster) # this variable switches autocommit of and sends standard commit at end of prepared statements. noAutoCommit <- TRUE channel <- odbcDriverConnect(connection = "DSN=MonetDB;UID=voc;PWD=voc;DATABASE=demo;host=localhost;port=50000;" , case="tolower" , believeNRows = TRUE , colQuote='"' , tabQuote = '"' , DBMSencoding = "" , rows_at_time = 1000 , bulk_add = "yes" ) if (noAutoCommit) odbcSetAutoCommit(channel, autoCommit=FALSE) # test connection #sqlTables(channel) #sqlColumns(channel, "databasetest") b <- 1L # number of batches n <- 10000L # batch size system.time({ for (i in 1:b){ cat("i=", i, "\n") x <- data.frame( id = ((i-1L)*n+1L):(i*n) # 4-byte integer record identifier , ubyte1 = sample(0:35, n, TRUE, prob=NULL) # signed byte columns (1 byte) , ubyte2 = sample(0:1, n, TRUE, prob=NULL) , ubyte3 = sample(0:3, n, TRUE, prob=NULL) , ubyte4 = sample(0:9, n, TRUE, prob=NULL) , ubyte5 = sample(0:24, n, TRUE, prob=NULL) , ubyte6 = sample(0:99, n, TRUE, prob=NULL) , ubyte7 = sample(0:127, n, TRUE, prob=NULL) , ubyte8 = sample(0:1, n, TRUE, prob=c(0.75, 0.25)) , ubyte9 = sample(0:1, n, TRUE, prob=c(0.9,0.1)) , ubyte10 = sample(0:1, n, TRUE, prob=c(0.96,0.04)) , ubyte11 = sample(0:1, n, TRUE, prob=c(0.99,0.01)) , ubyte12 = sample(0:1, n, TRUE, prob=c(0.996,0.004)) , ubyte13 = sample(0:1, n, TRUE, prob=c(0.999,0.001)) , ubyte14 = sample(0:1, n, TRUE, prob=c(0.9996,0.0004)) , ubyte15 = sample(0:1, n, TRUE, prob=c(0.9999,0.0001)) , smallint1 = sample(1:4000, n, TRUE, prob=NULL) # smallint columns (2 bytes) , smallint2 = sample(1:2, n, TRUE, prob=NULL) , smallint3 = sample(1:4, n, TRUE, prob=NULL) , smallint4 = sample(1:10, n, TRUE, prob=NULL) , smallint5 = sample(1:25, n, TRUE, prob=NULL) , smallint6 = sample(1:100, n, TRUE, prob=NULL) , smallint7 = sample(1:256, n, TRUE, prob=NULL) , smallint8 = sample(1:1000, n, TRUE, prob=NULL) , smallint9 = sample(1:32000, n, TRUE, prob=NULL) , smallint10 = sample(0:1, n, TRUE, prob=c(0.75, 0.25)) , smallint11 = sample(0:1, n, TRUE, prob=c(0.9,0.1)) , smallint12 = sample(0:1, n, TRUE, prob=c(0.96,0.04)) , smallint13 = sample(0:1, n, TRUE, prob=c(0.99,0.01)) , smallint14 = sample(0:1, n, TRUE, prob=c(0.996,0.004)) , smallint15 = sample(0:1, n, TRUE, prob=c(0.999,0.001)) , float1 = round(runif(n, 0, 100), 1) # float columns (4 bytes) , float2 = round(runif(n, 0, 100), 1) , float3 = round(runif(n, 0, 100), 1) , float4 = round(runif(n, 0, 100), 1) , float5 = round(runif(n, 0, 100), 1) , float6 = round(runif(n, 0, 100), 1) , float7 = round(runif(n, 0, 100), 1) , float8 = round(runif(n, 0, 100), 1) , float9 = round(runif(n, 0, 100), 1) , float10 = round(runif(n, 0, 100), 1) , float11 = round(runif(n, 0, 100), 1) , float12 = round(runif(n, 0, 100), 1) , float13 = round(runif(n, 0, 100), 1) , float14 = round(runif(n, 0, 100), 1) , float15 = round(runif(n, 0, 100), 1) , float16 = round(runif(n, 0, 100), 1) , float17 = round(runif(n, 0, 100), 1) , float18 = round(runif(n, 0, 100), 1) , float19 = round(runif(n, 0, 100), 1) , float20 = round(runif(n, 0, 100), 1) ) sqlSave(channel, x, tablename="databasetest", rownames=FALSE, append=TRUE, fast=TRUE) if (noAutoCommit) odbcEndTran(channel, commit=TRUE) print(sqlQuery(channel, 'SELECT count(*) as cc, min(id) as minid, max(id) as maxid from databasetest;')) } }) sqlQuery(channel, 'SELECT count(*) as cc, min(id) as minid, max(id) as maxid from databasetest;') close(channel)

1 0