- users-list - monetdb.org

BAT Buffer Pool functionality
by Hassan, Ahmad 04 Jun '13

04 Jun '13

Hi, I am reading Monetdb code to understand the functionality of BAT Buffer Pool (BBP). I understood that it doesn't allocate allocate memory for actual BAT but BATs are first instantiated on the HEAP memory using BATcreatedesc, BATnewstorage where BATcreatedesc also registers the entry of new BAT into BBP. BBP only allocates HEAP memory for maintaining the bookkeeping information for the BATs and keeping the directory structure information. I understood from the code that BBP keeps looking for the HOT/COLD state of BATs and performs unloading if memory is growing or descriptors are increasing using BBPTrim. Does this mean File descriptor here? Questions: 1) I want to know if MonetDB allocates a large heap space internally then reuse that space for allocating memory chunks to BATs on demand or every time MonetDB ask operating system to extend heap size when new BAT is created? 2) I want to understand how BBP manages the HEAP memory. When a BAT is no longer needed, then what BBP does with that memory. Does it return that memory to OS and shrinks the heap size used by MonetDB or it holds that space for future allocations? 3) A typical TPCH query generates 100's of intermediate BATs. Does MonetDB allocates memory for those BATs during query execution and returns it back to operating system when query finishes? 4) When we start MonetDB then how much initial heap memory is allocated by the MonetDB system. Does BBP allocates this memory on the HEAP? If not, then which part of the code knows which BATs to load at the start time and how much HEAP memory needs to be allocated for those BATs. I found the list of all BATs in Monetdb through bbp.get() mal call. I also understand that SQL catalog maintains the mapping of table to column names and column names to BAT_in_BBP mapping. But I couldn't able to find which BAT I should access through mal code to get the sql catalog mappings. I want to use MAL to see which table is composed of which column is linked to which BAT in BBP. Thanks for the help. Kind Regards, Ahmad

2 3

Possible to reset or flush out RAM usage in Windows without closing and re-opening mserver.exe?
by Anthony Damico 23 May '13

23 May '13

Hi, I'm running a two-day-long program on Windows x64 with MonetDB.R that breaks somewhere into the second day due to what I believe is overloaded RAM? I was under the impression that this should not really happen with MonetDB, since it's a database that doesn't load everything into RAM.. Here's what's happening: A big, complicated series of importations, merges, and recodes all work for the first two days on about 250,000,000 records of data.. but all the while my Windows Task Manager will slowly show the physical memory usage meter ticking up in the "Performance" tab. mserver.exe does NOT appear to be using RAM in the "Memory" column of the "Processes" tab, so it looks like there's a ghost using RAM. Then, after two days, the script will break (mserver output shown below) - returning a GDK error on a seemingly-benign line of code, just a simple recode: Error in .local(conn, statement, ..., async = async) : UPDATE ep10 SET rx_cso_cov = CVRD_D_PLAN_PD_AMT WHERE DRUG_CVRG_STUS_CD IN ( 'C','S','O' ) failed! Server says: !GDKerror:!ERROR: GDKmallocmax: failed for 589484032 bytes After that, if I close mserver.exe and open it up again, it takes about twenty minutes to get the ">" mserver prompt. But it does open up eventually.. and at that point, if I re-connect to the server with R and run the exact same line, it works without error. So it appears I just need to flush out that ghost/mystery RAM usage somehow? Does this error make sense? There is no way this is due to lack of hard disk space - this server's C drive has 15GB free and the attached drive actually hosting all of this data has 235GB free. I don't want to rule it out, but I also doubt this problem is related to MonetDB.R, since the scenario I described does not require that I restart my R console, just mserver.exe. I can work around the problem by manually killing and re-starting mserver.exe at a few logical breakpoints in my program, but I didn't think this was supposed to happen in the first place? Thanks!!! Anthony # MonetDB 5 server v11.15.7 "Feb2013-SP2" # Serving database 'medicare_sample', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 20.000 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:49800/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded ># SQL catalog created, loading sql scripts once # loading sql script: 09_like.sql # loading sql script: 10_math.sql # loading sql script: 11_times.sql # loading sql script: 12_url.sql # loading sql script: 13_date.sql # loading sql script: 14_inet.sql # loading sql script: 15_history.sql # loading sql script: 16_tracelog.sql # loading sql script: 17_compress.sql # loading sql script: 18_dictionary.sql # loading sql script: 19_cluster.sql # SQL catalog created, loading sql scripts once # loading sql script: 09_like.sql # loading sql script: 20_vacuum.sql # loading sql script: 10_math.sql # loading sql script: 11_times.sql # loading sql script: 12_url.sql # loading sql script: 21_dependency_functions.sql # loading sql script: 13_date.sql # loading sql script: 14_inet.sql # loading sql script: 15_history.sql # loading sql script: 16_tracelog.sql # loading sql script: 17_compress.sql # loading sql script: 18_dictionary.sql # loading sql script: 19_cluster.sql # loading sql script: 20_vacuum.sql # loading sql script: 21_dependency_functions.sql # loading sql script: 22_clients.sql # loading sql script: 23_skyserver.sql # loading sql script: 24_zorder.sql # loading sql script: 25_debug.sql # loading sql script: 22_clients.sql # loading sql script: 23_skyserver.sql # loading sql script: 39_analytics.sql # loading sql script: 24_zorder.sql # loading sql script: 25_debug.sql # loading sql script: 39_analytics.sql # loading sql script: 75_storagemodel.sql # loading sql script: 75_storagemodel.sql # loading sql script: 80_udf.sql # loading sql script: 80_udf.sql # loading sql script: 99_system.sql # loading sql script: 99_system.sql #GDKmalloc(589484032) fails, try to free up space [memory in use=71647756904,vir tual memory in use=72723595880] #GDKmalloc(589484032) result [mem=71615637640,vm=72198121608]

2 2

Fwd: Using the inet and url type functions
by Christopher Nelson 23 May '13

23 May '13

Hi, I noticed in the manual that there are many useful functions for the inet and url types. For example, I would like to convert an inet column to text for use in a Python script (since the python-monetdb module does not support the inet type.) I tried: sql>SELECT wlid, text(src), ts FROM web_log WHERE src IS NOT NULL ORDER BY src, ts LIMIT 5;syntax error, unexpected '(', expecting SCOLON in: "select wlid, text(" sql>SELECT wlid, inet:text(src), ts FROM web_log WHERE src IS NOT NULL ORDER BY src, ts LIMIT 5; unexpected symbol (:) sql>SELECT wlid, inet.text(src), ts FROM web_log WHERE src IS NOT NULL ORDER BY src, ts LIMIT 5; syntax error, unexpected '(', expecting SCOLON in: "select wlid, inet.text(" I have read through the manual sections on types and functions several times now, and I cannot see how to use the specified functions on these types. Any help would be greatly appreciated. Thanks! -={C}=-

2 8

Using something like "trace" within the MAL interpreter
by Masood Mortazavi 20 May '13

20 May '13

Hi, The "trace" commands works well when mclient is in the SQL interpreter mode. How about when mclient is in MAl interpreter mode? Is there a way to use the "trace" command in this mode? Best regards, Masood

3 6

support for non-median quantile functions?
by Anthony Damico 15 May '13

15 May '13

Hi, would it be possible to expand the median() function to be expanded work on any quantile? Martin's original request did say quantile, not just median :D http://osdir.com/ml/db.monetdb.bugs/2008-06/msg00206.html Given the MonetDB.R connection, I think a lot more statisticians are going to be using MonetDB -- and they'll find a lot of value in being able to calculate statistics like the 25th and 75th, percentiles quickly. I can submit a bugs.monetdb.org report if that's appropriate? Thanks for considering this!!

2 2

Re: Help for MonetDB source code (SQL Module)
by Stefan Manegold 14 May '13

14 May '13

Dear Khaled, for questions about MonetDB, I kindly ask you to use the respective mailing list(s); I Cc my answer to the users list. See also http://www.monetdb.org/Developers/Mailinglists On Mon, May 13, 2013 at 06:41:45PM +0200, Khaled Dehdouh wrote: > Dear sir, > > I am a student at university of LYON. Concerning MonetDB, I need your help, > please on some points. > > > - I want to do some optimizations in "GROUP BY" function of SQL module. > For this end, can you show me where is exactly the function code > responsible to perform the "Group By" operation on MonetDB source > codein order to modif > y it. I'm wondering how you know that "optimizations" are necessary/possible without knowing the actual code, but I guess you know. You find the respective code in gdk/gdk_group.c > - I would like to know also how to recompile the code after modification > of the "GROUP BY" function. See http://www.monetdb.org/Developers/SourceCompile http://dev.monetdb.org/hg/MonetDB/file/tip/HowToStart.rst http://dev.monetdb.org/hg/MonetDB/file/1272d67f24be/buildtools/doc/windowsb… > - How creating a new SQL command that performs function with parameters and > combines two existing functions in MONETDB. See http://www.monetdb.org/Documentation/SQLreference/Modules or your favorite SQL:2003 documentation. MonetDB is SQL standard compliant. Best, Stefan > Thanks in advance. > > Best regards, > Khaled DEHDOUH. -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

GROUP BY
by Bouslah Ahmed 14 May '13

14 May '13

Hello, I am a student at a university in France. I work on MonetDB, Windows platform. I have some questions: I want to optimize the "Group by" function of the SQL module in order to create a new function combining the "Group by" function and another function. For this I want to know: 1 - How to locate the class containing the code of the "Group by" function 2 - How to edit the code in question to change it 3 - Which editor should be used to compile the project. 4 - I tried to use "Visual Studio 2012" to open the project "MonetDB" but I have not managed to do. Regards Ahmed Bouslah.

3 2

Question on installing extensions
by Kaniska_Mandal＠DELL.com 12 May '13

12 May '13

Hi All, How to install extensions for Streams, XML and Json support ? Many thanks, Kaniska

2 1

TINYINT display problems in 3rd party query clients
by Diethard Steiner 10 May '13

10 May '13

Hi, I am wondering why TINYINT values are displayed with different values in 3rd party SQL clients like Squirrel (please see attached screenshot): So 1 is shown as 49 in example. I used following JDBC driver version: monetdb-jdbc-2.8.jar Any ideas? Thanks, Diethard

3 5

Feb2013-SP2 mserver loading time much longer than Feb2013 on Windows x86_64
by Anthony Damico 03 May '13

03 May '13

Hi, after upgrading to MonetDB Feb2013-SP2, I've noticed a huge increase in the loading time for the server to get the ">" shown in the last line below (indicating it's loaded and ready for mclient connections). It now takes between three and ten minutes. Has anything changed in the Windows version of the server between Feb2013 and Feb2013-SP2 that might cause this? Once loaded, the queries are still lightning-fast, but I'm debating whether I ought to re-create all of my databases from scratch using Feb2013-SP2, if there's any reason doing that might help? Any clues? :) Thanks!!! # MonetDB 5 server v11.15.7 "Feb2013-SP2" # Serving database 'medicare_sample', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 20.000 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:49800/ # MonetDB/JAQL module loaded >

1 0