- users-list - monetdb.org

UDF function with >3 parameters cannot benefit from using PYTHON_MAP
by Hanfeng Chen 15 Nov '17

15 Nov '17

Hi, I created a table with 9 columns and passed 6 columns from the table to a UDF. This UDF is fully vectorizable. However, we found only single thread was used during the execution. Then, we came to think about what the problem behind and we created the two simple functions python_min_map_3 and python_min_map_4 which takes 3 and 4 parameters separately. The MonetDB Linux version we used was July 2017. If PYTHON_MAP is enabled, it would return the results of each segment processed by parallel code. The number of results depends on how many threads are used. Otherwise, a single value should be returned (e.g. because of numpy.min). The result is - python_min_map_3 returns a couple of numbers - python_min_map_4 returns a single number Even when we increased the number of the arguments, a single number was always returned. Is it a restriction in using PYTHON_MAP when the number of arguments should be no more than 3? Here is our example code below. CREATE FUNCTION python_min_map_3(x0 FLOAT, x1 FLOAT, x2 FLOAT) RETURNS FLOAT LANGUAGE PYTHON_MAP { return numpy.min(x0) }; select python_min_map_3(x0, x1, x2) from table_0; CREATE FUNCTION python_min_map_4(x0 FLOAT, x1 FLOAT, x2 FLOAT, x3 FLOAT) RETURNS FLOAT LANGUAGE PYTHON_MAP { return numpy.min(x0) }; select python_min_map_4(x0, x1, x2,x3) from table_0; NumPy in MonetDB reference: https://www.monetdb.org/blog/embedded-pythonnumpy-monetdb Best regards, Hanfeng Chen

1 0

read BLOB column on tail heap and loop over rows
by Anton Kravchenko 13 Nov '17

13 Nov '17

I wonder how one would read BLOB column and loop over its rows? BAT *b = table_funcs.full_column(m->session->tr, blob_col_name); //blob *readblobrows = (blob*) b->T.heap.base; //unsigned char *readblobrows = (unsigned char*) b->T.heap.base; p.s. For INT column it would be go like this: int * readintrows = (int *) b->T.heap.base; for (irow=0; irow < nrow; irow++) { intvalue = *readintrows++; } Thanks, Anton

1 1

Querying JSON arrays
by Brian Ploetz 09 Nov '17

09 Nov '17

Hi, Suppose I have a table with a "tags" column that is of the JSON datatype ( https://www.monetdb.org/Documentation/Manuals/SQLreference/Types/JSON) The column will contain a JSON array of Strings. For example: ["foo","bar","baz"] Is it possible to query for rows that contain a specific value in the tags array? For example, if I wanted to find all of the rows where tags contains "foo", my query would look something like: select count(*) from my_table where tags ???? 'foo'; But it's not clear to me what ???? should be (or if it's even possible) based on the JSON documentation above. Thanks in advance. BP

2 3

large virtual memory spike on BLOB column select
by Anton Kravchenko 09 Nov '17

09 Nov '17

I am using MonetDB v11.27.9 on cent7. 1) 'select v_blob from blob_table' produces a virtual memory spike ~800Gb [using a 'limit' reproduces virtual memory spike too] >From 'select * from storage()' it looks a blob column size is ~25Gb. 2) And 'create table new_blob_table as (select v_blob from blob_table) with data;' produces only ~50Gb of virtual memory. 3) Also for a table with non-blob column both 'select * from nonblob_table' and 'create new_nonblobtable as (select * from nonblob_table) with data;' produce the ~same virtual memory consumption. p.s. Here are steps to reproduce this issue # to generate date by using Python hex_blob='47'*2000 hex_blob_chunk=(hex_blob+'\n')*1000 f=open('blob_data.txt','w') for irow in range(10000): f.write(hex_blob_chunk) f.close() # to load data into Monet blob_table create table blob_table(v_blob blob); COPY 10000000 RECORDS INTO blob_table FROM '/home/blob_data.txt' using delimiters ',','\n' NULL AS '' LOCKED; # to produce virtual memory spike of ~800GB select * from blob_table limit 50000; # to produce virtual memory of ~50Gb create table new_blob_table as (select * from blob_table) with data; Thanks, Anton

2 1

Problems creating json files with more than 10^3 table entries
by Giacomo Bergami 08 Nov '17

08 Nov '17

I am using MonetDB-11.27.9 on ubuntu 17.04. The client was compiled from the source code. I am trying to run the following query on your database select distinct T.src as author, T1.src as coauthor, json.tojsonarray(T1.dst) as papers from edges1000 as T, edges1000 as T1 where T.dst = T1.dst AND T.src <> T1.src group by T.src, T1.src ; When I run this query over a table with at most 10^4 entries, the mclient client quits. If I try to execute the same qury via JDBC, I receive an error that tells me "maybe mserver closed unexpectedly". On the other hand, the server is still alive and responsive if I afterwards run the query either oover mclient or JDBC over a table having less data (edges10). Is that a problem of memory allocation ? Is there a way to solve this issue? I'm using the default configuration, and compiled the source code using "autoreconf -if" first. Thanks in advance for your support, Bergami G

1 0

latest version question
by Lynn Carol Johnson 02 Nov '17

02 Nov '17

Hello monetdb team – We upgraded this afternoon to the latest version of monetdb (Jul2017-SP2). When testing access for my lab, I noticed the general login could no longer perform the command: \d <table_name> The errors were related to missing group access to sys.objects, sys.keys and sys.idxs. When, as monetdb admin, I granted “select” on these tables, the “\d <table_name>” command then worked. Is this something new, or does it indicate a problem with the db ? It was not necessary to grant general users permission to these system tables previously. Thanks - Lynn

2 4

Problem in installation
by Amit Ambekar 27 Oct '17

27 Oct '17

Hi, Sorry sending this email again because of limit in message body is 40 kb. I had installed monetdb and now wanted to start mclient. So first i started Mserver5.bat and then when i click on mclient.bat that time "api-ms-win-crt-runtime-l1-1-0.dll is missing issue" this error is coming in alert box. Is monetdb is only usefull for windows server 2003,vista and Windows 7 or it can be used with Windows 2012 server also? Installed Visual C++ Redistributable for Visual Studio 2015 , then too error coming . Please check attached screenshot. this error coming when tried to start mclient

3 4

Setting Up Tools
by Sandy 26 Oct '17

26 Oct '17

Just getting back into trying MonetDB after many years. Got the environment set up pretty well with the test DB `VOC` loaded up and able to do all expected in the command line tools. So I figured I would give a couple of the mentioned GUI tools a shot and see how they work against the server I just set up on a virtual box. I picked up the JDBC drivers and installed on my Window's 10 install of SQuirrel SQL and DBeaver. I got both of the drivers works sort of... On DBeaver I can execute and see queries in the tool but can't do much more than that. Overall looks like an issue with accessing the system table `sys.environment`, not sure if anything related to set up or rights. Using the `voc` user as created with the example db in the getting started. Any help or thought on where to start? Thanks! Sandy -------- Errors * DBeaver 4.0.8 On inspection of some of the logs I see things like - `Error loading object metadata from database` `SQL Error [22000]: SELECT: access denied for voc to table 'sys.environment' java.sql.SQLException: SELECT: access denied for voc to table 'sys.environment'` * SQuirreL SQL Client 3.7.1 java.sql.SQLException: SELECT: access denied for voc to table 'sys.environment' -------- Driver and DB versions I'm using this JAR for the drivers - C:\Users\Squidward\Downloads\monetdb-jdbc-2.25.jar, 2.25 (Liberica 20170330 based on MCL v1.15) MonetDB version - MonetDB v11.25.23 (Dec2016-SP5), 'mapi:monetdb://sandy-VirtualBox-18:50000/voc'

3 6

Announcement: New Jul2017-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 26 Oct '17

26 Oct '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017-SP2 bugfix release (11.27.9) MonetDB Common * A serious bug, possibly resulting in database corruption, having to do with appending data to a string BAT was fixed. Bug Fixes * 4017: server crashes when executing particular loopback query in embedded python * 6207: identifier ambiguous when grouping and selecting the same column twice * 6239: Incorrect profiling * 6261: New handling of delta tables hurts badly reusage of bats * 6287: should the CORR function return some numeric type that allows fractions? * 6321: Two-column aggregation on join result extremely slow. * 6335: Sqlitelogictest crash in complex SQL query * 6343: MERGE TABLE issue: unable to find * 6348: Interference of procedure/table name * 6350: Carriage return and form feed in TEXT fields are returned as 'r' and 'f' in jdbcclient and ResultSets * 6352: Scope resolution problem (sqlsmith) * 6353: implicit NULL value not propagated in distributed/remote query * 6374: Wrong answer from merge table after content changes * 6379: Table UDF: SEGV raised when invoking a non existing function * 6380: unable to create new databases from clean installation * 6381: Parser misses error messages in conditional * 6382: Can't set JSON fields via PreparedStatement * 6384: crash when setting a wrong listenaddr * 6385: AGGREGATE UDFs with more than 2 parameters incorrectly processed * 6386: Unexpected error from server for query with long floats * 6387: Performance degradation on multi column sort * 6388: JDBC Connection via user voc produces errors when fetching certain meta data information * 6392: SELECT EXISTS (empty table) returns 'true' * 6395: BAT leak of scalar result sets * 6397: Isolation of generating functions not correct * 6398: Null Matches in outer join are not supported * 6399: UDF crashes when subquery and scalar values are passed as pameters * 6400: getCharacterStream() currently not supported * 6404: COPY INTO crashes if table has primary key or foreign key constraint * 6405: Compilation error if DISABLE_PARENT_HASH not defined * 6409: sqllogictest crash on aggregation query with NOT IN clause in HAVING clause * 6410: Sqlitelogictest crash on aggregation query with IN clause * 6411: Sqlitelogictest crash in aggregation query * 6412: Overflow in sys.epoch * 6413: Wrong answer for TPC-H Q17 * 6414: Using RPAD returns: could not allocate space * 6416: Sqlitelogictest crash in aggregation query with a NOT IN clause * 6417: Segfault encountered (sqlsmith) * 6418: Segfault in renaming (sqlsmith) * 6419: segfault in rel_optimizer (sqlsmith) * 6420: Assertion error in mergetable task (sqlsmith) * 6422: Another assertion error in rel_or (sqlsmith) * 6423: Dereference null pointer (sqlsmith) * 6424: Assertion error in rel_rename_expr (sqlsmith) * 6425: Assertion error in exp_bin (sqlsmith) * 6426: Assertion error in rel_find_exp_ (sqlsmith) * 6427: Assertion error in eq_typeswitchloop (sqlsmith) * 6428: Sqlitelogictest crash in aggregation query * 6430: Assertion raised in another eq_typeswitch error (sqlsmith) * 6431: Sqlitelogictest crash in aggregation query with a long having clause * 6432: Assertion error in exp_bin (sqlsmith) * 6433: Sqlitelogictest crash in complex SELECT query with IN operator * 6435: Sqlitelogictest crash in simple select query * 6437: System schemas "profiler" and "json" shouldn't be allowed to be dropped. * 6438: Implement functionality to enforce the restrict option in: DROP SCHEMA xyz RESTRICT; * 6440: Faulty plan generated. Query returns more rows than expected or existing in the view sys.tables.

1 0

Proper syntax for a windowing function
by Omri Schwarz 25 Oct '17

25 Oct '17

Hi, all. I'm trying to figure out the proper syntax for queries such as a moving average over a window in MonetDB. THis works: sql>select avg(energy) from TABLENAME; +--------------------------+ | L3 | +==========================+ | 658.5049567388747 | +--------------------------+ This fails: select avg(energy) OVER () from TABLENAME; SELECT: function 'avg' not found (Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2017-SP1) Database: MonetDB v11.27.5 (Jul2017-SP1), 'mapi:monetdb://SERVER2:50000/scada' ) How do I write such a query? Many thanks.

4 5