July 2017 - users-list

how to know imprints take effect?
by integrity 19 Jul '17

19 Jul '17

Hi all, I am back, and I am using MonetDB Dec2016-SP5 with "enable-optimize" to do graph computing. I remember that imprints are suitable for all numeric types, however, after querying on the bigint column for many times, I found there is no imprints. What I want to achieve is to "make" MonetDB respond in milliseconds, although I know that in MonetDB it is not necessary to create index like imprints. So, on a table of 15,000,000 rows, can I expect to get response in milliseconds? and … [View More]

2 1

Announcement: New Jul2017 Feature release of MonetDB suite
by Sjoerd Mullender 18 Jul '17

18 Jul '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017 feature release MonetDB5 Server * The "sub" prefix of many … [View More]functions, both at the MAL and the C level, has been removed. * Changed the interfaces of the AUTH* functions: pass values, not pointers to values. * Removed calc.setoid(). * group.subgroup is now called group.group if it is not refining a group. Both group.group and group.subgroup now also have variants with a candidate list. * The allocation schemes for MAL blocks and Variables has been turned into block-based. This reduces the number of malloc()/free() calls. . * Added a new server-side protocol implementation. The new protocol is backwards compatible with the old protocol. Clients can choose whether they want to use the old or the new protocol during the initial handshake with the server. The new protocol is a binary column-based protocol that is significantly faster than the old protocol when transferring large result sets. In addition, the new protocol supports compression using Snappy or LZ4. * Moved the sphinx extension module to its own repository. See https://dev.monetdb.org/hg/MonetDB-sphinx/. * Removed GSL module: it's now a separate (extension) package. See https://dev.monetdb.org/hg/MonetDB-gsl/. * The PCRE library is now optional for systems that support POSIX regular expressions. * Added 5 new sys schema tables: function_languages, function_types, key_types, index_types and privilege_codes. They are pre-loaded with static content and contain descriptive names for the various integer type and code values. See also sql/scripts/51_sys_schema_extension.sql Merovingian * Added handling of a dbextra property per database at the daemon level. The user can set the dbextra property for a database using the command: $ monetdb set dbextra=<path> <database> and the daemon will make sure to start the new server using the correct --dbextra parameter. Client Package * The mclient and msqldump programs lost compatibility with old mserver5 versions (pre 2014) which didn't have a "system" column in the sys.schemas table. * The mclient and msqldump programs lost compatibility with ancient mserver5 versions (pre 2011) which didn't have the sys.systemfunctions table. * Removed the "array" and "quick" functions from the mapi library. To be precise, the removed functions are: mapi_execute_array, mapi_fetch_field_array, mapi_prepare_array, mapi_query_array, mapi_quick_query, mapi_quick_query_array, and mapi_quick_response. * Added a more elaborate \help command for SQL expressions. MonetDB Common * Improved error checking in the logger code (dealing with the write-ahead log); changed return types a several functions from int to gdk_return (i.e., they now return GDK_SUCCEED or GDK_FAIL). The logger no longer calls GDKfatal on error. Instead the caller is responsible for dealing with errors. * BATsort may now create an order index as a by product. * Quantile calculations now use the order index if available (and use BATsort otherwise, producing an order index). * Quantiles calculate a position in the sorted column. If this position is not an integer, we now choose the nearest position, favoring the lower if the distance to the two adjacent positions is equal (round down to nearest integer). * Removed function BATprintf. Use BATprint or BATprintcolumns instead. * Removed BATsave from the list of exported functions. * Replaced BBPincref/BBPdecref with BBPfix/BBPunfix for physical reference count and BBPretain/BBPrelease for logical reference count maintenance. * Removed automatic conversion of 32-bit OIDs to 64 bits on 64-bit architectures. * Removed functions OIDbase() and OIDnew(). * Removed talign field from BAT descriptor. * BATappend now takes an optional (NULL if not used) candidate list for the to-be-appended BAT. * New function BATkeyed(BAT *b) that determines (possibly using a hash table) whether all values in b are distinct. SQL * Made the operator precedence of % equal to those of * and /. All three are evaluated from left to right. * Removed table sys.connections. It was a remnant of an experimental change that had already been removed in 2012. * Protect against runaway profiler events If you hit a barrier block during profiling, the JSON event log may quickly become unwieldy. Event production is protected using a high water mark, which ensures that never within the single execution of MAL block the instruction causes excessive event records. Bug Fixes * 3465: Request: add support for CREATE VIEW with ORDER BY clause * 3545: monetdb commands don't work with -h -P -p options (locally and remotely) * 3996: select * from sys.connections always returns 0 rows. Expected to see at least one row for the active connection. * 6187: Nested WITH queries not supported * 6225: Order of evaluation of the modulo operator * 6289: Crashes and hangs with remote tables * 6292: Runaway SQL optimizer in too many nested operators * 6310: Name resolution error (sqlsmith) * 6312: Object not found in LIMIT clause (sqlsmith) * 6313: Null type resolution in disjunction fails (sqlsmith) * 6319: Server crash on LATERAL (sqlsmith) * 6322: Crash on disjunction with LIMIT (sqlsmith) * 6323: Deadlock calling sys.bbp() * 6324: Sqlitelogictest crash in a IN query (8th) * 6327: The daemon does not respect the actual name of the mserver5 executable * 6330: Sqlitelogictest crash on a complex SELECT query * 6331: sys.statistics column "nils" always contains 0. Expected a positive value for columns that have one or more nils/NULLs * 6332: Sqlitelogictest crash related to an undefined MAL function [View Less]

3 3

possible to change the decimal character from "." to ", " for the COPY INTO command?
by Anthony Damico 18 Jul '17

18 Jul '17

hi, i'm curious if this is possible to implement? not seeing it in the docs.. for example, in R you can use read.csv() which uses , as the delimiter and . as the decimal or you can use read.csv2 which uses ; as the delimiter and , as the decimal thanks all

1 0

Add entry in a table only if that entry does not already exist
by Stefano Piani 14 Jul '17

14 Jul '17

Dear monetdb users, I started to use MonetDB a few weeks ago and, until now, I have been able to use it proficiently. Unfortunately, I have faced a small issue that I have not been able to figure out how to fix. I collect data generated by different users. A unique id is associated to each user. I have a specific table (called "user") where I save the name and some other information about the user. In this table, the ID is a primary key. When I collect the data, I want to check if the … [View More]specific user who generated that data is already saved in the user table. If this is not the case, I want to add a row to the user table for this user. Unfortunately, I have not been able to accomplish this operation in one SQL query and, therefore, I have been forced to query for the user id and, if the answer is negative, add the user using another query. The following is the python script that perform the operations. def save_user_in_database(user, cursor): db_line = 'SELECT 1 FROM users WHERE user_id = {}'.format(user.id) LOGGER.info('Executing on DB: %s', db_line) cursor.execute(db_line) answer = cursor.fetchall() LOGGER.info('Get this answer %s', answer) if len(answer) == 0: LOGGER.info('Inserting new user') db_line = sql_cmd_from_template( USER_TABLE_INSERT_TEMPLATE, user.id, user.name, user.description, user.location ) LOGGER.info('Executing on DB: %s', db_line) cursor.execute(db_line) This function is executed inside a single threaded application. Moreover, there are no "DELETE" or similar operations inside this script (it only adds rows to the database). No user (human or software) but this script connects to my MonetDB instance. Unfortunately, after I while that the script is running (let's say one or two days), something unexpected happens. For example, this is the log generated for user 42847029 (there are a lot of other lines in between that I removed). As you can see, after a while, my check fails and I don't understand why. 2017/06/13 09:30:01 - INFO - save_user_in_database: Executing on DB: SELECT 1 FROM users WHERE user_id = 42847029 2017/06/13 09:30:01 - INFO - save_user_in_database: Get this answer [(1,)] 2017/06/13 10:00:04 - INFO - save_user_in_database: Executing on DB: SELECT 1 FROM users WHERE user_id = 42847029 2017/06/13 10:00:04 - INFO - save_user_in_database: Get this answer [(1,)] 2017/06/13 10:02:33 - INFO - save_user_in_database: Executing on DB: SELECT 1 FROM users WHERE user_id = 42847029 2017/06/13 10:02:33 - INFO - save_user_in_database: Get this answer [(1,)] 2017/06/13 10:30:36 - INFO - save_user_in_database: Executing on DB: SELECT 1 FROM users WHERE user_id = 42847029 2017/06/13 10:30:36 - INFO - save_user_in_database: Get this answer [] 2017/06/13 10:30:36 - INFO - save_user_in_database: Inserting new user 2017/06/13 10:30:36 - INFO - save_user_in_database: Executing on DB: INSERT INTO users(user_id, user_name, user_description, user_location) VALUES (42847029, 'XXXX', 'XXXX', 'XXXX') 2017/06/13 10:30:36 - WARNING - on_data: Error elaborating data Traceback (most recent call last): File "data_collector.py", line 267, in on_data return self.elaborate_data(data) File "data_collector.py", line 354, in elaborate_data save_user_in_database(user, cursor) File "data_collector.py", line 234, in save_user_in_database cursor.execute(db_line) File "/opt/Python-3.6.0/lib/python3.6/site-packages/pymonetdb-1.0.4-py3.6.egg/pymonetdb/sql/cursors.py", line 164, in execute block = self.connection.execute(query) File "/opt/Python-3.6.0/lib/python3.6/site-packages/pymonetdb-1.0.4-py3.6.egg/pymonetdb/sql/connections.py", line 133, in execute return self.command('s' + query + ';') File "/opt/Python-3.6.0/lib/python3.6/site-packages/pymonetdb-1.0.4-py3.6.egg/pymonetdb/sql/connections.py", line 138, in command return self.mapi.cmd(command) File "/opt/Python-3.6.0/lib/python3.6/site-packages/pymonetdb-1.0.4-py3.6.egg/pymonetdb/mapi.py", line 235, in cmd raise exception(string) pymonetdb.exceptions.IntegrityError: INSERT INTO: PRIMARY KEY constraint 'users.users_user_id_pkey' violated Have you any advice? Is there a way to perform the same check in just one SQL line? Is this way atomic (if a would ever move to a multithreading script)? Am I doing something wrong with my current code? Please, take into account that I do not have a deep experience with other SQL tools so it is perfectly possible that I miss something that for you is extremely obvious. Thank you in advance! Best regards, Stefano Piani [View Less]

2 3

Drop a local temporary table within a function
by Luis Larios 14 Jul '17

14 Jul '17

Hi, all, Consider the following scenario. I have a bunch of user defined functions, f_i, that create, populate, and use a local temporary table with the same name "res": create function f_i() returns table( ... ) begin # Result stats table. create local temporary table res( param1 int, param2 int ); /* Populate res */ /* Use res in some queries */ return table( /* Query involving res */ ); end; My first question is:… [View More] Does "create local temporary table x" create a table that is disposed/dropped as soon as the function where it was declared finishes execution? I assumed this was the case, but I'm now skeptical as we are running into all sorts of concurrency issues that point to the "res" table not being cleared out properly from memory. This takes me to the second question: How do I drop a temporary table if such table exists? The reason for this is to prevent "table create" exceptions (which we are currently running into). I'd like to achieve something along these lines: create function f_i() returns table( ... ) begin # Drop local temporary table if such table exists. drop local temporary table res; <-------- How do I do this in MonetDB? # Result stats table. create local temporary table res( param1 int, param2 int ); /* Populate res */ /* Use res in some queries */ return table( /* Query involving res */ ); end; Where, as you can see, I want to explicitly drop the local temporary table "res" if such exists. If I leave the "local temporary" markers in the drop statement, MonetDB complains that it found unexpected "LOCAL" or "TEMPORARY". I wouldn't like to remove these markers because a statement like: drop table res; will drop any table in the default function schema with that name. Any help and hints are well appreciated! Thanks for the help in advance ~ Luis Angel The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender. [View Less]

3 2

Jul2015-SP4 vs. Dec2016-SP5
by Anderson, David B 13 Jul '17

13 Jul '17

All, I know I've skipped multiple releases so this is a difficult question to answer, but should I expect the Dec2016-SP5 version of MonetDB to be faster than/equal to the Jul2015-SP4 version? I have a parallel set of servers (on the same box, 24 cores 192GB ram, listening to different ports) operating on identical data sets, and the Dec2016 version is slower than the Jul2015 version (~15-20%). My queries are basic joins between two tables with grouping. I have top running on the same box, and … [View More]

2 2

script-friendly prepare statement
by Roberto Cornacchia 13 Jul '17

13 Jul '17

Hi there, I'm trying to script the execution of (generated) prepared queries. It does not look possible, because the identifier to use for the EXEC statement is returned dynamically by the PREPARE statement: sql>prepare select 1; execute prepared statement using: EXEC 73(...) This can obviously not be scripted. If I tried to parse the output of the prepare statement to retrieve the identifier, this would split PREPARE and EXEC in two different mclient sessions, losing the prepared … [View More]

3 4

adding an "insert into" completely changes output from explain
by Anderson, David B 13 Jul '17

13 Jul '17

All, Is it expected behavior if adding "insert into xxxx" at the beginning of a sql query completely alters the output of explain? Without the "insert into", explain returns 1490 tuples, but when I add it, explain only returns 88. I was expecting that the number would go up, i.e. that the original plan would be wrapped in some additional commands which would implement the "insert into" part. Here is my sql: explain insert into aggr select 'CHANNEL', s.product, s.coupon, s.productionyear, s.… [View More]

2 1

Extremely slow order by queries
by Manish gupta 13 Jul '17

13 Jul '17

Hi, I have a scenario where data size is ~3M rows. My query is select C1, C2, C3 from T1 group by C1,C2,C3 This returns result in 30 Sec ( number of groups around 1 M ) While, if I modify the query to select C1, C2, C3 from T1 group by C1,C2,C3 Order by C1, C2 on the same dataset, it takes ~ 1 hour to get result. The platforms I tried is both widows 64 bit and Linux 64 bit, with latest monet installed. Both the machines have 4 cores. And the behavior is same on both the machines. I am … [View More]

2 1

Mserver returns with "memory exhausted in" error if query size exceeds 1.2MB
by Sharma, Sreejith 08 Jul '17

08 Jul '17

Hi, I have encountered scenario where SQL generated exceeds 3mb and it's failing in mserver with an error "memory exhausted in select .......". On further checks I have noticed that the SQL will go through only if SQL size is less than 1.2MB. Has anyone encountered this before? Or is anyone aware of any SQL size limitation in MonetDB? Please help. Regards, Sreejith

1 0