September 2013 - developers-list

TODO statement
by Gijs Molenaar 01 Oct '13

01 Oct '13

Hi MonetDB people, I encountered this TODO message while running a query: sql>SELECT objects.name more> FROM sys.keys, sys.objects, sys.tables, sys.schemas more> WHERE sys.keys.id = objects.id more> AND keys.table_id = tables.id more> AND tables.schema_id = schemas.id more> AND keys.type = 0 more> AND tables.id = 11323; TODO: column names of level >= 3 -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

3 3

timestamp microsecond precision
by Gijs Molenaar 30 Sep '13

30 Sep '13

Hi MonetDB people, Do the TIME(STAMP) types support microsecond precision? They accept a 'postint' argument, but no matter what I set it to it looks like the timestamp is rounded to milliseconds. greetings, -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

2 1

Announcement: New Feb2013-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 27 Sep '13

27 Sep '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013-SP4 bugfix release Merovingian * monetdb now no longer compresses output when not connected to a terminal, bug #3292 Bug Fixes * 2781: SQL localtime() returns GMT time * 2962: client timezone information needs to be handled also for modifications to timestamp values * 3232: The function weekofyear returns results not according to the iso 8601 standard * 3255: M5 tests inlineFunction & ifthencst fail since changeset 207ed7d7e03a * 3292: monetdb status should print full line * 3307: distinct multicolumn performance and implementation suggestion for count distinct * 3310: Cast value to integer gives wrong result * 3311: strLength assertion on join and concat * 3313: Assertion in rel2bin_project * 3315: Incorrect behavior with subqueries in DELETE * 3317: correlated subselect does not work * 3318: UPDATE does not work with DECLEAREd TABLE in FUNCTION * 3319: UNIQUE column in DECLAREd TABLE in FUNCTION results in segfault * 3329: Function AVG fails during runtime in a procedure * 3330: Corrupt mitosis plan with group-by query * 3332: MEDIAN: no such operator 'median(decimal)' * 3336: DB files not removed if all rows are deleted, even after restart * 3337: Monet ODBC gives wrong schema information * 3339: Segvault in complex function call * 3340: crash on seeming incorrect SQL query * 3341: segfault caused by select rank() on a newly-loaded table * 3342: incorrect automatic cast from decimal to tinyint * 3345: copy binary into fails with the wrong error message * 3346: Deadlock in mal_dataflow's q_dequeue() (SciQL-2 branch) * 3348: SQL: LIKE incorrect with _ * 3349: When selecting everything from a joined result, column contents are incorrect in one of my use cases. * 3351: gzip'ed files cannot be bulkloaded into DB * 3352: Median function with decimal type doesn't work * 3353: Readonly tables call for UNSET * 3354: Introduce query time-out * 3358: Remote client connection stays in CLOSE_WAIT state * 3360: uninitialized memory causing bad output in query plans * 3362: ALTER TABLE operations silently remove previously set READ ONLY property * 3363: a join causes SIGSEGV after creation of Unique Key * 3367: Fully qualified order by column gives "relational query without result" * 3369: \0 Byte appears in MAPI

1 0

Whats coming in upcoming build monetdb release
by Ashish Kumar Singh 21 Sep '13

21 Sep '13

Hi , During my periodic search for "Is a new monetdb build is going to be released soon" , I found one is placed under testing recently. Is there a place where it can be seen and reviewed what kind of features/fixes are expected in upcoming release? That will be a great help for some of the folks like me to plan ahead! Regards, Ashish

1 0

Disk space to store a table?
by baraa Mohamad 10 Sep '13

10 Sep '13

Dear MonetDB Developers, I'm doing some research about colmn oriented databases and I began with MonetDB. In fact I would like to know how can I get the real disk space used to store a given tabel?? After some research on the web I found that I can use the storage table as follow: select "schema", "table", sum("columnsize") as size from storage() group by "schema","table" having "schema" <> 'tmp'; But it seems to me that this is something like a "logical" storage space without any compression (all columns with type "int" have the same size), is there a way to know the actual disk space used by MonetDB to store a table?? Thank you in advance for your help, RegardsBaraa

1 0

Re: MonetDB: int128 - backing-out changeset 77c9c3da562c:
by Martin Kersten 08 Sep '13

08 Sep '13

On 9/8/13 8:45 PM, Stefan Manegold wrote: > Changeset: a331af9e20ea for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a331af9e20ea > Modified Files: > monetdb5/modules/mal/batcalc.mal > monetdb5/modules/mal/batcalc.mal.sh > Branch: int128 > Log Message: > > backing-out changeset 77c9c3da562c: > > Unfortunately, MonetDB / mserver5 / MAL parser / MAL interpreter > is/are unable to handle the new monetdb5/modules/mal/batcalc.mal > correctly. > > When starting the server, it complains about > !TypeException:jaqlfunc.count[7]:'batcalc.lng' undefined in: r:any := batcalc.lng(x:bat[:oid,:wrd]) > although the respective batcalc.lng(x:bat[:oid,:wrd]) > is in monetdb5/modules/mal/batcalc.mal . > Some tests indicate that the systems (MAL parser ?) > has some problems handling MAL files >= ~1 MB > correctly, but I did not manage to locate the > actual nature or origin of the problem. > > Any help to get this working is highly appreciated. Most likely it is in mal_import where the complete mal file is loaded first

1 0

Auto-vectorization
by ajonwa＠gmx.com 07 Sep '13

07 Sep '13

Hello All, Has anyone managed to auto-vectorize (automatic SIMDfication) any of the loops in the kernel? If yes, I would like to know the complier and the complier flags used. Thanks, Geoffrey

1 0

Re: MonetDB: default - add support for unix sockets
by Stefan Manegold 06 Sep '13

06 Sep '13

Hi Gijs, in case you did not notice this, yet: This checkin breaks several tests that do test or use the Python interface, resulting in errors like " Traceback (most recent call last): File "delete_all.SQL.py", line 9, in <module> dbh = monetdb.sql.Connection(port=port,database=db,autocommit=True) File "PREFIX/lib/python2.7/site-packages/monetdb/sql/connections.py", line 49, in __init__ unix_socket=unix_socket) File "PREFIX/lib/python2.7/site-packages/monetdb/mapi.py", line 96, in connect self.socket.connect(unix_socket) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 2] No such file or directory " or " Traceback (most recent call last): File "SOURCE/clients/examples/python/sqlsample.py", line 23, in <module> dbh = monetdb.sql.Connection(port=int(sys.argv[1]),database=sys.argv[2],autocommit=True) File "PREFIX/lib/python3.3/site-packages/monetdb/sql/connections.py", line 49, in __init__ unix_socket=unix_socket) File "PREFIX/lib/python3.3/site-packages/monetdb/mapi.py", line 97, in connect self.socket.connect(unix_socket) FileNotFoundError: [Errno 2] No such file or directory " See also, e.g., `Mtest.py sql/benchmarks/tpch/fileleak sql/test/concurrent sql/test/mapi` or our nightly TestWeb @ http://monetdb.cwi.nl/testweb/web/status.php?branch=default as of tomorrow morning. Best, Stefan On Thu, Sep 05, 2013 at 02:19:01PM +0200, Gijs Molenaar wrote: > Changeset: 0eaa07b061be for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0eaa07b061be > Modified Files: > clients/python2/monetdb/control.py > clients/python2/monetdb/mapi.py > clients/python2/monetdb/sql/connections.py > clients/python2/test/runtests.py > clients/python2/test/test_control.py > clients/python3/monetdb/control.py > clients/python3/monetdb/mapi.py > clients/python3/monetdb/sql/connections.py > clients/python3/test/runtests.py > clients/python3/test/test_control.py > Branch: default > Log Message: > > add support for unix sockets > > > diffs (truncated from 649 to 300 lines): > > diff --git a/clients/python2/monetdb/control.py b/clients/python2/monetdb/control.py > --- a/clients/python2/monetdb/control.py > +++ b/clients/python2/monetdb/control.py > @@ -8,10 +8,12 @@ def parse_statusline(line): > parses a sabdb format status line. Support v1 and v2. > > """ > - if not line.startswith('=sabdb:'): > + if line.startswith("="): > + line = line[1:] > + if not line.startswith('sabdb:'): > raise OperationalError('wrong result recieved') > > - prot_version, rest = line.split(":", 2)[1:] > + code, prot_version, rest = line.split(":", 2) > > if prot_version not in ["1", "2"]: > raise InterfaceError("unsupported sabdb protocol") > @@ -60,20 +62,26 @@ class Control: > Use this module to manage your MonetDB databases. You can create, start, > stop, lock, unlock, destroy your databases and request status information. > """ > - def __init__(self, hostname, port, passphrase): > + def __init__(self, hostname=None, port=None, passphrase=None, > + unix_socket="/tmp/.s.merovingian.50000"): > self.server = mapi.Connection() > self.hostname = hostname > self.port = port > self.passphrase = passphrase > + self.unix_socket= unix_socket > > # check connection > - self.server.connect(hostname, port, 'monetdb', passphrase, > - 'merovingian', 'control') > + self.server.connect(hostname=hostname, port=port, username='monetdb', > + password=passphrase, > + database='merovingian', language='control', > + unix_socket=unix_socket) > self.server.disconnect() > > def _send_command(self, database_name, command): > - self.server.connect(self.hostname, self.port, 'monetdb', > - self.passphrase, 'merovingian', 'control') > + self.server.connect(hostname=self.hostname, port=self.port, > + username='monetdb', password=self.passphrase, > + database='merovingian', language='control', > + unix_socket=self.unix_socket) > try: > return self.server.cmd("%s %s\n" % (database_name, command)) > finally: > @@ -165,8 +173,9 @@ class Control: > """ > properties = self._send_command(database_name, "get") > values = {} > - for dirty_line in properties.split("\n"): > - line = dirty_line[1:] > + for line in properties.split("\n"): > + if line.startswith("="): > + line = line[1:] > if not line.startswith("#"): > if "=" in line: > split = line.split("=") > diff --git a/clients/python2/monetdb/mapi.py b/clients/python2/monetdb/mapi.py > --- a/clients/python2/monetdb/mapi.py > +++ b/clients/python2/monetdb/mapi.py > @@ -70,8 +70,12 @@ class Connection(object): > self.database = "" > self.language = "" > > - def connect(self, hostname, port, username, password, database, language): > - """ setup connection to MAPI server""" > + def connect(self, database, username, password, language, hostname=None, > + port=None, unix_socket=None): > + """ setup connection to MAPI server > + > + unix_socket is used if hostname is not defined. > + """ > > self.hostname = hostname > self.port = port > @@ -79,24 +83,34 @@ class Connection(object): > self.password = password > self.database = database > self.language = language > + self.unix_socket = unix_socket > > - self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + if hostname: > + self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + # For performance, mirror MonetDB/src/common/stream.c socket settings. > + self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 0) > + self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > + self.socket.connect((hostname, port)) > + else: > + self.socket = socket.socket(socket.AF_UNIX) > + self.socket.connect(unix_socket) > + if self.language != 'control': > + self.socket.send('0') # don't know why, but we need to do this > > - # For performance, mirror MonetDB/src/common/stream.c socket settings. > - self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 0) > - self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > + if not (self.language == 'control' and not self.hostname): > + # control doesn't require authentication over socket > + self._login() > > - self.socket.connect((hostname, port)) > - self.__login() > + self.state = STATE_READY > > - def __login(self, iteration=0): > + def _login(self, iteration=0): > """ Reads challenge from line, generate response and check if > everything is okay """ > > - challenge = self.__getblock() > - response = self.__challenge_response(challenge) > - self.__putblock(response) > - prompt = self.__getblock().strip() > + challenge = self._getblock() > + response = self._challenge_response(challenge) > + self._putblock(response) > + prompt = self._getblock().strip() > > if len(prompt) == 0: > # Empty response, server is happy > @@ -117,7 +131,7 @@ class Connection(object): > if redirect[1] == "merovingian": > logger.debug("restarting authentication") > if iteration <= 10: > - self.__login(iteration=iteration + 1) > + self._login(iteration=iteration + 1) > else: > raise OperationalError("maximal number of redirects " > "reached (10)") > @@ -138,9 +152,6 @@ class Connection(object): > else: > raise ProgrammingError("unknown state: %s" % prompt) > > - self.state = STATE_READY > - return True > - > def disconnect(self): > """ disconnect from the monetdb server """ > self.state = STATE_INIT > @@ -153,8 +164,8 @@ class Connection(object): > if self.state != STATE_READY: > raise(ProgrammingError, "Not connected") > > - self.__putblock(operation) > - response = self.__getblock() > + self._putblock(operation) > + response = self._getblock() > if not len(response): > return "" > elif response.startswith(MSG_OK): > @@ -166,10 +177,16 @@ class Connection(object): > return response > elif response[0] == MSG_ERROR: > raise OperationalError(response[1:]) > + elif (self.language == 'control' and not self.hostname): > + if response.startswith("OK"): > + return response[2:].strip() or "" > + else: > + return response > else: > raise ProgrammingError("unknown state: %s" % response) > > - def __challenge_response(self, challenge): > + > + def _challenge_response(self, challenge): > """ generate a response to a mapi login challenge """ > challenges = challenge.split(':') > salt, identity, protocol, hashes, endian = challenges[:5] > @@ -204,19 +221,36 @@ class Connection(object): > return ":".join(["BIG", self.username, pwhash, self.language, > self.database]) + ":" > > - def __getblock(self): > + def _getblock(self): > """ read one mapi encoded block """ > + if (self.language == 'control' and not self.hostname): > + return self._getblock_socket() # control doesn't do block > + # splitting when using a socket > + else: > + return self._getblock_inet() > + > + def _getblock_inet(self): > result = StringIO() > last = 0 > while not last: > - flag = self.__getbytes(2) > + flag = self._getbytes(2) > unpacked = struct.unpack('<H', flag)[0] # little endian short > length = unpacked >> 1 > last = unpacked & 1 > - result.write(self.__getbytes(length)) > + result.write(self._getbytes(length)) > return result.getvalue() > > - def __getbytes(self, bytes_): > + def _getblock_socket(self): > + buffer = StringIO() > + while True: > + x = self.socket.recv(1) > + if len(x): > + buffer.write(x) > + else: > + break > + return buffer.getvalue().strip() > + > + def _getbytes(self, bytes_): > """Read an amount of bytes from the socket""" > result = StringIO() > count = bytes_ > @@ -228,8 +262,15 @@ class Connection(object): > result.write(recv) > return result.getvalue() > > - def __putblock(self, block): > + def _putblock(self, block): > """ wrap the line in mapi format and put it into the socket """ > + if (self.language == 'control' and not self.hostname): > + return self.socket.send(block) # control doesn't do block > + # splitting when using a socket > + else: > + self._putblock_inet(block) > + > + def _putblock_inet(self, block): > pos = 0 > last = 0 > while not last: > diff --git a/clients/python2/monetdb/sql/connections.py b/clients/python2/monetdb/sql/connections.py > --- a/clients/python2/monetdb/sql/connections.py > +++ b/clients/python2/monetdb/sql/connections.py > @@ -28,25 +28,25 @@ class Connection(object): > """A MonetDB SQL database connection""" > default_cursor = cursors.Cursor > > - def __init__(self, username="monetdb", password="monetdb", > - hostname="localhost", port=50000, database="demo", > - autocommit=False, user=None, host=None): > + def __init__(self, database, hostname=None, port=50000, username="monetdb", > + password="monetdb", unix_socket="/tmp/.s.monetdb.50000", > + autocommit=False): > """ Set up a connection to a MonetDB SQL database. > > - username -- username for connection (default: monetdb) > - password -- password for connection (default: monetdb) > - hostname -- hostname to connect to (default: localhost) > - port -- port to connect to (default: 50000) > - database -- name of the database (default: demo) > - autocommit -- enable/disable auto commit (default: False) > + database -- name of the database > + hostname -- Hostname where monetDB is running > + port -- port to connect to (default: 50000) > + username -- username for connection (default: "monetdb") > + password -- password for connection (default: "monetdb") > + unix_socket -- socket to connect to. used when hostname not set > + (default: "/tmp/.s.monetdb.50000") > + autocommit -- enable/disable auto commit (default: False) > + > """ > - if user is not None: > - username = user > - if host is not None: > - hostname = host > self.mapi = mapi.Connection() > self.mapi.connect(hostname=hostname, port=int(port), username=username, > - password=password, database=database, language="sql") > + password=password, database=database, language="sql", > + unix_socket=unix_socket) > self.set_autocommit(autocommit) > self.set_sizeheader(True) > self.set_replysize(100) > diff --git a/clients/python2/test/runtests.py b/clients/python2/test/runtests.py > --- a/clients/python2/test/runtests.py > +++ b/clients/python2/test/runtests.py > @@ -45,6 +45,8 @@ TSTDB = os.environ.get('TSTDB', 'demo') > TSTHOSTNAME = os.environ.get('TSTHOSTNAME', 'localhost') > TSTUSERNAME = os.environ.get('TSTUSERNAME', 'monetdb') > TSTPASSWORD = os.environ.get('TSTPASSWORD', 'monetdb') > +#TSTHOSTNAME = None # set to none if test a socket > + > > if os.environ.get("TSTDEBUG", "no") == "yes": > logging.basicConfig(level=logging.DEBUG) > diff --git a/clients/python2/test/test_control.py b/clients/python2/test/test_control.py > --- a/clients/python2/test/test_control.py > +++ b/clients/python2/test/test_control.py > @@ -15,6 +15,7 @@ > # Copyright August 2008-2013 MonetDB B.V. > # All Rights Reserved. > > +import os > import unittest > import logging > > @@ -33,6 +34,10 @@ from monetdb.exceptions import Operation > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

3 10