- developers-list - monetdb.org

timestamp microsecond precision
by Gijs Molenaar 30 Sep '13

30 Sep '13

Hi MonetDB people, Do the TIME(STAMP) types support microsecond precision? They accept a 'postint' argument, but no matter what I set it to it looks like the timestamp is rounded to milliseconds. greetings, -- Gijs Molenaar http://www.astro.uva.nl/people/gijs-molenaar/

2 1

Announcement: New Feb2013-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 27 Sep '13

27 Sep '13

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2013-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Feb 2013-SP4 bugfix release Merovingian * monetdb now no longer compresses output when not connected to a terminal, bug #3292 Bug Fixes * 2781: SQL localtime() returns GMT time * 2962: client timezone information needs to be handled also for modifications to timestamp values * 3232: The function weekofyear returns results not according to the iso 8601 standard * 3255: M5 tests inlineFunction & ifthencst fail since changeset 207ed7d7e03a * 3292: monetdb status should print full line * 3307: distinct multicolumn performance and implementation suggestion for count distinct * 3310: Cast value to integer gives wrong result * 3311: strLength assertion on join and concat * 3313: Assertion in rel2bin_project * 3315: Incorrect behavior with subqueries in DELETE * 3317: correlated subselect does not work * 3318: UPDATE does not work with DECLEAREd TABLE in FUNCTION * 3319: UNIQUE column in DECLAREd TABLE in FUNCTION results in segfault * 3329: Function AVG fails during runtime in a procedure * 3330: Corrupt mitosis plan with group-by query * 3332: MEDIAN: no such operator 'median(decimal)' * 3336: DB files not removed if all rows are deleted, even after restart * 3337: Monet ODBC gives wrong schema information * 3339: Segvault in complex function call * 3340: crash on seeming incorrect SQL query * 3341: segfault caused by select rank() on a newly-loaded table * 3342: incorrect automatic cast from decimal to tinyint * 3345: copy binary into fails with the wrong error message * 3346: Deadlock in mal_dataflow's q_dequeue() (SciQL-2 branch) * 3348: SQL: LIKE incorrect with _ * 3349: When selecting everything from a joined result, column contents are incorrect in one of my use cases. * 3351: gzip'ed files cannot be bulkloaded into DB * 3352: Median function with decimal type doesn't work * 3353: Readonly tables call for UNSET * 3354: Introduce query time-out * 3358: Remote client connection stays in CLOSE_WAIT state * 3360: uninitialized memory causing bad output in query plans * 3362: ALTER TABLE operations silently remove previously set READ ONLY property * 3363: a join causes SIGSEGV after creation of Unique Key * 3367: Fully qualified order by column gives "relational query without result" * 3369: \0 Byte appears in MAPI

1 0

Whats coming in upcoming build monetdb release
by Ashish Kumar Singh 21 Sep '13

21 Sep '13

Hi , During my periodic search for "Is a new monetdb build is going to be released soon" , I found one is placed under testing recently. Is there a place where it can be seen and reviewed what kind of features/fixes are expected in upcoming release? That will be a great help for some of the folks like me to plan ahead! Regards, Ashish

1 0

Disk space to store a table?
by baraa Mohamad 10 Sep '13

10 Sep '13

Dear MonetDB Developers, I'm doing some research about colmn oriented databases and I began with MonetDB. In fact I would like to know how can I get the real disk space used to store a given tabel?? After some research on the web I found that I can use the storage table as follow: select "schema", "table", sum("columnsize") as size from storage() group by "schema","table" having "schema" <> 'tmp'; But it seems to me that this is something like a "logical" storage space without any compression (all columns with type "int" have the same size), is there a way to know the actual disk space used by MonetDB to store a table?? Thank you in advance for your help, RegardsBaraa

1 0

Re: MonetDB: int128 - backing-out changeset 77c9c3da562c:
by Martin Kersten 08 Sep '13

08 Sep '13

On 9/8/13 8:45 PM, Stefan Manegold wrote: > Changeset: a331af9e20ea for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a331af9e20ea > Modified Files: > monetdb5/modules/mal/batcalc.mal > monetdb5/modules/mal/batcalc.mal.sh > Branch: int128 > Log Message: > > backing-out changeset 77c9c3da562c: > > Unfortunately, MonetDB / mserver5 / MAL parser / MAL interpreter > is/are unable to handle the new monetdb5/modules/mal/batcalc.mal > correctly. > > When starting the server, it complains about > !TypeException:jaqlfunc.count[7]:'batcalc.lng' undefined in: r:any := batcalc.lng(x:bat[:oid,:wrd]) > although the respective batcalc.lng(x:bat[:oid,:wrd]) > is in monetdb5/modules/mal/batcalc.mal . > Some tests indicate that the systems (MAL parser ?) > has some problems handling MAL files >= ~1 MB > correctly, but I did not manage to locate the > actual nature or origin of the problem. > > Any help to get this working is highly appreciated. Most likely it is in mal_import where the complete mal file is loaded first

1 0

Auto-vectorization
by ajonwa＠gmx.com 07 Sep '13

07 Sep '13

Hello All, Has anyone managed to auto-vectorize (automatic SIMDfication) any of the loops in the kernel? If yes, I would like to know the complier and the complier flags used. Thanks, Geoffrey

1 0

Re: MonetDB: default - add support for unix sockets
by Stefan Manegold 06 Sep '13

06 Sep '13

Hi Gijs, in case you did not notice this, yet: This checkin breaks several tests that do test or use the Python interface, resulting in errors like " Traceback (most recent call last): File "delete_all.SQL.py", line 9, in <module> dbh = monetdb.sql.Connection(port=port,database=db,autocommit=True) File "PREFIX/lib/python2.7/site-packages/monetdb/sql/connections.py", line 49, in __init__ unix_socket=unix_socket) File "PREFIX/lib/python2.7/site-packages/monetdb/mapi.py", line 96, in connect self.socket.connect(unix_socket) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 2] No such file or directory " or " Traceback (most recent call last): File "SOURCE/clients/examples/python/sqlsample.py", line 23, in <module> dbh = monetdb.sql.Connection(port=int(sys.argv[1]),database=sys.argv[2],autocommit=True) File "PREFIX/lib/python3.3/site-packages/monetdb/sql/connections.py", line 49, in __init__ unix_socket=unix_socket) File "PREFIX/lib/python3.3/site-packages/monetdb/mapi.py", line 97, in connect self.socket.connect(unix_socket) FileNotFoundError: [Errno 2] No such file or directory " See also, e.g., `Mtest.py sql/benchmarks/tpch/fileleak sql/test/concurrent sql/test/mapi` or our nightly TestWeb @ http://monetdb.cwi.nl/testweb/web/status.php?branch=default as of tomorrow morning. Best, Stefan On Thu, Sep 05, 2013 at 02:19:01PM +0200, Gijs Molenaar wrote: > Changeset: 0eaa07b061be for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0eaa07b061be > Modified Files: > clients/python2/monetdb/control.py > clients/python2/monetdb/mapi.py > clients/python2/monetdb/sql/connections.py > clients/python2/test/runtests.py > clients/python2/test/test_control.py > clients/python3/monetdb/control.py > clients/python3/monetdb/mapi.py > clients/python3/monetdb/sql/connections.py > clients/python3/test/runtests.py > clients/python3/test/test_control.py > Branch: default > Log Message: > > add support for unix sockets > > > diffs (truncated from 649 to 300 lines): > > diff --git a/clients/python2/monetdb/control.py b/clients/python2/monetdb/control.py > --- a/clients/python2/monetdb/control.py > +++ b/clients/python2/monetdb/control.py > @@ -8,10 +8,12 @@ def parse_statusline(line): > parses a sabdb format status line. Support v1 and v2. > > """ > - if not line.startswith('=sabdb:'): > + if line.startswith("="): > + line = line[1:] > + if not line.startswith('sabdb:'): > raise OperationalError('wrong result recieved') > > - prot_version, rest = line.split(":", 2)[1:] > + code, prot_version, rest = line.split(":", 2) > > if prot_version not in ["1", "2"]: > raise InterfaceError("unsupported sabdb protocol") > @@ -60,20 +62,26 @@ class Control: > Use this module to manage your MonetDB databases. You can create, start, > stop, lock, unlock, destroy your databases and request status information. > """ > - def __init__(self, hostname, port, passphrase): > + def __init__(self, hostname=None, port=None, passphrase=None, > + unix_socket="/tmp/.s.merovingian.50000"): > self.server = mapi.Connection() > self.hostname = hostname > self.port = port > self.passphrase = passphrase > + self.unix_socket= unix_socket > > # check connection > - self.server.connect(hostname, port, 'monetdb', passphrase, > - 'merovingian', 'control') > + self.server.connect(hostname=hostname, port=port, username='monetdb', > + password=passphrase, > + database='merovingian', language='control', > + unix_socket=unix_socket) > self.server.disconnect() > > def _send_command(self, database_name, command): > - self.server.connect(self.hostname, self.port, 'monetdb', > - self.passphrase, 'merovingian', 'control') > + self.server.connect(hostname=self.hostname, port=self.port, > + username='monetdb', password=self.passphrase, > + database='merovingian', language='control', > + unix_socket=self.unix_socket) > try: > return self.server.cmd("%s %s\n" % (database_name, command)) > finally: > @@ -165,8 +173,9 @@ class Control: > """ > properties = self._send_command(database_name, "get") > values = {} > - for dirty_line in properties.split("\n"): > - line = dirty_line[1:] > + for line in properties.split("\n"): > + if line.startswith("="): > + line = line[1:] > if not line.startswith("#"): > if "=" in line: > split = line.split("=") > diff --git a/clients/python2/monetdb/mapi.py b/clients/python2/monetdb/mapi.py > --- a/clients/python2/monetdb/mapi.py > +++ b/clients/python2/monetdb/mapi.py > @@ -70,8 +70,12 @@ class Connection(object): > self.database = "" > self.language = "" > > - def connect(self, hostname, port, username, password, database, language): > - """ setup connection to MAPI server""" > + def connect(self, database, username, password, language, hostname=None, > + port=None, unix_socket=None): > + """ setup connection to MAPI server > + > + unix_socket is used if hostname is not defined. > + """ > > self.hostname = hostname > self.port = port > @@ -79,24 +83,34 @@ class Connection(object): > self.password = password > self.database = database > self.language = language > + self.unix_socket = unix_socket > > - self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + if hostname: > + self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + # For performance, mirror MonetDB/src/common/stream.c socket settings. > + self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 0) > + self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > + self.socket.connect((hostname, port)) > + else: > + self.socket = socket.socket(socket.AF_UNIX) > + self.socket.connect(unix_socket) > + if self.language != 'control': > + self.socket.send('0') # don't know why, but we need to do this > > - # For performance, mirror MonetDB/src/common/stream.c socket settings. > - self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 0) > - self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > + if not (self.language == 'control' and not self.hostname): > + # control doesn't require authentication over socket > + self._login() > > - self.socket.connect((hostname, port)) > - self.__login() > + self.state = STATE_READY > > - def __login(self, iteration=0): > + def _login(self, iteration=0): > """ Reads challenge from line, generate response and check if > everything is okay """ > > - challenge = self.__getblock() > - response = self.__challenge_response(challenge) > - self.__putblock(response) > - prompt = self.__getblock().strip() > + challenge = self._getblock() > + response = self._challenge_response(challenge) > + self._putblock(response) > + prompt = self._getblock().strip() > > if len(prompt) == 0: > # Empty response, server is happy > @@ -117,7 +131,7 @@ class Connection(object): > if redirect[1] == "merovingian": > logger.debug("restarting authentication") > if iteration <= 10: > - self.__login(iteration=iteration + 1) > + self._login(iteration=iteration + 1) > else: > raise OperationalError("maximal number of redirects " > "reached (10)") > @@ -138,9 +152,6 @@ class Connection(object): > else: > raise ProgrammingError("unknown state: %s" % prompt) > > - self.state = STATE_READY > - return True > - > def disconnect(self): > """ disconnect from the monetdb server """ > self.state = STATE_INIT > @@ -153,8 +164,8 @@ class Connection(object): > if self.state != STATE_READY: > raise(ProgrammingError, "Not connected") > > - self.__putblock(operation) > - response = self.__getblock() > + self._putblock(operation) > + response = self._getblock() > if not len(response): > return "" > elif response.startswith(MSG_OK): > @@ -166,10 +177,16 @@ class Connection(object): > return response > elif response[0] == MSG_ERROR: > raise OperationalError(response[1:]) > + elif (self.language == 'control' and not self.hostname): > + if response.startswith("OK"): > + return response[2:].strip() or "" > + else: > + return response > else: > raise ProgrammingError("unknown state: %s" % response) > > - def __challenge_response(self, challenge): > + > + def _challenge_response(self, challenge): > """ generate a response to a mapi login challenge """ > challenges = challenge.split(':') > salt, identity, protocol, hashes, endian = challenges[:5] > @@ -204,19 +221,36 @@ class Connection(object): > return ":".join(["BIG", self.username, pwhash, self.language, > self.database]) + ":" > > - def __getblock(self): > + def _getblock(self): > """ read one mapi encoded block """ > + if (self.language == 'control' and not self.hostname): > + return self._getblock_socket() # control doesn't do block > + # splitting when using a socket > + else: > + return self._getblock_inet() > + > + def _getblock_inet(self): > result = StringIO() > last = 0 > while not last: > - flag = self.__getbytes(2) > + flag = self._getbytes(2) > unpacked = struct.unpack('<H', flag)[0] # little endian short > length = unpacked >> 1 > last = unpacked & 1 > - result.write(self.__getbytes(length)) > + result.write(self._getbytes(length)) > return result.getvalue() > > - def __getbytes(self, bytes_): > + def _getblock_socket(self): > + buffer = StringIO() > + while True: > + x = self.socket.recv(1) > + if len(x): > + buffer.write(x) > + else: > + break > + return buffer.getvalue().strip() > + > + def _getbytes(self, bytes_): > """Read an amount of bytes from the socket""" > result = StringIO() > count = bytes_ > @@ -228,8 +262,15 @@ class Connection(object): > result.write(recv) > return result.getvalue() > > - def __putblock(self, block): > + def _putblock(self, block): > """ wrap the line in mapi format and put it into the socket """ > + if (self.language == 'control' and not self.hostname): > + return self.socket.send(block) # control doesn't do block > + # splitting when using a socket > + else: > + self._putblock_inet(block) > + > + def _putblock_inet(self, block): > pos = 0 > last = 0 > while not last: > diff --git a/clients/python2/monetdb/sql/connections.py b/clients/python2/monetdb/sql/connections.py > --- a/clients/python2/monetdb/sql/connections.py > +++ b/clients/python2/monetdb/sql/connections.py > @@ -28,25 +28,25 @@ class Connection(object): > """A MonetDB SQL database connection""" > default_cursor = cursors.Cursor > > - def __init__(self, username="monetdb", password="monetdb", > - hostname="localhost", port=50000, database="demo", > - autocommit=False, user=None, host=None): > + def __init__(self, database, hostname=None, port=50000, username="monetdb", > + password="monetdb", unix_socket="/tmp/.s.monetdb.50000", > + autocommit=False): > """ Set up a connection to a MonetDB SQL database. > > - username -- username for connection (default: monetdb) > - password -- password for connection (default: monetdb) > - hostname -- hostname to connect to (default: localhost) > - port -- port to connect to (default: 50000) > - database -- name of the database (default: demo) > - autocommit -- enable/disable auto commit (default: False) > + database -- name of the database > + hostname -- Hostname where monetDB is running > + port -- port to connect to (default: 50000) > + username -- username for connection (default: "monetdb") > + password -- password for connection (default: "monetdb") > + unix_socket -- socket to connect to. used when hostname not set > + (default: "/tmp/.s.monetdb.50000") > + autocommit -- enable/disable auto commit (default: False) > + > """ > - if user is not None: > - username = user > - if host is not None: > - hostname = host > self.mapi = mapi.Connection() > self.mapi.connect(hostname=hostname, port=int(port), username=username, > - password=password, database=database, language="sql") > + password=password, database=database, language="sql", > + unix_socket=unix_socket) > self.set_autocommit(autocommit) > self.set_sizeheader(True) > self.set_replysize(100) > diff --git a/clients/python2/test/runtests.py b/clients/python2/test/runtests.py > --- a/clients/python2/test/runtests.py > +++ b/clients/python2/test/runtests.py > @@ -45,6 +45,8 @@ TSTDB = os.environ.get('TSTDB', 'demo') > TSTHOSTNAME = os.environ.get('TSTHOSTNAME', 'localhost') > TSTUSERNAME = os.environ.get('TSTUSERNAME', 'monetdb') > TSTPASSWORD = os.environ.get('TSTPASSWORD', 'monetdb') > +#TSTHOSTNAME = None # set to none if test a socket > + > > if os.environ.get("TSTDEBUG", "no") == "yes": > logging.basicConfig(level=logging.DEBUG) > diff --git a/clients/python2/test/test_control.py b/clients/python2/test/test_control.py > --- a/clients/python2/test/test_control.py > +++ b/clients/python2/test/test_control.py > @@ -15,6 +15,7 @@ > # Copyright August 2008-2013 MonetDB B.V. > # All Rights Reserved. > > +import os > import unittest > import logging > > @@ -33,6 +34,10 @@ from monetdb.exceptions import Operation > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

3 10

Re: MonetDB: Feb2013 - fixed bug 3232, use dayofweek, starting at Mo...
by Sjoerd Mullender 27 Aug '13

27 Aug '13

This checkin is not allowed on the release branch: it changes the ABI by introducing a function MTIMEdate_extract_dayofworkweek. What's wrong with changing the implementation of MTIMEdate_extract_dayofweek to do what MTIMEdate_extract_dayofworkweek does now (and dropping MTIMEdate_extract_dayofworkweek)? On 2013-08-25 16:23, Niels Nes wrote: > Changeset: 0620e9df2003 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0620e9df2003 > Added Files: > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.sql > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.stable.err > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.stable.out > Modified Files: > clients/mapiclient/mclient.c > monetdb5/modules/atoms/mtime.c > monetdb5/modules/atoms/mtime.h > monetdb5/modules/atoms/mtime.mal > sql/backends/monet5/sql.mx > sql/common/sql_types.c > sql/scripts/mtime.sql > sql/server/rel_select.c > sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > sql/test/BugTracker-2013/Tests/All > sql/test/leaks/Tests/check1.stable.out > sql/test/leaks/Tests/check2.stable.out > sql/test/leaks/Tests/check3.stable.out > sql/test/leaks/Tests/check4.stable.out > sql/test/leaks/Tests/check5.stable.out > sql/test/mapi/Tests/php_monetdb.stable.out > Branch: Feb2013 > Log Message: > > fixed bug 3232, use dayofweek, starting at Monday > fixed bug 2962, convert timestamp with time zone to local time zone, when used as input to the extract functions. > fixed bug 2781, use correct time zone in mclient > > > diffs (truncated from 761 to 300 lines): > > diff --git a/clients/mapiclient/mclient.c b/clients/mapiclient/mclient.c > --- a/clients/mapiclient/mclient.c > +++ b/clients/mapiclient/mclient.c > @@ -2559,6 +2559,7 @@ set_timezone(Mapi mid) > tmp = gmtime(&t); > gt = mktime(tmp); > tmp = localtime(&t); > + tmp->tm_isdst=0; /* We need the difference without dst */ > lt = mktime(tmp); > assert((lng) gt - (lng) lt >= (lng) INT_MIN && (lng) gt - (lng) lt <= (lng) INT_MAX); > tzone = (int) (gt - lt); > diff --git a/monetdb5/modules/atoms/mtime.c b/monetdb5/modules/atoms/mtime.c > --- a/monetdb5/modules/atoms/mtime.c > +++ b/monetdb5/modules/atoms/mtime.c > @@ -427,6 +427,7 @@ parse_substr(int *ret, str s, int min, s > return j; > } > > +/* Sunday = 1, Saturday = 7 */ > static int > date_dayofweek(date v) > { > @@ -434,6 +435,17 @@ date_dayofweek(date v) > return v <= 0 ? v + 7 : v; > } > > +/* Monday = 1, Sunday = 7 */ > +static int > +date_dayofworkweek(date v) > +{ > + int res = date_dayofweek(v); > + > + if (res >= 2) > + return res -1; > + return 7; > +} > + > #define SKIP_DAYS(d,w,i) d += i; w = (w + i)%7; if (w <= 0) w += 7; > > static date > @@ -1310,10 +1322,10 @@ date_extract_weekofyear(int *ret, date * > > fromdate((int) *v, &dummy, &dummy, &year); > year_jan_1 = todate(1, 1, year); > - dayofweek = date_dayofweek(year_jan_1); > + dayofweek = date_dayofworkweek(year_jan_1); > > if (dayofweek <= 4) { > - /* week of jan 1 belongs to this year */ > + /* 4 or more days in first week, ie week of jan 1 belongs to this year */ > *ret = (int) (1 + (*v - year_jan_1 + dayofweek - 1) / 7); > } else if (*v - year_jan_1 > 7 - dayofweek) { > /* week of jan 1 belongs to last year; but this is a later week */ > @@ -1340,6 +1352,18 @@ date_extract_dayofweek(int *ret, date *v > return MAL_SUCCEED; > } > > +/* Returns the current day of the week where 1=monday, .., 7=sunday */ > +static str > +date_extract_dayofworkweek(int *ret, date *v) > +{ > + if (*v == date_nil) { > + *ret = int_nil; > + } else { > + *ret = date_dayofworkweek(*v); > + } > + return MAL_SUCCEED; > +} > + > /* extracts hour from daytime (value between 0 and 23) */ > static str > daytime_extract_hours(int *ret, daytime *v) > @@ -1959,6 +1983,12 @@ MTIMEdate_extract_dayofweek(int *ret, da > } > > str > +MTIMEdate_extract_dayofworkweek(int *ret, date *v) > +{ > + return date_extract_dayofworkweek(ret, v); > +} > + > +str > MTIMEdaytime_extract_hours(int *ret, daytime *v) > { > return daytime_extract_hours(ret, v); > diff --git a/monetdb5/modules/atoms/mtime.h b/monetdb5/modules/atoms/mtime.h > --- a/monetdb5/modules/atoms/mtime.h > +++ b/monetdb5/modules/atoms/mtime.h > @@ -163,6 +163,7 @@ mtime_export str MTIMEdate_extract_day(i > mtime_export str MTIMEdate_extract_dayofyear(int *ret, date *v); > mtime_export str MTIMEdate_extract_weekofyear(int *ret, date *v); > mtime_export str MTIMEdate_extract_dayofweek(int *ret, date *v); > +mtime_export str MTIMEdate_extract_dayofworkweek(int *ret, date *v); > mtime_export str MTIMEdaytime_extract_hours(int *ret, daytime *v); > mtime_export str MTIMEdaytime_extract_minutes(int *ret, daytime *v); > mtime_export str MTIMEdaytime_extract_seconds(int *ret, daytime *v); > diff --git a/monetdb5/modules/atoms/mtime.mal b/monetdb5/modules/atoms/mtime.mal > --- a/monetdb5/modules/atoms/mtime.mal > +++ b/monetdb5/modules/atoms/mtime.mal > @@ -633,6 +633,10 @@ command dayofweek(d:date) :int > address MTIMEdate_extract_dayofweek > comment "Returns the current day of the week > where 1=sunday, .., 7=saturday"; > +command dayofworkweek(d:date) :int > +address MTIMEdate_extract_dayofworkweek > +comment "Returns the current day of the week > + where 1=monday, .., 7=sunday"; > > command add(v:timestamp, msecs:lng) :timestamp > address MTIMEtimestamp_add > diff --git a/sql/backends/monet5/sql.mx b/sql/backends/monet5/sql.mx > --- a/sql/backends/monet5/sql.mx > +++ b/sql/backends/monet5/sql.mx > @@ -714,6 +714,14 @@ comment "cast @1 to @1 and check for ove > @:mal_casttime(timestamp)@ > @:mal_casttime(daytime)@ > > +pattern current_time() :daytime > +address SQLcurrent_daytime > +comment "Get the clients current daytime"; > + > +pattern current_timestamp() :timestamp > +address SQLcurrent_timestamp > +comment "Get the clients current timestamp"; > + > command calc.date( v:void ) :date > address nil_2_date > comment "cast to date"; > @@ -6381,6 +6389,8 @@ sql5_export str second_interval_daytime( > sql5_export str second_interval_2_daytime( daytime *res, lng *s, int *d); > sql5_export str timestamp_2_daytime( daytime *res, timestamp *v, int *d); > sql5_export str date_2_timestamp( timestamp *res, date *v, int *d); > +str SQLcurrent_daytime(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > +str SQLcurrent_timestamp(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > @c > str > second_interval_daytime( lng *res, daytime *s, int *d, int *sk ) > @@ -6415,6 +6425,36 @@ second_interval_daytime( lng *res, dayti > return MAL_SUCCEED; > } > > +str SQLcurrent_daytime(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +{ > + mvc *m = NULL; > + str msg = getSQLContext(cntxt, mb, &m, NULL); > + daytime t, *res = (daytime *) getArgReference(stk, pci, 0); > + > + if (msg) > + return msg; > + > + if ((msg = MTIMEcurrent_time(&t)) == MAL_SUCCEED) > + *res = t+m->timezone; > + return msg; > +} > + > +str SQLcurrent_timestamp(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +{ > + mvc *m = NULL; > + str msg = getSQLContext(cntxt, mb, &m, NULL); > + timestamp t, *res = (timestamp *) getArgReference(stk, pci, 0); > + > + if (msg) > + return msg; > + > + if ((msg = MTIMEcurrent_timestamp(&t)) == MAL_SUCCEED) { > + lng offset = m->timezone; > + return MTIMEtimestamp_add(res, &t, &offset); > + } > + return msg; > +} > + > /* str dump_cache(int *r); */ > str > dump_cache(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > diff --git a/sql/common/sql_types.c b/sql/common/sql_types.c > --- a/sql/common/sql_types.c > +++ b/sql/common/sql_types.c > @@ -1457,8 +1457,8 @@ sqltypeinit( sql_allocator *sa) > sql_create_func(sa, "curtime", "mtime", "current_time", NULL, NULL, TMETZ, SCALE_NONE); > sql_create_func(sa, "current_time", "mtime", "current_time", NULL, NULL, TMETZ, SCALE_NONE); > sql_create_func(sa, "current_timestamp", "mtime", "current_timestamp", NULL, NULL, TMESTAMPTZ, SCALE_NONE); > - sql_create_func(sa, "localtime", "mtime", "current_time", NULL, NULL, TME, SCALE_NONE); > - sql_create_func(sa, "localtimestamp", "mtime", "current_timestamp", NULL, NULL, TMESTAMP, SCALE_NONE); > + sql_create_func(sa, "localtime", "sql", "current_time", NULL, NULL, TME, SCALE_NONE); > + sql_create_func(sa, "localtimestamp", "sql", "current_timestamp", NULL, NULL, TMESTAMP, SCALE_NONE); > > sql_create_func(sa, "sql_sub", "mtime", "date_sub_msec_interval", DTE, SECINT, DTE, SCALE_FIX); > sql_create_func(sa, "sql_sub", "mtime", "date_sub_month_interval", DTE, MONINT, DTE, SCALE_FIX); > @@ -1470,6 +1470,7 @@ sqltypeinit( sql_allocator *sa) > > sql_create_func(sa, "sql_sub", "mtime", "diff", DTE, DTE, INT, SCALE_FIX); > sql_create_func(sa, "sql_sub", "mtime", "diff", TMESTAMP, TMESTAMP, LNG, SCALE_FIX); > + sql_create_func(sa, "sql_sub", "mtime", "diff", TMESTAMPTZ, TMESTAMPTZ, LNG, SCALE_NONE); > > sql_create_func(sa, "sql_add", "mtime", "date_add_msec_interval", DTE, SECINT, DTE, SCALE_NONE); > sql_create_func(sa, "sql_add", "mtime", "addmonths", DTE, MONINT, DTE, SCALE_NONE); > @@ -1498,6 +1499,15 @@ sqltypeinit( sql_allocator *sa) > /* fix result type */ > f->res.scale = 3; > > + sql_create_func(sa, "year", "mtime", "year", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "month", "mtime", "month", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "day", "mtime", "day", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "hour", "mtime", "hours", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "minute", "mtime", "minutes", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + f = sql_create_func(sa, "second", "mtime", "sql_seconds", TMESTAMPTZ, NULL, DEC, SCALE_NONE); > + /* fix result type */ > + f->res.scale = 3; > + > sql_create_func(sa, "year", "mtime", "year", MONINT, NULL, INT, SCALE_NONE); > sql_create_func(sa, "month", "mtime", "month", MONINT, NULL, INT, SCALE_NONE); > sql_create_func(sa, "day", "mtime", "day", SECINT, NULL, LNG, SCALE_NONE); > @@ -1507,7 +1517,7 @@ sqltypeinit( sql_allocator *sa) > > sql_create_func(sa, "dayofyear", "mtime", "dayofyear", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "weekofyear", "mtime", "weekofyear", DTE, NULL, INT, SCALE_FIX); > - sql_create_func(sa, "dayofweek", "mtime", "dayofweek", DTE, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "dayofweek", "mtime", "dayofworkweek", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "dayofmonth", "mtime", "day", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "week", "mtime", "weekofyear", DTE, NULL, INT, SCALE_FIX); > > diff --git a/sql/scripts/mtime.sql b/sql/scripts/mtime.sql > --- a/sql/scripts/mtime.sql > +++ b/sql/scripts/mtime.sql > @@ -29,16 +29,16 @@ create function "current_timestamp"( ) r > external name mtime."current_timestamp"; > > create function "localtime"( ) returns TIME > - external name mtime."current_time"; > + external name sql."current_time"; > + > create function "localtimestamp"( ) returns TIMESTAMP > - external name mtime."current_timestamp"; > + external name sql."current_timestamp"; > > create type second_interval as interval day to sec; > > create function local_timezone( ) returns second_interval > external name mtime.local_timezone; > > - > create function sql_sub( date, second_interval ) returns date > external name mtime.date_sub_sec_interval; > create function sql_sub( date, month_interval ) returns date > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -3290,6 +3290,17 @@ rel_unop(mvc *sql, sql_rel **rel, symbol > f = sql_bind_func(sql->sa, s, fname, t, NULL, F_AGGR); > if (f && IS_AGGR(f->func)) > return _rel_aggr(sql, rel, 0, s, fname, l->next, fs); > + > + if (f && type_has_tz(t) && f->func->fix_scale == SCALE_FIX) { > + /* set timezone (using msec) */ > + sql_subtype *intsec = sql_bind_subtype(sql->sa, "sec_interval", 10 /*hour to second */, 0); > + sql_exp *tz = exp_atom_lng(sql->sa, sql->timezone); > + > + tz = exp_convert(sql->sa, tz, exp_subtype(tz), intsec); > + e = rel_binop_(sql, e, tz, NULL, "sql_add", ek.card); > + if (!e) > + return NULL; > + } > return rel_unop_(sql, e, s, fname, ek.card); > } > > diff --git a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > --- a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > +++ b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > @@ -30,9 +30,9 @@ stderr of test 'predicate_select.Bug-309 > # 18:53:13 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" "--host=niels" "--port=32323" > # 18:53:13 > > > -MAPI = (monetdb) /var/tmp/mtest-16462/.s.monetdb.36324 > +MAPI = (monetdb) /var/tmp/mtest-4932/.s.monetdb.34449 > QUERY = select * from tables where name; > -ERROR = !conversion of string 'r' to type bit failed. > +ERROR = !conversion of string 'storagemodelinput' to type bit failed. > > # 18:53:13 > > # 18:53:13 > "Done." > diff --git a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > --- a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > +++ b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > @@ -47,19 +47,19 @@ Ready. > [ 2130, "idxs", 2102, NULL, 0, true, 2, false, 0 ] > [ 2135, "triggers", 2102, NULL, 0, true, 2, false, 0 ] > [ 2146, "objects", 2102, NULL, 0, true, 2, false, 0 ] > -[ 5060, "tables", 2000, "SELECT * FROM (SELECT p.*, 0 AS \"temporary\" FROM \"sys\".\"_tables\" AS p UNION ALL SELECT t.*, 1 AS \"temporary\" FROM \"tmp\".\"_tables\" AS t) AS tables where tables.type <> 2;", 1, true, 0, false, 0 ] > -[ 5070, "columns", 2000, "SELECT * FROM (SELECT p.* FROM \"sys\".\"_columns\" AS p UNION ALL SELECT t.* FROM \"tmp\".\"_columns\" AS t) AS columns;", 1, true, 0, false, 0 ] > -[ 5090, "db_user_info", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5098, "users", 2000, "SELECT u.\"name\" AS \"name\", ui.\"fullname\", ui.\"default_schema\" FROM db_users() AS u LEFT JOIN \"sys\".\"db_user_info\" AS ui ON u.\"name\" = ui.\"name\" ;", 1, true, 0, false, 0 ] > -[ 5102, "user_role", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5105, "auths", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5109, "privileges", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5262, "queryhistory", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5276, "callhistory", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5293, "querylog", 2000, "create view querylog as\nselect qd.*, ql.ctime, ql.arguments, ql.exec, ql.result, ql.foot, ql.memory, ql.tuples, ql.inblock, ql.oublock from queryhistory qd, callhistory ql\nwhere qd.id = ql.id;", 1, true, 0, false, 0 ] > -[ 6059, "storagemodelinput", 2000, NULL, 0, true, 0, false, 0 ] > -[ 6100, "tablestoragemodel", 2000, "-- A summary of the table storage requirement is is available as a table view.\n-- The auxillary column denotes the maximum space if all non-sorted columns\n-- would be augmented with a hash (rare situation)\ncreate view sys.tablestoragemodel\nas select \"schema\",\"table\",max(count) as \"count\",\n\tsum(columnsize) as columnsize,\n\tsum(heapsize) as heapsize,\n\tsum(indices) as indices,\n\tsum(case when sorted = false then 8 * count else 0 end) as auxillary\nfrom sys.storagemodel() group by \"schema\",\"table\";", 1, true, 0, false, 0 ] > -[ 6122, "systemfunctions", 2000, NULL, 0, true, 0, false, 0 ] > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

Re: MonetDB: Feb2013 - Organize parallelism per worker pool
by Stefan Manegold 21 Aug '13

21 Aug '13

Martin, could you please also add sql/test/BugTracker-2013/Tests/nestedcalls.sql ? And could you plase also add the test scripts and stable output for tests sql/test/BugTracker-2013/Tests/oid_handling sql/test/BugTracker-2013/Tests/constraint_checking.Bug_3335 sql/test/BugTracker-2013/Tests/pivot.Bug-3339 sql/test/BugTracker-2013/Tests/recursion or alternatively remove them from sql/test/BugTracker-2013/Tests/All Thanks! In any case, propagation of this will checking will result in (expected/unavoidable) conflicts in monetdb5/mal/mal_dataflow.[ch] and (avoidable) conflicts in sql/test/BugTracker-2013/Tests All of these will need to be solved by hand ... Stefan ----- Original Message ----- > Changeset: 489815265a61 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=489815265a61 > Added Files: > sql/test/BugTracker-2013/Tests/nestedcalls.stable.err > sql/test/BugTracker-2013/Tests/nestedcalls.stable.out > Modified Files: > monetdb5/mal/mal_dataflow.c > monetdb5/mal/mal_dataflow.h > monetdb5/modules/mal/language.c > monetdb5/modules/mal/language.h > sql/test/BugTracker-2013/Tests/All > Branch: Feb2013 > Log Message: > > Organize parallelism per worker pool > Each (recursive) dataflow block is handed a worker pool now. > Once we run out of worker pools or fail to create a worker, > we continue in sequential mode. > > Cleaning up the threads is implicit, like all other non-pool > interpreters that might be active at system shutdown. > > > diffs (truncated from 762 to 300 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -35,6 +35,7 @@ > * The flow graphs should be organized such that parallel threads can > * access it mostly without expensive locking. > */ > +#include "monetdb_config.h" > #include "mal_dataflow.h" > #include "mal_client.h" > > @@ -82,10 +83,10 @@ typedef struct DATAFLOW { > Queue *done; /* instructions handled */ > } *DataFlow, DataFlowRec; > > -#define MAXQ 1024 > -static MT_Id workers[THREADS] = {0}; > -static int workerqueue[THREADS] = {0}; /* maps workers towards the todo > queues */ > -static Queue *todo[MAXQ] = {0}; /* pending instructions organized by user > MAXTODO > #users */ > +#define MAXQ 256 > +static Queue *todos[MAXQ] = {0}; /* pending instructions organized by > dataflow block */ > +static bit occupied[MAXQ]={0}; /* worker pool is in use? */ > +static int volatile exiting = 0; > > /* > * Calculate the size of the dataflow dependency graph. > @@ -108,9 +109,8 @@ DFLOWgraphSize(MalBlkPtr mb, int start, > */ > > static Queue* > -q_create(int sz) > +q_create(int sz, const char *name) > { > - const char* name = "q_create"; > Queue *q = (Queue*)GDKmalloc(sizeof(Queue)); > > if (q == NULL) > @@ -208,6 +208,8 @@ q_dequeue(Queue *q) > > assert(q); > MT_sema_down(&q->s, "q_dequeue"); > + if (exiting) > + return NULL; > MT_lock_set(&q->l, "q_dequeue"); > assert(q->last > 0); > if (q->last > 0) { > @@ -255,25 +257,23 @@ DFLOWworker(void *t) > { > DataFlow flow; > FlowEvent fe = 0, fnxt = 0; > - int id = (int) ((MT_Id *) t - workers), last = 0; > - int wq; > Thread thr; > str error = 0; > - > - int i; > - lng usec = 0; > + Queue *todo = *(Queue **) t; > + int i,last; > > thr = THRnew("DFLOWworker"); > > GDKsetbuf(GDKmalloc(GDKMAXERRLEN)); /* where to leave errors */ > GDKerrbuf[0] = 0; > while (1) { > - assert(workerqueue[id] > 0); > - wq = workerqueue[id] - 1; > if (fnxt == 0) > - fe = q_dequeue(todo[wq]); > + fe = q_dequeue(todo); > else > fe = fnxt; > + if (exiting) { > + break; > + } > fnxt = 0; > assert(fe); > flow = fe->flow; > @@ -285,22 +285,20 @@ DFLOWworker(void *t) > continue; > } > > - usec = GDKusec(); > /* skip all instructions when we have encontered an error */ > if (flow->error == 0) { > #ifdef USE_MAL_ADMISSION > if (MALadmission(fe->argclaim, fe->hotclaim)) { > fe->hotclaim = 0; /* don't assume priority anymore */ > - assert(todo[wq]); > - if (todo[wq]->last == 0) > + if (todo->last == 0) > MT_sleep_ms(DELAYUNIT); > - q_requeue(todo[wq], fe); > + q_requeue(todo, fe); > continue; > } > #endif > error = runMALsequence(flow->cntxt, flow->mb, fe->pc, fe->pc + 1, > flow->stk, 0, 0); > PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= %d claim= " LLFMT > "," LLFMT " %s\n", > - fe->pc, id, fe->argclaim, fe->hotclaim, error ? error : ""); > + fe->pc, (int)((Queue **)t - todos), fe->argclaim, fe->hotclaim, > error ? error : ""); > #ifdef USE_MAL_ADMISSION > /* release the memory claim */ > MALadmission(-fe->argclaim, -fe->hotclaim); > @@ -331,8 +329,8 @@ DFLOWworker(void *t) > InstrPtr p = getInstrPtr(flow->mb, fe->pc); > assert(p); > fe->hotclaim = 0; > - for (i = 0; i < p->retc; i++) > - fe->hotclaim += getMemoryClaim(flow->mb, flow->stk, fe->pc, i, FALSE); > + //for (i = 0; i < p->retc; i++) > + //fe->hotclaim += getMemoryClaim(flow->mb, flow->stk, p, i, FALSE); > } > #endif > MT_lock_set(&flow->flowlock, "MALworker"); > @@ -351,56 +349,64 @@ DFLOWworker(void *t) > > q_enqueue(flow->done, fe); > if ( fnxt == 0) { > - assert(todo[wq]); > - if (todo[wq]->last == 0) > + if (todo->last == 0) > profilerHeartbeatEvent("wait"); > - else > - MALresourceFairness(NULL, NULL, usec); > } > } > GDKfree(GDKerrbuf); > GDKsetbuf(0); > - workerqueue[wq] = 0; > - workers[wq] = 0; > THRdel(thr); > } > > /* > - * Create a set of DFLOW interpreters. > + * Create an interpreter pool. > * One worker will adaptively be available for each client. > * The remainder are taken from the GDKnr_threads argument and > - * typically is equal to the number of cores > + * typically is equal to the number of cores. > + * A recursive MAL function call would make for one worker less, > + * which limits the number of cores for parallel processing. > * The workers are assembled in a local table to enable debugging. > + * > + * BEWARE, failure to create a new worker thread is not an error > + * but would lead to serial execution. > */ > -static void > -DFLOWinitialize(int index) > +static int > +DFLOWinitialize(void) > { > - int i, worker, limit; > + int i, threads, grp; > + MT_Id worker; > > - assert(index >= 0); > - assert(index < THREADS); > + threads = GDKnr_threads ? GDKnr_threads : 1; > MT_lock_set(&mal_contextLock, "DFLOWinitialize"); > - if (todo[index]) { > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > - return; > + for(grp = 0; grp< MAXQ; grp++) > + if ( occupied[grp] == FALSE){ > + occupied[grp] = TRUE; > + break; > + } > + MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > + if (grp > THREADS) { > + // continue non-parallel > + return -1; > } > - todo[index] = q_create(2048); > - assert(todo[index]); > - limit = GDKnr_threads ? GDKnr_threads : 1; > - assert(limit <= THREADS); > - for (worker = 0, i = 0; i < limit; i++){ > - for (; worker < THREADS; worker++) > - if( workers[worker] == 0) > - break; > - assert(worker < THREADS); > - if (worker < THREADS) { > - assert(workers[worker] == 0); > - MT_create_thread(&workers[worker], DFLOWworker, (void *) > &workers[worker], MT_THR_JOINABLE); > - assert(workers[worker] > 0); > - workerqueue[worker] = index + 1; > + if ( todos[grp] ) > + return grp; > + > + todos[grp] = q_create(2048, "todo"); > + if (todos[grp] == NULL) > + return -1; > + > + // associate a set of workers with the pool > + for (i = 0; grp>= 0 && i < threads; i++){ > + if (MT_create_thread(&worker, DFLOWworker, (void *) &todos[grp], > MT_THR_JOINABLE) < 0) { > + //Can not create interpreter thread > + grp = -1; > + } > + if (worker == 0) { > + //Failed to create interpreter thread > + grp = -1; > } > } > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > + return grp; > } > > /* > @@ -409,18 +415,28 @@ DFLOWinitialize(int index) > * For each instruction we keep a list of instructions whose > * blocking counter should be decremented upon finishing it. > */ > -static void > +static str > DFLOWinitBlk(DataFlow flow, MalBlkPtr mb, int size) > { > int pc, i, j, k, l, n, etop = 0; > int *assign; > InstrPtr p; > > + if (flow == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Called with flow == NULL"); > + if (mb == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Called with mb == NULL"); > PARDEBUG printf("Initialize dflow block\n"); > assign = (int *) GDKzalloc(mb->vtop * sizeof(int)); > + if (assign == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Failed to allocate assign"); > etop = flow->stop - flow->start; > for (n = 0, pc = flow->start; pc < flow->stop; pc++, n++) { > p = getInstrPtr(mb, pc); > + if (p == NULL) { > + GDKfree(assign); > + throw(MAL, "dataflow", "DFLOWinitBlk(): getInstrPtr() returned NULL"); > + } > > /* initial state, ie everything can run */ > flow->status[n].flow = flow; > @@ -501,6 +517,7 @@ DFLOWinitBlk(DataFlow flow, MalBlkPtr mb > #ifdef USE_MAL_ADMISSION > memorypool = memoryclaims = 0; > #endif > + return MAL_SUCCEED; > } > > /* > @@ -528,18 +545,17 @@ static void showFlowEvent(DataFlow flow, > */ > > static str > -DFLOWscheduler(DataFlow flow) > +DFLOWscheduler(DataFlow flow, Queue *todo) > { > int last; > int i; > #ifdef USE_MAL_ADMISSION > - int j; > + //int j; > InstrPtr p; > #endif > int tasks=0, actions; > str ret = MAL_SUCCEED; > FlowEvent fe, f = 0; > - int wq; > > if (flow == NULL) > throw(MAL, "dataflow", "DFLOWscheduler(): Called with flow == NULL"); > @@ -549,19 +565,19 @@ DFLOWscheduler(DataFlow flow) > /* initialize the eligible statements */ > fe = flow->status; > > - if (fe[0].flow->cntxt->flags & timerFlag) > - fe[0].flow->cntxt->timer = GDKusec(); > - > MT_lock_set(&flow->flowlock, "MALworker"); > - wq = flow->cntxt->idx; > for (i = 0; i < actions; i++) > if (fe[i].blocks == 0) { > #ifdef USE_MAL_ADMISSION > p = getInstrPtr(flow->mb,fe[i].pc); > - for (j = p->retc; j < p->argc; j++) > - fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, > fe[i].pc, j, FALSE); > + if (p == NULL) { > + MT_lock_unset(&flow->flowlock, "MALworker"); > + throw(MAL, "dataflow", "DFLOWscheduler(): getInstrPtr(flow->mb,fe[i].pc) > returned NULL"); > + } > + //for (j = p->retc; j < p->argc; j++) > + //fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, p, j, > FALSE); > #endif > - q_enqueue(todo[wq], flow->status + i); > + q_enqueue(todo, flow->status + i); > flow->status[i].state = DFLOWrunning; > PARDEBUG mnstr_printf(GDKstdout, "#enqueue pc=%d claim=" LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } > @@ -571,6 +587,10 @@ DFLOWscheduler(DataFlow flow) > > while (actions != tasks ) { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 2

default vs. Feb2013: opt_evaluate.c & mal_dataflow.c
by Stefan Manegold 19 Aug '13

19 Aug '13

Martin, I just merged recent changes from Feb2013 into default, and while doing so resolved --- hopefully correctly; see below --- some conflicts that occurred in opt_evaluate.c & mal_dataflow.c . In fact, (1) I first merged up until before your changeset that introduced the new multiple worker pools, cf., http://dev.monetdb.org/hg/MonetDB/rev/10125a9c3976 (2) then merged only that very changeset, cf., http://dev.monetdb.org/hg/MonetDB/rev/1eea7fa4cc7c (3) and finally merge the rest, cf., http://dev.monetdb.org/hg/MonetDB/rev/3e3d7d5f8dc9 During (1), I resolved some conflicts in opt_evaluate.c due to http://dev.monetdb.org/hg/MonetDB/rev/62ed11377998 I hope, I did this correctly. However, there are still some differences in opt_evaluate.c between Feb2013 & default; cf., hg diff -p -r Feb2013 -r default monetdb5/optimizer/opt_evaluate.c Could you please check, whether these are OK? During (2), I resolved some conflicts in mal_dataflow.c due to http://dev.monetdb.org/hg/MonetDB/rev/4c47797192f4 I hope, I did this correctly. However, there are still some differences in mal_dataflow.c between Feb2013 & default; cf., hg diff -p -r Feb2013 -r default monetdb5/mal/mal_dataflow.c Could you please check, whether these are OK? It appears, that after (2), some tests in sql/jdbc/tests/Tests/ run into a timeout (only when run multi-threaded, i.e., not when run single-threaded) --- I have not yet been able to find out why ... Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0