- developers-list - monetdb.org

Whats coming in upcoming build monetdb release
by Ashish Kumar Singh 21 Sep '13

21 Sep '13

Hi , During my periodic search for "Is a new monetdb build is going to be released soon" , I found one is placed under testing recently. Is there a place where it can be seen and reviewed what kind of features/fixes are expected in upcoming release? That will be a great help for some of the folks like me to plan ahead! Regards, Ashish

1 0

Disk space to store a table?
by baraa Mohamad 10 Sep '13

10 Sep '13

Dear MonetDB Developers, I'm doing some research about colmn oriented databases and I began with MonetDB. In fact I would like to know how can I get the real disk space used to store a given tabel?? After some research on the web I found that I can use the storage table as follow: select "schema", "table", sum("columnsize") as size from storage() group by "schema","table" having "schema" <> 'tmp'; But it seems to me that this is something like a "logical" storage space without any compression (all columns with type "int" have the same size), is there a way to know the actual disk space used by MonetDB to store a table?? Thank you in advance for your help, RegardsBaraa

1 0

Re: MonetDB: int128 - backing-out changeset 77c9c3da562c:
by Martin Kersten 08 Sep '13

08 Sep '13

On 9/8/13 8:45 PM, Stefan Manegold wrote: > Changeset: a331af9e20ea for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a331af9e20ea > Modified Files: > monetdb5/modules/mal/batcalc.mal > monetdb5/modules/mal/batcalc.mal.sh > Branch: int128 > Log Message: > > backing-out changeset 77c9c3da562c: > > Unfortunately, MonetDB / mserver5 / MAL parser / MAL interpreter > is/are unable to handle the new monetdb5/modules/mal/batcalc.mal > correctly. > > When starting the server, it complains about > !TypeException:jaqlfunc.count[7]:'batcalc.lng' undefined in: r:any := batcalc.lng(x:bat[:oid,:wrd]) > although the respective batcalc.lng(x:bat[:oid,:wrd]) > is in monetdb5/modules/mal/batcalc.mal . > Some tests indicate that the systems (MAL parser ?) > has some problems handling MAL files >= ~1 MB > correctly, but I did not manage to locate the > actual nature or origin of the problem. > > Any help to get this working is highly appreciated. Most likely it is in mal_import where the complete mal file is loaded first

1 0

Auto-vectorization
by ajonwa＠gmx.com 07 Sep '13

07 Sep '13

Hello All, Has anyone managed to auto-vectorize (automatic SIMDfication) any of the loops in the kernel? If yes, I would like to know the complier and the complier flags used. Thanks, Geoffrey

1 0

Re: MonetDB: default - add support for unix sockets
by Stefan Manegold 06 Sep '13

06 Sep '13

Hi Gijs, in case you did not notice this, yet: This checkin breaks several tests that do test or use the Python interface, resulting in errors like " Traceback (most recent call last): File "delete_all.SQL.py", line 9, in <module> dbh = monetdb.sql.Connection(port=port,database=db,autocommit=True) File "PREFIX/lib/python2.7/site-packages/monetdb/sql/connections.py", line 49, in __init__ unix_socket=unix_socket) File "PREFIX/lib/python2.7/site-packages/monetdb/mapi.py", line 96, in connect self.socket.connect(unix_socket) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 2] No such file or directory " or " Traceback (most recent call last): File "SOURCE/clients/examples/python/sqlsample.py", line 23, in <module> dbh = monetdb.sql.Connection(port=int(sys.argv[1]),database=sys.argv[2],autocommit=True) File "PREFIX/lib/python3.3/site-packages/monetdb/sql/connections.py", line 49, in __init__ unix_socket=unix_socket) File "PREFIX/lib/python3.3/site-packages/monetdb/mapi.py", line 97, in connect self.socket.connect(unix_socket) FileNotFoundError: [Errno 2] No such file or directory " See also, e.g., `Mtest.py sql/benchmarks/tpch/fileleak sql/test/concurrent sql/test/mapi` or our nightly TestWeb @ http://monetdb.cwi.nl/testweb/web/status.php?branch=default as of tomorrow morning. Best, Stefan On Thu, Sep 05, 2013 at 02:19:01PM +0200, Gijs Molenaar wrote: > Changeset: 0eaa07b061be for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0eaa07b061be > Modified Files: > clients/python2/monetdb/control.py > clients/python2/monetdb/mapi.py > clients/python2/monetdb/sql/connections.py > clients/python2/test/runtests.py > clients/python2/test/test_control.py > clients/python3/monetdb/control.py > clients/python3/monetdb/mapi.py > clients/python3/monetdb/sql/connections.py > clients/python3/test/runtests.py > clients/python3/test/test_control.py > Branch: default > Log Message: > > add support for unix sockets > > > diffs (truncated from 649 to 300 lines): > > diff --git a/clients/python2/monetdb/control.py b/clients/python2/monetdb/control.py > --- a/clients/python2/monetdb/control.py > +++ b/clients/python2/monetdb/control.py > @@ -8,10 +8,12 @@ def parse_statusline(line): > parses a sabdb format status line. Support v1 and v2. > > """ > - if not line.startswith('=sabdb:'): > + if line.startswith("="): > + line = line[1:] > + if not line.startswith('sabdb:'): > raise OperationalError('wrong result recieved') > > - prot_version, rest = line.split(":", 2)[1:] > + code, prot_version, rest = line.split(":", 2) > > if prot_version not in ["1", "2"]: > raise InterfaceError("unsupported sabdb protocol") > @@ -60,20 +62,26 @@ class Control: > Use this module to manage your MonetDB databases. You can create, start, > stop, lock, unlock, destroy your databases and request status information. > """ > - def __init__(self, hostname, port, passphrase): > + def __init__(self, hostname=None, port=None, passphrase=None, > + unix_socket="/tmp/.s.merovingian.50000"): > self.server = mapi.Connection() > self.hostname = hostname > self.port = port > self.passphrase = passphrase > + self.unix_socket= unix_socket > > # check connection > - self.server.connect(hostname, port, 'monetdb', passphrase, > - 'merovingian', 'control') > + self.server.connect(hostname=hostname, port=port, username='monetdb', > + password=passphrase, > + database='merovingian', language='control', > + unix_socket=unix_socket) > self.server.disconnect() > > def _send_command(self, database_name, command): > - self.server.connect(self.hostname, self.port, 'monetdb', > - self.passphrase, 'merovingian', 'control') > + self.server.connect(hostname=self.hostname, port=self.port, > + username='monetdb', password=self.passphrase, > + database='merovingian', language='control', > + unix_socket=self.unix_socket) > try: > return self.server.cmd("%s %s\n" % (database_name, command)) > finally: > @@ -165,8 +173,9 @@ class Control: > """ > properties = self._send_command(database_name, "get") > values = {} > - for dirty_line in properties.split("\n"): > - line = dirty_line[1:] > + for line in properties.split("\n"): > + if line.startswith("="): > + line = line[1:] > if not line.startswith("#"): > if "=" in line: > split = line.split("=") > diff --git a/clients/python2/monetdb/mapi.py b/clients/python2/monetdb/mapi.py > --- a/clients/python2/monetdb/mapi.py > +++ b/clients/python2/monetdb/mapi.py > @@ -70,8 +70,12 @@ class Connection(object): > self.database = "" > self.language = "" > > - def connect(self, hostname, port, username, password, database, language): > - """ setup connection to MAPI server""" > + def connect(self, database, username, password, language, hostname=None, > + port=None, unix_socket=None): > + """ setup connection to MAPI server > + > + unix_socket is used if hostname is not defined. > + """ > > self.hostname = hostname > self.port = port > @@ -79,24 +83,34 @@ class Connection(object): > self.password = password > self.database = database > self.language = language > + self.unix_socket = unix_socket > > - self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + if hostname: > + self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > + # For performance, mirror MonetDB/src/common/stream.c socket settings. > + self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 0) > + self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > + self.socket.connect((hostname, port)) > + else: > + self.socket = socket.socket(socket.AF_UNIX) > + self.socket.connect(unix_socket) > + if self.language != 'control': > + self.socket.send('0') # don't know why, but we need to do this > > - # For performance, mirror MonetDB/src/common/stream.c socket settings. > - self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 0) > - self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > + if not (self.language == 'control' and not self.hostname): > + # control doesn't require authentication over socket > + self._login() > > - self.socket.connect((hostname, port)) > - self.__login() > + self.state = STATE_READY > > - def __login(self, iteration=0): > + def _login(self, iteration=0): > """ Reads challenge from line, generate response and check if > everything is okay """ > > - challenge = self.__getblock() > - response = self.__challenge_response(challenge) > - self.__putblock(response) > - prompt = self.__getblock().strip() > + challenge = self._getblock() > + response = self._challenge_response(challenge) > + self._putblock(response) > + prompt = self._getblock().strip() > > if len(prompt) == 0: > # Empty response, server is happy > @@ -117,7 +131,7 @@ class Connection(object): > if redirect[1] == "merovingian": > logger.debug("restarting authentication") > if iteration <= 10: > - self.__login(iteration=iteration + 1) > + self._login(iteration=iteration + 1) > else: > raise OperationalError("maximal number of redirects " > "reached (10)") > @@ -138,9 +152,6 @@ class Connection(object): > else: > raise ProgrammingError("unknown state: %s" % prompt) > > - self.state = STATE_READY > - return True > - > def disconnect(self): > """ disconnect from the monetdb server """ > self.state = STATE_INIT > @@ -153,8 +164,8 @@ class Connection(object): > if self.state != STATE_READY: > raise(ProgrammingError, "Not connected") > > - self.__putblock(operation) > - response = self.__getblock() > + self._putblock(operation) > + response = self._getblock() > if not len(response): > return "" > elif response.startswith(MSG_OK): > @@ -166,10 +177,16 @@ class Connection(object): > return response > elif response[0] == MSG_ERROR: > raise OperationalError(response[1:]) > + elif (self.language == 'control' and not self.hostname): > + if response.startswith("OK"): > + return response[2:].strip() or "" > + else: > + return response > else: > raise ProgrammingError("unknown state: %s" % response) > > - def __challenge_response(self, challenge): > + > + def _challenge_response(self, challenge): > """ generate a response to a mapi login challenge """ > challenges = challenge.split(':') > salt, identity, protocol, hashes, endian = challenges[:5] > @@ -204,19 +221,36 @@ class Connection(object): > return ":".join(["BIG", self.username, pwhash, self.language, > self.database]) + ":" > > - def __getblock(self): > + def _getblock(self): > """ read one mapi encoded block """ > + if (self.language == 'control' and not self.hostname): > + return self._getblock_socket() # control doesn't do block > + # splitting when using a socket > + else: > + return self._getblock_inet() > + > + def _getblock_inet(self): > result = StringIO() > last = 0 > while not last: > - flag = self.__getbytes(2) > + flag = self._getbytes(2) > unpacked = struct.unpack('<H', flag)[0] # little endian short > length = unpacked >> 1 > last = unpacked & 1 > - result.write(self.__getbytes(length)) > + result.write(self._getbytes(length)) > return result.getvalue() > > - def __getbytes(self, bytes_): > + def _getblock_socket(self): > + buffer = StringIO() > + while True: > + x = self.socket.recv(1) > + if len(x): > + buffer.write(x) > + else: > + break > + return buffer.getvalue().strip() > + > + def _getbytes(self, bytes_): > """Read an amount of bytes from the socket""" > result = StringIO() > count = bytes_ > @@ -228,8 +262,15 @@ class Connection(object): > result.write(recv) > return result.getvalue() > > - def __putblock(self, block): > + def _putblock(self, block): > """ wrap the line in mapi format and put it into the socket """ > + if (self.language == 'control' and not self.hostname): > + return self.socket.send(block) # control doesn't do block > + # splitting when using a socket > + else: > + self._putblock_inet(block) > + > + def _putblock_inet(self, block): > pos = 0 > last = 0 > while not last: > diff --git a/clients/python2/monetdb/sql/connections.py b/clients/python2/monetdb/sql/connections.py > --- a/clients/python2/monetdb/sql/connections.py > +++ b/clients/python2/monetdb/sql/connections.py > @@ -28,25 +28,25 @@ class Connection(object): > """A MonetDB SQL database connection""" > default_cursor = cursors.Cursor > > - def __init__(self, username="monetdb", password="monetdb", > - hostname="localhost", port=50000, database="demo", > - autocommit=False, user=None, host=None): > + def __init__(self, database, hostname=None, port=50000, username="monetdb", > + password="monetdb", unix_socket="/tmp/.s.monetdb.50000", > + autocommit=False): > """ Set up a connection to a MonetDB SQL database. > > - username -- username for connection (default: monetdb) > - password -- password for connection (default: monetdb) > - hostname -- hostname to connect to (default: localhost) > - port -- port to connect to (default: 50000) > - database -- name of the database (default: demo) > - autocommit -- enable/disable auto commit (default: False) > + database -- name of the database > + hostname -- Hostname where monetDB is running > + port -- port to connect to (default: 50000) > + username -- username for connection (default: "monetdb") > + password -- password for connection (default: "monetdb") > + unix_socket -- socket to connect to. used when hostname not set > + (default: "/tmp/.s.monetdb.50000") > + autocommit -- enable/disable auto commit (default: False) > + > """ > - if user is not None: > - username = user > - if host is not None: > - hostname = host > self.mapi = mapi.Connection() > self.mapi.connect(hostname=hostname, port=int(port), username=username, > - password=password, database=database, language="sql") > + password=password, database=database, language="sql", > + unix_socket=unix_socket) > self.set_autocommit(autocommit) > self.set_sizeheader(True) > self.set_replysize(100) > diff --git a/clients/python2/test/runtests.py b/clients/python2/test/runtests.py > --- a/clients/python2/test/runtests.py > +++ b/clients/python2/test/runtests.py > @@ -45,6 +45,8 @@ TSTDB = os.environ.get('TSTDB', 'demo') > TSTHOSTNAME = os.environ.get('TSTHOSTNAME', 'localhost') > TSTUSERNAME = os.environ.get('TSTUSERNAME', 'monetdb') > TSTPASSWORD = os.environ.get('TSTPASSWORD', 'monetdb') > +#TSTHOSTNAME = None # set to none if test a socket > + > > if os.environ.get("TSTDEBUG", "no") == "yes": > logging.basicConfig(level=logging.DEBUG) > diff --git a/clients/python2/test/test_control.py b/clients/python2/test/test_control.py > --- a/clients/python2/test/test_control.py > +++ b/clients/python2/test/test_control.py > @@ -15,6 +15,7 @@ > # Copyright August 2008-2013 MonetDB B.V. > # All Rights Reserved. > > +import os > import unittest > import logging > > @@ -33,6 +34,10 @@ from monetdb.exceptions import Operation > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

3 10

Re: MonetDB: Feb2013 - fixed bug 3232, use dayofweek, starting at Mo...
by Sjoerd Mullender 27 Aug '13

27 Aug '13

This checkin is not allowed on the release branch: it changes the ABI by introducing a function MTIMEdate_extract_dayofworkweek. What's wrong with changing the implementation of MTIMEdate_extract_dayofweek to do what MTIMEdate_extract_dayofworkweek does now (and dropping MTIMEdate_extract_dayofworkweek)? On 2013-08-25 16:23, Niels Nes wrote: > Changeset: 0620e9df2003 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=0620e9df2003 > Added Files: > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.sql > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.stable.err > sql/test/BugTracker-2013/Tests/iso8601_weekofyear.Bug-3232.stable.out > Modified Files: > clients/mapiclient/mclient.c > monetdb5/modules/atoms/mtime.c > monetdb5/modules/atoms/mtime.h > monetdb5/modules/atoms/mtime.mal > sql/backends/monet5/sql.mx > sql/common/sql_types.c > sql/scripts/mtime.sql > sql/server/rel_select.c > sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > sql/test/BugTracker-2013/Tests/All > sql/test/leaks/Tests/check1.stable.out > sql/test/leaks/Tests/check2.stable.out > sql/test/leaks/Tests/check3.stable.out > sql/test/leaks/Tests/check4.stable.out > sql/test/leaks/Tests/check5.stable.out > sql/test/mapi/Tests/php_monetdb.stable.out > Branch: Feb2013 > Log Message: > > fixed bug 3232, use dayofweek, starting at Monday > fixed bug 2962, convert timestamp with time zone to local time zone, when used as input to the extract functions. > fixed bug 2781, use correct time zone in mclient > > > diffs (truncated from 761 to 300 lines): > > diff --git a/clients/mapiclient/mclient.c b/clients/mapiclient/mclient.c > --- a/clients/mapiclient/mclient.c > +++ b/clients/mapiclient/mclient.c > @@ -2559,6 +2559,7 @@ set_timezone(Mapi mid) > tmp = gmtime(&t); > gt = mktime(tmp); > tmp = localtime(&t); > + tmp->tm_isdst=0; /* We need the difference without dst */ > lt = mktime(tmp); > assert((lng) gt - (lng) lt >= (lng) INT_MIN && (lng) gt - (lng) lt <= (lng) INT_MAX); > tzone = (int) (gt - lt); > diff --git a/monetdb5/modules/atoms/mtime.c b/monetdb5/modules/atoms/mtime.c > --- a/monetdb5/modules/atoms/mtime.c > +++ b/monetdb5/modules/atoms/mtime.c > @@ -427,6 +427,7 @@ parse_substr(int *ret, str s, int min, s > return j; > } > > +/* Sunday = 1, Saturday = 7 */ > static int > date_dayofweek(date v) > { > @@ -434,6 +435,17 @@ date_dayofweek(date v) > return v <= 0 ? v + 7 : v; > } > > +/* Monday = 1, Sunday = 7 */ > +static int > +date_dayofworkweek(date v) > +{ > + int res = date_dayofweek(v); > + > + if (res >= 2) > + return res -1; > + return 7; > +} > + > #define SKIP_DAYS(d,w,i) d += i; w = (w + i)%7; if (w <= 0) w += 7; > > static date > @@ -1310,10 +1322,10 @@ date_extract_weekofyear(int *ret, date * > > fromdate((int) *v, &dummy, &dummy, &year); > year_jan_1 = todate(1, 1, year); > - dayofweek = date_dayofweek(year_jan_1); > + dayofweek = date_dayofworkweek(year_jan_1); > > if (dayofweek <= 4) { > - /* week of jan 1 belongs to this year */ > + /* 4 or more days in first week, ie week of jan 1 belongs to this year */ > *ret = (int) (1 + (*v - year_jan_1 + dayofweek - 1) / 7); > } else if (*v - year_jan_1 > 7 - dayofweek) { > /* week of jan 1 belongs to last year; but this is a later week */ > @@ -1340,6 +1352,18 @@ date_extract_dayofweek(int *ret, date *v > return MAL_SUCCEED; > } > > +/* Returns the current day of the week where 1=monday, .., 7=sunday */ > +static str > +date_extract_dayofworkweek(int *ret, date *v) > +{ > + if (*v == date_nil) { > + *ret = int_nil; > + } else { > + *ret = date_dayofworkweek(*v); > + } > + return MAL_SUCCEED; > +} > + > /* extracts hour from daytime (value between 0 and 23) */ > static str > daytime_extract_hours(int *ret, daytime *v) > @@ -1959,6 +1983,12 @@ MTIMEdate_extract_dayofweek(int *ret, da > } > > str > +MTIMEdate_extract_dayofworkweek(int *ret, date *v) > +{ > + return date_extract_dayofworkweek(ret, v); > +} > + > +str > MTIMEdaytime_extract_hours(int *ret, daytime *v) > { > return daytime_extract_hours(ret, v); > diff --git a/monetdb5/modules/atoms/mtime.h b/monetdb5/modules/atoms/mtime.h > --- a/monetdb5/modules/atoms/mtime.h > +++ b/monetdb5/modules/atoms/mtime.h > @@ -163,6 +163,7 @@ mtime_export str MTIMEdate_extract_day(i > mtime_export str MTIMEdate_extract_dayofyear(int *ret, date *v); > mtime_export str MTIMEdate_extract_weekofyear(int *ret, date *v); > mtime_export str MTIMEdate_extract_dayofweek(int *ret, date *v); > +mtime_export str MTIMEdate_extract_dayofworkweek(int *ret, date *v); > mtime_export str MTIMEdaytime_extract_hours(int *ret, daytime *v); > mtime_export str MTIMEdaytime_extract_minutes(int *ret, daytime *v); > mtime_export str MTIMEdaytime_extract_seconds(int *ret, daytime *v); > diff --git a/monetdb5/modules/atoms/mtime.mal b/monetdb5/modules/atoms/mtime.mal > --- a/monetdb5/modules/atoms/mtime.mal > +++ b/monetdb5/modules/atoms/mtime.mal > @@ -633,6 +633,10 @@ command dayofweek(d:date) :int > address MTIMEdate_extract_dayofweek > comment "Returns the current day of the week > where 1=sunday, .., 7=saturday"; > +command dayofworkweek(d:date) :int > +address MTIMEdate_extract_dayofworkweek > +comment "Returns the current day of the week > + where 1=monday, .., 7=sunday"; > > command add(v:timestamp, msecs:lng) :timestamp > address MTIMEtimestamp_add > diff --git a/sql/backends/monet5/sql.mx b/sql/backends/monet5/sql.mx > --- a/sql/backends/monet5/sql.mx > +++ b/sql/backends/monet5/sql.mx > @@ -714,6 +714,14 @@ comment "cast @1 to @1 and check for ove > @:mal_casttime(timestamp)@ > @:mal_casttime(daytime)@ > > +pattern current_time() :daytime > +address SQLcurrent_daytime > +comment "Get the clients current daytime"; > + > +pattern current_timestamp() :timestamp > +address SQLcurrent_timestamp > +comment "Get the clients current timestamp"; > + > command calc.date( v:void ) :date > address nil_2_date > comment "cast to date"; > @@ -6381,6 +6389,8 @@ sql5_export str second_interval_daytime( > sql5_export str second_interval_2_daytime( daytime *res, lng *s, int *d); > sql5_export str timestamp_2_daytime( daytime *res, timestamp *v, int *d); > sql5_export str date_2_timestamp( timestamp *res, date *v, int *d); > +str SQLcurrent_daytime(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > +str SQLcurrent_timestamp(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci); > @c > str > second_interval_daytime( lng *res, daytime *s, int *d, int *sk ) > @@ -6415,6 +6425,36 @@ second_interval_daytime( lng *res, dayti > return MAL_SUCCEED; > } > > +str SQLcurrent_daytime(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +{ > + mvc *m = NULL; > + str msg = getSQLContext(cntxt, mb, &m, NULL); > + daytime t, *res = (daytime *) getArgReference(stk, pci, 0); > + > + if (msg) > + return msg; > + > + if ((msg = MTIMEcurrent_time(&t)) == MAL_SUCCEED) > + *res = t+m->timezone; > + return msg; > +} > + > +str SQLcurrent_timestamp(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > +{ > + mvc *m = NULL; > + str msg = getSQLContext(cntxt, mb, &m, NULL); > + timestamp t, *res = (timestamp *) getArgReference(stk, pci, 0); > + > + if (msg) > + return msg; > + > + if ((msg = MTIMEcurrent_timestamp(&t)) == MAL_SUCCEED) { > + lng offset = m->timezone; > + return MTIMEtimestamp_add(res, &t, &offset); > + } > + return msg; > +} > + > /* str dump_cache(int *r); */ > str > dump_cache(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) > diff --git a/sql/common/sql_types.c b/sql/common/sql_types.c > --- a/sql/common/sql_types.c > +++ b/sql/common/sql_types.c > @@ -1457,8 +1457,8 @@ sqltypeinit( sql_allocator *sa) > sql_create_func(sa, "curtime", "mtime", "current_time", NULL, NULL, TMETZ, SCALE_NONE); > sql_create_func(sa, "current_time", "mtime", "current_time", NULL, NULL, TMETZ, SCALE_NONE); > sql_create_func(sa, "current_timestamp", "mtime", "current_timestamp", NULL, NULL, TMESTAMPTZ, SCALE_NONE); > - sql_create_func(sa, "localtime", "mtime", "current_time", NULL, NULL, TME, SCALE_NONE); > - sql_create_func(sa, "localtimestamp", "mtime", "current_timestamp", NULL, NULL, TMESTAMP, SCALE_NONE); > + sql_create_func(sa, "localtime", "sql", "current_time", NULL, NULL, TME, SCALE_NONE); > + sql_create_func(sa, "localtimestamp", "sql", "current_timestamp", NULL, NULL, TMESTAMP, SCALE_NONE); > > sql_create_func(sa, "sql_sub", "mtime", "date_sub_msec_interval", DTE, SECINT, DTE, SCALE_FIX); > sql_create_func(sa, "sql_sub", "mtime", "date_sub_month_interval", DTE, MONINT, DTE, SCALE_FIX); > @@ -1470,6 +1470,7 @@ sqltypeinit( sql_allocator *sa) > > sql_create_func(sa, "sql_sub", "mtime", "diff", DTE, DTE, INT, SCALE_FIX); > sql_create_func(sa, "sql_sub", "mtime", "diff", TMESTAMP, TMESTAMP, LNG, SCALE_FIX); > + sql_create_func(sa, "sql_sub", "mtime", "diff", TMESTAMPTZ, TMESTAMPTZ, LNG, SCALE_NONE); > > sql_create_func(sa, "sql_add", "mtime", "date_add_msec_interval", DTE, SECINT, DTE, SCALE_NONE); > sql_create_func(sa, "sql_add", "mtime", "addmonths", DTE, MONINT, DTE, SCALE_NONE); > @@ -1498,6 +1499,15 @@ sqltypeinit( sql_allocator *sa) > /* fix result type */ > f->res.scale = 3; > > + sql_create_func(sa, "year", "mtime", "year", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "month", "mtime", "month", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "day", "mtime", "day", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "hour", "mtime", "hours", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "minute", "mtime", "minutes", TMESTAMPTZ, NULL, INT, SCALE_FIX); > + f = sql_create_func(sa, "second", "mtime", "sql_seconds", TMESTAMPTZ, NULL, DEC, SCALE_NONE); > + /* fix result type */ > + f->res.scale = 3; > + > sql_create_func(sa, "year", "mtime", "year", MONINT, NULL, INT, SCALE_NONE); > sql_create_func(sa, "month", "mtime", "month", MONINT, NULL, INT, SCALE_NONE); > sql_create_func(sa, "day", "mtime", "day", SECINT, NULL, LNG, SCALE_NONE); > @@ -1507,7 +1517,7 @@ sqltypeinit( sql_allocator *sa) > > sql_create_func(sa, "dayofyear", "mtime", "dayofyear", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "weekofyear", "mtime", "weekofyear", DTE, NULL, INT, SCALE_FIX); > - sql_create_func(sa, "dayofweek", "mtime", "dayofweek", DTE, NULL, INT, SCALE_FIX); > + sql_create_func(sa, "dayofweek", "mtime", "dayofworkweek", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "dayofmonth", "mtime", "day", DTE, NULL, INT, SCALE_FIX); > sql_create_func(sa, "week", "mtime", "weekofyear", DTE, NULL, INT, SCALE_FIX); > > diff --git a/sql/scripts/mtime.sql b/sql/scripts/mtime.sql > --- a/sql/scripts/mtime.sql > +++ b/sql/scripts/mtime.sql > @@ -29,16 +29,16 @@ create function "current_timestamp"( ) r > external name mtime."current_timestamp"; > > create function "localtime"( ) returns TIME > - external name mtime."current_time"; > + external name sql."current_time"; > + > create function "localtimestamp"( ) returns TIMESTAMP > - external name mtime."current_timestamp"; > + external name sql."current_timestamp"; > > create type second_interval as interval day to sec; > > create function local_timezone( ) returns second_interval > external name mtime.local_timezone; > > - > create function sql_sub( date, second_interval ) returns date > external name mtime.date_sub_sec_interval; > create function sql_sub( date, month_interval ) returns date > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -3290,6 +3290,17 @@ rel_unop(mvc *sql, sql_rel **rel, symbol > f = sql_bind_func(sql->sa, s, fname, t, NULL, F_AGGR); > if (f && IS_AGGR(f->func)) > return _rel_aggr(sql, rel, 0, s, fname, l->next, fs); > + > + if (f && type_has_tz(t) && f->func->fix_scale == SCALE_FIX) { > + /* set timezone (using msec) */ > + sql_subtype *intsec = sql_bind_subtype(sql->sa, "sec_interval", 10 /*hour to second */, 0); > + sql_exp *tz = exp_atom_lng(sql->sa, sql->timezone); > + > + tz = exp_convert(sql->sa, tz, exp_subtype(tz), intsec); > + e = rel_binop_(sql, e, tz, NULL, "sql_add", ek.card); > + if (!e) > + return NULL; > + } > return rel_unop_(sql, e, s, fname, ek.card); > } > > diff --git a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > --- a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > +++ b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.err > @@ -30,9 +30,9 @@ stderr of test 'predicate_select.Bug-309 > # 18:53:13 > "mclient" "-lsql" "-ftest" "-Eutf-8" "-i" "-e" "--host=niels" "--port=32323" > # 18:53:13 > > > -MAPI = (monetdb) /var/tmp/mtest-16462/.s.monetdb.36324 > +MAPI = (monetdb) /var/tmp/mtest-4932/.s.monetdb.34449 > QUERY = select * from tables where name; > -ERROR = !conversion of string 'r' to type bit failed. > +ERROR = !conversion of string 'storagemodelinput' to type bit failed. > > # 18:53:13 > > # 18:53:13 > "Done." > diff --git a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > --- a/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > +++ b/sql/test/BugTracker-2012/Tests/predicate_select.Bug-3090.stable.out > @@ -47,19 +47,19 @@ Ready. > [ 2130, "idxs", 2102, NULL, 0, true, 2, false, 0 ] > [ 2135, "triggers", 2102, NULL, 0, true, 2, false, 0 ] > [ 2146, "objects", 2102, NULL, 0, true, 2, false, 0 ] > -[ 5060, "tables", 2000, "SELECT * FROM (SELECT p.*, 0 AS \"temporary\" FROM \"sys\".\"_tables\" AS p UNION ALL SELECT t.*, 1 AS \"temporary\" FROM \"tmp\".\"_tables\" AS t) AS tables where tables.type <> 2;", 1, true, 0, false, 0 ] > -[ 5070, "columns", 2000, "SELECT * FROM (SELECT p.* FROM \"sys\".\"_columns\" AS p UNION ALL SELECT t.* FROM \"tmp\".\"_columns\" AS t) AS columns;", 1, true, 0, false, 0 ] > -[ 5090, "db_user_info", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5098, "users", 2000, "SELECT u.\"name\" AS \"name\", ui.\"fullname\", ui.\"default_schema\" FROM db_users() AS u LEFT JOIN \"sys\".\"db_user_info\" AS ui ON u.\"name\" = ui.\"name\" ;", 1, true, 0, false, 0 ] > -[ 5102, "user_role", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5105, "auths", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5109, "privileges", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5262, "queryhistory", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5276, "callhistory", 2000, NULL, 0, true, 0, false, 0 ] > -[ 5293, "querylog", 2000, "create view querylog as\nselect qd.*, ql.ctime, ql.arguments, ql.exec, ql.result, ql.foot, ql.memory, ql.tuples, ql.inblock, ql.oublock from queryhistory qd, callhistory ql\nwhere qd.id = ql.id;", 1, true, 0, false, 0 ] > -[ 6059, "storagemodelinput", 2000, NULL, 0, true, 0, false, 0 ] > -[ 6100, "tablestoragemodel", 2000, "-- A summary of the table storage requirement is is available as a table view.\n-- The auxillary column denotes the maximum space if all non-sorted columns\n-- would be augmented with a hash (rare situation)\ncreate view sys.tablestoragemodel\nas select \"schema\",\"table\",max(count) as \"count\",\n\tsum(columnsize) as columnsize,\n\tsum(heapsize) as heapsize,\n\tsum(indices) as indices,\n\tsum(case when sorted = false then 8 * count else 0 end) as auxillary\nfrom sys.storagemodel() group by \"schema\",\"table\";", 1, true, 0, false, 0 ] > -[ 6122, "systemfunctions", 2000, NULL, 0, true, 0, false, 0 ] > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

Re: MonetDB: Feb2013 - Organize parallelism per worker pool
by Stefan Manegold 21 Aug '13

21 Aug '13

Martin, could you please also add sql/test/BugTracker-2013/Tests/nestedcalls.sql ? And could you plase also add the test scripts and stable output for tests sql/test/BugTracker-2013/Tests/oid_handling sql/test/BugTracker-2013/Tests/constraint_checking.Bug_3335 sql/test/BugTracker-2013/Tests/pivot.Bug-3339 sql/test/BugTracker-2013/Tests/recursion or alternatively remove them from sql/test/BugTracker-2013/Tests/All Thanks! In any case, propagation of this will checking will result in (expected/unavoidable) conflicts in monetdb5/mal/mal_dataflow.[ch] and (avoidable) conflicts in sql/test/BugTracker-2013/Tests All of these will need to be solved by hand ... Stefan ----- Original Message ----- > Changeset: 489815265a61 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=489815265a61 > Added Files: > sql/test/BugTracker-2013/Tests/nestedcalls.stable.err > sql/test/BugTracker-2013/Tests/nestedcalls.stable.out > Modified Files: > monetdb5/mal/mal_dataflow.c > monetdb5/mal/mal_dataflow.h > monetdb5/modules/mal/language.c > monetdb5/modules/mal/language.h > sql/test/BugTracker-2013/Tests/All > Branch: Feb2013 > Log Message: > > Organize parallelism per worker pool > Each (recursive) dataflow block is handed a worker pool now. > Once we run out of worker pools or fail to create a worker, > we continue in sequential mode. > > Cleaning up the threads is implicit, like all other non-pool > interpreters that might be active at system shutdown. > > > diffs (truncated from 762 to 300 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -35,6 +35,7 @@ > * The flow graphs should be organized such that parallel threads can > * access it mostly without expensive locking. > */ > +#include "monetdb_config.h" > #include "mal_dataflow.h" > #include "mal_client.h" > > @@ -82,10 +83,10 @@ typedef struct DATAFLOW { > Queue *done; /* instructions handled */ > } *DataFlow, DataFlowRec; > > -#define MAXQ 1024 > -static MT_Id workers[THREADS] = {0}; > -static int workerqueue[THREADS] = {0}; /* maps workers towards the todo > queues */ > -static Queue *todo[MAXQ] = {0}; /* pending instructions organized by user > MAXTODO > #users */ > +#define MAXQ 256 > +static Queue *todos[MAXQ] = {0}; /* pending instructions organized by > dataflow block */ > +static bit occupied[MAXQ]={0}; /* worker pool is in use? */ > +static int volatile exiting = 0; > > /* > * Calculate the size of the dataflow dependency graph. > @@ -108,9 +109,8 @@ DFLOWgraphSize(MalBlkPtr mb, int start, > */ > > static Queue* > -q_create(int sz) > +q_create(int sz, const char *name) > { > - const char* name = "q_create"; > Queue *q = (Queue*)GDKmalloc(sizeof(Queue)); > > if (q == NULL) > @@ -208,6 +208,8 @@ q_dequeue(Queue *q) > > assert(q); > MT_sema_down(&q->s, "q_dequeue"); > + if (exiting) > + return NULL; > MT_lock_set(&q->l, "q_dequeue"); > assert(q->last > 0); > if (q->last > 0) { > @@ -255,25 +257,23 @@ DFLOWworker(void *t) > { > DataFlow flow; > FlowEvent fe = 0, fnxt = 0; > - int id = (int) ((MT_Id *) t - workers), last = 0; > - int wq; > Thread thr; > str error = 0; > - > - int i; > - lng usec = 0; > + Queue *todo = *(Queue **) t; > + int i,last; > > thr = THRnew("DFLOWworker"); > > GDKsetbuf(GDKmalloc(GDKMAXERRLEN)); /* where to leave errors */ > GDKerrbuf[0] = 0; > while (1) { > - assert(workerqueue[id] > 0); > - wq = workerqueue[id] - 1; > if (fnxt == 0) > - fe = q_dequeue(todo[wq]); > + fe = q_dequeue(todo); > else > fe = fnxt; > + if (exiting) { > + break; > + } > fnxt = 0; > assert(fe); > flow = fe->flow; > @@ -285,22 +285,20 @@ DFLOWworker(void *t) > continue; > } > > - usec = GDKusec(); > /* skip all instructions when we have encontered an error */ > if (flow->error == 0) { > #ifdef USE_MAL_ADMISSION > if (MALadmission(fe->argclaim, fe->hotclaim)) { > fe->hotclaim = 0; /* don't assume priority anymore */ > - assert(todo[wq]); > - if (todo[wq]->last == 0) > + if (todo->last == 0) > MT_sleep_ms(DELAYUNIT); > - q_requeue(todo[wq], fe); > + q_requeue(todo, fe); > continue; > } > #endif > error = runMALsequence(flow->cntxt, flow->mb, fe->pc, fe->pc + 1, > flow->stk, 0, 0); > PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= %d claim= " LLFMT > "," LLFMT " %s\n", > - fe->pc, id, fe->argclaim, fe->hotclaim, error ? error : ""); > + fe->pc, (int)((Queue **)t - todos), fe->argclaim, fe->hotclaim, > error ? error : ""); > #ifdef USE_MAL_ADMISSION > /* release the memory claim */ > MALadmission(-fe->argclaim, -fe->hotclaim); > @@ -331,8 +329,8 @@ DFLOWworker(void *t) > InstrPtr p = getInstrPtr(flow->mb, fe->pc); > assert(p); > fe->hotclaim = 0; > - for (i = 0; i < p->retc; i++) > - fe->hotclaim += getMemoryClaim(flow->mb, flow->stk, fe->pc, i, FALSE); > + //for (i = 0; i < p->retc; i++) > + //fe->hotclaim += getMemoryClaim(flow->mb, flow->stk, p, i, FALSE); > } > #endif > MT_lock_set(&flow->flowlock, "MALworker"); > @@ -351,56 +349,64 @@ DFLOWworker(void *t) > > q_enqueue(flow->done, fe); > if ( fnxt == 0) { > - assert(todo[wq]); > - if (todo[wq]->last == 0) > + if (todo->last == 0) > profilerHeartbeatEvent("wait"); > - else > - MALresourceFairness(NULL, NULL, usec); > } > } > GDKfree(GDKerrbuf); > GDKsetbuf(0); > - workerqueue[wq] = 0; > - workers[wq] = 0; > THRdel(thr); > } > > /* > - * Create a set of DFLOW interpreters. > + * Create an interpreter pool. > * One worker will adaptively be available for each client. > * The remainder are taken from the GDKnr_threads argument and > - * typically is equal to the number of cores > + * typically is equal to the number of cores. > + * A recursive MAL function call would make for one worker less, > + * which limits the number of cores for parallel processing. > * The workers are assembled in a local table to enable debugging. > + * > + * BEWARE, failure to create a new worker thread is not an error > + * but would lead to serial execution. > */ > -static void > -DFLOWinitialize(int index) > +static int > +DFLOWinitialize(void) > { > - int i, worker, limit; > + int i, threads, grp; > + MT_Id worker; > > - assert(index >= 0); > - assert(index < THREADS); > + threads = GDKnr_threads ? GDKnr_threads : 1; > MT_lock_set(&mal_contextLock, "DFLOWinitialize"); > - if (todo[index]) { > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > - return; > + for(grp = 0; grp< MAXQ; grp++) > + if ( occupied[grp] == FALSE){ > + occupied[grp] = TRUE; > + break; > + } > + MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > + if (grp > THREADS) { > + // continue non-parallel > + return -1; > } > - todo[index] = q_create(2048); > - assert(todo[index]); > - limit = GDKnr_threads ? GDKnr_threads : 1; > - assert(limit <= THREADS); > - for (worker = 0, i = 0; i < limit; i++){ > - for (; worker < THREADS; worker++) > - if( workers[worker] == 0) > - break; > - assert(worker < THREADS); > - if (worker < THREADS) { > - assert(workers[worker] == 0); > - MT_create_thread(&workers[worker], DFLOWworker, (void *) > &workers[worker], MT_THR_JOINABLE); > - assert(workers[worker] > 0); > - workerqueue[worker] = index + 1; > + if ( todos[grp] ) > + return grp; > + > + todos[grp] = q_create(2048, "todo"); > + if (todos[grp] == NULL) > + return -1; > + > + // associate a set of workers with the pool > + for (i = 0; grp>= 0 && i < threads; i++){ > + if (MT_create_thread(&worker, DFLOWworker, (void *) &todos[grp], > MT_THR_JOINABLE) < 0) { > + //Can not create interpreter thread > + grp = -1; > + } > + if (worker == 0) { > + //Failed to create interpreter thread > + grp = -1; > } > } > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > + return grp; > } > > /* > @@ -409,18 +415,28 @@ DFLOWinitialize(int index) > * For each instruction we keep a list of instructions whose > * blocking counter should be decremented upon finishing it. > */ > -static void > +static str > DFLOWinitBlk(DataFlow flow, MalBlkPtr mb, int size) > { > int pc, i, j, k, l, n, etop = 0; > int *assign; > InstrPtr p; > > + if (flow == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Called with flow == NULL"); > + if (mb == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Called with mb == NULL"); > PARDEBUG printf("Initialize dflow block\n"); > assign = (int *) GDKzalloc(mb->vtop * sizeof(int)); > + if (assign == NULL) > + throw(MAL, "dataflow", "DFLOWinitBlk(): Failed to allocate assign"); > etop = flow->stop - flow->start; > for (n = 0, pc = flow->start; pc < flow->stop; pc++, n++) { > p = getInstrPtr(mb, pc); > + if (p == NULL) { > + GDKfree(assign); > + throw(MAL, "dataflow", "DFLOWinitBlk(): getInstrPtr() returned NULL"); > + } > > /* initial state, ie everything can run */ > flow->status[n].flow = flow; > @@ -501,6 +517,7 @@ DFLOWinitBlk(DataFlow flow, MalBlkPtr mb > #ifdef USE_MAL_ADMISSION > memorypool = memoryclaims = 0; > #endif > + return MAL_SUCCEED; > } > > /* > @@ -528,18 +545,17 @@ static void showFlowEvent(DataFlow flow, > */ > > static str > -DFLOWscheduler(DataFlow flow) > +DFLOWscheduler(DataFlow flow, Queue *todo) > { > int last; > int i; > #ifdef USE_MAL_ADMISSION > - int j; > + //int j; > InstrPtr p; > #endif > int tasks=0, actions; > str ret = MAL_SUCCEED; > FlowEvent fe, f = 0; > - int wq; > > if (flow == NULL) > throw(MAL, "dataflow", "DFLOWscheduler(): Called with flow == NULL"); > @@ -549,19 +565,19 @@ DFLOWscheduler(DataFlow flow) > /* initialize the eligible statements */ > fe = flow->status; > > - if (fe[0].flow->cntxt->flags & timerFlag) > - fe[0].flow->cntxt->timer = GDKusec(); > - > MT_lock_set(&flow->flowlock, "MALworker"); > - wq = flow->cntxt->idx; > for (i = 0; i < actions; i++) > if (fe[i].blocks == 0) { > #ifdef USE_MAL_ADMISSION > p = getInstrPtr(flow->mb,fe[i].pc); > - for (j = p->retc; j < p->argc; j++) > - fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, > fe[i].pc, j, FALSE); > + if (p == NULL) { > + MT_lock_unset(&flow->flowlock, "MALworker"); > + throw(MAL, "dataflow", "DFLOWscheduler(): getInstrPtr(flow->mb,fe[i].pc) > returned NULL"); > + } > + //for (j = p->retc; j < p->argc; j++) > + //fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, p, j, > FALSE); > #endif > - q_enqueue(todo[wq], flow->status + i); > + q_enqueue(todo, flow->status + i); > flow->status[i].state = DFLOWrunning; > PARDEBUG mnstr_printf(GDKstdout, "#enqueue pc=%d claim=" LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } > @@ -571,6 +587,10 @@ DFLOWscheduler(DataFlow flow) > > while (actions != tasks ) { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 2

default vs. Feb2013: opt_evaluate.c & mal_dataflow.c
by Stefan Manegold 19 Aug '13

19 Aug '13

Martin, I just merged recent changes from Feb2013 into default, and while doing so resolved --- hopefully correctly; see below --- some conflicts that occurred in opt_evaluate.c & mal_dataflow.c . In fact, (1) I first merged up until before your changeset that introduced the new multiple worker pools, cf., http://dev.monetdb.org/hg/MonetDB/rev/10125a9c3976 (2) then merged only that very changeset, cf., http://dev.monetdb.org/hg/MonetDB/rev/1eea7fa4cc7c (3) and finally merge the rest, cf., http://dev.monetdb.org/hg/MonetDB/rev/3e3d7d5f8dc9 During (1), I resolved some conflicts in opt_evaluate.c due to http://dev.monetdb.org/hg/MonetDB/rev/62ed11377998 I hope, I did this correctly. However, there are still some differences in opt_evaluate.c between Feb2013 & default; cf., hg diff -p -r Feb2013 -r default monetdb5/optimizer/opt_evaluate.c Could you please check, whether these are OK? During (2), I resolved some conflicts in mal_dataflow.c due to http://dev.monetdb.org/hg/MonetDB/rev/4c47797192f4 I hope, I did this correctly. However, there are still some differences in mal_dataflow.c between Feb2013 & default; cf., hg diff -p -r Feb2013 -r default monetdb5/mal/mal_dataflow.c Could you please check, whether these are OK? It appears, that after (2), some tests in sql/jdbc/tests/Tests/ run into a timeout (only when run multi-threaded, i.e., not when run single-threaded) --- I have not yet been able to find out why ... Thanks! Stefan -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Re: MonetDB: default - Report correct 100% sample size
by Niels Nes 16 Aug '13

16 Aug '13

On Fri, Aug 16, 2013 at 09:56:49PM +0200, Martin Kersten wrote: > Changeset: afa6da5dd7bd for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=afa6da5dd7bd > Modified Files: > sql/backends/monet5/sql_statistics.c > Branch: default > Log Message: > > Report correct 100% sample size > > > diffs (12 lines): > > diff --git a/sql/backends/monet5/sql_statistics.c b/sql/backends/monet5/sql_statistics.c > --- a/sql/backends/monet5/sql_statistics.c > +++ b/sql/backends/monet5/sql_statistics.c > @@ -169,7 +169,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > } > snprintf(query,8192, > "insert into sys.statistics values('%s','%s','%s','%s',%d,now(),"LLFMT","LLFMT","LLFMT","LLFMT",'%s','%s',%s);", > - b->name, bt->name, bc->name, c->type.type->sqlname,width, samplesize, sz, uniq, nils, minval,maxval, sorted?"true":"false"); > + b->name, bt->name, bc->name, c->type.type->sqlname,width, (samplesize?samplesize:sz), sz, uniq, nils, minval,maxval, sorted?"true":"false"); > #ifdef DEBUG_SQL_STATISTICS > mnstr_printf(cntxt->fdout,"%s\n",dquery); > mnstr_printf(cntxt->fdout,"%s\n",query); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list Martin Volgens mij wil je de volgende verzameling van analyze calls: analyze([opt_sample_size]) -- full db analyze(schema, [opt_sample_size]) -- full schema analyze(schema, table, [opt_sample_size]) -- full table analyze(schema, table, column, [opt_sample_size]) -- full table ie dus niet analyze(table, ...) voor die is het onduidelijk welk schema. Niels -- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl

1 0

Re: MonetDB: Feb2013 - New thread group per dataflow block
by Stefan Manegold 16 Aug '13

16 Aug '13

Martin, this checkins appears to make several tests run into timeouts; I have not yet been able to analyze why ... A complete Mtest.py run results in * 142 out of 1863 tests could not be executed 1 out of 1863 tests produced slightly different output sql/test/BugTracker-2012/predicate_select.Bug-3090 * 6 out of 1863 tests ran into timeout * sql/benchmarks/ATIS/load * sql/benchmarks/arno/insert_ATOM * sql/benchmarks/arno/insert_BOND * sql/benchmarks/arno_flt/init * sql/benchmarks/moa/load * sql/benchmarks/wisconsin/load 1 out of 1863 tests caused an abort (assertion failure) monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667 7 out of 1863 tests produced SIGNIFICANTLY different output monetdb5/optimizer/inlineFunction monetdb5/optimizer/ifthencst sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073 sql/test/BugTracker-2012/create_function.Bug-3172 sql/test/BugTracker-2012/currenttime.Bug-2781 sql/test/BugTracker-2012/null_except_null.Bug-3040 sql/test/BugTracker-2012/day-of-month-localization.Bug-2962 while with the version before this checkin, it results in 41 out of 1863 tests could not be executed 1 out of 1863 tests produced slightly different output sql/test/BugTracker-2012/predicate_select.Bug-3090 1 out of 1863 tests caused an abort (assertion failure) monetdb5/tests/BugTracker/kunion-and-nil.Bug-1667 7 out of 1863 tests produced SIGNIFICANTLY different output monetdb5/optimizer/inlineFunction monetdb5/optimizer/ifthencst sql/test/BugTracker-2012/conditions_when_for_triggers_do_not_work.Bug-2073 sql/test/BugTracker-2012/create_function.Bug-3172 sql/test/BugTracker-2012/currenttime.Bug-2781 sql/test/BugTracker-2012/null_except_null.Bug-3040 sql/test/BugTracker-2012/day-of-month-localization.Bug-2962 Best, Stefan ----- Original Message ----- > Changeset: 3669ddd28bf0 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=3669ddd28bf0 > Modified Files: > monetdb5/mal/mal_dataflow.c > Branch: Feb2013 > Log Message: > > New thread group per dataflow block > The centralized worker thread group could lead to an unacceptable > situation. If a user is heavily processing complex queries, then > no other user could even log into the system, for its MAL statements > ended up at the end of the shared queues. > > The problem has been resolved by introducing a thread group per dataflow > block. This may lead to a large number of processes, whose resources > are managed by the OS. > > It solves bug 3258 > > > diffs (235 lines): > > diff --git a/monetdb5/mal/mal_dataflow.c b/monetdb5/mal/mal_dataflow.c > --- a/monetdb5/mal/mal_dataflow.c > +++ b/monetdb5/mal/mal_dataflow.c > @@ -78,12 +78,12 @@ typedef struct DATAFLOW { > int *nodes; /* dependency graph nodes */ > int *edges; /* dependency graph */ > MT_Lock flowlock; /* lock to protect the above */ > + queue *todo; /* pending instructions */ > queue *done; /* instructions handled */ > + int threads; /* worker threads active */ > + MT_Id workers[THREADS]; > } *DataFlow, DataFlowRec; > > -static MT_Id workers[THREADS]; > -static queue *todo = 0; /* pending instructions */ > - > /* > * Calculate the size of the dataflow dependency graph. > */ > @@ -138,7 +138,6 @@ q_destroy(queue *q) > static void > q_enqueue_(queue *q, FlowEvent d) > { > - assert(d); > if (q->last == q->size) { > q->size <<= 1; > q->data = GDKrealloc(q->data, sizeof(FlowEvent) * q->size); > @@ -214,7 +213,6 @@ q_dequeue(queue *q) > */ > > MT_lock_unset(&q->l, "q_dequeue"); > - assert(r); > return r; > } > > @@ -239,14 +237,15 @@ q_dequeue(queue *q) > static void > DFLOWworker(void *t) > { > - DataFlow flow; > + DataFlow flow = (DataFlow) t; > FlowEvent fe = 0, fnxt = 0; > - int id = (int) ((MT_Id *) t - workers), last = 0; > + MT_Id id = MT_getpid(); > + int last = 0; > Thread thr; > str error = 0; > > int i; > - lng usec = 0; > + //lng usec = 0; > > thr = THRnew("DFLOWworker"); > > @@ -254,8 +253,10 @@ DFLOWworker(void *t) > GDKerrbuf[0] = 0; > while (1) { > if (fnxt == 0) > - fe = q_dequeue(todo); > + fe = q_dequeue(flow->todo); > else fe = fnxt; > + if ( fe == 0) > + break; > fnxt = 0; > assert(fe); > flow = fe->flow; > @@ -266,20 +267,20 @@ DFLOWworker(void *t) > continue; > } > > - usec = GDKusec(); > + //usec = GDKusec(); > /* skip all instructions when we have encontered an error */ > if (flow->error == 0) { > #ifdef USE_MAL_ADMISSION > if (MALadmission(fe->argclaim, fe->hotclaim)) { > fe->hotclaim = 0; /* don't assume priority anymore */ > - if (todo->last == 0) > + if (flow->todo->last == 0) > MT_sleep_ms(DELAYUNIT); > - q_requeue(todo, fe); > + q_requeue(flow->todo, fe); > continue; > } > #endif > error = runMALsequence(flow->cntxt, flow->mb, fe->pc, fe->pc + 1, > flow->stk, 0, 0); > - PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= %d claim= " LLFMT > "," LLFMT " %s\n", > + PARDEBUG mnstr_printf(GDKstdout, "#executed pc= %d wrk= "SZFMT" claim= " > LLFMT "," LLFMT " %s\n", > fe->pc, id, fe->argclaim, fe->hotclaim, error ? error : ""); > #ifdef USE_MAL_ADMISSION > /* release the memory claim */ > @@ -330,12 +331,15 @@ DFLOWworker(void *t) > > q_enqueue(flow->done, fe); > if ( fnxt == 0) { > - if (todo->last == 0) > + if (flow->todo->last == 0) > profilerHeartbeatEvent("wait"); > - else > - MALresourceFairness(NULL, NULL, usec); > + //else > + //MALresourceFairness(NULL, NULL, usec); > } > } > + for( i = 0; i< flow->threads; i++) > + if ( flow->workers[i] == id) > + flow->workers[i] = 0; > GDKfree(GDKerrbuf); > GDKsetbuf(0); > THRdel(thr); > @@ -349,22 +353,51 @@ DFLOWworker(void *t) > * The workers are assembled in a local table to enable debugging. > */ > static void > -DFLOWinitialize(void) > +DFLOWinitialize(DataFlow flow, int size) > { > - int i, limit; > + int i; > > - MT_lock_set(&mal_contextLock, "DFLOWinitialize"); > - if (todo) { > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > - return; > + MT_lock_init(&flow->flowlock, "DFLOWworker"); > + flow->todo = q_create(size); > + flow->done = q_create(size); > + flow->threads = GDKnr_threads ? GDKnr_threads :1; > + for (i = 0; i < flow->threads; i++){ > + MT_create_thread(&flow->workers[i], DFLOWworker, (void *) flow, > MT_THR_JOINABLE); > + /* upon failure of starting threads we reduce the count */ > + if ( flow->workers[i]== 0){ > + flow->threads --; > + i--; > + } > } > - todo = q_create(2048); > - limit = GDKnr_threads ? GDKnr_threads : 1; > - for (i = 0; i < limit; i++) > - MT_create_thread(&workers[i], DFLOWworker, (void *) &workers[i], > MT_THR_JOINABLE); > - MT_lock_unset(&mal_contextLock, "DFLOWinitialize"); > } > > +static str > +DFLOWfinalize(DataFlow flow) > +{ > + int i, cnt= flow->threads, runs =0; > + > + for( i = 0; i< cnt; i++) > + q_enqueue(flow->todo, 0); > + /* time out when threads are already killed */ > + do{ > + runs++; > + cnt = 0; > + MT_sleep_ms(1); > + for( i = 0; i < flow->threads; i++) > + cnt += flow->workers[i] ==0; > + } while( cnt != flow->threads && runs <5000); > + > + if ( runs == 5000) > + throw(MAL,"dataflow","Timeout on thread termination"); > + GDKfree(flow->status); > + GDKfree(flow->edges); > + GDKfree(flow->nodes); > + q_destroy(flow->done); > + q_destroy(flow->todo); > + MT_lock_destroy(&flow->flowlock); > + GDKfree(flow); > + return MAL_SUCCEED; > +} > /* > * The dataflow administration is based on administration of > * how many variables are still missing before it can be executed. > @@ -518,7 +551,7 @@ DFLOWscheduler(DataFlow flow) > for (j = p->retc; j < p->argc; j++) > fe[i].argclaim = getMemoryClaim(fe[0].flow->mb, fe[0].flow->stk, > fe[i].pc, j, FALSE); > #endif > - q_enqueue(todo, flow->status + i); > + q_enqueue(flow->todo, flow->status + i); > flow->status[i].state = DFLOWrunning; > PARDEBUG mnstr_printf(GDKstdout, "#enqueue pc=%d claim=" LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } > @@ -543,7 +576,7 @@ DFLOWscheduler(DataFlow flow) > if (flow->status[i].blocks == 1 ) { > flow->status[i].state = DFLOWrunning; > flow->status[i].blocks--; > - q_enqueue(todo, flow->status + i); > + q_enqueue(flow->todo, flow->status + i); > PARDEBUG > mnstr_printf(GDKstdout, "#enqueue pc=%d claim= " LLFMT "\n", > flow->status[i].pc, flow->status[i].argclaim); > } else { > @@ -579,13 +612,11 @@ runMALdataflow(Client cntxt, MalBlkPtr m > > assert(stoppc > startpc); > > - /* check existence of workers */ > - if (workers[0] == 0) > - DFLOWinitialize(); > - assert(workers[0]); > - assert(todo); > - > flow = (DataFlow)GDKzalloc(sizeof(DataFlowRec)); > + > + DFLOWinitialize(flow, stoppc- startpc +1); > + assert(flow->todo); > + assert(flow->done); > > flow->cntxt = cntxt; > flow->mb = mb; > @@ -596,9 +627,6 @@ runMALdataflow(Client cntxt, MalBlkPtr m > flow->start = startpc + 1; > flow->stop = stoppc; > > - MT_lock_init(&flow->flowlock, "DFLOWworker"); > - flow->done = q_create(stoppc- startpc+1); > - > flow->status = (FlowEvent)GDKzalloc((stoppc - startpc + 1) * > sizeof(FlowEventRec)); > size = DFLOWgraphSize(mb, startpc, stoppc); > size += stoppc - startpc; > @@ -608,11 +636,9 @@ runMALdataflow(Client cntxt, MalBlkPtr m > > ret = DFLOWscheduler(flow); > > - GDKfree(flow->status); > - GDKfree(flow->edges); > - GDKfree(flow->nodes); > - q_destroy(flow->done); > - MT_lock_destroy(&flow->flowlock); > - GDKfree(flow); > + if( ret == MAL_SUCCEED) > + ret = DFLOWfinalize(flow); > + else (void) > + DFLOWfinalize(flow); > return ret; > } > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0