- developers-list - monetdb.org

Fwd: lib_pyapi complains about undefined symbol PyInt_Type
by Dean De Leo 06 Oct '16

06 Oct '16

Hi, on my personal machine, when starting mserver5, monetdb complains with the following message: !SyntaxException:parseError: !LoaderException:loadLibrary:Loading error failed to open library pyapi (from within file '/linux/workspace/monetdb/build/lib/monetdb5/lib_pyapi.so'): /linux/workspace/monetdb/build/lib/monetdb5/lib_pyapi.so: undefined symbol: PyInt_Type !SyntaxException:parseError: !<identifier> expected I guess this is another case of conflict Python 2 vs 3. I am using ArchLinux where the default is v3. In my case a workaround seems to be forcing back python2 (see patch). Dean

2 1

GDK_mmap_minsize again
by Hannes Mühleisen 24 Sep '16

24 Sep '16

Hello list, we were wondering about the purpose of GDK_mmap_minsize when creating transient columns. The attached patch will always *try* to malloc/realloc a transient column but still fall back to memory-mapped files if malloc should fail. This dramatically improves performance. Any good reason why this should not be the default behaviour? Thanks, Mark and Hannes

5 5

eviction in Monetdb
by Narendra Pradeep 15 Sep '16

15 Sep '16

Hello , Need to know about monetdb eviction policy ? My understanding is Monetdb creates a buffer pool {BBP} and inserts all the bats that are used . But not sure when does Monetdb deletes the bat from the buffer pool and what eviction policy does it use (like LRU or based on some threshold) ? Thanks

2 1

Re: MonetDB: graph1 - Add the operator bat.getCount() to retrieve th...
by Stefan Manegold 14 Sep '16

14 Sep '16

Hi Dean, how is bat.getCount() different from aggr.count() ? ;-) Thanks! Stefan ----- On Sep 14, 2016, at 6:30 PM, Dean De Leo commits+dleo=cwi.nl(a)monetdb.org wrote: > Changeset: 9ac831deacb3 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9ac831deacb3 > Modified Files: > monetdb5/modules/kernel/bat5.c > monetdb5/modules/kernel/bat5.h > monetdb5/modules/kernel/bat5.mal > Branch: graph1 > Log Message: > > Add the operator bat.getCount() to retrieve the cardinality of a bat > > > diffs (48 lines): > > diff --git a/monetdb5/modules/kernel/bat5.c b/monetdb5/modules/kernel/bat5.c > --- a/monetdb5/modules/kernel/bat5.c > +++ b/monetdb5/modules/kernel/bat5.c > @@ -496,6 +496,19 @@ BKCgetColumnType(str *res, const bat *bi > } > > str > +BKCgetCount(lng *res, const bat *bid) > +{ > + *res = lng_nil; > + if (BBPcheck(*bid, "bat.getCount")) { > + BAT *b = BBPquickdesc(*bid, 0); > + > + if (b != NULL) > + *res = (lng) BATcount(b); > + } > + return MAL_SUCCEED; > +} > + > +str > BKCgetRole(str *res, const bat *bid) > { > BAT *b; > diff --git a/monetdb5/modules/kernel/bat5.h b/monetdb5/modules/kernel/bat5.h > --- a/monetdb5/modules/kernel/bat5.h > +++ b/monetdb5/modules/kernel/bat5.h > @@ -29,6 +29,7 @@ mal_export str BKCbun_inplace_force(bat > mal_export str BKCbat_inplace_force(bat *r, const bat *bid, const bat *rid, > const bat *uid, const bit *force); > mal_export str BKCgetCapacity(lng *res, const bat *bid); > mal_export str BKCgetColumnType(str *res, const bat *bid); > +mal_export str BKCgetCount(lng *res, const bat *bid); > mal_export str BKCgetRole(str *res, const bat *bid); > mal_export str BKCsetkey(bat *res, const bat *bid, const bit *param); > mal_export str BKCisSorted(bit *res, const bat *bid); > diff --git a/monetdb5/modules/kernel/bat5.mal b/monetdb5/modules/kernel/bat5.mal > --- a/monetdb5/modules/kernel/bat5.mal > +++ b/monetdb5/modules/kernel/bat5.mal > @@ -77,6 +77,10 @@ command getCapacity(b:bat[:any_1]):lng > address BKCgetCapacity > comment "Returns the current allocation size (in max number of elements) of a > BAT."; > > +command getCount(b:bat[:any_1]):lng > +address BKCgetCount > +comment "Returns the current tuple count of a BAT."; > + > command getColumnType( b:bat[:any_1] ) :str > address BKCgetColumnType > comment "Returns the type of the tail column of a BAT, as an integer type > number."; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

BAT in memory
by Narendra Pradeep 30 Aug '16

30 Aug '16

Hello , I am reading monetdb source code . I need to know which function check whether the bat of the particular column lies in memory or not ? And how does it maps the bat with corresponding column ? Thanks . Regars , Naren

1 0

Monetdb printing values of variables in console or terminal
by Narendra Pradeep 23 Aug '16

23 Aug '16

Hai everyone , How to print messages in console or to write something in log file of the Monetdb . can someone help me with exact function ? I cant even write a word in logfile and console is there something with my build ? Regards , Pradeep

1 0

compression in monetdb
by Narendra Pradeep 11 Aug '16

11 Aug '16

Hai , Read that dictionary encoding occurs in monetdb . so 1) what is dictionary compression technique ? 2) Does it works for strings alone or other data structures too ? 3) I a mailing list it was stated that it works only till the size of the column in 128 MB for older release? Is this the same for current release ? 4) Does Monetdb uses any other compression technique ? 5) If yes where can I find the source of it (Precisely , which file in source ) Thanks in advance regards , Pradeep

1 0

Parameters in bind function
by Narendra Pradeep 08 Aug '16

08 Aug '16

Hello everyone , I am trying to understand the MAL statements for joining three tables . On which I came across bind function in MAL statement . I have a table lineitem with l_quantity in it . So what does the following statements means 1 ) ( X_94:bat[:oid,:oid] ,X_95:bat[:oid,:int] ) := sql.bind(X_4,"sys","lineitem","l_quantity",2,0,4); 2 ) (X_96:bat[:oid,:oid] ,X_97:bat[:oid,:int] ) := sql.bind(X_4,"sys","lineitem","l_quantity",2,1,4); what does the 5 th and 6 th argument means ? Regards , Pradeep

1 0

jdbc exception
by Panagiotis Koutsourakis 05 Aug '16

05 Aug '16

Hello, has anyone encountered something like the following exception before? java.lang.AssertionError: Server sent a block larger than BLOCKsize: 13456 > 8193 at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.readBlock(MapiSocket.java:917) at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.read(MapiSocket.java:966) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at nl.cwi.monetdb.mcl.io.BufferedMCLReader.readLine(BufferedMCLReader.java:119) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:278) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:398) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:252) at nl.cwi.monetdb.jdbc.MonetConnection.<init>(MonetConnection.java:232) at nl.cwi.monetdb.jdbc.MonetDriver.connect(MonetDriver.java:171) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) Does anyone has any tips? The jdbc driver version used is 2.13.

2 1

Re: MonetDB: default - We were severely underquoting according to th...
by Niels Nes 01 Aug '16

01 Aug '16

Sjoerd Kunnen we wel voor perl-DBI testen? Nu draaien somige tests niet (geven errors) omdat perl-DBI mist. Niels On Mon, Aug 01, 2016 at 03:55:06PM +0200, Sjoerd Mullender wrote: > Changeset: a64440dcd8bf for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a64440dcd8bf > Modified Files: > configure.ag > Branch: default > Log Message: > > We were severely underquoting according to the autoconf documentation. > > > diffs (truncated from 2554 to 300 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -9,7 +9,7 @@ dnl Copyright 1997 - July 2008 CWI, Augu > > # Process this file with autoconf to produce a configure script. > > -AC_PREREQ(2.60) > +AC_PREREQ([2.60]) > AC_INIT([MonetDB], [11.24.0], [info(a)monetdb.org] [MonetDB], [http://www.monetdb.org/] > AC_CONFIG_AUX_DIR([buildtools/conf]) > AC_CONFIG_MACRO_DIR([buildtools/conf]) > @@ -22,13 +22,13 @@ AC_CONFIG_HEADERS([monetdb_config.h]) > AC_SUBST([CONFIG_H], [monetdb_config.h]) > > HOST=[$target] > -AC_DEFINE_UNQUOTED(HOST, "$HOST", [Host identifier]) > +AC_DEFINE_UNQUOTED([HOST], ["$HOST"], [Host identifier]) > > if test x"$target_os" = xcygwin; then > - AC_DEFINE(WIN32, 1, [Define on Microsoft Windows (also under Cygwin)]) > + AC_DEFINE([WIN32], 1, [Define on Microsoft Windows (also under Cygwin)]) > fi > -AM_CONDITIONAL(NOT_WIN32, test x"$target_os" != xcygwin) > -AM_CONDITIONAL(NATIVE_WIN32, false) > +AM_CONDITIONAL([NOT_WIN32], [test x"$target_os" != xcygwin]) > +AM_CONDITIONAL([NATIVE_WIN32], [false]) > > m4_ifndef([PKG_PROG_PKG_CONFIG], > [m4_fatal([Could not locate the pkg-config autoconf > @@ -57,16 +57,16 @@ AC_DEFUN([MONETDB_VERSION_TO_NUMBER], > # except set the variables. The command to set them in case we do > # need to escape the \'s is Qvar=`echo "$var" | sed 's/\\\\/\\\\\\\\/g'` > BUILD=[`pwd`] > -AC_SUBST(BUILD) > +AC_SUBST([BUILD]) > > SOURCE=[`(cd $srcdir && pwd)`] > -AC_SUBST(SOURCE) > +AC_SUBST([SOURCE]) > > . $srcdir/libversions > -AC_SUBST(GDK_VERSION) > -AC_SUBST(MAPI_VERSION) > -AC_SUBST(MONETDB5_VERSION) > -AC_SUBST(STREAM_VERSION) > +AC_SUBST([GDK_VERSION]) > +AC_SUBST([MAPI_VERSION]) > +AC_SUBST([MONETDB5_VERSION]) > +AC_SUBST([STREAM_VERSION]) > > # if no --prefix option, we must set Qprefix to the default value > # the problem is that prefix is set very late in the process if it > @@ -77,14 +77,14 @@ else > Qprefix="${prefix}" > fi > Qprefix="`echo "$Qprefix" | sed 's/\\\\/\\\\\\\\/g'`" > -AC_SUBST(Qprefix) > +AC_SUBST([Qprefix]) > # exec_prefix is created very late, so let make do the expansion > if test "${exec_prefix}x" = "NONEx"; then > Qexec_prefix='${prefix}' > else > Qexec_prefix="`echo "$exec_prefix" | sed 's/\\\\/\\\\\\\\/g'`" > fi > -AC_SUBST(Qexec_prefix) > +AC_SUBST([Qexec_prefix]) > > if test -f "$srcdir"/vertoo.data; then > dnl Developers compiling from Mercurial: > @@ -124,21 +124,21 @@ case "$CC" in > ;; > esac > > -AC_DEFINE(MONETDB_RELEASE, "unreleased", [Release name or "unreleased"]) > +AC_DEFINE([MONETDB_RELEASE], ["unreleased"], [Release name or "unreleased"]) > > dft_gdk=yes > -AC_ARG_ENABLE(gdk, > - AS_HELP_STRING([--enable-gdk], > - [enable support for GDK (default=yes)]), > - enable_gdk=$enableval, > - enable_gdk=$dft_gdk) > +AC_ARG_ENABLE([gdk], > + [AS_HELP_STRING([--enable-gdk], > + [enable support for GDK (default=yes)])], > + [enable_gdk=$enableval], > + [enable_gdk=$dft_gdk]) > > dft_monetdb5=yes > -AC_ARG_ENABLE(monetdb5, > - AS_HELP_STRING([--enable-monetdb5], > - [enable support for MonetDB5 (default=yes)]), > - enable_monetdb5=$enableval, > - enable_monetdb5=$dft_monetdb5) > +AC_ARG_ENABLE([monetdb5], > + [AS_HELP_STRING([--enable-monetdb5], > + [enable support for MonetDB5 (default=yes)])], > + [enable_monetdb5=$enableval], > + [enable_monetdb5=$dft_monetdb5]) > case "$enable_gdk-$enable_monetdb5" in > no-yes) > AC_MSG_ERROR([MonetDB5 requires GDK.]) > @@ -150,32 +150,32 @@ no-auto) > esac > > dft_fits=auto > -AC_ARG_ENABLE(fits, > - AS_HELP_STRING([--enable-fits], > - [enable support for FITS (default=auto)]), > - enable_fits=$enableval, > - enable_fits=$dft_fits) > +AC_ARG_ENABLE([fits], > + [AS_HELP_STRING([--enable-fits], > + [enable support for FITS (default=auto)])], > + [enable_fits=$enableval], > + [enable_fits=$dft_fits]) > > dft_netcdf=auto > -AC_ARG_ENABLE(netcdf, > - AS_HELP_STRING([--enable-netcdf], > - [enable support for netcdf (default=auto)]), > - enable_netcdf=$enableval, > - enable_netcdf=$dft_netcdf) > +AC_ARG_ENABLE([netcdf], > + [AS_HELP_STRING([--enable-netcdf], > + [enable support for netcdf (default=auto)])], > + [enable_netcdf=$enableval], > + [enable_netcdf=$dft_netcdf]) > > dft_lidar=auto > -AC_ARG_ENABLE(lidar, > - AS_HELP_STRING([--enable-lidar], > - [enable support for LiDAR data (default=auto)]), > - have_lidar=$enableval, > - have_lidar=$dft_lidar) > +AC_ARG_ENABLE([lidar], > + [AS_HELP_STRING([--enable-lidar], > + [enable support for LiDAR data (default=auto)])], > + [have_lidar=$enableval], > + [have_lidar=$dft_lidar]) > > dft_sql=yes > -AC_ARG_ENABLE(sql, > - AS_HELP_STRING([--enable-sql], > - [enable support for MonetDB/SQL (default=yes)]), > - enable_sql=$enableval, > - enable_sql=$dft_sql) > +AC_ARG_ENABLE([sql], > + [AS_HELP_STRING([--enable-sql], > + [enable support for MonetDB/SQL (default=yes)])], > + [enable_sql=$enableval], > + [enable_sql=$dft_sql]) > case "$enable_monetdb5-$enable_sql" in > no-yes) > AC_MSG_ERROR([MonetDB/SQL requires MonetDB5.]) > @@ -187,11 +187,11 @@ no-auto) > esac > > dft_geom=auto > -AC_ARG_ENABLE(geom, > - AS_HELP_STRING([--enable-geom], > - [enable support for geom module (default=auto)]), > - enable_geom=$enableval, > - enable_geom=$dft_geom) > +AC_ARG_ENABLE([geom], > + [AS_HELP_STRING([--enable-geom], > + [enable support for geom module (default=auto)])], > + [enable_geom=$enableval], > + [enable_geom=$dft_geom]) > case "$enable_monetdb5-$enable_geom" in > no-yes) > AC_MSG_ERROR([geom module requires MonetDB5]) > @@ -203,11 +203,11 @@ no-auto) > esac > > dft_shp=auto > -AC_ARG_ENABLE(shp, > - AS_HELP_STRING([--enable-shp], > - [enable support for ESRI Shapefiles (default=auto)]), > - have_shp=$enableval, > - have_shp=$dft_shp) > +AC_ARG_ENABLE([shp], > + [AS_HELP_STRING([--enable-shp], > + [enable support for ESRI Shapefiles (default=auto)])], > + [have_shp=$enableval], > + [have_shp=$dft_shp]) > case "$enable_geom-$have_shp" in > no-yes) > AC_MSG_ERROR([ESRI Shapefile vault requires the geom module]) > @@ -219,85 +219,85 @@ no-auto) > esac > > dft_mapi=yes > -AC_ARG_ENABLE(mapi, > - AS_HELP_STRING([--enable-mapi], > - [Enable MAPI (default=yes)]), > - enable_mapi=$enableval, > - enable_mapi=$dft_mapi) > -AM_CONDITIONAL(HAVE_MAPI, test x$enable_mapi != xno) > +AC_ARG_ENABLE([mapi], > + [AS_HELP_STRING([--enable-mapi], > + [Enable MAPI (default=yes)])], > + [enable_mapi=$enableval], > + [enable_mapi=$dft_mapi]) > +AM_CONDITIONAL([HAVE_MAPI], [test x$enable_mapi != xno]) > if test "x$enable_mapi" != xno; then > - AC_DEFINE(HAVE_MAPI, 1, [Enable MAPI]) > + AC_DEFINE([HAVE_MAPI], 1, [Enable MAPI]) > fi > > dft_embedded=no > -AC_ARG_ENABLE(embedded, > - AS_HELP_STRING([--enable-embedded], > - [enable support for running MonetDB as a library (default=no)]), > - enable_embedded=$enableval, > - enable_embedded=$dft_embedded) > -AM_CONDITIONAL(HAVE_EMBEDDED, test x$enable_embedded != xno) > +AC_ARG_ENABLE([embedded], > + [AS_HELP_STRING([--enable-embedded], > + [enable support for running MonetDB as a library (default=no)])], > + [enable_embedded=$enableval], > + [enable_embedded=$dft_embedded]) > +AM_CONDITIONAL([HAVE_EMBEDDED], [test x$enable_embedded != xno]) > if test "x$enable_embedded" != xno; then > - AC_DEFINE(HAVE_EMBEDDED, 1, [Support for MonetDB as a library]) > + AC_DEFINE([HAVE_EMBEDDED], 1, [Support for MonetDB as a library]) > fi > > dft_embedded_r=no > -AC_ARG_ENABLE(embedded-r, > - AS_HELP_STRING([--enable-embedded-r], > - [enable support for running MonetDB as a library for R (default=no)]), > - enable_embedded_r=$enableval, > - enable_embedded_r=$dft_embedded_r) > -AM_CONDITIONAL(HAVE_EMBEDDED_R, test x$enable_embedded_r != xno) > +AC_ARG_ENABLE([embedded-r], > + [AS_HELP_STRING([--enable-embedded-r], > + [enable support for running MonetDB as a library for R (default=no)])], > + [enable_embedded_r=$enableval], > + [enable_embedded_r=$dft_embedded_r]) > +AM_CONDITIONAL([HAVE_EMBEDDED_R], [test x$enable_embedded_r != xno]) > if test "x$enable_embedded_r" != xno; then > - AC_DEFINE(HAVE_EMBEDDED_R, 1, [Support for MonetDB as a library for R]) > + AC_DEFINE([HAVE_EMBEDDED_R], 1, [Support for MonetDB as a library for R]) > fi > > dft_gsl=auto > -AC_ARG_ENABLE(gsl, > - AS_HELP_STRING([--enable-gsl], > - [enable support for GSL (default=auto)]), > - enable_gsl=$enableval, > - enable_gsl=$dft_gsl) > +AC_ARG_ENABLE([gsl], > + [AS_HELP_STRING([--enable-gsl], > + [enable support for GSL (default=auto)])], > + [enable_gsl=$enableval], > + [enable_gsl=$dft_gsl]) > > dft_rintegration=auto > -AC_ARG_ENABLE(rintegration, > - AS_HELP_STRING([--enable-rintegration], > - [enable support for R integration into MonetDB (default=auto)]), > - enable_rintegration=$enableval, > - enable_rintegration=$dft_rintegration) > +AC_ARG_ENABLE([rintegration], > + [AS_HELP_STRING([--enable-rintegration], > + [enable support for R integration into MonetDB (default=auto)])], > + [enable_rintegration=$enableval], > + [enable_rintegration=$dft_rintegration]) > > dft_pyintegration=auto > -AC_ARG_ENABLE(pyintegration, > - AS_HELP_STRING([--enable-pyintegration], > - [enable support for Python integration into MonetDB (default=auto)]), > - enable_pyintegration=$enableval, > - enable_pyintegration=$dft_pyintegration) > +AC_ARG_ENABLE([pyintegration], > + [AS_HELP_STRING([--enable-pyintegration], > + [enable support for Python integration into MonetDB (default=auto)])], > + [enable_pyintegration=$enableval], > + [enable_pyintegration=$dft_pyintegration]) > > dft_odbc=auto > -AC_ARG_ENABLE(odbc, > - AS_HELP_STRING([--enable-odbc], > - [compile the MonetDB ODBC driver (default=auto)]), > - enable_odbc=$enableval, > - enable_odbc=$dft_odbc) > +AC_ARG_ENABLE([odbc], > + [AS_HELP_STRING([--enable-odbc], > + [compile the MonetDB ODBC driver (default=auto)])], > + [enable_odbc=$enableval], > + [enable_odbc=$dft_odbc]) > > dft_testing=auto > -AC_ARG_ENABLE(testing, > - AS_HELP_STRING([--enable-testing], > - [enable support for testing (default=auto)]), > - enable_testing=$enableval, > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- Niels Nes, Manager ITF, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: https://www.cwi.nl/people/niels e-mail: Niels.Nes(a)cwi.nl

1 0