- developers-list - monetdb.org

Re: MonetDB: default - Cleanup code upon request.
by Fabian Groffen 15 Nov '12

15 Nov '12

On 15-11-2012 08:31:51 +0100, Martin Kersten wrote: > +/* There is just a single queue for the workers */ > static void > MRqueueCreate(int sz) > { > @@ -54,14 +55,21 @@ MRqueueCreate(int sz) > MT_lock_init(&mrqlock, "q_create"); > MT_lock_set(&mrqlock, "q_create"); > MT_sema_init(&mrqsema, 0, "q_create"); > - sz = ((sz << 1) >> 1); /* we want a multiple of 2 */ > + if ( mrqueue ) { > + GDKerror("One map-reduce queue allowed"); > + return; > + } > + sz *= 2; > mrqueue = (MRqueue *) GDKzalloc(sizeof(MRqueue) * sz); > - assert(mrqueue); > + if ( mrqueue == 0) { > + GDKerror("Could not create the map-reduce queue"); > + return; shouldn't you unset the mrqlock here? > + } _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

BugID=3178: A patch to compute stdev using one scan of BAT is available
by Yinhe Cao 13 Nov '12

13 Nov '12

Hi, The current implementation of ALGstdev_@1 in monetdb5/modules/kernel/algebra.mx takes two scans of the BAT. First scan to compute the sum (average) Second scan to compute stdev. By using a well-known mathematical property of the stdev, one scan is enough: during the scan, the sum of the value and the square of the value can be computed together, and the stdev can be computed then. The new implementation would roughly cut the running time by half for most of meaningful cases, especially when the bat is big. Link to the patch: http://bugs.monetdb.org/attachment.cgi?id=158&action=edit Link to the bug: http://bugs.monetdb.org/show_bug.cgi?id=3178 Any comments or suggestions will be greatly appreciated. Thanks, Yinhe _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Clean install gcc pragma warnings
by Stefan de Konink 10 Nov '12

10 Nov '12

Hi, I just try to install MonetDB on a new system. Used hg and updated to Oct2013. y.tab.c: In function 'sqlparse': y.tab.c:6609:3: error: unknown option after '#pragma GCC diagnostic' kind [-Werror=pragmas] y.tab.c:13496:3: error: unknown option after '#pragma GCC diagnostic' kind [-Werror=pragmas] cc1: all warnings being treated as errors I'm currently running gcc (Gentoo 4.6.3 p1.7, pie-0.5.2) 4.6.3 Is this known? Stefan _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

2 2

RubyGem on Jul2012
by Charalampos Nikolaou 08 Nov '12

08 Nov '12

Hi there, it's my first time playing around with the Jul2012 version of MonetDB and during "make install" I encountered the following error: mkdir -p '/Library/Ruby/Site/1.8' gem install --local --install-dir '/Library/Ruby/Site/1.8' --force --rdoc 'activerecord-monetdb-adapter-0.1.gem' ERROR: While executing gem ... (Gem::FilePermissionError) You don't have write permissions into the /Library/Ruby/Site/1.8 directory. As you can see, I am on a Mac (10.7) and Ruby is installed in /Library/Ruby as indicated above, BUT since I work with many versions of MonetDB, I install all of them "locally", in my home directory. Thus, I would prefer not to install the MonetDB gem file in a system directory. Is that possible? By reading the help output of the configuration, I saw that this is not possible, or might be? Moreover, I understood by the commit messages that this addition (Ruby) is needed for providing another client for MonetDB server. While I do not need such a client, is there an option to avoid building/using/producing it in the compilation process? I didn't see any option in the configuration as well. Thanks a lot, Babis _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

3 5

Re: MonetDB: default - Low-level task scheduler.
by Sjoerd Mullender 07 Nov '12

07 Nov '12

There are a few things wrong with this code: - sz·=·((sz·<<·1)·>>·1); does *not* turn sz into a multiple of two (as suggested by the comment). This statement basically is a no-op. - when you create joinable threads, you should join them. - it's not a great idea to use assert to make sure that GDKmalloc succeeds. Better is to return an error. - The include of monet_options.h should be in the C file, not in the include file (which would be included elsewhere where monet_options.h will already be included). On 2012-11-07 13:03, Martin Kersten wrote: > Changeset: 5ff3c16e865f for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=5ff3c16e865f > Added Files: > gdk/gdk_mapreduce.c > gdk/gdk_mapreduce.h > Modified Files: > gdk/Makefile.ag > monetdb5/modules/mal/groups.c > Branch: default > Log Message: > > Low-level task scheduler. > This module provide a lightweight map-reduce scheduler for multicore systems. > A limited number of workers are initialized upfront, which take the tasks > from a central queue. The header of these task descriptors should comply > with the MRtask structure. > > > diffs (239 lines): > > diff --git a/gdk/Makefile.ag b/gdk/Makefile.ag > --- a/gdk/Makefile.ag > +++ b/gdk/Makefile.ag > @@ -36,7 +36,7 @@ lib_gdk = { > gdk_private.h gdk_delta.h gdk_logger.h gdk_posix.h \ > gdk_system.h gdk_tm.h gdk_storage.h \ > gdk_calc.c gdk_calc.h gdk_calc_compare.h gdk_calc_private.h \ > - gdk_aggr.c gdk_group.c \ > + gdk_aggr.c gdk_group.c gdk_mapreduce.c gdk_mapreduce.h \ > bat.feps bat1.feps bat2.feps \ > libbat.rc > LIBS = ../common/options/libmoptions \ > diff --git a/gdk/gdk_mapreduce.c b/gdk/gdk_mapreduce.c > new file mode 100644 > --- /dev/null > +++ b/gdk/gdk_mapreduce.c > @@ -0,0 +1,141 @@ > +/* > + * The contents of this file are subject to the MonetDB Public License > + * Version 1.1 (the "License"); you may not use this file except in > + * compliance with the License. You may obtain a copy of the License at > + * http://www.monetdb.org/Legal/MonetDBLicense > + * > + * Software distributed under the License is distributed on an "AS IS" > + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > + * License for the specific language governing rights and limitations > + * under the License. > + * > + * The Original Code is the MonetDB Database System. > + * > + * The Initial Developer of the Original Code is CWI. > + * Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > + * Copyright August 2008-2012 MonetDB B.V. > + * All Rights Reserved. > + */ > + > +/* > + * (co) Martin L. Kersten > + * This module provide a lightweight map-reduce scheduler for multicore systems. > + * A limited number of workers are initialized upfront, which take the tasks > + * from a central queue. The header of these task descriptors should comply > + * with the MRtask structure. > + * > + */ > +#include "monetdb_config.h" > +#include "gdk.h" > +#include "gdk_mapreduce.h" > + > +/* each entry in the queue contains a list of tasks */ > +typedef struct MRQUEUE { > + MRtask **tasks; > + int index; /* next available task */ > + int size; /* number of tasks */ > +} MRqueue; > + > +static MRqueue *mrqueue; > +static int mrqsize= -1; /* size of queue */ > +static int mrqlast= -1; > +static MT_Lock mrqlock; /* its a shared resource, ie we need locks */ > +static MT_Sema mrqsema; /* threads wait on empty queues */ > + > + > +static void MRworker(void *); > + > +static void > +MRqueueCreate(int sz) > +{ > + int i; > + MT_Id tid; > + > + MT_lock_init(&mrqlock, "q_create"); > + MT_lock_set(&mrqlock,"q_create"); > + MT_sema_init(&mrqsema, 0, "q_create"); > + sz = ((sz << 1) >> 1); /* we want a multiple of 2 */ > + mrqueue = (MRqueue*)GDKzalloc(sizeof(MRqueue) *sz); > + assert(mrqueue); > + mrqsize = sz; > + mrqlast = 0; > + /* create a worker thread for each core as specified as system parameter*/ > + for ( i =0; i < GDKnr_threads; i++) > + MT_create_thread(&tid, MRworker, (void *) 0, MT_THR_JOINABLE); > + MT_lock_unset(&mrqlock,"q_create"); > +} > + > +static void > +MRenqueue(int taskcnt, MRtask **tasks) > +{ > + assert(taskcnt > 0); > + MT_lock_set(&mrqlock, "mrqlock"); > + if (mrqlast == mrqsize) { > + mrqsize <<= 1; > + mrqueue = (MRqueue*) GDKrealloc(mrqueue, sizeof(MRqueue) * mrqsize); > + } > + mrqueue[mrqlast].index = 0; > + mrqueue[mrqlast].tasks = tasks; > + mrqueue[mrqlast].size = taskcnt; > + mrqlast++; > + MT_lock_unset(&mrqlock, "mrqlock"); > + /* a task list is added for consumption*/ > + while (taskcnt-- > 0) > + MT_sema_up(&mrqsema, "mrqsema"); > +} > + > +static MRtask * > +MRdequeue(void) > +{ > + MRtask *r = NULL; > + int idx; > + > + MT_sema_down(&mrqsema, "mrqsema"); > + assert(mrqlast); > + MT_lock_set(&mrqlock, "mrqlock"); > + if (mrqlast > 0) { > + idx = mrqueue[mrqlast-1].index; > + r = mrqueue[mrqlast-1].tasks[idx++]; > + if ( mrqueue[mrqlast-1].size == idx) > + mrqlast--; > + else > + mrqueue[mrqlast-1].index = idx; > + } > + MT_lock_unset(&mrqlock, "mrqlock"); > + assert(r); > + return r; > +} > + > +static void > +MRworker(void * arg) > +{ > + MRtask *task; > + (void) arg; > + do{ > + task= MRdequeue(); > + (task->cmd)(task); > + MT_sema_up(task->sema, "mrqsema"); > + } while (1); > +} > + > +/* schedule the tasks and return when all are done */ > +void > +MRschedule(int taskcnt, void **arg, void (*cmd)(void*p)) > +{ > + int i; > + MT_Sema sema; > + MRtask **task = (MRtask**) arg; > + > + if ( mrqueue == 0) > + MRqueueCreate(1024); > + > + MT_sema_init(&sema, 0, "q_create"); > + for ( i= 0; i < taskcnt; i++){ > + task[i]->sema = & sema; > + task[i]->cmd = cmd; > + } > + MRenqueue(taskcnt,task); > + /* waiting for all report result */ > + for ( i= 0; i < taskcnt; i++) > + MT_sema_down(&sema, "mrqsema"); > +} > diff --git a/gdk/gdk_mapreduce.h b/gdk/gdk_mapreduce.h > new file mode 100644 > --- /dev/null > +++ b/gdk/gdk_mapreduce.h > @@ -0,0 +1,32 @@ > +/* > + * The contents of this file are subject to the MonetDB Public License > + * Version 1.1 (the "License"); you may not use this file except in > + * compliance with the License. You may obtain a copy of the License at > + * http://www.monetdb.org/Legal/MonetDBLicense > + * > + * Software distributed under the License is distributed on an "AS IS" > + * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > + * License for the specific language governing rights and limitations > + * under the License. > + * > + * The Original Code is the MonetDB Database System. > + * > + * The Initial Developer of the Original Code is CWI. > + * Portions created by CWI are Copyright (C) 1997-July 2008 CWI. > + * Copyright August 2008-2012 MonetDB B.V. > + * All Rights Reserved. > + */ > + > +#ifndef _GDK_MAPREDUCE_H_ > +#define _GDK_MAPREDUCE_H_ > + > +#include <monet_options.h> > + > +typedef struct{ > + MT_Sema *sema; /* micro scheduler handle */ > + void (*cmd)(void *); /* the function to be executed */ > +}MRtask; > + > +gdk_export void MRschedule(int taskcnt, void **arg, void (*cmd)(void *p)); > + > +#endif /* _GDK_MAPREDUCE_H_ */ > diff --git a/monetdb5/modules/mal/groups.c b/monetdb5/modules/mal/groups.c > --- a/monetdb5/modules/mal/groups.c > +++ b/monetdb5/modules/mal/groups.c > @@ -66,11 +66,15 @@ GRPmulticolumngroup(Client cntxt, MalBlk > /* sort order may have influences */ > /* SF100 Q16 showed < ordering is 2 times faster as > ordering */ > for ( i = 3; i< pci->argc; i++) > - for ( j = i+1; j<pci->argc; j++) > - if ( sizes[j] < sizes[i]){ > - l = sizes[j]; sizes[j]= sizes[i]; sizes[i]= l; > - bi = bid[j]; bid[j]= bid[i]; bid[i]= bi; > - } > + for ( j = i+1; j<pci->argc; j++) > + if ( sizes[j] < sizes[i]){ > + l = sizes[j]; > + sizes[j]= sizes[i]; > + sizes[i]= l; > + bi = bid[j]; > + bid[j]= bid[i]; > + bid[i]= bi; > + } > /* for (i=2; i<pci->argc; i++) > mnstr_printf(cntxt->fdout,"# after [%d] "LLFMT"\n",i, sizes[i]); */ > > @@ -82,8 +86,6 @@ GRPmulticolumngroup(Client cntxt, MalBlk > i = 4; > if (msg == MAL_SUCCEED && pci->argc > 4 ) > do { > - if (*ext) > - BBPdecref(*ext, TRUE); > /* early break when there are as many groups as histogram entries */ > b = BATdescriptor(*hist); > if ( b ){ > @@ -91,8 +93,8 @@ GRPmulticolumngroup(Client cntxt, MalBlk > BBPreleaseref(*hist); > if ( j) break; > } > - if (*hist) > - BBPdecref(*hist, TRUE); > + BBPdecref(*ext, TRUE); > + BBPdecref(*hist, TRUE); > > /* (grp,ext,hist) := group.subgroupdone(arg,grp) */ > oldgrp= *grp; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Where can I add my join algorithm in MonetDB?
by Xiaojun Chang 05 Nov '12

05 Nov '12

Hi, I am one Ph.D student and right now I am working on IMDB. Could anyone tell me where I can add my join algorithm in MonetDB? I cannot find such document. Thanks. Xiaojun Chang _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

3 2

Re: cloning a repository from release MonetDB-11.9.7
by Chi-Young Ku 02 Nov '12

02 Nov '12

Hi, I would like to clone a repository from MonetDB-11.9.7. What would be the branch or rev number for: hg clone http://dev.monetdb.org/hg/MonetDB -b Thanks in advance. Chi ________________________________ From: Chi-Young Ku <chi(a)querymetrica.com> To: "developers-list(a)monetdb.org" <developers-list(a)monetdb.org> Cc: Chi- Young Ku <chi(a)querymetrica.com> Sent: Wednesday, October 17, 2012 5:47 PM Subject: Generating gprof report Hi, I used the "--enable_profile" option to configure my MonetDB and ran "make clean; make; make install". I, then, started my MonetDB instance and ran the following query: select count(*) from lineitem where l_partkey > 9000; Lastly, I quit from the "mclient" session and issued "monetdbd stop ...". I observed that there are several gmon.out files in different directories in my top build directory. Therefore, I issued the following gprof command: gprof /usr/local/bin/mserver5 ./sql/backends/monet5/gmon.out ./gdk/gmon.out ./monetdb5/extras/mal_optimizer_template/gmon.out ./monetdb5/modules/atoms/gmon.out ./monetdb5/modules/kernel/gmon.out ./monetdb5/modules/mal/gmon.out ./monetdb5/scheduler/gmon.out ./monetdb5/optimizer/gmon.out ./monetdb5/mal/gmon.out However, the output of the above command did not show any calling sequence containing the execution of MAL instructions. Did I miss anything? Thanks in advance. Chi _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

3 2

Re: MonetDB: Oct2012 - Ported to Ubuntu Quantal Quetzal.
by Fabian Groffen 01 Nov '12

01 Nov '12

On 01-11-2012 08:53:47 +0100, Sjoerd Mullender wrote: > Changeset: 4c6c6b484d34 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4c6c6b484d34 > Modified Files: > configure.ag > Branch: Oct2012 > Log Message: > > Ported to Ubuntu Quantal Quetzal. > Whether it is the newer geos library (3.3.3) or something > quantal-specific, I don't know, but we need to use the --clibs option > to geos-config. This option does not exist on our Fedora system. It is an option from newer geos. However, my 3.3.5 has both libs and clibs options. _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

2 1

Re: MonetDB: default - The storage cost calculator
by Stefan Manegold 25 Oct '12

25 Oct '12

This checkin --- strangely dated Sep 21 2012; cf. http://dev.monetdb.org/hg/MonetDB/rev/523120729e8d !?? --- breaks compilation: http://monetdb.cwi.nl/testweb/web/status.php Martin Kersten <commits(a)monetdb.org> wrote: Changeset: 523120729e8d for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=523120729e8d Added Files: sql/scripts/75_storagemodel.sql Modified Files: monetdb5/mal/mal_profiler.c sql/backends/monet5/sql.mx sql/scripts/Makefile.ag Branch: default Log Message: The storage cost calculator This script gives the database administrator insight in the actual footprint of the persistent tables and the maximum playground used when indices are introduced upon them. By chancing the storagemodelinput table directly, the footprint for yet to be loaded databases can be assessed. The actual storage footprint of an existing database can be obtained by the table procuding function storage() It represents the actual state of affairs, i.e. storage on disk of columns and foreign key indices, and possible temporary hash indices. For strings we take a sample to determine their average length. diffs (truncated from 425 to 300 lines): diff --git a/monetdb5/mal/mal_profiler.c b/monetdb5/mal/mal_profiler.c --- a/monetdb5/mal/mal_profiler.c +++ b/monetdb5/mal/mal_profiler.c @@ -182,6 +182,10 @@ profilerEvent(int idx, MalBlkPtr mb, Mal } if (profileCounter[PROFstart].status == 0 && start) return; + if (myname == 0) + myname = putName("profiler", 8); + if (getModuleId(getInstrPtr(mb, pc)) == myname) + return; if (offlineProfiling) offlineProfilerEvent(idx, mb, stk, pc,start); if (cachedProfiling) diff --git a/sql/backends/monet5/sql.mx b/sql/backends/monet5/sql.mx --- a/sql/backends/monet5/sql.mx +++ b/sql/backends/monet5/sql.mx @@ -504,12 +504,13 @@ pattern storage()( schema:bat[:oid,:str], table:bat[:oid,:str], column:bat[:oid,:str], + type:bat[:oid,:str], location:bat[:oid,:str], count:bat[:oid,:lng], - capacity:bat[:oid,:lng], - width:bat[:oid,:int], - size:bat[:oid,:lng], - hashsize:bat[:oid,:lng], + atomwidth:bat[:oid,:int], + columnsize:bat[:oid,:lng], + heap:bat[:oid,:lng], + indices:bat[:oid,:lng], sorted:bat[:oid,:bit]) address sql_storage comment "return a table with storage information "; @@ -7347,13 +7348,13 @@ str SQLoptimizersUpdate(Client cntxt, Ma * Inspection of the actual storage footprint is a recurring question of users. * This is modelled as a generic SQL table producing function. * create function storage() - * returns table ("schema" string, "table" string, "column" string, location string, "count" bigint, capacity bigint, width int, size bigint, hashsize bigint, sorted int) + * returns table ("schema" string, "table" string, "column" string, "type" string, location string, "count" bigint, width int, columnsize bigint, heapsize bigint indices bigint, sorted int) * external name sql.storage; */ str sql_storage(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci) { - BAT *sch, *tab, *col, *loc, *cnt, *cap, *atom, *size, *aux, *sort; + BAT *sch, *tab, *col, *type, *loc, *cnt, *atom, *size, *heap, *indices, *sort; mvc *m = NULL; str msg = getSQLContext(cntxt,mb, &m, NULL); sql_trans *tr = m->session->tr; @@ -7362,13 +7363,14 @@ sql_storage(Client cntxt, MalBlkPtr mb, int *rsch = (int*) getArgReference(stk,pci,0); int *rtab = (int*) getArgReference(stk,pci,1); int *rcol = (int*) getArgReference(stk,pci,2); - int *rloc = (int*) getArgReference(stk,pci,3); - int *rcnt = (int*) getArgReference(stk,pci,4); - int *rcap = (int*) getArgReference(stk,pci,5); + int *rtype = (int*) getArgReference(stk,pci,3); + int *rloc = (int*) getArgReference(stk,pci,4); + int *rcnt = (int*) getArgReference(stk,pci,5); int *ratom = (int*) getArgReference(stk,pci,6); int *rsize = (int*) getArgReference(stk,pci,7); - int *raux = (int*) getArgReference(stk,pci,8); - int *rsort = (int*) getArgReference(stk,pci,9); + int *rheap = (int*) getArgReference(stk,pci,8); + int *rindices = (int*) getArgReference(stk,pci,9); + int *rsort = (int*) getArgReference(stk,pci,10); if (msg) return msg; @@ -7379,31 +7381,34 @@ sql_storage(Client cntxt, MalBlkPtr mb, BATseqbase(tab, 0); col = BATnew(TYPE_void,TYPE_str, 0); BATseqbase(col, 0); + type = BATnew(TYPE_void,TYPE_str, 0); + BATseqbase(type, 0); loc = BATnew(TYPE_void,TYPE_str, 0); BATseqbase(loc, 0); cnt = BATnew(TYPE_void,TYPE_lng, 0); BATseqbase(cnt, 0); - cap = BATnew(TYPE_void,TYPE_lng, 0); - BATseqbase(cap, 0); atom = BATnew(TYPE_void,TYPE_int, 0); BATseqbase(atom, 0); size = BATnew(TYPE_void,TYPE_lng, 0); BATseqbase(size, 0); - aux = BATnew(TYPE_void,TYPE_lng, 0); - BATseqbase(aux, 0); + heap = BATnew(TYPE_void,TYPE_lng, 0); + BATseqbase(heap, 0); + indices = BATnew(TYPE_void,TYPE_lng, 0); + BATseqbase(indices, 0); sort = BATnew(TYPE_void,TYPE_bit, 0); BATseqbase(sort, 0); - if ( sch == NULL || tab == NULL || col == NULL || loc == NULL || sort == NULL || - cnt == NULL || cap == NULL || atom == NULL || size == NULL || aux == NULL){ + if ( sch == NULL || tab == NULL || col == NULL || type == NULL || loc == NULL || sort == NULL || + cnt == NULL || atom == NULL || size == NULL || heap == NULL ||indices == NULL){ if ( sch ) BBPreleaseref(sch->batCacheid); if ( tab ) BBPreleaseref(tab->batCacheid); if ( col ) BBPreleaseref(col->batCacheid); if ( loc ) BBPreleaseref(loc->batCacheid); if ( cnt ) BBPreleaseref(cnt->batCacheid); - if ( cap ) BBPreleaseref(cap->batCacheid); + if ( type ) BBPreleaseref(type->batCacheid); if ( atom ) BBPreleaseref(atom->batCacheid); if ( size ) BBPreleaseref(size->batCacheid); - if ( aux ) BBPreleaseref(aux->batCacheid); + if ( heap ) BBPreleaseref(heap->batCacheid); + if ( indices ) BBPreleaseref(indices->batCacheid); if ( sort ) BBPreleaseref(sort->batCacheid); throw(SQL,"sql.storage", MAL_MALLOC_FAIL); } @@ -7412,7 +7417,8 @@ sql_storage(Client cntxt, MalBlkPtr mb, sql_schema *s= (sql_schema*) nsch->data; if ( isalpha((int)b->name[0]) ) - if (s->tables.set) for(ntab= (s)->tables.set->h ;ntab; ntab= ntab->next){ + if (s->tables.set) + for(ntab= (s)->tables.set->h ;ntab; ntab= ntab->next){ sql_base *bt= ntab->data; sql_table *t= (sql_table*) bt; if (isTable(t)) @@ -7426,28 +7432,49 @@ sql_storage(Client cntxt, MalBlkPtr mb, sch = BUNappend(sch, b->name, FALSE); tab = BUNappend(tab, bt->name, FALSE); col = BUNappend(col, bc->name, FALSE); + type = BUNappend(type, c->type.type->sqlname, FALSE); /*printf(" cnt "BUNFMT, BATcount(bn));*/ sz= BATcount(bn); cnt = BUNappend(cnt, &sz, FALSE); - /*printf(" cap "BUNFMT, BATcapacity(bn));*/ - sz= BATcapacity(bn); - cap = BUNappend(cap, &sz, FALSE); /*printf(" loc %s", BBP_physical(bn->batCacheid));*/ loc = BUNappend(loc, BBP_physical(bn->batCacheid), FALSE); /*printf(" width %d", bn->T->width);*/ w= bn->T->width; + if ( bn->ttype == TYPE_str){ + BUN p,q; + double sum=0; + BATiter bi = bat_iterator(bn); + lng cnt1,cnt2= cnt1=(lng) BATcount(bn); + + /* just take a sample */ + if ( cnt1 > 512) + cnt1 = cnt2 = 512; + BATloop(bn,p,q){ + str s = BUNtail(bi,p); + if( s != NULL && strcmp(s, str_nil)) + sum += (int) strlen(s); + if ( --cnt1 <= 0) + break; + } + if ( cnt2) + w = (int) (sum/cnt2); + } atom = BUNappend(atom, &w, FALSE); - /*printf(" size "BUNFMT, tailsize(bn,BATcount(bn)) + (bn->T->vheap? bn->T->vheap->size:0));*/ - sz = tailsize(bn,BATcapacity(bn)) + (bn->T->vheap? bn->T->vheap->size:0); - sz += headsize(bn,BATcapacity(bn)) + (bn->H->vheap? bn->H->vheap->size:0); + + sz = tailsize(bn,BATcount(bn)); + sz += headsize(bn,BATcount(bn)); size = BUNappend(size, &sz, FALSE); + sz = bn->T->vheap? bn->T->vheap->size:0; + sz += bn->H->vheap? bn->H->vheap->size:0; + heap = BUNappend(heap, &sz, FALSE); + sz = bn->T->hash?bn->T->hash->heap->size:0; sz += bn->H->hash?bn->H->hash->heap->size:0; - aux = BUNappend(aux, &sz, FALSE); - /*printf(" auxsize "BUNFMT, bn->T->hash?bn->T->hash->heap->size:0);*/ + indices = BUNappend(indices, &sz, FALSE); + /*printf(" indices "BUNFMT, bn->T->hash?bn->T->hash->heap->size:0);*/ /*printf("\n");*/ w = BATtordered(bn); @@ -7468,28 +7495,49 @@ sql_storage(Client cntxt, MalBlkPtr mb, sch = BUNappend(sch, b->name, FALSE); tab = BUNappend(tab, bt->name, FALSE); col = BUNappend(col, bc->name, FALSE); + type = BUNappend(type, "oid", FALSE); /*printf(" cnt "BUNFMT, BATcount(bn));*/ sz= BATcount(bn); cnt = BUNappend(cnt, &sz, FALSE); - /*printf(" cap "BUNFMT, BATcapacity(bn));*/ - sz= BATcapacity(bn); - cap = BUNappend(cap, &sz, FALSE); /*printf(" loc %s", BBP_physical(bn->batCacheid));*/ loc = BUNappend(loc, BBP_physical(bn->batCacheid), FALSE); /*printf(" width %d", bn->T->width);*/ w= bn->T->width; + if ( bn->ttype == TYPE_str){ + BUN p,q; + double sum=0; + BATiter bi = bat_iterator(bn); + lng cnt1, cnt2= cnt1 = BATcount(bn); + + /* just take a sample */ + if ( cnt1 > 512) + cnt1 = cnt2 = 512; + BATloop(bn,p,q){ + str s = BUNtail(bi,p); + if( s != NULL && strcmp(s, str_nil)) + sum += (int) strlen(s); + if ( --cnt1 <= 0) + break; + } + if ( cnt2) + w = (int) (sum/cnt2); + } atom = BUNappend(atom, &w, FALSE); /*printf(" size "BUNFMT, tailsize(bn,BATcount(bn)) + (bn->T->vheap? bn->T->vheap->size:0));*/ - sz = tailsize(bn,BATcapacity(bn)) + (bn->T->vheap? bn->T->vheap->size:0); - sz += headsize(bn,BATcapacity(bn)) + (bn->H->vheap? bn->H->vheap->size:0); + sz = tailsize(bn,BATcount(bn)); + sz += headsize(bn,BATcount(bn)); size = BUNappend(size, &sz, FALSE); + sz = bn->T->vheap? bn->T->vheap->size:0; + sz += bn->H->vheap? bn->H->vheap->size:0; + heap = BUNappend(heap, &sz, FALSE); + sz = bn->T->hash?bn->T->hash->heap->size:0; sz += bn->H->hash?bn->H->hash->heap->size:0; - aux = BUNappend(aux, &sz, FALSE); - /*printf(" auxsize "BUNFMT, bn->T->hash?bn->T->hash->heap->size:0);*/ + indices = BUNappend(indices, &sz, FALSE); + /*printf(" indices "BUNFMT, bn->T->hash?bn->T->hash->heap->size:0);*/ /*printf("\n");*/ w = BATtordered(bn); sort = BUNappend(sort, &w, FALSE); @@ -7504,11 +7552,12 @@ sql_storage(Client cntxt, MalBlkPtr mb, BBPkeepref(*rtab = tab->batCacheid); BBPkeepref(*rcol = col->batCacheid); BBPkeepref(*rloc = loc->batCacheid); + BBPkeepref(*rtype = type->batCacheid); BBPkeepref(*rcnt = cnt->batCacheid); - BBPkeepref(*rcap = cap->batCacheid); BBPkeepref(*ratom = atom->batCacheid); BBPkeepref(*rsize = size->batCacheid); - BBPkeepref(*raux = aux->batCacheid); + BBPkeepref(*rheap = heap->batCacheid); + BBPkeepref(*rindices = indices->batCacheid); BBPkeepref(*rsort = sort->batCacheid); return MAL_SUCCEED; } diff --git a/sql/scripts/75_storagemodel.sql b/sql/scripts/75_storagemodel.sql new file mode 100644 --- /dev/null +++ b/sql/scripts/75_storagemodel.sql @@ -0,0 +1,154 @@ +-- The contents of this file are subject to the MonetDB Public License +-- Version 1.1 (the "License"); you may not use this file except in +-- compliance with the License. You may obtain a copy of the License at +-- http://www.monetdb.org/Legal/MonetDBLicense +-- +-- Software distributed under the License is distributed on an "AS IS" +-- basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the +-- License for the specific language governing rights and limitations +-- under the License. +-- +-- The Original Code is the MonetDB Database System. +-- +-- The Initial Developer of the Original Code is CWI. +-- Copyright August 2008-2012 MonetDB B.V. +-- All Rights Reserved. + + +-- Author M.Kersten +-- This script gives the database administrator insight in the actual +-- footprint of the persistent tables and the maximum playground used +-- when indices are introduced upon them. +-- By chancing the storagemodelinput table directly, the footprint for +-- yet to be loaded databases can be assessed. + +-- The actual storage footprint of an existing database can be +-- obtained by the table procuding function storage() +-- It represents the actual state of affairs, i.e. storage on disk +-- of columns and foreign key indices, and possible temporary hash indices. +-- For strings we take a sample to determine their average length. + +create function storage() +returns table ("schema" string, "table" string, "column" string, "type" string, location string, "count" bigint, typewidth int, columnsize bigint, heapsize bigint, indices bigint, sorted boolean) +external name sql.storage; + +-- To determine the footprint of an arbitrary database, we first have +-- to define its schema, followed by an indication of the properties of each column. +-- A storage model input table for the size prediction is shown below: +create table storagemodelinput( + "schema" string, + "table" string, + "column" string, _______________________________________________ checkin-list mailing list checkin-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/checkin-list _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0

Re: MonetDB Development Environment
by Stefan Manegold 24 Oct '12

24 Oct '12

Fedora, joe, gcc S. ----- Reply message ----- From: "Fabian Groffen" <Fabian.Groffen(a)cwi.nl> To: "Communication channel for developers of the MonetDB suite." <developers-list(a)monetdb.org> Subject: MonetDB Development Environment Date: Wed, Oct 24, 2012 10:08 On 23-10-2012 14:19:42 -0700, Masood Mortazavi wrote: > I'm wondering�what development environments (OS, editors, etc.)�or IDEs, if > any, MonetDB development team are using. Vim and Emacs. Fabian _______________________________________________ developers-list mailing list developers-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list

1 0