Re: [MonetDB-users] Embedded Use: Persisting and Metadata

13 Feb 2008


      Yuvaraj Athur Raghuvir wrote:
...
I have the following calls of interest:
    bat.getSpaceReserved
    bat.getSpaceUsed
    bbp.getDiskSpace
Is there sample code that shows how to invoke these MAL calls and
interpret the results from C/C++ code?
looking at the thread, i see that you use an embedded SQL version. This 
means all interactions
with the kernel are through SQL statements and JDBC.
Internally, the SQL compiler produces an intermediate code, called MAL, 
which is interpreted
by the kernel. The operations mentioned above are internal MAL calls, 
which are bound
to specific C routines which you can find out using the MonetDB help 
functions, e.g.
...
mserver5
?bat.getSpace()
command bat.getSpaceUsed(b:bat[:any_1,:any_2]):lng
address BKCgetSpaceUsed;
comment Determine the total space (in bytes) occupied by a BAT.
command bat.getSpaceReserved(b:bat[:any_1,:any_2]):lng
address BKCgetSpaceReserved;
comment Determine the total space (in bytes) reserved for a BAT.
...
?bbp.getDiskSpace()
command bbp.getDiskSpace():int
address CMDbbpDiskSpace;
comment Estimate the amount of diskspace occupied by dbfarm
However, calling those functions directly from you C environment 
requires deep
knowledge of the total architecture. The easiest and safest is

bbp_export str CMDbbpDiskSpace(int *ret);

the others require BAT identifiers, which you don't have under control.
...
My search in the lib files I have used so far do not have these calls.
Do let me how to proceed here.
~Yuva
On 11/02/2008, Martin Kersten  wrote:
...
Yuvaraj Athur Raghuvir wrote:
...
...
From a C/C++ code, how do I access the MAL level storage operators? I
to find operators of interest, you may start at the appendix of the manual:
http://monetdb.cwi.nl/projects/monetdb/MonetDB/Documentation/Instruction-Hel...
and perform a grep/find
...
would assume that using the right libraries I should be able to get this
data into my C/C++ code. Could you please point me to the right
resources here? Thanks.
Further, I want to know the storage cost on disk for the entire
database. As I see the data is stored as a "farm". Does this mean that I
can sum the sizes of all files under the root folder to get the disk
storage cost?
yes, that is the persistent storage requirement of the database
...
~Yuva
On Feb 9, 2008 12:10 AM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:
Yuvaraj Athur Raghuvir wrote:
     > Thanks.
     >
     > Regarding db size, I would expect that the storage strategy will
have
...
> additional overheads due to bookkeeping requirements necessary for
     > reconstruction of the relations between the data elements. Since
    there
     > are no indices (I assume this due to columnar storage) I can
discount
...
> on the space required for indices.
     >
    Indices are automatically created and dropped as a side effect of
    queries.
    They disappear at the end of a session.
> Are there Mapi calls that I can use to determine the size of the
     > tables? I would like to horizontally partition the data across
     > multiple stores based on table size and growth estimates.
     >
    the SQL level does not open up the kernel information about storage
    requirements.
    At the MAL level several operators are available to assess storage
cost.
...
> ~Yuva
     >
     >
     > On 08/02/2008, Yuvaraj Athur Raghuvir mailto:yuvaraj.a.r@gmail.com> wrote:
     >
     >> Finally, I have managed to get the Mbeddded example working in
    VC++ 2005
     >> Express Edition on Windows XP ! Thanks to the folks in the
    mailing list.
     >>
     >> Use Case: MonetDB5 , Embedded Mode , SQL
     >>
     >> To move towards the application I intend to experiment on, I
    need some more
     >> information:
     >> 1) How to make data durable?  In the simple example,  I tried
    the following:
     >> a)  First Run: Create table and insert rows. Select to check if
    everything
     >> is ok.
     >> b) Second Run: Only perform selection
     >>
     >>>> I get an error in the second run.
     >>>>
     >> 2) How to get meta data of the database?
     >> a) I want to know the size of the tables and meta tables in the
    data base.
     >> In particular I am interested in seeing the growth of the
    database on
     >> addition of content.
     >> b) Typically, there is a master table in the database that records
     >> information of the schema persisted. What is the name of the
    Master Table
     >> and its schema in MonetDB?
     >> c) SQLite has an independent analyzer tool which dumps the state
    of the
     >> storage from which I can extract most of the data I need. Does
    MonetDB have
     >> a similar mechanism?
     >>
     >> ~Yuva
     >>
     >>
     >>>>>> code snippet >>>>>
     >>>>>>
     >>     dbh = embedded_sql (set, setlen);
     >>     if (dbh == NULL || mapi_error(dbh))
     >>         die(dbh, hdl);
     >>     /* switch off autocommit */
     >>     if (mapi_setAutocommit(dbh, 0) != MOK || mapi_error(dbh))
     >>         die(dbh,NULL);
     >>     if ((hdl = mapi_query(dbh, "create table emp"
     >>     " (name varchar(20),age int)")) == NULL || mapi_error(dbh))
     >>         die(dbh, hdl);
     >>     close_handle(dbh,hdl);
     >>     for(i=0; i< 1000; i++) {
     >>         char query[100];
     >>         _snprintf(query, 100,"insert into emp values(\'user%d\',
    %d)", i, (i
     >> % 82));
     >>         if ((hdl = mapi_query(dbh, query)) == NULL ||
    mapi_error(dbh))
     >>             die(dbh, hdl);
     >>         close_handle(dbh,hdl);
     >>     }
     >>     if ((hdl = mapi_query(dbh, "select * from emp")) == NULL ||
     >> mapi_error(dbh))
     >>         die(dbh, hdl);
     >>     i=0;
     >>     while (mapi_fetch_row(hdl)) {
     >>         char *age = mapi_fetch_field(hdl, 1);
     >>         i= i+ atoi(age);
     >>     }
     >>     if (mapi_error(dbh))
     >>         die(dbh, hdl);
     >>     close_handle(dbh,hdl);
     >>     printf("The sum is %d \n",i);
     >>     mapi_disconnect(dbh);
     >>     return 0;
     >>
     >>
     >
     >
-------------------------------------------------------------------------
...
> This SF.net email is sponsored by: Microsoft
     > Defy all challenges. Microsoft(R) Visual Studio 2008.
     > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
     > _______________________________________________
     > MonetDB-users mailing list
     > MonetDB-users@lists.sourceforge.net
    mailto:MonetDB-users@lists.sourceforge.net
     > https://lists.sourceforge.net/lists/listinfo/monetdb-users
     >
-------------------------------------------------------------------------
...
This SF.net email is sponsored by: Microsoft
    Defy all challenges. Microsoft(R) Visual Studio 2008.
    http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
    _______________________________________________
    MonetDB-users mailing list
    MonetDB-users@lists.sourceforge.net
    mailto:MonetDB-users@lists.sourceforge.net
    https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
------------------------------------------------------------------------
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users