Yuvaraj Athur Raghuvir wrote:
> Thanks.
>
> Regarding db size, I would expect that the storage strategy will have
> additional overheads due to bookkeeping requirements necessary for
> reconstruction of the relations between the data elements. Since there
> are no indices (I assume this due to columnar storage) I can discount
> on the space required for indices.
>   
Indices are automatically created and dropped as a side effect of queries.
They disappear at the end of a session.

> Are there Mapi calls that I can use to determine the size of the
> tables? I would like to horizontally partition the data across
> multiple stores based on table size and growth estimates.
>   
the SQL level does not open up the kernel information about storage 
requirements.
At the MAL level several operators are available to assess storage cost.
> ~Yuva
>
>
> On 08/02/2008, Yuvaraj Athur Raghuvir  wrote:
>   
>> Finally, I have managed to get the Mbeddded example working in VC++ 2005
>> Express Edition on Windows XP ! Thanks to the folks in the mailing list.
>>
>> Use Case: MonetDB5 , Embedded Mode , SQL
>>
>> To move towards the application I intend to experiment on, I need some more
>> information:
>> 1) How to make data durable?  In the simple example,  I tried the following:
>> a)  First Run: Create table and insert rows. Select to check if everything
>> is ok.
>> b) Second Run: Only perform selection
>>     
>>>> I get an error in the second run.
>>>>         
>> 2) How to get meta data of the database?
>> a) I want to know the size of the tables and meta tables in the data base.
>> In particular I am interested in seeing the growth of the database on
>> addition of content.
>> b) Typically, there is a master table in the database that records
>> information of the schema persisted. What is the name of the Master Table
>> and its schema in MonetDB?
>> c) SQLite has an independent analyzer tool which dumps the state of the
>> storage from which I can extract most of the data I need. Does MonetDB have
>> a similar mechanism?
>>
>> ~Yuva
>>
>>     
>>>>>> code snippet >>>>>
>>>>>>             
>>     dbh = embedded_sql (set, setlen);
>>     if (dbh == NULL || mapi_error(dbh))
>>         die(dbh, hdl);
>>     /* switch off autocommit */
>>     if (mapi_setAutocommit(dbh, 0) != MOK || mapi_error(dbh))
>>         die(dbh,NULL);
>>     if ((hdl = mapi_query(dbh, "create table emp"
>>     " (name varchar(20),age int)")) == NULL || mapi_error(dbh))
>>         die(dbh, hdl);
>>     close_handle(dbh,hdl);
>>     for(i=0; i< 1000; i++) {
>>         char query[100];
>>         _snprintf(query, 100,"insert into emp values(\'user%d\', %d)", i, (i
>> % 82));
>>         if ((hdl = mapi_query(dbh, query)) == NULL || mapi_error(dbh))
>>             die(dbh, hdl);
>>         close_handle(dbh,hdl);
>>     }
>>     if ((hdl = mapi_query(dbh, "select * from emp")) == NULL ||
>> mapi_error(dbh))
>>         die(dbh, hdl);
>>     i=0;
>>     while (mapi_fetch_row(hdl)) {
>>         char *age = mapi_fetch_field(hdl, 1);
>>         i= i+ atoi(age);
>>     }
>>     if (mapi_error(dbh))
>>         die(dbh, hdl);
>>     close_handle(dbh,hdl);
>>     printf("The sum is %d \n",i);
>>     mapi_disconnect(dbh);
>>     return 0;
>>
>>     
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-users
>   

Yuvaraj Athur Raghuvir wrote:

...

From a C/C++ code, how do I access the MAL level storage operators? I

to find operators of interest, you may start at the appendix of the manual: http://monetdb.cwi.nl/projects/monetdb/MonetDB/Documentation/Instruction-Hel... and perform a grep/find

would assume that using the right libraries I should be able to get this data into my C/C++ code. Could you please point me to the right resources here? Thanks.

Further, I want to know the storage cost on disk for the entire database. As I see the data is stored as a "farm". Does this mean that I can sum the sizes of all files under the root folder to get the disk storage cost? yes, that is the persistent storage requirement of the database

~Yuva

On Feb 9, 2008 12:10 AM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:

Yuvaraj Athur Raghuvir wrote: > Thanks. > > Regarding db size, I would expect that the storage strategy will have > additional overheads due to bookkeeping requirements necessary for > reconstruction of the relations between the data elements. Since there > are no indices (I assume this due to columnar storage) I can discount > on the space required for indices. > Indices are automatically created and dropped as a side effect of queries. They disappear at the end of a session.

> Are there Mapi calls that I can use to determine the size of the > tables? I would like to horizontally partition the data across > multiple stores based on table size and growth estimates. > the SQL level does not open up the kernel information about storage requirements. At the MAL level several operators are available to assess storage cost. > ~Yuva > > > On 08/02/2008, Yuvaraj Athur Raghuvir mailto:yuvaraj.a.r@gmail.com> wrote: > >> Finally, I have managed to get the Mbeddded example working in VC++ 2005 >> Express Edition on Windows XP ! Thanks to the folks in the mailing list. >> >> Use Case: MonetDB5 , Embedded Mode , SQL >> >> To move towards the application I intend to experiment on, I need some more >> information: >> 1) How to make data durable? In the simple example, I tried the following: >> a) First Run: Create table and insert rows. Select to check if everything >> is ok. >> b) Second Run: Only perform selection >> >>>> I get an error in the second run. >>>> >> 2) How to get meta data of the database? >> a) I want to know the size of the tables and meta tables in the data base. >> In particular I am interested in seeing the growth of the database on >> addition of content. >> b) Typically, there is a master table in the database that records >> information of the schema persisted. What is the name of the Master Table >> and its schema in MonetDB? >> c) SQLite has an independent analyzer tool which dumps the state of the >> storage from which I can extract most of the data I need. Does MonetDB have >> a similar mechanism? >> >> ~Yuva >> >> >>>>>> code snippet >>>>> >>>>>> >> dbh = embedded_sql (set, setlen); >> if (dbh == NULL || mapi_error(dbh)) >> die(dbh, hdl); >> /* switch off autocommit */ >> if (mapi_setAutocommit(dbh, 0) != MOK || mapi_error(dbh)) >> die(dbh,NULL); >> if ((hdl = mapi_query(dbh, "create table emp" >> " (name varchar(20),age int)")) == NULL || mapi_error(dbh)) >> die(dbh, hdl); >> close_handle(dbh,hdl); >> for(i=0; i< 1000; i++) { >> char query[100]; >> _snprintf(query, 100,"insert into emp values(\'user%d\', %d)", i, (i >> % 82)); >> if ((hdl = mapi_query(dbh, query)) == NULL || mapi_error(dbh)) >> die(dbh, hdl); >> close_handle(dbh,hdl); >> } >> if ((hdl = mapi_query(dbh, "select * from emp")) == NULL || >> mapi_error(dbh)) >> die(dbh, hdl); >> i=0; >> while (mapi_fetch_row(hdl)) { >> char *age = mapi_fetch_field(hdl, 1); >> i= i+ atoi(age); >> } >> if (mapi_error(dbh)) >> die(dbh, hdl); >> close_handle(dbh,hdl); >> printf("The sum is %d \n",i); >> mapi_disconnect(dbh); >> return 0; >> >> > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users >

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Yuvaraj Athur Raghuvir wrote:

I have the following calls of interest: bat.getSpaceReserved bat.getSpaceUsed bbp.getDiskSpace

Is there sample code that shows how to invoke these MAL calls and interpret the results from C/C++ code?

looking at the thread, i see that you use an embedded SQL version. This means all interactions with the kernel are through SQL statements and JDBC. Internally, the SQL compiler produces an intermediate code, called MAL, which is interpreted by the kernel. The operations mentioned above are internal MAL calls, which are bound to specific C routines which you can find out using the MonetDB help functions, e.g.

mserver5 ?bat.getSpace() command bat.getSpaceUsed(b:bat[:any_1,:any_2]):lng address BKCgetSpaceUsed; comment Determine the total space (in bytes) occupied by a BAT.

command bat.getSpaceReserved(b:bat[:any_1,:any_2]):lng address BKCgetSpaceReserved; comment Determine the total space (in bytes) reserved for a BAT.

?bbp.getDiskSpace() command bbp.getDiskSpace():int address CMDbbpDiskSpace; comment Estimate the amount of diskspace occupied by dbfarm

However, calling those functions directly from you C environment requires deep knowledge of the total architecture. The easiest and safest is bbp_export str CMDbbpDiskSpace(int *ret); the others require BAT identifiers, which you don't have under control.

My search in the lib files I have used so far do not have these calls.

Do let me how to proceed here. ~Yuva

On 11/02/2008, Martin Kersten wrote:

...

Yuvaraj Athur Raghuvir wrote:

...

...

From a C/C++ code, how do I access the MAL level storage operators? I

to find operators of interest, you may start at the appendix of the manual: http://monetdb.cwi.nl/projects/monetdb/MonetDB/Documentation/Instruction-Hel... and perform a grep/find

...

would assume that using the right libraries I should be able to get this data into my C/C++ code. Could you please point me to the right resources here? Thanks.

Further, I want to know the storage cost on disk for the entire database. As I see the data is stored as a "farm". Does this mean that I can sum the sizes of all files under the root folder to get the disk storage cost?

yes, that is the persistent storage requirement of the database

...

~Yuva

On Feb 9, 2008 12:10 AM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:

Yuvaraj Athur Raghuvir wrote: > Thanks. > > Regarding db size, I would expect that the storage strategy will

have

...

> additional overheads due to bookkeeping requirements necessary for > reconstruction of the relations between the data elements. Since there > are no indices (I assume this due to columnar storage) I can

discount

...

> on the space required for indices. > Indices are automatically created and dropped as a side effect of queries. They disappear at the end of a session.

> Are there Mapi calls that I can use to determine the size of the > tables? I would like to horizontally partition the data across > multiple stores based on table size and growth estimates. > the SQL level does not open up the kernel information about storage requirements. At the MAL level several operators are available to assess storage

cost.

...

> ~Yuva > > > On 08/02/2008, Yuvaraj Athur Raghuvir mailto:yuvaraj.a.r@gmail.com> wrote: > >> Finally, I have managed to get the Mbeddded example working in VC++ 2005 >> Express Edition on Windows XP ! Thanks to the folks in the mailing list. >> >> Use Case: MonetDB5 , Embedded Mode , SQL >> >> To move towards the application I intend to experiment on, I need some more >> information: >> 1) How to make data durable? In the simple example, I tried the following: >> a) First Run: Create table and insert rows. Select to check if everything >> is ok. >> b) Second Run: Only perform selection >> >>>> I get an error in the second run. >>>> >> 2) How to get meta data of the database? >> a) I want to know the size of the tables and meta tables in the data base. >> In particular I am interested in seeing the growth of the database on >> addition of content. >> b) Typically, there is a master table in the database that records >> information of the schema persisted. What is the name of the Master Table >> and its schema in MonetDB? >> c) SQLite has an independent analyzer tool which dumps the state of the >> storage from which I can extract most of the data I need. Does MonetDB have >> a similar mechanism? >> >> ~Yuva >> >> >>>>>> code snippet >>>>> >>>>>> >> dbh = embedded_sql (set, setlen); >> if (dbh == NULL || mapi_error(dbh)) >> die(dbh, hdl); >> /* switch off autocommit */ >> if (mapi_setAutocommit(dbh, 0) != MOK || mapi_error(dbh)) >> die(dbh,NULL); >> if ((hdl = mapi_query(dbh, "create table emp" >> " (name varchar(20),age int)")) == NULL || mapi_error(dbh)) >> die(dbh, hdl); >> close_handle(dbh,hdl); >> for(i=0; i< 1000; i++) { >> char query[100]; >> _snprintf(query, 100,"insert into emp values(\'user%d\', %d)", i, (i >> % 82)); >> if ((hdl = mapi_query(dbh, query)) == NULL || mapi_error(dbh)) >> die(dbh, hdl); >> close_handle(dbh,hdl); >> } >> if ((hdl = mapi_query(dbh, "select * from emp")) == NULL || >> mapi_error(dbh)) >> die(dbh, hdl); >> i=0; >> while (mapi_fetch_row(hdl)) { >> char *age = mapi_fetch_field(hdl, 1); >> i= i+ atoi(age); >> } >> if (mapi_error(dbh)) >> die(dbh, hdl); >> close_handle(dbh,hdl); >> printf("The sum is %d \n",i); >> mapi_disconnect(dbh); >> return 0; >> >> > >

-------------------------------------------------------------------------

...

> This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users >

-------------------------------------------------------------------------

...

This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users