BAT Data structures of a Schema
Hi Martin, I have the following list of tables in database schema: sql>\d TABLE sys.customer TABLE sys.lineitem TABLE sys.nation TABLE sys.orders TABLE sys.part TABLE sys.partsupp TABLE sys.region TABLE sys.supplier Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please? I want to debug and analyse specifically that part of the code region that declares, initialises and maintains the list of BAT structures for the tables. The following query is providing the information about the tables and I am assuming that this query also access this information from all the individual BAT structures of a database schema. select * from storage() where "table" = 'lineitem'; Secondly, Is it possible to list of get the list of all the BAT structures (Column BATS and Delta BATs) in the running database please? Thanks. Kind Regards, Ahmad
On 01/21/13 14:15, Hassan, Ahmad wrote:
Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please?
Do you mean the "wrapper code" that implements SQL storage (MonetDB/sql/storage/bat) or the underlying that does the raw processing? Stefan
Hi Stefan, Firstly, thanks for the email. I wish to identify the variable names of the BAT structures that constitute all the tables of the database schema and the place in the code where these BAT variables are declared/initialized using helper/wrapper functions for a database schema. The reason I need this is, I want to add compiler level instrumentation to analyse the memory access pattern of the BAT data structures of a database schema. The only way I can identify various individual data structures at the compiler level is through the name of the variables in the source code. If you can point to me this information, it will be very helpful for the understanding and analyses. So I would particularly need the variable names in C code that represents the each individual table and then the variable name of each BAT structure with in each table. The variable names will be correlated/mapped to the virtual addresses at the compiler level. Thanks. Kind Regards, Ahmad -----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Stefan de Konink Sent: 21 January 2013 13:24 To: users-list@monetdb.org Subject: Re: BAT Data structures of a Schema On 01/21/13 14:15, Hassan, Ahmad wrote:
Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please?
Do you mean the "wrapper code" that implements SQL storage (MonetDB/sql/storage/bat) or the underlying that does the raw processing? Stefan _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi Ahmad, Your question requires an answer way beyond the typical message in this forum. (A couple of months studying the code base ;)) Data structures are everywhere. They may play a short term role, e.g. during query parsing, an intermediate role, e.g. a temporary hash index, or a persistent role. All of them may be crucial in achieving the required performance. The code is in this respect your main (only) resource and you need ample of time to pin the area you really want to monitor. A tool to analyse access/instruction behavior is "valgrind --tool=callgrind mserver5...." success, Martin On 1/21/13 2:55 PM, Hassan, Ahmad wrote:
Hi Stefan,
Firstly, thanks for the email. I wish to identify the variable names of the BAT structures that constitute all the tables of the database schema and the place in the code where these BAT variables are declared/initialized using helper/wrapper functions for a database schema.
The reason I need this is, I want to add compiler level instrumentation to analyse the memory access pattern of the BAT data structures of a database schema. The only way I can identify various individual data structures at the compiler level is through the name of the variables in the source code. If you can point to me this information, it will be very helpful for the understanding and analyses. So I would particularly need the variable names in C code that represents the each individual table and then the variable name of each BAT structure with in each table.
The variable names will be correlated/mapped to the virtual addresses at the compiler level.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Stefan de Konink Sent: 21 January 2013 13:24 To: users-list@monetdb.org Subject: Re: BAT Data structures of a Schema
On 01/21/13 14:15, Hassan, Ahmad wrote:
Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please?
Do you mean the "wrapper code" that implements SQL storage (MonetDB/sql/storage/bat) or the underlying that does the raw processing?
Stefan _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi Martin, Thanks for the advice. Ignoring all the rest of the intermediate/index/delta data structures and focussing only on the core data structures that hold Table's real/core data that is accessed over time to fetch and store data in memory. This part of the code would be in gdk or sql directory? Or in both directories please? Thanks. Kind Regards, Ahmad -----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Martin Kersten Sent: 21 January 2013 14:06 To: users-list@monetdb.org Subject: Re: BAT Data structures of a Schema Hi Ahmad, Your question requires an answer way beyond the typical message in this forum. (A couple of months studying the code base ;)) Data structures are everywhere. They may play a short term role, e.g. during query parsing, an intermediate role, e.g. a temporary hash index, or a persistent role. All of them may be crucial in achieving the required performance. The code is in this respect your main (only) resource and you need ample of time to pin the area you really want to monitor. A tool to analyse access/instruction behavior is "valgrind --tool=callgrind mserver5...." success, Martin On 1/21/13 2:55 PM, Hassan, Ahmad wrote:
Hi Stefan,
Firstly, thanks for the email. I wish to identify the variable names of the BAT structures that constitute all the tables of the database schema and the place in the code where these BAT variables are declared/initialized using helper/wrapper functions for a database schema.
The reason I need this is, I want to add compiler level instrumentation to analyse the memory access pattern of the BAT data structures of a database schema. The only way I can identify various individual data structures at the compiler level is through the name of the variables in the source code. If you can point to me this information, it will be very helpful for the understanding and analyses. So I would particularly need the variable names in C code that represents the each individual table and then the variable name of each BAT structure with in each table.
The variable names will be correlated/mapped to the virtual addresses at the compiler level.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Stefan de Konink Sent: 21 January 2013 13:24 To: users-list@monetdb.org Subject: Re: BAT Data structures of a Schema
On 01/21/13 14:15, Hassan, Ahmad wrote:
Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please?
Do you mean the "wrapper code" that implements SQL storage (MonetDB/sql/storage/bat) or the underlying that does the raw processing?
Stefan _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi Ahmad That information can be seen from the storage module http://www.monetdb.org/Documentation/Cookbooks/SQLrecipies/storage-model Its implementation directs you to the kind of analysis made to come up with this information. Note, though, that these size do not say anything about how much/what is actually in memory during query processing. That's up to the OS memory manager and subject to any auxiliary temporary data structure produced on the fly, e.g. indices and views The minimal data required to access the persistent data is stored in the BBP datastructures, which is schema oblivious. The persistent part of the schema can be analysed like a normal relational table/query. regards, Martin On 1/21/13 4:12 PM, Hassan, Ahmad wrote:
Hi Martin,
Thanks for the advice. Ignoring all the rest of the intermediate/index/delta data structures and focussing only on the core data structures that hold Table's real/core data that is accessed over time to fetch and store data in memory. This part of the code would be in gdk or sql directory? Or in both directories please?
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Martin Kersten Sent: 21 January 2013 14:06 To: users-list@monetdb.org Subject: Re: BAT Data structures of a Schema
Hi Ahmad,
Your question requires an answer way beyond the typical message in this forum. (A couple of months studying the code base ;)) Data structures are everywhere. They may play a short term role, e.g. during query parsing, an intermediate role, e.g. a temporary hash index, or a persistent role. All of them may be crucial in achieving the required performance. The code is in this respect your main (only) resource and you need ample of time to pin the area you really want to monitor.
A tool to analyse access/instruction behavior is "valgrind --tool=callgrind mserver5...."
success, Martin
On 1/21/13 2:55 PM, Hassan, Ahmad wrote:
Hi Stefan,
Firstly, thanks for the email. I wish to identify the variable names of the BAT structures that constitute all the tables of the database schema and the place in the code where these BAT variables are declared/initialized using helper/wrapper functions for a database schema.
The reason I need this is, I want to add compiler level instrumentation to analyse the memory access pattern of the BAT data structures of a database schema. The only way I can identify various individual data structures at the compiler level is through the name of the variables in the source code. If you can point to me this information, it will be very helpful for the understanding and analyses. So I would particularly need the variable names in C code that represents the each individual table and then the variable name of each BAT structure with in each table.
The variable names will be correlated/mapped to the virtual addresses at the compiler level.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Stefan de Konink Sent: 21 January 2013 13:24 To: users-list@monetdb.org Subject: Re: BAT Data structures of a Schema
On 01/21/13 14:15, Hassan, Ahmad wrote:
Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please?
Do you mean the "wrapper code" that implements SQL storage (MonetDB/sql/storage/bat) or the underlying that does the raw processing?
Stefan _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi Ahmad, to answer your question literally --- provided I understand it correctly --- in most places of the code, the variable holding a BAT (the very BAT that the respective code currently works on) is called "b" --- but maybe, that's not what you wish to know ... Other than that, be aware that THE prime data structure used in MonetDB are simple C arrays; thus, the data structure by itself does not define any memory access pattern at all. The memory access patterns are determined by the operations that work on the data stored in arrays. Most of these are found in .../gdk/ and .../monetdb5/modules// . For memory access patters, also see http://repository.cwi.nl/search/searchrepository.php?id=14301 http://repository.cwi.nl/search/searchrepository.php?id=19993 http://repository.cwi.nl/search/searchrepository.php?id=20680 Best, Stefan On Mon, Jan 21, 2013 at 01:55:20PM +0000, Hassan, Ahmad wrote:
Hi Stefan,
Firstly, thanks for the email. I wish to identify the variable names of the BAT structures that constitute all the tables of the database schema and the place in the code where these BAT variables are declared/initialized using helper/wrapper functions for a database schema.
The reason I need this is, I want to add compiler level instrumentation to analyse the memory access pattern of the BAT data structures of a database schema. The only way I can identify various individual data structures at the compiler level is through the name of the variables in the source code. If you can point to me this information, it will be very helpful for the understanding and analyses. So I would particularly need the variable names in C code that represents the each individual table and then the variable name of each BAT structure with in each table.
The variable names will be correlated/mapped to the virtual addresses at the compiler level.
Thanks.
Kind Regards, Ahmad
-----Original Message----- From: users-list-bounces+ahmad.hassan=sap.com@monetdb.org [mailto:users-list-bounces+ahmad.hassan=sap.com@monetdb.org] On Behalf Of Stefan de Konink Sent: 21 January 2013 13:24 To: users-list@monetdb.org Subject: * Re: BAT Data structures of a Schema
On 01/21/13 14:15, Hassan, Ahmad wrote:
Please can you point me to the C code file(s) that hold the BAT objects for each column of the above tables please?
Do you mean the "wrapper code" that implements SQL storage (MonetDB/sql/storage/bat) or the underlying that does the raw processing?
Stefan _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
participants (4)
-
Hassan, Ahmad -
Martin Kersten -
Stefan de Konink -
Stefan Manegold