[Monetdb-developers] About MonetDB's modules
Hello, I am PhD student at EPFL, Switzerland, and I am working on database replication. I just started looking at MonetDB and I have a simple doubt. Are modules like the SQL module written in MIL or in C/C++? I downloaded the sources of the SQL front end and I found some files whose extension is ".mx" and which seem to implement the SQL translator into MIL. The code looks pretty much like C++. Moreover, I read somewhere in your web page that two-phase-locking is implemented outside the database kernel. Where exactly is it implemented for the SQL front end? Thank you very much in advance! Best regards, Rodrigo Schmidt
Hello, A quick reply Rodrigo Schmidt wrote:
Hello,
I am PhD student at EPFL, Switzerland, and I am working on database replication. I just started looking at MonetDB and I have a simple doubt. Are modules like the SQL module written in MIL or in C/C++?
Modules are mostly written in C, but they require more programming experience than MIL. MIL is primarilly used to quickly produce code, without all the worries you have when writing a C-based module.
I downloaded the sources of the SQL front end and I found some files whose extension is ".mx" and which seem to implement the SQL translator into MIL. The code looks pretty much like C++.
The SQL to MIL translator is written in C. The code and documentation are packaged in 'mx' format, which supports combining code and documentation. The Mx tool is provided in the distribution.
Moreover, I read somewhere in your web page that two-phase-locking is implemented outside the database kernel. Where exactly is it implemented for the SQL front end?
It is handled in the catalog management area. Basically, the kernel provides a module to implement locks and semaphores. However, the relational algebra engine does not include these calls to separate the users, it is expected that the front-end compiler provides these calls. This is in line with what some modern DBMS do.
Thank you very much in advance!
Success, keep us informed on your progress and findings. Martin Kersten
Best regards, Rodrigo Schmidt
------------------------------------------------------- This SF.Net email is sponsored by Yahoo. Introducing Yahoo! Search Developer Network - Create apps using Yahoo! Search APIs Find out how you can build Yahoo! directly into your own Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Wed, Jun 01, 2005 at 06:02:05PM +0200, Martin Kersten wrote:
Hello,
Rodrigo Schmidt wrote: Seems I made a mistake in my earlier reply.
Hello,
I am PhD student at EPFL, Switzerland, and I am working on database replication. I just started looking at MonetDB and I have a simple doubt. Are modules like the SQL module written in MIL or in C/C++?
Modules are mostly written in C, but they require more programming experience than MIL. MIL is primarilly used to quickly produce code, without all the worries you have when writing a C-based module.
Indeed SQL server module is writen in C. The SQL server module translates SQL into MIL-statements which are then executed on MonetDB/MIL server. Some small procedures are done directly in mil.
I downloaded the sources of the SQL front end and I found some files whose extension is ".mx" and which seem to implement the SQL translator into MIL. The code looks pretty much like C++.
The SQL to MIL translator is written in C. The code and documentation are packaged in 'mx' format, which supports combining code and documentation. The Mx tool is provided in the distribution.
Moreover, I read somewhere in your web page that two-phase-locking is implemented outside the database kernel. Where exactly is it implemented for the SQL front end?
It is handled in the catalog management area. Basically, the kernel provides a module to implement locks and semaphores. However, the relational algebra engine does not include these calls to separate the users, it is expected that the front-end compiler provides these calls. This is in line with what some modern DBMS do.
The MonetDB/MIL server indeed only comes with the handles to build transaction support (such as locking etc). The SQL server complements this with full transaction support. This is based on Optimistic Concurrency Control. A transaction simply starts and at the end checks the used columns where not changed by other transactions during its lifetime. All changes are then logged to a writeahead log and once this is completed the 'commit' is done. Niels Nes
Thank you very much in advance!
Success, keep us informed on your progress and findings.
Martin Kersten
Best regards, Rodrigo Schmidt
------------------------------------------------------- This SF.Net email is sponsored by Yahoo. Introducing Yahoo! Search Developer Network - Create apps using Yahoo! Search APIs Find out how you can build Yahoo! directly into your own Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
------------------------------------------------------- This SF.Net email is sponsored by Yahoo. Introducing Yahoo! Search Developer Network - Create apps using Yahoo! Search APIs Find out how you can build Yahoo! directly into your own Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Hi Niels (and others), Thank you very much for the help.
The MonetDB/MIL server indeed only comes with the handles to build transaction support (such as locking etc). The SQL server complements this with full transaction support. This is based on Optimistic Concurrency Control. A transaction simply starts and at the end checks the used columns where not changed by other transactions during its lifetime. All changes are then logged to a writeahead log and once this is completed the 'commit' is done.
Does this mean that a transaction can be aborted by the time the client issues a commit request?? Wasn't it better to implement classic 2PL (two-phase locking) for an MMDB? At least the early papers in main memory databases advocate using simple 2PL with coarse lock granularity, since transactions execute faster and concurrency becomes less important. Thanks, Rodrigo
On Wed, Jun 01, 2005 at 08:40:35PM +0200, Rodrigo Schmidt wrote:
Hi Niels (and others),
Thank you very much for the help.
The MonetDB/MIL server indeed only comes with the handles to build transaction support (such as locking etc). The SQL server complements this with full transaction support. This is based on Optimistic Concurrency Control. A transaction simply starts and at the end checks the used columns where not changed by other transactions during its lifetime. All changes are then logged to a writeahead log and once this is completed the 'commit' is done.
Does this mean that a transaction can be aborted by the time the client issues a commit request??
Indeed, but also 'classic 2PL' can still abort (allthough only because of completely different reasons).
Wasn't it better to implement classic 2PL (two-phase locking) for an MMDB? At least the early papers in main memory databases advocate using simple 2PL with coarse lock granularity, since transactions execute faster and concurrency becomes less important.
The current 'occ' usually works well, ie no lock overhead and no deadlock detection. Also the current scheme has a very simple 'abort',ie just throw away the current copies. But I agree its far from perfect and we have started some work on a 'simple' lock based version. Niels
Thanks, Rodrigo
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Hi,
Indeed, but also 'classic 2PL' can still abort (allthough only because of completely different reasons).
Sure, but the spurious aborts should happen more often with OCC. Moreover, the reason why transactions are aborted using OCC renders a database replication algorithm more complex.
The current 'occ' usually works well, ie no lock overhead and no deadlock detection. Also the current scheme has a very simple 'abort',ie just throw away the current copies. But I agree its far from perfect and we have started some work on a 'simple' lock based version.
Do you have any idea of when this new module will be finished? I will probably have to write some simple 2PL mechanism to use MonetDB with our replication protocols. I do not intend to cover deadlock detection but since I am not used to Monet's structure and sources, it might take a while until I get it finished working alone. If you have plans to finish it soon, I could try to help you to get it done faster. BTW, here are two questions I sent to Martin Kersten and forgot to forward to the developers list. Actually I changed them a little bit as a result of your previous answers First, is it easy (feasible) to write a module to extend the kernel and provide consistent snapshots for read-only transactions while update transactions execute without blocking? Second, is there a simple way to bypass Mapi and submit SQL requests directly to the server's SQL front-end without the cost of a socket connection? Thanks again for the help. All the best, Rodrigo
participants (3)
-
Martin Kersten -
Niels Nes -
Rodrigo Schmidt