Hi Fabian, happy to ear you so soon, I'm requested to build a huge database containing meteo observations. There are about 200 meteostations in a given geographic area. Observations are collected by a system that after some software procedures (aggregation, interpolation, ...) generates some mappings in a grid of about 30000 points to feed then model runs. There are 24 such grids for each day during the year, each point of the grid is related to about 20 meteo parameters. The first need we have is to store all these info in a structured way, in the most efficient way. Then there is the need of making some analisys on these data, mainly statistical analisys but also to be able to do spatial queries on them. Time range of analisys could be from one month or less but could also arrive to cover many years, let's say 20 years ..., in the case of climatological analisys. The analisys should be done in the most effective way. To support these activities we need to write software on our own because we need to integrate an existing system. I then began to search for database management systems that could be able to manage such data burden and amongst others I've found monetdb that seems to fit the problem we have. It's oriented to scientific data/observations, it has programming bindings in languages as java and C that could make it a profitable solution for us, it's open source (we are using only open source products ...) has some kind of spatial extensions, ... and here we are ... These are the reasons I'm asking for C API. In this context though I need to evaluate its usage and so I'm searching for informattions. Best regards. -------------------------------------------------------------- -- Dott. Ing. Luca Paganotti -- Via dei Giardini 9 -- 21035 Cunardo (VA) -- 393 1346898 -------------------------------------------------------------- -- sourceforge email: -- lucapaganotti@users.sourceforge.net -- enml.org email: -- luca.paganotti@enml.org -- skype name: luca.paganotti -------------------------------------------------------------- -- On Fri, Nov 18, 2011 at 8:28 AM, Fabian Groffen wrote:

Hi Luca,

On 18-11-2011 08:20:11 +0100, luca paganotti wrote:

...

I'm interested in learning more about monetdb, but I've some difficulties in finding the informations I'm searching for.

I would like to learn more about the C API, so I'm searching for some coding examples. The only thing I found was presumably an old example (as Stefan told me) about mapi usage that unfortunately doesn't work for me with monetdb current release (tried both linux and windows releases). I wrote two times to the developer list but at the moment no meaningful answer.

I'm affraid the MAPI C-library isn't the best example of a clean and clear interface. It has many (hidden) complexities that are mastered currently only be a few of us.

Perhaps you could explain us in what way you intend to use MAPI. Maybe we can give you a start, or suggest you an alternative way to achieve your goal.

------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

Hi lefteris, (is this your name?) some comments below. On Sat, Nov 19, 2011 at 9:54 AM, Lefteris wrote:

Hi Luca,

monetdb is for sure the right choice for such application that you are describing.

Good to know.

Monetdb is a database system, composed of many software layers that optimize the execution of a task in mane levels. As such, you should use monetdb as a database system and not as a library that provides a data structure. what I am trying to advocate is that you don't need to bind in the c code! Your task is not to implement new features or extent monetdb, but to use its features to perform a task.

You're probably right, programmers (like me) tend to concentrate only on source code ... ;-) Anyway, to have the example on monetdb web site correctly built and run took me about 20 minutes on my mobile fedora 16.

What you need is to first use the mclient interface with the sql support to perform some preliminary tests for your application, and then hook your applications to monetdb by opening a connection and issuing sql statements to load and manipulate your data. A very important added bonus for your application is that we are currently also developing sciql, an extension to the sql language to support scientific query demands!

I took a look at SciQL web site yesterday as suggested by Martin Kersten. Have to read SciQL.pdf I've downloaded. So does monetdb implement SciQl or it will do in the future as you're currently developing? I view also SciDb web site, are SciQL and SciDB related in some way?

In short, my advice is to use monetdb as a database system and not as some c library for a "nosql" kind of infrastructure:)

Was not my intention to use a a dbms as a c library. My interest in the C API is only from an application perspective. I can make you a couple examples: 1) let's say I want to write a unix daemon that searches the web for METAR files. Whenever it finds a new file the daemon parses it and then stores structured info in a dbms. It's very likely that this daemon will be written in C, or at least I would make it this way. The C API can make me connect to a db and then to use any kind of SQL statement as I need it. 2) If I would write the bindings for a language not currently supported by monetdb like eiffel the best way is to have a C API. Best regards.

lefteris

ps. your application is very interesting for our research too, maybe we can have a more extensive offline discussion about the solutions that are available.

On Fri, Nov 18, 2011 at 9:11 AM, luca paganotti wrote:

...

Hi Fabian,

happy to ear you so soon,

I'm requested to build a huge database containing meteo observations. There are about 200 meteostations in a given geographic area. Observations are collected by a system that after some software procedures (aggregation, interpolation, ...) generates some mappings in a grid of about 30000 points to feed then model runs. There are 24 such grids for each day during the year, each point of the grid is related to about 20 meteo parameters. The first need we have is to store all these info in a structured way, in the most efficient way. Then there is the need of making some analisys on these data, mainly statistical analisys but also to be able to do spatial queries on them. Time range of analisys could be from one month or less but could also arrive to cover many years, let's say 20 years ..., in the case of climatological analisys. The analisys should be done in the most effective way.

To support these activities we need to write software on our own because we need to integrate an existing system.

I then began to search for database management systems that could be able to manage such data burden and amongst others I've found monetdb that seems to fit the problem we have. It's oriented to scientific data/observations, it has programming bindings in languages as java and C that could make it a profitable solution for us, it's open source (we are using only open source products ...) has some kind of spatial extensions, ... and here we are ...

These are the reasons I'm asking for C API.

In this context though I need to evaluate its usage and so I'm searching for informattions.

Best regards.

-------------------------------------------------------------- -- Dott. Ing. Luca Paganotti -- Via dei Giardini 9 -- 21035 Cunardo (VA) -- 393 1346898 -------------------------------------------------------------- -- sourceforge email: -- lucapaganotti@users.sourceforge.net -- enml.org email: -- luca.paganotti@enml.org -- skype name: luca.paganotti -------------------------------------------------------------- --

On Fri, Nov 18, 2011 at 8:28 AM, Fabian Groffen wrote:

...

Hi Luca,

On 18-11-2011 08:20:11 +0100, luca paganotti wrote:

...

I'm interested in learning more about monetdb, but I've some difficulties in finding the informations I'm searching for.

I would like to learn more about the C API, so I'm searching for some coding examples. The only thing I found was presumably an old example (as Stefan told me) about mapi usage that unfortunately doesn't work for me with monetdb current release (tried both linux and windows releases). I wrote two times to the developer list but at the moment no meaningful answer.

I'm affraid the MAPI C-library isn't the best example of a clean and clear interface. It has many (hidden) complexities that are mastered currently only be a few of us.

Perhaps you could explain us in what way you intend to use MAPI. Maybe we can give you a start, or suggest you an alternative way to achieve your goal.

------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

On Sun, Nov 20, 2011 at 5:19 AM, Fabian Groffen wrote:

...

Was not my intention to use a a dbms as a c library. My interest in the C API is only from an application perspective. I can make you a couple examples: 1) let's say I want to write a unix daemon that searches the web for METAR files. Whenever it finds a new file the daemon parses it and

On 19-11-2011 22:12:09 +0100, luca paganotti wrote: then

...

stores structured info in a dbms. It's very likely that this daemon will be written in C, or at least I would make it this way. The C API can make me connect to a db and then to use any kind of SQL statement as I need it. 2) If I would write the bindings for a language not currently supported by monetdb like eiffel the best way is to have a C API.

We prefer bindings to be written in the language itself, as much as possible. I'm almost sure wrapping the MAPI C-API in Eiffel will result in something foreign to the language (if you can figure out pre and post conditions anyway).

Hi Fabian, the intermediate results of an eiffel system build are c source files, then to have the final results you use a C compiler, I don't see any language mismatch. Best regards.

Perhaps you better look at the Python, Perl, PHP, Ruby and JDBC bindings instead, since they all implement the MAPI wire protocol in the language of the binding itself.

------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

Hi Fabian, the link you gave me is rather outdated, take a look at http://docs.eiffel.com/book/solutions/eiffelstore for the current docs about eiffelstore. Surely I'm talking about interfacing with monetdb which for me is a black box, I don't know its internals. But taking for example the matisse eiffel bindings, they do something similar to what I described, besides the results were not so bad. I'm not saying that I want now to write eiffel bindings for monetdb, it was only an example trying to explore some possibilities. I think too that extending eiffelstore to manage also monetdb could be interesting but has to be done carefully ... I'm interested in the C API because I feel much confident with C but if this can cause troubles I'll fall back to java and jdbc. Have a nice day. On Sun, Nov 20, 2011 at 9:45 PM, Fabian Groffen wrote:

...

On Sun, Nov 20, 2011 at 5:19 AM, Fabian Groffen <[1] Fabian.Groffen@cwi.nl> wrote:

...

We prefer bindings to be written in the language itself, as much as possible. I'm almost sure wrapping the MAPI C-API in Eiffel will result in something foreign to the language (if you can figure out pre and post conditions anyway).

the intermediate results of an eiffel system build are c source files,

On 20-11-2011 09:50:31 +0100, luca paganotti wrote: then

...

to have the final results you use a C compiler, I don't see any language mismatch.

Sure, but mapping each C function into a C-wrapper for Eiffel will make a mess. Swig is a well known example of that.

I would more expect something like: http://archive.eiffel.com/products/store.html

------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers