OK. I figured that must be the command after digging through the manuals. But for the vertically partitioned approach demonstrated in various research papers, I believe the end user has to manually load the RDF data split according to the predicates into different tables, which also requires that the SQL query being issued specifies these tables and joins across these tables as opposed to one big triple table. Correct me if I am wrong. Thanks. Medha On Tue, Oct 6, 2009 at 5:06 PM, Sjoerd Mullender wrote:

On 2009-10-06 17:59, Medha Atre wrote:

...

Hi,

Thanks for the reply.

At this point, all I am looking for is to load RDF NTriples data -- treated as a 3 column table -- into MonetDB using "vertical partitioning" to index the data nicely (as I read about MonetDB being used for the same in different research papers recently.. e.g. http://portal.acm.org/citation.cfm?id=1453856.1453927 RDF-3X etc). I am not really looking for a great support for different RDF formats (as the data has already been cleaned into NTriples format using rdflib), I want to check the correctness of the results generated using my own querying tool.

So can you please suggest me a method of treating the given RDF NTriples file as a simple "space separated" 3-column data file from which I can load the data into MonetDB creating the required indexes or "vertical partitions"?

Perhaps something like COPY <n> RECORDS INTO <table> FROM <file> USING DELIMITERS ' ','\n'; where <n> is the number of tuples, <table> is the name of a table that you created for this, and <file> is the full path name of the file (in single quotes).

...

Querying with SQL front end is fine with me!

Looking forward to your help.

Medha

...

------------------------------

Message: 5 Date: Tue, 6 Oct 2009 10:19:57 +0200 From: Lefteris Subject: Re: [MonetDB-users] loading RDF data in MonetDB To: Communication channel for MonetDB users         Message-ID:         <852badbc0910060119x4f125a06qd5fc7ed0c56035c@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1

Hi,

there is a highly experimental module for MonetDB to load RDF data. This is not activated in the stable release, and in order to use it you will need to do some compilation on your own and possible report back any problems you have. If you are willing to jump deeper in the MonetDB experience I can guide you on how to do this. The module will read any RDF format supported by the rdflib library and create a triple store with appropriate indices. However, the only way to query the triple store at this point is through the SQL front-end.

regrads,

lefteris

On Tue, Oct 6, 2009 at 8:59 AM, Medha Atre  wrote:

...

Hi,

I intend to use MonetDB for some of my experimental work using RDF data. I was wondering if there is an easy way to load RDF data (in NTriples format) in MonetDB making use of the column-wise vertical partitioning of the RDF data, based on properties/predicates in the RDF data.

Some pointers or help regarding this is much appreciated.

I am sorry if this is not the correct forum to post this message.

Thanks. Medha

Sorry for the typo, before MonetDB-SQL, I am trying to compile MonetDB5-xx.tar.gz and I am getting the Mapi error while compiling MonetDB5 Thanks. Medha On Tue, Oct 6, 2009 at 8:43 PM, Medha Atre wrote:

I tried installing MonetDB-SQL-xx.tar.gz from sources.

Make fails giving error -lMapi not found.

I was wondering why does it need Mapi library when all I want to do is simply build a database and load some data and query it!!

Can someone help please? The READMEs in the sources are of no help and there is no step-by-step setup procedure given on the web-documentation.

Thanks in anticipation.

Medha

On Tue, Oct 6, 2009 at 8:26 PM, Medha Atre wrote:

...

OK. I figured that must be the command after digging through the manuals. But for the vertically partitioned approach demonstrated in various research papers, I believe the end user has to manually load the RDF data split according to the predicates into different tables, which also requires that the SQL query being issued specifies these tables and joins across these tables as opposed to one big triple table.

Correct me if I am wrong.

Thanks.

Medha

On Tue, Oct 6, 2009 at 5:06 PM, Sjoerd Mullender wrote:

...

On 2009-10-06 17:59, Medha Atre wrote:

...

Hi,

Thanks for the reply.

At this point, all I am looking for is to load RDF NTriples data -- treated as a 3 column table -- into MonetDB using "vertical partitioning" to index the data nicely (as I read about MonetDB being used for the same in different research papers recently.. e.g. http://portal.acm.org/citation.cfm?id=1453856.1453927 RDF-3X etc). I am not really looking for a great support for different RDF formats (as the data has already been cleaned into NTriples format using rdflib), I want to check the correctness of the results generated using my own querying tool.

So can you please suggest me a method of treating the given RDF NTriples file as a simple "space separated" 3-column data file from which I can load the data into MonetDB creating the required indexes or "vertical partitions"?

Perhaps something like COPY <n> RECORDS INTO <table> FROM <file> USING DELIMITERS ' ','\n'; where <n> is the number of tuples, <table> is the name of a table that you created for this, and <file> is the full path name of the file (in single quotes).

...

Querying with SQL front end is fine with me!

Looking forward to your help.

Medha

...

------------------------------

Message: 5 Date: Tue, 6 Oct 2009 10:19:57 +0200 From: Lefteris Subject: Re: [MonetDB-users] loading RDF data in MonetDB To: Communication channel for MonetDB users         Message-ID:         <852badbc0910060119x4f125a06qd5fc7ed0c56035c@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1

Hi,

there is a highly experimental module for MonetDB to load RDF data. This is not activated in the stable release, and in order to use it you will need to do some compilation on your own and possible report back any problems you have. If you are willing to jump deeper in the MonetDB experience I can guide you on how to do this. The module will read any RDF format supported by the rdflib library and create a triple store with appropriate indices. However, the only way to query the triple store at this point is through the SQL front-end.

regrads,

lefteris

On Tue, Oct 6, 2009 at 8:59 AM, Medha Atre  wrote:

...

Hi,

I intend to use MonetDB for some of my experimental work using RDF data. I was wondering if there is an easy way to load RDF data (in NTriples format) in MonetDB making use of the column-wise vertical partitioning of the RDF data, based on properties/predicates in the RDF data.

Some pointers or help regarding this is much appreciated.

I am sorry if this is not the correct forum to post this message.

Thanks. Medha

Ahh.. thanks a lot. It worked. But I still have some other hiccups -- After following the short documentation given on http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Start-and-Stop.html I tried following $ merovingian $ monetdb create rdf $ monetdb start rdf *starting 'rdf' failed: database 'rdf' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities* If I do "monetdb status" it shows me "rdf" running and health 100% but then why this error message? Also if I try connecting to the rdf database using mclient, it says -- $ mclient -lsql --database=rdf !MALException:setScenario:Scenario not initialized 'sql' But when I do "ps" mserver is shown to be running with "include sql" construct -- $ ps -aef | grep <myuserid> | grep mserver <myuserid> 12219 11936 0 15:48 ? 00:00:00 /path/to/MonetDB5/bin/mserver5 --config=/path/to/MonetDB5/etc/monetdb5.conf --dbname=rdf1 --dbinit=include sql; --set monet_daemon yes --set mapi_open false --set mapi_autosense true --set mapi_port 50001 --set monet_vault_key /path/to/MonetDB5/var/MonetDB5/dbfarm/rdf/.vaultkey Below I am giving last few lines from merovingian.log 2009-10-06 15:47:02 MSG rdf[12197]: # MonetDB server v5.14.2, based on kernel v1.32.2 2009-10-06 15:47:02 MSG rdf[12197]: # Serving database 'rdf', using 8 threads 2009-10-06 15:47:02 MSG rdf[12197]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2009-10-06 15:47:02 MSG rdf[12197]: # Copyright (c) 1993-July 2008 CWI. 2009-10-06 15:47:02 MSG rdf[12197]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved 2009-10-06 15:47:02 MSG rdf[12197]: # Visit http://monetdb.cwi.nl/ for further information 2009-10-06 15:47:02 MSG rdf[12197]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ 2009-10-06 15:47:11 ERR merovingian[11936]: failed to fork mserver: database 'rdf' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities 2009-10-06 15:47:29 MSG merovingian[11936]: proxying client localhost.localdomain:38697 for database 'rdf' to mapi:monetdb://127.0.0.1:50001/rdf 2009-10-06 15:47:29 MSG merovingian[11936]: client localhost.localdomain:38697 has disconnected from proxy 2009-10-06 15:47:50 MSG rdf[12197]: !MALException:malInclude:could not open file: sql ------------------------------------------------------------------------------------------- Also regarding your comment about PSO reordering below, does it mean that if I just reorganize my triples as Predicate, Subject, Object and simply load using "copy into" into MonetDB it will take care of vertically partitioning them? Or do I still have to batch load into different tables according to different properties? Thanks. Medha On Tue, Oct 6, 2009 at 8:55 PM, Peter Boncz wrote:

Hi,

to compile from sources the easiest is the supersource tarball, this is the first clickable item you encounter on the download page monetdb.cwi.nl/projects/monetdb/Download

if you insist on getting the individual components one by one, you need: - MonetDB-1 - MonetDB-client (this one has the Mapi library you were missing) - MonetDB5-server - MonetDB-SQL

ah, and that idea of vertically decomposed RDF, it is essentially the same as PSO ordering, which helps some queries, but then other queries profit from other orderings (POS,SPO,SOP,OSP,OPS); nothing new under the sun. The RDF parser Lefteris made effectively creates all these orders.

Peter

-----Original Message----- From: Medha Atre [mailto:medha.atre@gmail.com] Sent: Wednesday, October 07, 2009 2:45 AM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] loading RDF data in MonetDB

Sorry for the typo, before MonetDB-SQL, I am trying to compile MonetDB5-xx.tar.gz and I am getting the Mapi error while compiling MonetDB5

Thanks. Medha

On Tue, Oct 6, 2009 at 8:43 PM, Medha Atre wrote:

...

I tried installing MonetDB-SQL-xx.tar.gz from sources.

Make fails giving error -lMapi not found.

I was wondering why does it need Mapi library when all I want to do is simply build a database and load some data and query it!!

Can someone help please? The READMEs in the sources are of no help and there is no step-by-step setup procedure given on the web-documentation.

Thanks in anticipation.

Medha

On Tue, Oct 6, 2009 at 8:26 PM, Medha Atre wrote:

...

OK. I figured that must be the command after digging through the manuals. But for the vertically partitioned approach demonstrated in various research papers, I believe the end user has to manually load the RDF data split according to the predicates into different tables, which also requires that the SQL query being issued specifies these tables and joins across these tables as opposed to one big triple table.

Correct me if I am wrong.

Thanks.

Medha

On Tue, Oct 6, 2009 at 5:06 PM, Sjoerd Mullender wrote:

...

On 2009-10-06 17:59, Medha Atre wrote:

...

Hi,

Thanks for the reply.

At this point, all I am looking for is to load RDF NTriples data -- treated as a 3 column table -- into MonetDB using "vertical partitioning" to index the data nicely (as I read about MonetDB being used for the same in different research papers recently.. e.g. http://portal.acm.org/citation.cfm?id=1453856.1453927 RDF-3X etc). I am not really looking for a great support for different RDF formats (as the data has already been cleaned into NTriples format using rdflib), I want to check the correctness of the results generated using my own querying tool.

So can you please suggest me a method of treating the given RDF NTriples file as a simple "space separated" 3-column data file from which I can load the data into MonetDB creating the required indexes or "vertical partitions"?

Perhaps something like COPY <n> RECORDS INTO <table> FROM <file> USING DELIMITERS ' ','\n'; where <n> is the number of tuples, <table> is the name of a table that you created for this, and <file> is the full path name of the file (in single quotes).

...

Querying with SQL front end is fine with me!

Looking forward to your help.

Medha

...

------------------------------

Message: 5 Date: Tue, 6 Oct 2009 10:19:57 +0200 From: Lefteris Subject: Re: [MonetDB-users] loading RDF data in MonetDB To: Communication channel for MonetDB users         Message-ID:         <852badbc0910060119x4f125a06qd5fc7ed0c56035c@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1

Hi,

there is a highly experimental module for MonetDB to load RDF data. This is not activated in the stable release, and in order to use it you will need to do some compilation on your own and possible report back any problems you have. If you are willing to jump deeper in the MonetDB experience I can guide you on how to do this. The module will read any RDF format supported by the rdflib library and create a triple store with appropriate indices. However, the only way to query the triple store at this point is through the SQL front-end.

regrads,

lefteris

On Tue, Oct 6, 2009 at 8:59 AM, Medha Atre  wrote: > Hi, > > I intend to use MonetDB for some of my experimental work using RDF > data. I was wondering if there is an easy way to load RDF data (in > NTriples format) in MonetDB making use of the column-wise vertical > partitioning of the RDF data, based on properties/predicates in the > RDF data. > > Some pointers or help regarding this is much appreciated. > > I am sorry if this is not the correct forum to post this message. > > Thanks. > Medha

I installed in the following order as suggested by Peter Boncz in his previous mail: http://monetdb.cwi.nl/downloads/sources/Aug2009-SP1/MonetDB-1.32.2.tar.gz http://monetdb.cwi.nl/downloads/sources/Aug2009-SP1/MonetDB-client-1.32.2.ta... http://monetdb.cwi.nl/downloads/sources/Aug2009-SP1/MonetDB5-server-5.14.2.t... http://monetdb.cwi.nl/downloads/sources/Aug2009-SP1/MonetDB-SQL-2.32.2.tar.g... Every time after installing previous library I put its "bin" dir in the $PATH so that for the next one's installation it can pick up the required dependency libraries from the earlier one's installation. E.g. $ tar -xzf MonetDB-1.32.2.tar.gz $ cd MonetDB-1.32.2 $ ./configure --prefix=/path/to/MonetDB $ make install $ export PATH=$PATH:/path/to/MonetDB/bin $ tar -xzf MonetDB-client-1.32.2.tar.gz $ cd MonetDB-client-1.32.2 $ ./configure --prefix=/path/to/MonetDB-client $ make install $ export PATH=$PATH:/path/to/MonetDB-client/bin $ tar -xzf MonetDB5-server-5.14.2.tar.gz $ cd MonetDB5-server-5.14.2 $ ./configure --prefix=/path/to/MonetDB5 $ make install $ export PATH=$PATH:/path/to/MonetDB5/bin $ tar -xzf MonetDB-SQL-2.32.2.tar.gz $ cd MonetDB-SQL-2.32.2 $ ./configure --prefix=/path/to/MonetDB-SQL $ make install $ export PATH=$PATH:/path/to/MonetDB-SQL/bin And then I ran the other commands I already mentioned below (e.g. merovingian etc). There were no error messages in the "make install" of any of the 4 packages I have listed above. Could you please help me understand what might be going wrong? Thanks. Medha On Wed, Oct 7, 2009 at 4:42 AM, Fabian Groffen wrote:

On 06-10-2009 22:14:22 -0400, Medha Atre wrote:

...

$ monetdb start rdf *starting 'rdf' failed: database 'rdf' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities*

...

If I do "monetdb status" it shows me "rdf" running and health 100% but then why this error message? Also if I try connecting to the rdf database using mclient, it says --

$ mclient -lsql --database=rdf !MALException:setScenario:Scenario not initialized 'sql'

...

2009-10-06 15:47:50 MSG rdf[12197]: !MALException:malInclude:could not open file: sql

How did you install MonetDB/SQL?  This indicates something went wrong with the installation.

Thanks Fabian. This worked. I just have small another question -- is there a way to store the output of a "select" query run on "mclient" into a file? I would like to compare the results with the ones generated by some other tool. Thanks. Medha On Wed, Oct 7, 2009 at 2:26 PM, Fabian Groffen wrote:

On 07-10-2009 14:17:47 -0400, Medha Atre wrote: [snip installation of all packages in a separate prefix]

...

And then I ran the other commands I already mentioned below (e.g. merovingian etc). There were no error messages in the "make install" of any of the 4 packages I have listed above.

Could you please help me understand what might be going wrong?

any reason you choose not to use  monetdb-install.sh --enable-sql --prefix=/path/to/MonetDB ?  If you don't have it, download the latest version of this script from http://monetdb.cwi.nl/projects/monetdb/Assets/monetdb-install.sh

Alternatively, just download a Super Source Tarball, which has this script included.

That basically solves all your problems.  If you want to have all components installed in their own prefix, you'll have to do some post-fixing to get the modules installed where mserver expects them to be.

On 2009-10-08 08:05, Niels Nes wrote:

On Wed, Oct 07, 2009 at 10:48:33PM -0400, Medha Atre wrote:

...

Thanks Fabian. This worked.

I just have small another question -- is there a way to store the output of a "select" query run on "mclient" into a file? I would like to compare the results with the ones generated by some other tool.

see http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Data-Manipulation.h... section about copy into file

If you're working interactively with mclient, you can also use \> file select ... \> The first line redirects subsequent output to the file, the last line directs it back to the terminal. Check out the \? command.

Niels

...

Thanks. Medha

On Wed, Oct 7, 2009 at 2:26 PM, Fabian Groffen wrote:

...

On 07-10-2009 14:17:47 -0400, Medha Atre wrote: [snip installation of all packages in a separate prefix]

...

And then I ran the other commands I already mentioned below (e.g. merovingian etc). There were no error messages in the "make install" of any of the 4 packages I have listed above.

Could you please help me understand what might be going wrong?

any reason you choose not to use monetdb-install.sh --enable-sql --prefix=/path/to/MonetDB ? If you don't have it, download the latest version of this script from http://monetdb.cwi.nl/projects/monetdb/Assets/monetdb-install.sh

Alternatively, just download a Super Source Tarball, which has this script included.

That basically solves all your problems. If you want to have all components installed in their own prefix, you'll have to do some post-fixing to get the modules installed where mserver expects them to be.

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Sjoerd Mullender

Hi, Thanks all for the valuable inputs. I could compile and successfully run MonetDB on one m/c, but I am facing issues in loading data into it on another m/c. The other m/c is actually much higher end m/c with 16G memory (server architecture) as opposed the first one in which I loaded the same data. Every time I try to load the data (RDF data in integer ID form), it keeps giving "connection timeout" error. Note: the data is really large (>1 billion triples). I am running "mclient" on the same m/c where monetdb server is running. I tried it couple of times until now, but the issue persists. Can someone throw some light on what might be going wrong? Medha On Thu, Oct 8, 2009 at 3:23 AM, Stefan Manegold wrote:

On Thu, Oct 08, 2009 at 08:56:35AM +0200, Sjoerd Mullender wrote:

...

On 2009-10-08 08:05, Niels Nes wrote:

...

On Wed, Oct 07, 2009 at 10:48:33PM -0400, Medha Atre wrote:

...

Thanks Fabian. This worked.

I just have small another question -- is there a way to store the output of a "select" query run on "mclient" into a file? I would like to compare the results with the ones generated by some other tool.

see http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Data-Manipulation.h... section about copy into file

If you're working interactively with mclient, you can also use \> file select ... \> The first line redirects subsequent output to the file, the last line directs it back to the terminal.

Check out the \? command.

You can also simply use your shell's features:

       mclient -lsql -s'select ...' > output-file        echo 'select ...' | mclient -lsql > output-file        mclient -lsql < query-file > output-file        mclient -lsql query-file > output-file

Stefan

...

...

Niels

...

Thanks. Medha

On Wed, Oct 7, 2009 at 2:26 PM, Fabian Groffen  wrote:

...

On 07-10-2009 14:17:47 -0400, Medha Atre wrote: [snip installation of all packages in a separate prefix]

...

And then I ran the other commands I already mentioned below (e.g. merovingian etc). There were no error messages in the "make install" of any of the 4 packages I have listed above.

Could you please help me understand what might be going wrong?

any reason you choose not to use   monetdb-install.sh --enable-sql --prefix=/path/to/MonetDB ?  If you don't have it, download the latest version of this script from http://monetdb.cwi.nl/projects/monetdb/Assets/monetdb-install.sh

Alternatively, just download a Super Source Tarball, which has this script included.

That basically solves all your problems.  If you want to have all components installed in their own prefix, you'll have to do some post-fixing to get the modules installed where mserver expects them to be.

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Sjoerd Mullender

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  | | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       | | The Netherlands     | Fax : +31 (20) 592-4312       |

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

The error response produced is not MonetDB related. It is a result from the compute environment, which closes a connection is it has been silent for a long time. It may be as simple as timeout settings of your basic communication channel. There is nothing we can do about that. This may happen if you start any query (or dataload with integrity checking) that takes a long time. Lefteris wrote:

Since you are loading triples in ID integer form, a billion triples should be no problem in a 16gig memory machine. Could you please give more information on what commands/queries you use to load the data, and the exact full error message. Which OS are you running in the machine and which version of monet you installed? (stable or current?)

regards,

lefteris

On Fri, Oct 16, 2009 at 7:20 PM, Medha Atre wrote:

...

Hi,

Thanks all for the valuable inputs. I could compile and successfully run MonetDB on one m/c, but I am facing issues in loading data into it on another m/c. The other m/c is actually much higher end m/c with 16G memory (server architecture) as opposed the first one in which I loaded the same data. Every time I try to load the data (RDF data in integer ID form), it keeps giving "connection timeout" error. Note: the data is really large (>1 billion triples). I am running "mclient" on the same m/c where monetdb server is running.

I tried it couple of times until now, but the issue persists.

Can someone throw some light on what might be going wrong?

Medha

On Thu, Oct 8, 2009 at 3:23 AM, Stefan Manegold wrote:

...

On Thu, Oct 08, 2009 at 08:56:35AM +0200, Sjoerd Mullender wrote:

...

...

...

Thanks Fabian. This worked.

I just have small another question -- is there a way to store the output of a "select" query run on "mclient" into a file? I would like to compare the results with the ones generated by some other tool. see http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Data-Manipulation.h...

On Wed, Oct 07, 2009 at 10:48:33PM -0400, Medha Atre wrote: section about copy into file If you're working interactively with mclient, you can also use \> file select ... \> The first line redirects subsequent output to the file, the last line

On 2009-10-08 08:05, Niels Nes wrote: directs it back to the terminal.

Check out the \? command. You can also simply use your shell's features:

mclient -lsql -s'select ...' > output-file echo 'select ...' | mclient -lsql > output-file mclient -lsql < query-file > output-file mclient -lsql query-file > output-file

Stefan

...

...

Niels

...

Thanks. Medha

On Wed, Oct 7, 2009 at 2:26 PM, Fabian Groffen wrote: > On 07-10-2009 14:17:47 -0400, Medha Atre wrote: > [snip installation of all packages in a separate prefix] >> And then I ran the other commands I already mentioned below (e.g. >> merovingian etc). >> There were no error messages in the "make install" of any of the 4 >> packages I have listed above. >> >> Could you please help me understand what might be going wrong? > any reason you choose not to use > monetdb-install.sh --enable-sql --prefix=/path/to/MonetDB > ? If you don't have it, download the latest version of this script from > http://monetdb.cwi.nl/projects/monetdb/Assets/monetdb-install.sh > > Alternatively, just download a Super Source Tarball, which has this > script included. > > That basically solves all your problems. If you want to have all > components installed in their own prefix, you'll have to do some > post-fixing to get the modules installed where mserver expects them to > be. ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Sjoerd Mullender

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

On 16-10-2009 15:54:57 -0400, Medha Atre wrote:

Actually, I did a typo in my previous message. The response is not "connection *timeout*" but it's "connection *terminated*"! Since I don't own this machine, i will have to contact the administrators of the machine to check with them, but in the meanwhile searching on the web for the same error brought to my notice that several a few users have also reported the same problem and a bug was filed in MonetDB sometime early this year. I was just wondering if it's the same thing?

Connection terminated is definitely a MonetDB server problem/feature.

For more information, here are the last few lines of merovingian log

2009-10-16 07:13:52 MSG rdf[26793]: # MonetDB server v5.14.2, based on kernel v1.32.2 2009-10-16 07:13:52 MSG rdf[26793]: # Serving database 'rdf', using 8 threads 2009-10-16 07:13:52 MSG rdf[26793]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2009-10-16 07:13:52 MSG rdf[26793]: # Copyright (c) 1993-July 2008 CWI. 2009-10-16 07:13:52 MSG rdf[26793]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved 2009-10-16 07:13:52 MSG rdf[26793]: # Visit http://monetdb.cwi.nl/ for further information 2009-10-16 07:13:52 MSG rdf[26793]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ 2009-10-16 07:13:52 MSG rdf[26793]: # MonetDB/SQL module v2.32.2 loaded 2009-10-16 07:14:22 MSG merovingian[26784]: proxying client localhost.localdomain:43986 for database 'rdf' to mapi:monetdb://127.0.0.1:50001/rdf 2009-10-16 07:49:59 MSG merovingian[26784]: database 'rdf' (26793) was killed by signal 9

^^^ and signal 9 is a SIGKILL, so is your OS prehaps killing your mserver because it would bring down the system as a whole? OOM most probably. Can you check dmesg and the messages log for messages about victim processes being killed?

Yes. Actually I did observe that after starting to load the data, within few minutes, the server m/c would freeze and I was not even able to SSH into it. But I wonder why this would happen on one m/c and not on the other which is using exactly the same sources of MonetDB server and client, and same data to load! The m/c where mserver is getting killed is running CentOS, following is the output of "uname -a" (masking out the server domain name for security)! "Linux <servername>.<inst>.edu 2.6.18-128.4.1.el5 #1 SMP Tue Aug 4 20:19:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux" I checked "dmesg". The process was indeed killed by the OS. Here's the o/p from dmesg -- --------------------------------------- Out of memory: Killed process 5857 (mserver5). Out of memory: Killed process 5858 (mserver5). mserver5: page allocation failure. order:0, mode:0x200d2 Out of memory: Killed process 14891 (mserver5). Out of memory: Killed process 14892 (mserver5). mserver5: page allocation failure. order:0, mode:0x200d2 mserver5: page allocation failure. order:0, mode:0x200d2 VM: killing process mserver5 mserver5: page allocation failure. order:0, mode:0x200d2 Out of memory: Killed process 16950 (mserver5). Out of memory: Killed process 16951 (mserver5). mserver5: page allocation failure. order:0, mode:0x200d2 mserver5: page allocation failure. order:0, mode:0x200d2 mserver5 invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Out of memory: Killed process 26793 (mserver5). Out of memory: Killed process 26794 (mserver5). mserver5: page allocation failure. order:0, mode:0x200d2 mserver5: page allocation failure. order:0, mode:0x200d2 mserver5: page allocation failure. order:0, mode:0x200d2 --------------------------------------- Medha On Fri, Oct 16, 2009 at 4:00 PM, Sjoerd Mullender wrote:

On 2009-10-16 21:54, Medha Atre wrote:

...

Actually, I did a typo in my previous message. The response is not "connection *timeout*" but it's "connection *terminated*"! Since I don't own this machine, i will have to contact the administrators of the machine to check with them, but in the meanwhile searching on the web for the same error brought to my notice that several a few users have also reported the same problem and a bug was filed in MonetDB sometime early this year. I was just wondering if it's the same thing?

For more information, here are the last few lines of merovingian log

2009-10-16 07:13:52 MSG rdf[26793]: # MonetDB server v5.14.2, based on kernel v1.32.2 2009-10-16 07:13:52 MSG rdf[26793]: # Serving database 'rdf', using 8 threads 2009-10-16 07:13:52 MSG rdf[26793]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2009-10-16 07:13:52 MSG rdf[26793]: # Copyright (c) 1993-July 2008 CWI. 2009-10-16 07:13:52 MSG rdf[26793]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved 2009-10-16 07:13:52 MSG rdf[26793]: # Visit http://monetdb.cwi.nl/ for further information 2009-10-16 07:13:52 MSG rdf[26793]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ 2009-10-16 07:13:52 MSG rdf[26793]: # MonetDB/SQL module v2.32.2 loaded 2009-10-16 07:14:22 MSG merovingian[26784]: proxying client localhost.localdomain:43986 for database 'rdf' to mapi:monetdb://127.0.0.1:50001/rdf 2009-10-16 07:49:59 MSG merovingian[26784]: database 'rdf' (26793) was killed by signal 9

This killed by signal 9 is a hint that the OS thought the process was taking too many resources (presumably memory resources) and killed the process.

...

2009-10-16 07:50:00 MSG merovingian[26784]: client localhost.localdomain:43986 has disconnected from proxy

I will also look into the connection timeout problem if that's the cause.

Thanks for the help. Medha

On Fri, Oct 16, 2009 at 3:15 PM, Martin Kersten  wrote:

...

The error response produced is not MonetDB related. It is a result from the compute environment, which closes a connection is it has been silent for a long time. It may be as simple as timeout settings of your basic communication channel. There is nothing we can do about that. This may happen if you start any query (or dataload with integrity checking) that takes a long time.

Lefteris wrote:

...

Since you are loading triples in ID integer form, a billion triples should be no problem in a 16gig memory machine. Could you please give more information on what commands/queries you use to load the data, and the exact full error message. Which OS are you running in the machine and which version of monet you installed? (stable or current?)

regards,

lefteris

On Tue, Oct 06, 2009 at 08:43:46PM -0400, Medha Atre wrote:

I tried installing MonetDB-SQL-xx.tar.gz from sources.

Make fails giving error -lMapi not found.

I was wondering why does it need Mapi library when all I want to do is simply build a database and load some data and query it!!

Can someone help please? The READMEs in the sources are of no help and there is no step-by-step setup procedure given on the web-documentation.

On http://monetdb.cwi.nl/ "Download" leads to http://monetdb.cwi.nl/Download/ from where "Installation" leads to http://monetdb.cwi.nl/Download/Packages/ which explains to how to install pre-built binary packages, build from the super source ball, and point to the "documentation" about how to build from individual source tarballs at http://monetdb.cwi.nl/MonetDB/Documentation/Experts.html I hope this provides sufficent choices and information ... Stefan

Thanks in anticipation.

Medha

On Tue, Oct 6, 2009 at 8:26 PM, Medha Atre wrote:

...

OK. I figured that must be the command after digging through the manuals. But for the vertically partitioned approach demonstrated in various research papers, I believe the end user has to manually load the RDF data split according to the predicates into different tables, which also requires that the SQL query being issued specifies these tables and joins across these tables as opposed to one big triple table.

Correct me if I am wrong.

Thanks.

Medha

On Tue, Oct 6, 2009 at 5:06 PM, Sjoerd Mullender wrote:

...

On 2009-10-06 17:59, Medha Atre wrote:

...

Hi,

Thanks for the reply.

At this point, all I am looking for is to load RDF NTriples data -- treated as a 3 column table -- into MonetDB using "vertical partitioning" to index the data nicely (as I read about MonetDB being used for the same in different research papers recently.. e.g. http://portal.acm.org/citation.cfm?id=1453856.1453927 RDF-3X etc). I am not really looking for a great support for different RDF formats (as the data has already been cleaned into NTriples format using rdflib), I want to check the correctness of the results generated using my own querying tool.

So can you please suggest me a method of treating the given RDF NTriples file as a simple "space separated" 3-column data file from which I can load the data into MonetDB creating the required indexes or "vertical partitions"?

Perhaps something like COPY <n> RECORDS INTO <table> FROM <file> USING DELIMITERS ' ','\n'; where <n> is the number of tuples, <table> is the name of a table that you created for this, and <file> is the full path name of the file (in single quotes).

...

Querying with SQL front end is fine with me!

Looking forward to your help.

Medha

...

------------------------------

Message: 5 Date: Tue, 6 Oct 2009 10:19:57 +0200 From: Lefteris Subject: Re: [MonetDB-users] loading RDF data in MonetDB To: Communication channel for MonetDB users         Message-ID:         <852badbc0910060119x4f125a06qd5fc7ed0c56035c@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1

Hi,

there is a highly experimental module for MonetDB to load RDF data. This is not activated in the stable release, and in order to use it you will need to do some compilation on your own and possible report back any problems you have. If you are willing to jump deeper in the MonetDB experience I can guide you on how to do this. The module will read any RDF format supported by the rdflib library and create a triple store with appropriate indices. However, the only way to query the triple store at this point is through the SQL front-end.

regrads,

lefteris

On Tue, Oct 6, 2009 at 8:59 AM, Medha Atre  wrote:

...

Hi,

I intend to use MonetDB for some of my experimental work using RDF data. I was wondering if there is an easy way to load RDF data (in NTriples format) in MonetDB making use of the column-wise vertical partitioning of the RDF data, based on properties/predicates in the RDF data.

Some pointers or help regarding this is much appreciated.

I am sorry if this is not the correct forum to post this message.

Thanks. Medha

------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

Thanks for the information Lefteris. Regards. Medha On Wed, Oct 7, 2009 at 4:25 AM, Lefteris wrote:

Hi,

so the story goes like that:

you need to first load a triple table with the command:  create table triples_tables(subject integer, property integer, object integer);  copy <n> records into triples_table from <file> using (...) as described by Sjoerd.

For "forcing" Monet to use a vertical partition schema you have to do the following: for each property you will define a new table as follows:

create table prop1(subject integer, object integer); create table prop2(subject integer, object integer); create table prop3(subject integer, object integer); ...

then to populate these tables you do for each property:

INSERT into prop1 SELECT subject, object FROM triples_table where property=prop1 ORDER BY subject, object; ...

Now monet will have vertical partitioned tables and will know about ordering thus doing merge joins whenever possible etc.

However, notice that I am using integers and not strings. I assume that you actually have a dictionary where integers point to the strings, otherwise nothing of this will work fast at query time because you will always need to join string columns and the comparison is too slow. So you will need to process a bit more your data before you add them to monet:)

Take a look also at our paper http://www.cwi.nl/htbin/ins1/publications?request=pdf&key=SiGoKeNeMa:VLDB:08 where we did a similar comparison.

Hope this helped, good luck

lefteris

On Wed, Oct 7, 2009 at 2:26 AM, Medha Atre wrote:

...

OK. I figured that must be the command after digging through the manuals. But for the vertically partitioned approach demonstrated in various research papers, I believe the end user has to manually load the RDF data split according to the predicates into different tables, which also requires that the SQL query being issued specifies these tables and joins across these tables as opposed to one big triple table.

Correct me if I am wrong.

Thanks.

Medha

On Tue, Oct 6, 2009 at 5:06 PM, Sjoerd Mullender wrote:

...

On 2009-10-06 17:59, Medha Atre wrote:

...

Hi,

Thanks for the reply.

At this point, all I am looking for is to load RDF NTriples data -- treated as a 3 column table -- into MonetDB using "vertical partitioning" to index the data nicely (as I read about MonetDB being used for the same in different research papers recently.. e.g. http://portal.acm.org/citation.cfm?id=1453856.1453927 RDF-3X etc). I am not really looking for a great support for different RDF formats (as the data has already been cleaned into NTriples format using rdflib), I want to check the correctness of the results generated using my own querying tool.

So can you please suggest me a method of treating the given RDF NTriples file as a simple "space separated" 3-column data file from which I can load the data into MonetDB creating the required indexes or "vertical partitions"?

Perhaps something like COPY <n> RECORDS INTO <table> FROM <file> USING DELIMITERS ' ','\n'; where <n> is the number of tuples, <table> is the name of a table that you created for this, and <file> is the full path name of the file (in single quotes).

...

Querying with SQL front end is fine with me!

Looking forward to your help.

Medha

...

------------------------------

Message: 5 Date: Tue, 6 Oct 2009 10:19:57 +0200 From: Lefteris Subject: Re: [MonetDB-users] loading RDF data in MonetDB To: Communication channel for MonetDB users         Message-ID:         <852badbc0910060119x4f125a06qd5fc7ed0c56035c@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1

Hi,

there is a highly experimental module for MonetDB to load RDF data. This is not activated in the stable release, and in order to use it you will need to do some compilation on your own and possible report back any problems you have. If you are willing to jump deeper in the MonetDB experience I can guide you on how to do this. The module will read any RDF format supported by the rdflib library and create a triple store with appropriate indices. However, the only way to query the triple store at this point is through the SQL front-end.

regrads,

lefteris

On Tue, Oct 6, 2009 at 8:59 AM, Medha Atre  wrote:

...

Hi,

I intend to use MonetDB for some of my experimental work using RDF data. I was wondering if there is an easy way to load RDF data (in NTriples format) in MonetDB making use of the column-wise vertical partitioning of the RDF data, based on properties/predicates in the RDF data.

Some pointers or help regarding this is much appreciated.

I am sorry if this is not the correct forum to post this message.

Thanks. Medha