[MonetDB-users] How to set the memory limit for monetDB.

newer
[MonetDB-users] Performance issues...

Kang Zhang

30 Apr 2008 30 Apr '08

5:11 a.m.

I am using the latest version of monetDB from the web-site(the super source tarball). My machine is a X86-64 machine with a red-hat installed on it. When I insert data into monetDB, it always requires memory from the system. Then it will be crashed with running out of memory. Is there a way to set the limit for memory used by monetDB ? I've read the docs on the web site and can't get what I need from that. Thank you. -- ------------------------------------------------------------ Kang Zhang Computer Science Dept. Shanghai Jiao Tong University

Attachments:

attachment.html (text/html — 614 bytes)

Show replies by date

Martin Kersten

30 Apr 30 Apr

5:21 a.m.

Kang Zhang wrote:

...

I am using the latest version of monetDB from the web-site(the super source tarball). My machine is a X86-64 machine with a red-hat installed on it. When I insert data into monetDB, it always requires memory from the system. Then it will be crashed with running out of memory. Is there a way to set the limit for memory used by monetDB ? I've read the docs on the web site and can't get what I need from that. Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns.

Thank you.

-- ------------------------------------------------------------ Kang Zhang

Computer Science Dept. Shanghai Jiao Tong University ------------------------------------------------------------------------

------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... ------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Kang Zhang

5:47 a.m.

The data's size is about 3 Gigabytes. 21,852,641 triples need to be inserted in total. My machine is 64-bit and only 8G memory for that. Thank you for your immediate reply. Kang. On Wed, Apr 30, 2008 at 1:21 PM, Martin Kersten wrote:

...

Kang Zhang wrote:

...
I am using the latest version of monetDB from the web-site(the super source tarball). My machine is a X86-64 machine with a red-hat installed on it. When I insert data into monetDB, it always requires memory from the system. Then it will be crashed with running out of memory. Is there a way to set the limit for memory used by monetDB ? I've read the docs on the web site and can't get what I need from that. Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns.

Thank you.

-- ------------------------------------------------------------ Kang Zhang

Computer Science Dept. Shanghai Jiao Tong University ------------------------------------------------------------------------

-------------------------------------------------------------------------

...
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...

...
------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- ------------------------------------------------------------ Kang Zhang Computer Science Dept. Shanghai Jiao Tong University

Martin Kersten

6:19 a.m.

Kang Zhang wrote:

...

The data's size is about 3 Gigabytes. 21,852,641 triples need to be inserted in total. My machine is 64-bit and only 8G memory for that. Can you say a little more when and how it crashed. Did you use the SQL COPY command? Is it a victim process selected by the OS, due to excessive virtual memory use? Do you have an gdb stack trace where it happen?

Thank you for your immediate reply. Kang.

On Wed, Apr 30, 2008 at 1:21 PM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:

Kang Zhang wrote: > I am using the latest version of monetDB from the web-site(the super > source tarball). > My machine is a X86-64 machine with a red-hat installed on it. When I > insert data into monetDB, > it always requires memory from the system. Then it will be crashed > with running out of memory. > Is there a way to set the limit for memory used by monetDB ? I've read > the docs on the web site > and can't get what I need from that. Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns. > > Thank you. > > -- > ------------------------------------------------------------ > Kang Zhang > > Computer Science Dept. > Shanghai Jiao Tong University > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... > ------------------------------------------------------------------------ > > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users >

------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- ------------------------------------------------------------ Kang Zhang

Computer Science Dept. Shanghai Jiao Tong University ------------------------------------------------------------------------

------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... ------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Kang Zhang

8:31 a.m.

In fact, I've wrapped monetDB using JENA's api. And tried to load DBPedia's RDF data with it. It runs OK on small dataset. But it always requires memory during the insertion. So it failed when there's no memory to be allocated. Since it has been wrapped by JENA, I know few of the background schema of the database. I go with you it may caused by the os's vm strategy. I'll check it. I've seen that there are some debug option for the config file. Can I use some of them to log something that can let you know my problem? Which one should I use ? On Wed, Apr 30, 2008 at 2:19 PM, Martin Kersten wrote:

...

Kang Zhang wrote:

...
The data's size is about 3 Gigabytes. 21,852,641 triples need to be inserted in total. My machine is 64-bit and only 8G memory for that. Can you say a little more when and how it crashed. Did you use the SQL COPY command? Is it a victim process selected by the OS, due to excessive virtual memory use? Do you have an gdb stack trace where it happen?

Thank you for your immediate reply. Kang.

On Wed, Apr 30, 2008 at 1:21 PM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:

Kang Zhang wrote: > I am using the latest version of monetDB from the web-site(the super > source tarball). > My machine is a X86-64 machine with a red-hat installed on it. When I > insert data into monetDB, > it always requires memory from the system. Then it will be crashed > with running out of memory. > Is there a way to set the limit for memory used by monetDB ? I've read > the docs on the web site > and can't get what I need from that. Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns. > > Thank you. > > -- > ------------------------------------------------------------ > Kang Zhang > > Computer Science Dept. > Shanghai Jiao Tong University >

------------------------------------------------------------------------

...
> >

-------------------------------------------------------------------------

...
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. >

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...

...
>

------------------------------------------------------------------------

...
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users >

-------------------------------------------------------------------------

...
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...

...
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- ------------------------------------------------------------ Kang Zhang

Computer Science Dept. Shanghai Jiao Tong University ------------------------------------------------------------------------

-------------------------------------------------------------------------

...
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...

...
------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- ------------------------------------------------------------ Kang Zhang Computer Science Dept. Shanghai Jiao Tong University

Kang Zhang

8:40 a.m.

I've found that there are no swap file for this machine. :( I guess this is the reason for that. On Wed, Apr 30, 2008 at 4:31 PM, Kang Zhang wrote:

...

In fact, I've wrapped monetDB using JENA's api. And tried to load DBPedia's RDF data with it. It runs OK on small dataset. But it always requires memory during the insertion. So it failed when there's no memory to be allocated. Since it has been wrapped by JENA, I know few of the background schema of the database. I go with you it may caused by the os's vm strategy. I'll check it. I've seen that there are some debug option for the config file. Can I use some of them to log something that can let you know my problem? Which one should I use ?

On Wed, Apr 30, 2008 at 2:19 PM, Martin Kersten wrote:

...
Kang Zhang wrote:

...
The data's size is about 3 Gigabytes. 21,852,641 triples need to be inserted in total. My machine is 64-bit and only 8G memory for that. Can you say a little more when and how it crashed. Did you use the SQL COPY command? Is it a victim process selected by the OS, due to excessive virtual memory use? Do you have an gdb stack trace where it happen?

Thank you for your immediate reply. Kang.

On Wed, Apr 30, 2008 at 1:21 PM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:

Kang Zhang wrote: > I am using the latest version of monetDB from the web-site(the super > source tarball). > My machine is a X86-64 machine with a red-hat installed on it. When I > insert data into monetDB, > it always requires memory from the system. Then it will be crashed > with running out of memory. > Is there a way to set the limit for memory used by monetDB ? I've read > the docs on the web site > and can't get what I need from that. Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns. > > Thank you. > > -- > ------------------------------------------------------------ > Kang Zhang > > Computer Science Dept. > Shanghai Jiao Tong University >

------------------------------------------------------------------------

...
> >

-------------------------------------------------------------------------

...
> This SF.net email is sponsored by the 2008 JavaOne(SM)

Conference

...
> Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. >

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...

...
>

------------------------------------------------------------------------

...
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users >

-------------------------------------------------------------------------

...
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...

...
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- ------------------------------------------------------------ Kang Zhang

Computer Science Dept. Shanghai Jiao Tong University

------------------------------------------------------------------------

...
-------------------------------------------------------------------------

...
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...

...
------------------------------------------------------------------------

...
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.

http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- ------------------------------------------------------------ Kang Zhang

Computer Science Dept. Shanghai Jiao Tong University

-- ------------------------------------------------------------ Kang Zhang Computer Science Dept. Shanghai Jiao Tong University

McKennirey.Matthew

12:44 p.m.

New subject: [MonetDB-users] Memory limit for monetDB.

Is there any strategy that would allow MonetDB to operate in a situation where the size of the columns to be read significantly exceeds the available memory on the machine? On Wednesday 30 April 2008 01:21:56 Martin Kersten wrote:

...

indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns.

Martin Kersten

2:56 p.m.

New subject: [MonetDB-users] Memory limit for monetDB.

McKennirey.Matthew wrote:

...

Is there any strategy that would allow MonetDB to operate in a situation where the size of the columns to be read significantly exceeds the available memory on the machine?

It is part of ongoing research works on partitioned and distributed versions. The primitives to build a distributed version are in the kernel already, e.g. remote execution of MAL blocks. However, this is all experimental. The current state of affair is that also the SQL compiler can handle partitioned tables, i.e. smaller BATs, which means that it results in better handling of memory on a single processor, but also enables a parallel and distributed execution. We need, however, more confidence on the stability of the often novel approaches taken here. On Linux systems, the system includes a thread that keeps an eye on the memory consumption and throws pages out when necessary. Often, the performance you see in out-of-memory situations is what you can expect from any system in that case. See the TPC-H performance figures in http://monetdb.cwi.nl/projects/monetdb//SQL/Benchmark/TPCH/ An update of this table is already available and will be brought online shortly. The message remains the same.

...

On Wednesday 30 April 2008 01:21:56 Martin Kersten wrote:

...
indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns.

------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

6085

Age (days ago)

6085

Last active (days ago)

List overview

Download

7 comments

3 participants

participants (3)

Kang Zhang
Martin Kersten
McKennirey.Matthew