[MonetDB-users] Memory Issues

Venks

25 Apr 2007 25 Apr '07

1:27 a.m.

Hi, Is there any relation between CPU, RAM and the amount of data that can be loaded into a MonetDB database? I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM. I had similar issues on a Windows box too? Thanks, - Venkatesh

Attachments:

attachment.html (text/html — 474 bytes)

Show replies by date

Niels Nes

25 Apr 25 Apr

6:31 a.m.

On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks wrote:

...

Hi,

Is there any relation between CPU, RAM and the amount of data that can be loaded into a MonetDB database?

I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.

I had similar issues on a Windows box too?

Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger than about 320 bytes 13.4 Million rows cannot fit. Niels

...

Thanks,

- Venkatesh

...

------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

Fabian Groffen

7:11 a.m.

On 25-04-2007 08:31:13 +0200, Niels Nes wrote:

...

On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks wrote:

...
I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.

...

Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger than about 320 bytes 13.4 Million rows cannot fit.

Still I think "Segmentation Fault" is the wrong message to convey here.

Venks

11:17 a.m.

Thanks very much for your reply confirming what I understood based on MonetDB architecture. Please allow me another follow-up question Will there be an issue on a 64 bit system? Is there an algorithm/logic that I can use to identify the maximum number rows that I can load in one table based on the size of the row? Regards, - Venkatesh On 4/25/07, Niels Nes wrote:

...

On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks wrote:

...
Hi,

Is there any relation between CPU, RAM and the amount of data that can be loaded into a MonetDB database?

I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.

I had similar issues on a Windows box too?

Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger than about 320 bytes 13.4 Million rows cannot fit.

Niels

...
Thanks,

- Venkatesh

...
-------------------------------------------------------------------------

...
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

--

Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

Colin Foss

4:50 p.m.

On 64bit I have two large tables with 49M and 250M rows respectively. --- Venks wrote:

...

Thanks very much for your reply confirming what I understood based on MonetDB architecture. Please allow me another follow-up question

Will there be an issue on a 64 bit system? Is there an algorithm/logic that I can use to identify the maximum number rows that I can load in one table based on the size of the row?

Regards,

- Venkatesh

On 4/25/07, Niels Nes wrote:

...
On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks

...
...
Hi,

Is there any relation between CPU, RAM and

...
...
can be loaded into a MonetDB database?

I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.

I had similar issues on a Windows box too?

Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger

wrote: the amount of data that than about 320 bytes

...
13.4 Million rows cannot fit.

Niels

...
Thanks,

- Venkatesh

...

-------------------------------------------------------------------------

...

...
...
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

...
--

Niels Nes, Centre for Mathematics and Computer

Science (CWI)

...
Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

...

This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/>

...

MonetDB-users mailing list MonetDB-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

Venks

4:54 p.m.

Colin, Do you know the size of each row? I have a table with 50 columns and could grow to 100 columns for our testing. Does this matter? On 4/25/07, Colin Foss wrote:

...

On 64bit I have two large tables with 49M and 250M rows respectively.

--- Venks wrote:

...
Thanks very much for your reply confirming what I understood based on MonetDB architecture. Please allow me another follow-up question

Will there be an issue on a 64 bit system? Is there an algorithm/logic that I can use to identify the maximum number rows that I can load in one table based on the size of the row?

Regards,

- Venkatesh

On 4/25/07, Niels Nes wrote:

...
On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks

...
...
Hi,

Is there any relation between CPU, RAM and

...
...
can be loaded into a MonetDB database?

I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.

I had similar issues on a Windows box too?

Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger

wrote: the amount of data that than about 320 bytes

...
13.4 Million rows cannot fit.

Niels

...
Thanks,

- Venkatesh

...
-------------------------------------------------------------------------

...
...
...
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
--

Niels Nes, Centre for Mathematics and Computer

Science (CWI)

...
Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

...
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/>

...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

Colin Foss

5:02 p.m.

Venks, Number of columns is limited only by disk space. I have columns which are approx 2GB on disk but I believe column size is limited to the maximum size of a file in your given file system, or maybe it is 2GB. The developers are authority on max column size. Colin --- Venks wrote:

...

Colin,

Do you know the size of each row? I have a table with 50 columns and could grow to 100 columns for our testing. Does this matter?

On 4/25/07, Colin Foss wrote:

...
On 64bit I have two large tables with 49M and 250M rows respectively.

--- Venks wrote:

...
Thanks very much for your reply confirming what

...
...
understood based on MonetDB architecture. Please allow me another follow-up question

Will there be an issue on a 64 bit system? Is

...
...
an algorithm/logic that I can use to identify the maximum number rows

I there that I

...
...
can load in one table based on the size of the row?

Regards,

- Venkatesh

On 4/25/07, Niels Nes wrote:

...
On Tue, Apr 24, 2007 at 09:27:13PM -0400,

...
...
Hi,

Is there any relation between CPU, RAM

and

...
...
can be loaded into a MonetDB database?

I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.

I had similar issues on a Windows box too?

Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger

Venks wrote: the amount of data that than about 320 bytes

...
13.4 Million rows cannot fit.

Niels

...
Thanks,

- Venkatesh

...

-------------------------------------------------------------------------

...

...
...
...
...
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/

...
...
...
...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

...
...
...
--

Niels Nes, Centre for Mathematics and Computer

Science (CWI)

...
Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

-------------------------------------------------------------------------

...

...
...
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/>

...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

...
...

Venks

5:08 p.m.

Colin, Thanks for your reply. My question was specific to your data based on "Neils" response regarding addressable space and the byte size of a record. Anyhow, I think I need a 64 Bit machine for sure to load huge amounts of data for testing. Thanks, - Venkatesh On 4/25/07, Colin Foss wrote:

...

Venks,

Number of columns is limited only by disk space. I have columns which are approx 2GB on disk but I believe column size is limited to the maximum size of a file in your given file system, or maybe it is 2GB.

The developers are authority on max column size.

Colin

--- Venks wrote:

...
Colin,

Do you know the size of each row? I have a table with 50 columns and could grow to 100 columns for our testing. Does this matter?

On 4/25/07, Colin Foss wrote:

...
On 64bit I have two large tables with 49M and 250M rows respectively.

--- Venks wrote:

...
Thanks very much for your reply confirming what

...
...
understood based on MonetDB architecture. Please allow me another follow-up question

Will there be an issue on a 64 bit system? Is

...
...
an algorithm/logic that I can use to identify the maximum number rows

I there that I

...
...
can load in one table based on the size of the row?

Regards,

- Venkatesh

On 4/25/07, Niels Nes wrote:

...
On Tue, Apr 24, 2007 at 09:27:13PM -0400,

...
...
Hi,

Is there any relation between CPU, RAM

and

...
...
can be loaded into a MonetDB database?

I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.

I had similar issues on a Windows box too?

Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger

Venks wrote: the amount of data that than about 320 bytes

...
13.4 Million rows cannot fit.

Niels

...
Thanks,

- Venkatesh

...
-------------------------------------------------------------------------

...
...
...
...
...
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/

...
...
...
...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
...
...
--

Niels Nes, Centre for Mathematics and Computer

Science (CWI)

...
Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

-------------------------------------------------------------------------

...
...
...
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/>

...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
...

Niels Nes

8:12 p.m.

On Wed, Apr 25, 2007 at 01:08:28PM -0400, Venks wrote:

...

Colin,

Thanks for your reply. My question was specific to your data based on "Neils" response regarding addressable space and the byte size of a record.

Intels 64 bit machines have 40 bit adress space (ie Up to one terabyte (TB) of platform address space) Amd's 64 bit machines have 48 bit adress space (ie 256 TB). A crude measure for the maximum number of rows is to simply divide thes numbers by your record width (even tough we store each column separately).

...

Anyhow, I think I need a 64 Bit machine for sure to load huge amounts of data for testing.

For huge tables 64 bit is indeed needed. Niels

...

Thanks,

- Venkatesh

On 4/25/07, Colin Foss <[1]cgfoss@yahoo.com> wrote:

Venks, Number of columns is limited only by disk space. I have columns which are approx 2GB on disk but I believe column size is limited to the maximum size of a file in your given file system, or maybe it is 2GB. The developers are authority on max column size. Colin --- Venks <[2] venkatesh.mantha@gmail.com> wrote: > Colin, > > Do you know the size of each row? I have a table > with 50 columns and could > grow to 100 columns for our testing. Does this > matter? > > > On 4/25/07, Colin Foss <[3]cgfoss@yahoo.com> wrote: > > > > On 64bit I have two large tables with 49M and 250M > > rows respectively. > > > > > > --- Venks <[4]venkatesh.mantha@gmail.com> wrote: > > > > > Thanks very much for your reply confirming what > I > > > understood based on > > > MonetDB architecture. Please allow me another > > > follow-up question > > > > > > Will there be an issue on a 64 bit system? Is > there > > > an algorithm/logic that > > > I can use to identify the maximum number rows > that I > > > can load in one table > > > based on the size of the row? > > > > > > Regards, > > > > > > - Venkatesh > > > > > > > > > On 4/25/07, Niels Nes <[5]Niels.Nes@cwi.nl > wrote: > > > > > > > > On Tue, Apr 24, 2007 at 09:27:13PM -0400, > Venks > > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Is there any relation between CPU, RAM > and > > > the amount of data that > > > > > can be loaded into a MonetDB database? > > > > > > > > > > > > > > > > > > > > I am trying to load 13.4 million rows of > data > > > and I am getting > > > > > "Segmentation Fault" on a Linux box > running > > > on 32 bit CPU with 4GB > > > > > RAM. > > > > > > > > > > > > > > > > > > > > I had similar issues on a Windows box > too? > > > > > > > > > Yes there are relations between the > 'addressable' > > > memory and the > > > > amount of data MonetDB can handle. With a 32 > bit > > > system your adress > > > > space is at best 4G. So if your records are > larger > > > than about 320 bytes > > > > 13.4 Million rows cannot fit. > > > > > > > > Niels > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > - Venkatesh > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ ------- > > > > > This SF.net email is sponsored by DB2 > Express > > > > > Download DB2 Express C - the FREE version of > DB2 > > > express and take > > > > > control of your XML. No limits. Just data. > Click > > > to get it now. > > > > > [6]http://sourceforge.net/powerbar/db2/ > > > > > > _______________________________________________ > > > > > MonetDB-users mailing list > > > > > [7]MonetDB-users@lists.sourceforge.net > > > > > > > > > > > [8]https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > > > > > > > -- > > > > > > > > Niels Nes, Centre for Mathematics and Computer > > > Science (CWI) > > > > Kruislaan 413, 1098 SJ Amsterdam, The > Netherlands > > > > room C0.02, phone ++31 20 592-4098, fax ++31 > 20 > > > 592-4312 > > > > url: [9]http://www.cwi.nl/~niels e-mail: > > > [10]Niels.Nes@cwi.nl > > > > > > > > > > > ------------------------------------------------------------------ ------- > > > This SF.net email is sponsored by DB2 Express > > > Download DB2 Express C - the FREE version of DB2 > > > express and take > > > control of your XML. No limits. Just data. Click > to > > > get it now. > > > [11]http://sourceforge.net/powerbar/db2/> > > _______________________________________________ > > > MonetDB-users mailing list > > > [12]MonetDB-users@lists.sourceforge.net > > > > > > [13]https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > > >

References

1. mailto:cgfoss@yahoo.com 2. mailto:venkatesh.mantha@gmail.com 3. mailto:cgfoss@yahoo.com 4. mailto:venkatesh.mantha@gmail.com 5. mailto:Niels.Nes@cwi.nl 6. http://sourceforge.net/powerbar/db2/ 7. mailto:MonetDB-users@lists.sourceforge.net 8. https://lists.sourceforge.net/lists/listinfo/monetdb-users 9. http://www.cwi.nl/~niels 10. mailto:Niels.Nes@cwi.nl 11. http://sourceforge.net/powerbar/db2/ 12. mailto:MonetDB-users@lists.sourceforge.net 13. https://lists.sourceforge.net/lists/listinfo/monetdb-users

6456

Age (days ago)

6456

Last active (days ago)

List overview

Download

8 comments

4 participants

participants (4)

Colin Foss
Fabian Groffen
Niels Nes
Venks