[MonetDB-users] Large Databases
I am trying to create a large database almost 10GB (TPC-H). When I try to bulk load a 7GB file the server does not work anymore. First, the server quits in the middle of the bulk load, then when I try to re-start the server it does not start showing a message about the OS not having enough space. Then I split the 7GB file into individuals 1GB files, after I upload around 4GB individually, the server crashes. Then when I re-start the server it does not start stating that the BACKUP folder cannot be accessed and not enough permissions! I am using MonetDB 5 on Windows XP 1GB RAM. Any ideas? Thank You. alan -- View this message in context: http://www.nabble.com/Large-Databases-tf4710020.html#a13462531 Sent from the monetdb-users mailing list archive at Nabble.com.
algatt wrote: I guess you are working on a 32-bit machine, which incurs an addressing limitation of 2GB. Most tables in MonetDB are memory mapped files, which indeed means that at some point the OS can not allocate (virtual) memory anymore and simply kills the process. Since it is in the middle of a load, recovery records are likely incomplete.
I am trying to create a large database almost 10GB (TPC-H).
When I try to bulk load a 7GB file the server does not work anymore. First, the server quits in the middle of the bulk load, then when I try to re-start the server it does not start showing a message about the OS not having enough space.
Then I split the 7GB file into individuals 1GB files, after I upload around 4GB individually, the server crashes. Then when I re-start the server it does not start stating that the BACKUP folder cannot be accessed and not enough permissions!
I am using MonetDB 5 on Windows XP 1GB RAM.
Any ideas?
Thank You.
alan
Is there any solution to this?
Also, I have an AMD64, so if I use a Linux Distro 64Bit does that change
anthing?
Thanks
alan
On 29/10/2007, Martin Kersten
algatt wrote: I guess you are working on a 32-bit machine, which incurs an addressing limitation of 2GB. Most tables in MonetDB are memory mapped files, which indeed means that at some point the OS can not allocate (virtual) memory anymore and simply kills the process. Since it is in the middle of a load, recovery records are likely incomplete.
I am trying to create a large database almost 10GB (TPC-H).
When I try to bulk load a 7GB file the server does not work anymore. First, the server quits in the middle of the bulk load, then when I try to re-start the server it does not start showing a message about the OS not having enough space.
Then I split the 7GB file into individuals 1GB files, after I upload around 4GB individually, the server crashes. Then when I re-start the server it does not start stating that the BACKUP folder cannot be accessed and not enough permissions!
I am using MonetDB 5 on Windows XP 1GB RAM.
Any ideas?
Thank You.
alan
------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Alan Gatt wrote:
Is there any solution to this?
Also, I have an AMD64, so if I use a Linux Distro 64Bit does that change anthing? yes, 64 bit addressing really makes a difference :-))
Thanks
alan
On 29/10/2007, * Martin Kersten*
mailto:Martin.Kersten@cwi.nl> wrote: algatt wrote: I guess you are working on a 32-bit machine, which incurs an addressing limitation of 2GB. Most tables in MonetDB are memory mapped files, which indeed means that at some point the OS can not allocate (virtual) memory anymore and simply kills the process. Since it is in the middle of a load, recovery records are likely incomplete. > I am trying to create a large database almost 10GB (TPC-H). > > When I try to bulk load a 7GB file the server does not work anymore. First, > the server quits in the middle of the bulk load, then when I try to re-start > the server it does not start showing a message about the OS not having > enough space. > > Then I split the 7GB file into individuals 1GB files, after I upload around > 4GB individually, the server crashes. Then when I re-start the server it > does not start stating that the BACKUP folder cannot be accessed and not > enough permissions! > > I am using MonetDB 5 on Windows XP 1GB RAM. > > Any ideas? > > Thank You. > > alan >
------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ http://get.splunk.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------
------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ------------------------------------------------------------------------
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (3)
-
Alan Gatt
-
algatt
-
Martin Kersten