Re: [MonetDB-users] Huge tables

I think your problem right now is that there is still a "log" directory. Remove both dbfarm and log directories and try again. This should at least revive MonetDB a bit ;) Andrei Martsinchyk wrote:

Thanks for response, I do not believe that 2GB BAT size limit was hit. Largest column in the table was defined as varchar(44), I had 12,000,000 records loaded and wanted to add 6,000,000 more records. So, there should be 18 * 44=792MBytes. It depends how BAT is represented internally, though. I am afraid I ran out of memory or disk space. Anyway the error has not been handled and now I can not start server to run tests once again and clarify the problem. Could anybody help me to find out what happened to server ?

2005/10/3, Fabian :

...

Andrei,

Sorry that your previous post (seemed to be) ignored. Unfortunately nobody was able to respond in time.

I am not sure on this answer, but if I recall correctly MonetDB cannot possibly process columns which are larger than the addresseable memory. This is due to the fact that MonetDB is a main-memory database.

Please someone correct me if I'm wrong here.

Regards, Fabian

Andrei Martsinchyk wrote:

...

Hello,

It seems MonetDB has problem with handling of huge tables (larger than amount of memory available). I tried to load large amount of data into a table using COPY command. I loaded same data (6 millions records, 700+MBytes) few times. Loading was successful two times, on third time server crushed. After crush I could not start the server again, it output messages

!WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !FATAL: BBPinit: cannot properly process bat/BACKUP/.

and exited. Removing of the database data directory did not help much - server output messages

!ERROR:mvc_init: unable to create system tables !WARNING: BBPincref: range error 410

Couple minutes later it output

!ERROR: BATSIGcrash: Mserver internal error (Segmentation fault), please restart . !ERROR: (One potential cause could be that your disk might be full...)

Well, insufficient disk space could be a cause, before server crashed I had approximately 2GB free. But I have cleaned up 10 GB and problem persists.

How can I troubleshoot and fix the server ? What are memory / disk space requirements when bulk loading data ?

I tested with 32-bit system, 512 MB RAM + 1GB swap, Ubuntu Linux, MonetDB-4.9.3, nightly build dated September 30.

-- Best regards, Andrei Martsinchyk mailto:andrei.martsinchyk@gmail.com

------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------- This SF.Net email is sponsored by: Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Best regards, Andrei Martsinchyk mailto:andrei.martsinchyk@gmail.com