Re: [MonetDB-users] Res: Res: MonetDB aborts when importing over 10 million rows

On Sat, Jun 26, 2010 at 07:15:46AM -0700, Luis Fernando Kauer wrote:

Hi Stefan.

Thanks again for aswering so quickly. I had intented to use it on user machines and so I'm stuck to Windows XP 32-bit.

You might want to consider trying the following: - load the data of each year (each file) into an individual table; - create a view that unions all tables into one (views are not materialized in MonetDB) - phrase your queries using this view as if it was a single large table containing all data If (a) each query uses only a subset of all columns or (b) your queries contain rather selective predicates and our optimizer successfully manages to automatically push them beneath the view, i.e., before the data from the different tables is actually combined, this *might* work on 32-bit systems.

It crashes and closes the server. Usually when I restart the server after the crash it shows messages about the memory problem but it's able to start. But sometimes it would not start anymore and I had to reinstall the server, since I don't know what I could do to make it start again.

It would be helpful for us (and hence eventually also for you ;-)) if you could report such crashes as bugs via our bugtracker: http://bugs.monetdb.org/ Stefan

Regards, Luis Fernando

________________________________ De: Stefan Manegold Para: Communication channel for MonetDB users Enviadas: Sexta-feira, 25 de Junho de 2010 18:20:30 Assunto: Re: [MonetDB-users] Res: MonetDB aborts when importing over 10 million rows

Hi Luis,

on a 32-bit system, MonetDB is limited to database of at most 4 GB; on Windows, the limit is usually at most 3 GB or even only 2 GB, as Windows does not allow larger processes on a 32-bit system; see also http://zone.ni.com/reference/en-XX/help/371361D-01/lvhowto/enable_lrg_ad_awa...

In your case, with 8 files (years) of >300 MB you have >2.4 GB of raw data, and I'm afraid you will not be able to load this amount into MonetDB on a 32-bit Windows system; surely not in the dfault configuration that does not allow processes to grow beyond 2 GB, but evenn if you configure it to allow 3 GB (cf., http://zone.ni.com/reference/en-XX/help/371361D-01/lvhowto/enable_lrg_ad_awa...) it might not work.

To load all the data into MonetDB, you'd need to consider a 64-bit system (and 64-bit MonetDB).

BTW, does the server actually crash, or does it "merely" report an error, aborting the load, but the server keeps running?

Stefan orge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |