Re: [MonetDB-users] predictable crashes of MonetDB5 SQL server under Win32bit
Thanks Stefan,
with loading ~1.5 GB on a 32 bit machine, there is a reasonable chance that the loading process (that needs to be able to address all data concurrently) comes close to or even hits the 32-bit (i.e., max. 4 GB, possibly only 2 GB) address space limit.
Does this limitation refers to the size of the data that I am adding per COPY INTO (was 0.2 GB) or the final size of the table (was at 1.9). Assuming the latter this would give a hard limit for table size under Win32. If this is the case, a workaround could be partioning the data into several tables. What is more efficient in MonetDB: chunks of rows at full columns width or chunks of columns at full row length?
Would it be possible that you could provide us with your data? That would help us a lot to analyze, locate and hopefully fix the crash...
I give you access by seperate mail, you can also create the test data file using the R script snippet in my previous mail. Have a nice weekend Jens
On Fri, Jan 23, 2009 at 04:52:54PM +0100, Jens Oehlschlägel wrote:
Thanks Stefan,
with loading ~1.5 GB on a 32 bit machine, there is a reasonable chance that the loading process (that needs to be able to address all data concurrently) comes close to or even hits the 32-bit (i.e., max. 4 GB, possibly only 2 GB) address space limit.
Does this limitation refers to the size of the data that I am adding per COPY INTO (was 0.2 GB) or the final size of the table (was at 1.9). Assuming the latter this would give a hard limit for table size under Win32.
Indeed, the limitation refers to the final table size, as all COPY INTO queries add data to the same table --- and the limitation of course holds for all 32-bit systems, not only for Win32 (although Win32 often appear to be particularly sensitive to memory / address space fragmentation) ...
If this is the case, a workaround could be partioning the data into several tables. What is more efficient in MonetDB: chunks of rows at full columns width or chunks of columns at full row length?
Indeed, partitioning the data into multiple tables would help with the loading, and there is should not matter (much) whether you partition horizontally or vertically. Of course, for querying the data, it depends on your query load, which partitioning is more efficient. Basically, the partitioning that requires the least the partitions per query would be the most efficient one. But also the way that your queries re-combine the partition that are required --- we don't have generic automatic partitining *yet*, hence, when partitioning your table, you also need to "partition" your queries accordingly --- has an impact on the performance. It is hard to make a general statement, here. We might be able to give more advice for more concrete cases, i.e., if your could provide us with some more details about your typical / anticipated query load, we could have a look at that.
Would it be possible that you could provide us with your data? That would help us a lot to analyze, locate and hopefully fix the crash...
I give you access by seperate mail, you can also create the test data file using the R script snippet in my previous mail.
Ok, I dind't realize that. Thanks. We'll try to have a closer look at the problem next week. Have a nice weekend! Stefan
Have a nice weekend
Jens
------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Hi, I was creating a table thru sql client. Then I've decided to stop that process - It was taking lots of time. Now when I open sql client it does not do anything - what do I do now? Thanks. Dariusz.
Dear MonetDB users, In the Monet 4 distribution rpms neither Mserver nor mclient seems to have an option to find out its version information. Is this true? I would be very pleased if this option will be added in the near future. "Mserver --version" Greetings, Hans van Rijswijk.
Hans van Rijswijk (DT) wrote:
Dear MonetDB users,
In the Monet 4 distribution rpms neither Mserver nor mclient seems to have an option to find out its version information. Is this true? I would be very pleased if this option will be added in the near future. "Mserver --version"
The server produces the following information. Mclient does not request this information from the server. It should, indeed, to make it easily accessible. mk@eir::~> mserver5 --version MonetDB server v5.9.0 (64-bit), based on kernel v1.27.0 (64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Configured for prefix: /ufs/mk/monet5//Linux Libraries: libpcre: 7.3 2007-08-28 (compiled with 7.3) openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) libxml2: 2.7.2 (compiled with 2.7.2) Compiled by: mk@eir.ins.cwi.nl Compilation: gcc -Wall -Wextra -std=c99 -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 Linking : ld -IPA -m elf_x86_64
Greetings, Hans van Rijswijk.
------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Jan 26, 2009 at 06:05:22PM +0100, Martin Kersten wrote:
Hans van Rijswijk (DT) wrote:
Dear MonetDB users,
In the Monet 4 distribution rpms neither Mserver nor mclient seems to have an option to find out its version information. Is this true? I would be very pleased if this option will be added in the near future. "Mserver --version"
The server produces the following information. Mclient does not request this information from the server. It should, indeed, to make it easily accessible.
mserver5 indeed provides a --version switch, but Mserver (v4) indeed does not; neither does mclient. If at all, mclient should not (only) request the server's version number, but rather (also) reveal its own version number. For the time being, one of the following might help: ======== $ Mserver http://monetdb.cwi.nl/ for further information. ======== and in case you have compiled from sources or the *-devel RPMs installed (AFAIK not included in the Windows installers): ======== $ monetdb4-config --version -------- 4.26.5 ======== $ monetdb5-config --version -------- 5.8.5 ======== $ monetdb-clients-config --version -------- 1.26.5 ======== To keep us reminded, please feel free to file a feature request via http://sourceforge.net/tracker/?atid=482471&group_id=56967&func=browse Stefan
mk@eir::~> mserver5 --version MonetDB server v5.9.0 (64-bit), based on kernel v1.27.0 (64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Configured for prefix: /ufs/mk/monet5//Linux Libraries: libpcre: 7.3 2007-08-28 (compiled with 7.3) openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) libxml2: 2.7.2 (compiled with 2.7.2) Compiled by: mk@eir.ins.cwi.nl Compilation: gcc -Wall -Wextra -std=c99 -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 Linking : ld -IPA -m elf_x86_64
Greetings, Hans van Rijswijk.
------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
participants (5)
-
dariuszs -
Hans van Rijswijk (DT) -
Jens Oehlschlägel -
Martin Kersten -
Stefan Manegold