- users-list - monetdb.org

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

1 0

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

------------------ Original ------------------ From: "Martin Kersten"<Martin.Kersten(a)cwi.nl>; Date: Fri, Jul 26, 2013 04:46 PM To: "users-list"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Hi, You should plot the times of each of your runs to see the trend. This single run may be an outlier, which could come from anything in your system environment. Even a seemingly harmless concurrent program using significant memory could compete with MonetDB. And indeed, at some point you will see disk IO. regards, Martin

1 0

Re: Monetdb copy binary time varys very much!
by integrity 26 Jul '13

26 Jul '13

------------------ Original ------------------ From: "Martin Kersten"<Martin.Kersten(a)cwi.nl>; Date: Fri, Jul 26, 2013 04:46 PM To: "users-list"<users-list(a)monetdb.org>; Subject: Re: Monetdb copy binary time varys very much! Hi, You should plot the times of each of your runs to see the trend. This single run may be an outlier, which could come from anything in your system environment. Even a seemingly harmless concurrent program using significant memory could compete with MonetDB. And indeed, at some point you will see disk IO. regards, Martin On 7/26/13 10:40 AM, integrity wrote: > Hi dear all, > > I did a test of insert 2,000,000,000 rows into MonetDB with "COPY BINARY > INTO FROM binary_file", > in this test > 1. i generate 21 files, each file represents a table column and has > 200000 rows. > I created them with rand() number and use fwrite() binary write method. > > the table creation sql command is : > *create table tmatch(id bigint,a float,b float,c float, d float, e float,* > *f float,g float,h float, i float,j float,* > *k float,l float,m float, n float,o float,* > *p float,q float,r float, s float,t float);* > the table has 21 columns,each column has 8 bytes, so each column file is > c1=200000*21*8 Byte= 268800000 Byte=3.2MB > 2. I use "COPY BINARY INTO FROM above_binary" to load each binary file > into tmatch. > The test was run 10000 times repeatedly. > > the average time of 10000 times is only 1.0635589558955727, > but when at 9043th time, it cost 227m36.136 ,some times later the time > value increase to a large number, is it because of flush data from > cache into database after the cache is full? > The problem is since we have to control the total process time within 15 > seconds , I am wondering if you can help me reduce the maximum time to a > lower point? > > > Thanks very much! > Meng > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

1 0

ALTER in procedure
by Radovan Bičiště 23 Jul '13

23 Jul '13

Hello, This is my first post here. I tried to search the archive but did not find anything related. We build application and would like disable foreign keys in a procedure. Something like: CREATE PROCEDURE crmkartaaktivita_drop_cons() BEGIN -- hlavička procedury DECLARE v_proc VARCHAR(250); SET v_proc = 'crmkartaaktivita_drop_cons'; -- drop ALTER TABLE crmkartastitek DROP CONSTRAINT crmkartastitek_crmkartaaktivita_FK; ALTER TABLE crmkartastitekm DROP CONSTRAINT crmkartastitekm_crmkartaakitvita_FK; END; / I'm getting "Statement 'ALTER_TABLE' is not a valid flow control statement". Is there any way around please? Our algorithm should like this: 1. disable/drop foreign keys. 2. load data. 3. enable/create constraints. Thanks, Radovan -- Radovan Biciste Czech Republic

2 7

If MonetDB could use table partition ?
by integrity 22 Jul '13

22 Jul '13

I am using "COPY BINARY INTO ...FROM ..." to insert 1000 FITS files into a table and the table has 21 columns, every FITS file has 200000 records. After some FITS files, the loading becoming very slowly, so If MonetDB could use table partition to accelerating the copy? or if Data Vaults could be loading more quickly? Thanks! Meng

2 2

Re: Could i binary bulk load to a CERTAIN column?
by integrity 21 Jul '13

21 Jul '13

my table test4 is : create table test4(id bigint, a float, b float, c float); so i didnt selectively load to test4 but it crashed. my code is : #include <stdio.h> #include <stdlib.h> #include <time.h> #include <mapi.h> void itoa (int n,char s[]) { int i,j,sign,k; char tmp; if((sign=n)<0)//记录符号 n=-n;//使n成为正数 i=0; do{ s[i++]=n%10+'0';//取下一个数字 }while ((n/=10)>0);//删除该数字 if(sign<0) s[i++]='-'; s[i]='\0'; for(j=i-1,k=0;j>k;j--,k++) { tmp = s[k]; s[k] = s[j]; s[j] = tmp; } } int writetoFile() { unsigned int seed; int k,j,i=0; long al[200000]; double af[200000]; FILE *fp; char fname[5]; fp = fopen("id","wb"); //创建二进制文件 for(k=1;k<=200000;k++) { al[k] = rand(); } fwrite(al, sizeof(long), 200000,fp); //fwrite以二进制形式对文件操作 fclose(fp); for(j=1;j<=20;j++) { for(k=1;k<=200000;k++) { af[k] = (rand() /(double)(RAND_MAX)); } itoa(j,fname); fp=fopen(fname,"wb"); fwrite(af, sizeof(double), 200000,fp); fclose(fp); } } void die(Mapi dbh, MapiHdl hdl) { if (hdl != NULL) { mapi_explain_query(hdl, stderr); do { if (mapi_result_error(hdl) != NULL) mapi_explain_result(hdl, stderr); } while (mapi_next_result(hdl) == 1); mapi_close_handle(hdl); mapi_destroy(dbh); } else if (dbh != NULL) { mapi_explain(dbh, stderr); mapi_destroy(dbh); } else { fprintf(stderr, "command failed\n"); } exit(-1); } MapiHdl query(Mapi dbh, char *q) { MapiHdl ret = NULL; if ((ret = mapi_query(dbh, q)) == NULL || mapi_error(dbh) != MOK) die(dbh, ret); return(ret); } void update(Mapi dbh, char *q) { MapiHdl ret = query(dbh, q); if (mapi_close_handle(ret) != MOK) die(dbh, ret); } int filetoMonetDB() { Mapi dbh; MapiHdl hdl = NULL; dbh = mapi_connect("localhost",50000, "monetdb", "monetdb", "sql","mydb2" ); if(mapi_error(dbh)) die(dbh,hdl); char *sql = NULL; sql = (char *)malloc(100); memset(sql,0,100); //sprintf(sql, "copy binary into tmatch from('/home/data1/writeBinary/id','/home/data1/writeBinary/1','/home/data1/writeBinary/2','/home/data1/writeBinary/3','/home/data1/writeBinary/4')"); // '/home/data1/writeBinary/5','/home/data1/writeBinary/6','/home/data1/writeBinary/7','/home/data1/writeBinary/8','/home/data1/writeBinary/9','/home/data1/writeBinary/10', \ // '/home/data1/writeBinary/11', '/home/data1/writeBinary/12', '/home/data1/writeBinary/13', '/home/data1/writeBinary/14', '/home/data1/writeBinary/15', \ // '/home/data1/writeBinary/16', '/home/data1/writeBinary/17', '/home/data1/writeBinary/18', '/home/data1/writeBinary/19', '/home/data1/writeBinary/20' )"); // sprintf(sql,"insert into test2 values(2)"); sprintf(sql, "copy binary into test3 from('/home/data1/writeBinary/id','/home/data1/writeBinary/1')"); //sprintf(sql,"copy binary into test4 from('/home/data1/writeBinary/id','/home/data1/writeBinary/1','home/data1/writeBinary/2','home/data1/writeBinary/3')"); //sprintf(sql,"copy binary into test4(id) from('/home/data1/writeBinary/id')"); update(dbh, sql); mapi_destroy(dbh); free(sql); return 1; } int main() { struct timeval tpstart,tpend; double timeuse; gettimeofday(&tpstart,NULL); writetoFile(); gettimeofday(&tpend,NULL); timeuse=1000000*(tpend.tv_sec-tpstart.tv_sec)+tpend.tv_usec-tpstart.tv_usec;//unit:us timeuse /= 1000000;//unit:s printf("21 columns write to file used %f seconds.\n",timeuse) ; gettimeofday(&tpstart,NULL); filetoMonetDB(); gettimeofday(&tpend,NULL); timeuse=1000000*(tpend.tv_sec-tpstart.tv_sec)+tpend.tv_usec-tpstart.tv_usec;//unit:us timeuse /= 1000000;//unit:s printf("Files load to MonetDB used %f seconds.\n",timeuse) ; return 1; }

3 3

Could i binary bulk load to a CERTAIN column?
by integrity 21 Jul '13

21 Jul '13

Hi all, when i use : copy binary into test4 from('/home/data1/writeBinary/id','/home/data1/writeBinary/1','home/data1/writeBinary/2','home/data1/writeBinary/3'); it fails: 21 columns write to file used 0.195698 seconds. *** glibc detected *** ./fitscolumntodb: malloc(): memory corruption: 0x0000000001d743e0 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3efde7d982] /lib64/libc.so.6(__libc_malloc+0x63)[0x3efde7f313] /lib64/libc.so.6(__strdup+0x22)[0x3efde85c42] /home/data1/opt/MonetDB/lib/libmapi.so.6(+0xb24f)[0x7f77d4ca724f] /home/data1/opt/MonetDB/lib/libmapi.so.6(mapi_query+0xa0)[0x7f77d4cac410] ./fitscolumntodb[0x400fc5] ./fitscolumntodb[0x40101c] ./fitscolumntodb[0x401198] ./fitscolumntodb[0x40127d] /lib64/libc.so.6(__libc_start_main+0xf5)[0x3efde21a05] ./fitscolumntodb[0x400bc9] ======= Memory map: ======== 00400000-00402000 r-xp 00000000 08:01 18874373 /home/data1/writeBinary/fitscolumntodb 00601000-00602000 r--p 00001000 08:01 18874373 /home/data1/writeBinary/fitscolumntodb 00602000-00603000 rw-p 00002000 08:01 18874373 /home/data1/writeBinary/fitscolumntodb 01d74000-01d95000 rw-p 00000000 00:00 0 [heap] 3efd600000-3efd620000 r-xp 00000000 fd:01 700425 /usr/lib64/ld-2.16.so 3efd820000-3efd821000 r--p 00020000 fd:01 700425 /usr/lib64/ld-2.16.so 3efd821000-3efd822000 rw-p 00021000 fd:01 700425 /usr/lib64/ld-2.16.so 3efd822000-3efd823000 rw-p 00000000 00:00 0 3efde00000-3efdfad000 r-xp 00000000 fd:01 700426 /usr/lib64/libc-2.16.so 3efdfad000-3efe1ad000 ---p 001ad000 fd:01 700426 /usr/lib64/libc-2.16.so 3efe1ad000-3efe1b1000 r--p 001ad000 fd:01 700426 /usr/lib64/libc-2.16.so 3efe1b1000-3efe1b3000 rw-p 001b1000 fd:01 700426 /usr/lib64/libc-2.16.so 3efe1b3000-3efe1b8000 rw-p 00000000 00:00 0 3efe200000-3efe216000 r-xp 00000000 fd:01 700427 /usr/lib64/libpthread-2.16.so 3efe216000-3efe416000 ---p 00016000 fd:01 700427 /usr/lib64/libpthread-2.16.so 3efe416000-3efe417000 r--p 00016000 fd:01 700427 /usr/lib64/libpthread-2.16.so 3efe417000-3efe418000 rw-p 00017000 fd:01 700427 /usr/lib64/libpthread-2.16.so 3efe418000-3efe41c000 rw-p 00000000 00:00 0 3efe600000-3efe603000 r-xp 00000000 fd:01 700430 /usr/lib64/libdl-2.16.so 3efe603000-3efe802000 ---p 00003000 fd:01 700430 /usr/lib64/libdl-2.16.so 3efe802000-3efe803000 r--p 00002000 fd:01 700430 /usr/lib64/libdl-2.16.so 3efe803000-3efe804000 rw-p 00003000 fd:01 700430 /usr/lib64/libdl-2.16.so 3efea00000-3efea15000 r-xp 00000000 fd:01 672671 /usr/lib64/libz.so.1.2.7 3efea15000-3efec14000 ---p 00015000 fd:01 672671 /usr/lib64/libz.so.1.2.7 3efec14000-3efec15000 r--p 00014000 fd:01 672671 /usr/lib64/libz.so.1.2.7 3efec15000-3efec16000 rw-p 00015000 fd:01 672671 /usr/lib64/libz.so.1.2.7 3efee00000-3efee07000 r-xp 00000000 fd:01 700431 /usr/lib64/librt-2.16.so 3efee07000-3eff006000 ---p 00007000 fd:01 700431 /usr/lib64/librt-2.16.so 3eff006000-3eff007000 r--p 00006000 fd:01 700431 /usr/lib64/librt-2.16.so 3eff007000-3eff008000 rw-p 00007000 fd:01 700431 /usr/lib64/librt-2.16.so 3eff600000-3eff615000 r-xp 00000000 fd:01 700435 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 3eff615000-3eff814000 ---p 00015000 fd:01 700435 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 3eff814000-3eff815000 r--p 00014000 fd:01 700435 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 3eff815000-3eff816000 rw-p 00015000 fd:01 700435 /usr/lib64/libgcc_s-4.7.2-20121109.so.1 3effa00000-3effa5d000 r-xp 00000000 fd:01 700439 /usr/lib64/libpcre.so.1.0.1 3effa5d000-3effc5c000 ---p 0005d000 fd:01 700439 /usr/lib64/libpcre.so.1.0.1 3effc5c000-3effc5d000 r--p 0005c000 fd:01 700439 /usr/lib64/libpcre.so.1.0.1 3effc5d000-3effc5e000 rw-p 0005d000 fd:01 700439 /usr/lib64/libpcre.so.1.0.1 3effe00000-3effe1f000 r-xp 00000000 fd:01 700440 /usr/lib64/libselinux.so.1 3effe1f000-3f0001e000 ---p 0001f000 fd:01 700440 /usr/lib64/libselinux.so.1 3f0001e000-3f0001f000 r--p 0001e000 fd:01 700440 /usr/lib64/libselinux.so.1 3f0001f000-3f00020000 rw-p 0001f000 fd:01 700440 /usr/lib64/libselinux.so.1 3f00020000-3f00022000 rw-p 00000000 00:00 0 3f00200000-3f00216000 r-xp 00000000 fd:01 700438 /usr/lib64/libresolv-2.16.so 3f00216000-3f00415000 ---p 00016000 fd:01 700438 /usr/lib64/libresolv-2.16.so 3f00415000-3f00416000 r--p 00015000 fd:01 700438 /usr/lib64/libresolv-2.16.so 3f00416000-3f00417000 rw-p 00016000 fd:01 700438 /usr/lib64/libresolv-2.16.so 3f00417000-3f00419000 rw-p 00000000 00:00 0 3f06200000-3f06203000 r-xp 00000000 fd:01 700443 /usr/lib64/libcom_err.so.2.1 3f06203000-3f06402000 ---p 00003000 fd:01 700443 /usr/lib64/libcom_err.so.2.1 3f06402000-3f06403000 r--p 00002000 fd:01 700443 /usr/lib64/libcom_err.so.2.1 3f06403000-3f06404000 rw-p 00003000 fd:01 700443 /usr/lib64/libcom_err.so.2.1 3f06600000-3f06603000 r-xp 00000000 fd:01 665783 /usr/lib64/libkeyutils.so.1.4 3f06603000-3f06802000 ---p 00003000 fd:01 665783 /usr/lib64/libkeyutils.so.1.4 3f06802000-3f06803000 r--p 00002000 fd:01 665783 /usr/lib64/libkeyutils.so.1.4 3f06803000-3f06804000 rw-p 00003000 fd:01 665783 /usr/lib64/libkeyutils.so.1.4 3f06a00000-3f06a0a000 r-xp 00000000 fd:01 700441 /usr/lib64/libkrb5support.so.0.1 3f06a0a000-3f06c09000 ---p 0000a000 fd:01 700441 /usr/lib64/libkrb5support.so.0.1 3f06c09000-3f06c0a000 r--p 00009000 fd:01 700441 /usr/lib64/libkrb5support.so.0.1 3f06c0a000-3f06c0b000 rw-p 0000a000 fd:01 700441 /usr/lib64/libkrb5support.so.0.1 3f06e00000-3f06e28000 r-xp 00000000 fd:01 700442 /usr/lib64/libk5crypto.so.3.1 3f06e28000-3f07028000 ---p 00028000 fd:01 700442 /usr/lib64/libk5crypto.so.3.1 3f07028000-3f07029000 r--p 00028000 fd:01 700442 /usr/lib64/libk5crypto.so.3.1 3f07029000-3f0702a000 rw-p 00029000 fd:01 700442 /usr/lib64/libk5crypto.so.3.1 3f0702a000-3f0702b000 rw-p 00000000 00:00 0 3f07200000-3f072d9000 r-xp 00000000 fd:01 700444 /usr/lib64/libkrb5.so.3.3 3f072d9000-3f074d9000 ---p 000d9000 fd:01 700444 /usr/lib64/libkrb5.so.3.3 3f074d9000-3f074e3000 r--p 000d9000 fd:01 700444 /usr/lib64/libkrb5.so.3.3 3f074e3000-3f074e5000 rw-p 000e3000 fd:01 700444 /usr/lib64/libkrb5.so.3.3 3f07600000-3f07641000 r-xp 00000000 fd:01 700445 /usr/lib64/libgssapi_krb5.so.2.2 3f07641000-3f07840000 ---p 00041000 fd:01 700445 /usr/lib64/libgssapi_krb5.so.2.2 3f07840000-3f07841000 r--p 00040000 fd:01 700445 /usr/lib64/libgssapi_krb5.so.2.2 3f07841000-3f07843000 rw-p 00041000 fd:01 700445 /usr/lib64/libgssapi_krb5.so.2.2 3f07e00000-3f07f9b000 r-xp 00000000 fd:01 700446 /usr/lib64/libcrypto.so.1.0.1c 3f07f9b000-3f0819a000 ---p 0019b000 fd:01 700446 /usr/lib64/libcrypto.so.1.0.1c 3f0819a000-3f081b4000 r--p 0019a000 fd:01 700446 /usr/lib64/libcrypto.so.1.0.1c 3f081b4000-3f081bf000 rw-p 001b4000 fd:01 700446 /usr/lib64/libcrypto.so.1.0.1c 3f081bf000-3f081c4000 rw-p 00000000 00:00 0 3f08e00000-3f08e5b000 r-xp 00000000 fd:01 700447 /usr/lib64/libssl.so.1.0.1c 3f08e5b000-3f0905a000 ---p 0005b000 fd:01 700447 /usr/lib64/libssl.so.1.0.1c 3f0905a000-3f0905e000 r--p 0005a000 fd:01 700447 /usr/lib64/libssl.so.1.0.1c 3f0905e000-3f09064000 rw-p 0005e000 fd:01 700447 /usr/lib64/libssl.so.1.0.1c 3f0fa00000-3f0fa0f000 r-xp 00000000 fd:01 665786 /usr/lib64/libbz2.so.1.0.6 3f0fa0f000-3f0fc0e000 ---p 0000f000 fd:01 665786 /usr/lib64/libbz2.so.1.0.6 3f0fc0e000-3f0fc0f000 r--p 0000e000 fd:01 665786 /usr/lib64/libbz2.so.1.0.6 3f0fc0f000-3f0fc10000 rw-p 0000f000 fd:01 665786 /usr/lib64/libbz2.so.1.0.6 3f11600000-3f11663000 r-xp 00000000 fd:01 700543 /usr/lib64/libfreebl3.soAborted (core dumped) but if i load fewer columns, it succeed: sprintf(sql, "copy binary into test3 from('/home/data1/writeBinary/id','/home/data1/writeBinary/1')"); so if i can load into table with a certain column ? Thanks! Meng

2 1

XQuery support in MonetDB
by Stephen Cameron 20 Jul '13

20 Jul '13

Hello List, Is there anyone making use of XQuery with MonetDB? Why was this XQuery work terminated/frozen, obviously its now a low priority, but why? People seem to resist saying that something is not as good an idea as first thought! Thanks for your insights. Steve Cameron collinta.com.au

2 1

Must the file be deleted after "copy binary into table from file"？
by integrity 19 Jul '13

19 Jul '13

can i have the file reserved? Thanks Meng

4 3

Re: Must the file be deleted after "copy binary into table from file"？
by integrity 19 Jul '13

19 Jul '13

Thanks very much for explaining these to me, but is it possible to provide two options of move, one do the move and reserve, one just move? That will be very convenient for uses.^_^ Sincerely, Meng PS,google mail in China is very unstable, often got disconnected from internet for some reason, so I have to applie another account here. ------------------ Original ------------------ From: "Stefan Manegold"<Stefan.Manegold(a)cwi.nl>; Date: Fri, Jul 19, 2013 02:15 PM To: "Communication channel for MonetDB users"<users-list(a)monetdb.org>; Cc: "Niels Nes"<Niels.Nes(a)cwi.nl>; "sjoerd"<Sjoerd.Mullender(a)cwi.nl>; Subject: Re: Must the file be deleted after "copy binary into table from file"？ Hi Meng, for efficiently, i.e., to avoid a file copy, MonetDB's binary copy into simply moves(^) the user-provided and to be loaded files into its dbfarm. Thus, if you need to preserve a copy of the files you provide, you need to do so yourself before executing a binary copy into. Best, Stefan (^) 1) The actual behavior (move) is actually different than the copy that the documentation at http://www.monetdb.org/Documentation/Cookbooks/SQLrecipes/BinaryBulkLoad suggests. We need to fix the latter. 2) The actual behavior currently differs between data types; for numerical types, the user-provided files are moved as explained above; for stings, the files are read and processed, i.e., the used-provided file remains in place as-is. We need to consider unifying this behavior, e.g., by deleting also the user provided files for strings once they have been processed/loaded. 3) In case the copy into file, e.g., due to a failing sanity check after the file is moved, that data is factually lost, i.e., the files are gone, but no data is loaded into the database; we need to either document this clearly, or make the implementation behave like a transaction. ----- Original Message ----- > can i have the file reserved? > > > > Thanks > Meng > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/users-list > -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list .

2 1