[Monetdb-developers] COPY, terminating connection
Hi, When i run the COPY command from a batch script and after few minutes of processing (2,3minutes), and few thousand lines inserted in the DB, im getting : Connection terminated merovingian log is showing: MSG demo[5084]: # MonetDB server v5.6.0, based on kernel v1.24.0 MSG demo[5084]: # Serving database 'demo' MSG demo[5084]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked MSG demo[5084]: # Copyright (c) 1993-2008 CWI, all rights reserved MSG demo[5084]: # Visit http://monetdb.cwi.nl/ for further information MSG demo[5084]: # Listening for connection requests on mapi:monetdb://ulys:50001/ MSG merovingian[3981]: proxying client 127.0.0.1:41420 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41420 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41422 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41422 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41424 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41424 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41426 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41426 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41428 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41428 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41430 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41430 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41432 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41434 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41432 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41436 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41434 disconnected from proxy MSG merovingian[3981]: client has 127.0.0.1:41436 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41438 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41438 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41440 for database 'demo' to mapi:monetdb://ulys:50001/ TME 2008-09-12 11:56:46 MSG merovingian[3981]: client has 127.0.0.1:41440 disconnected from proxy This is what my script does : #!/bin/sh FILE=$1 awk -F "\"*;\"*" '{print ";",$1,";",$2,";",$3,";",$4,";",$5,";",$6,";",$8,";",$9,";",$10,";",$13,";",$15,";",$18,";",$22,";",$24,";",$25,";",$26,";",$27,";",$28,";",$29,";",$30,";",$31,";",$32,";",$33,";",$35,";",$36,";",$38,";",$39,";",$42,";",$47,";",$48,";",$49,";",$53,";",$54,";",$55,";","FILE"}' $1 | sed -e "s/ ; /;/g; s/; /;/g" | mclient -lsql -d demo -i -s "COPY INTO cdrwholesaler FROM STDIN USING DELIMITERS ';' NULL AS ''" and this is how i call it : find /usr/ftp -name '*.CDR' -exec /var/www/consolecdr/batch/test_monetdb/awk_csv_generator.sh {} \; Is there anything im doing wrong ? I have the feeling there must be too many connections crashing the system .. coz after that i can't use mclient anymore, i have to restart the DB. Please advise, Thanks SB
On Fri, Sep 12, 2008 at 04:48:43AM -0700, sylver_b wrote:
Hi,
When i run the COPY command from a batch script and after few minutes of processing (2,3minutes), and few thousand lines inserted in the DB, im getting :
Connection terminated
merovingian log is showing:
MSG demo[5084]: # MonetDB server v5.6.0, based on kernel v1.24.0 MSG demo[5084]: # Serving database 'demo' MSG demo[5084]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked MSG demo[5084]: # Copyright (c) 1993-2008 CWI, all rights reserved MSG demo[5084]: # Visit http://monetdb.cwi.nl/ for further information MSG demo[5084]: # Listening for connection requests on mapi:monetdb://ulys:50001/ MSG merovingian[3981]: proxying client 127.0.0.1:41420 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41420 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41422 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41422 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41424 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41424 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41426 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41426 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41428 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41428 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41430 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41430 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41432 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41434 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41432 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41436 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41434 disconnected from proxy MSG merovingian[3981]: client has 127.0.0.1:41436 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41438 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41438 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41440 for database 'demo' to mapi:monetdb://ulys:50001/ TME 2008-09-12 11:56:46 MSG merovingian[3981]: client has 127.0.0.1:41440 disconnected from proxy
This is what my script does :
#!/bin/sh
FILE=$1 awk -F "\"*;\"*" '{print ";",$1,";",$2,";",$3,";",$4,";",$5,";",$6,";",$8,";",$9,";",$10,";",$13,";",$15,";",$18,";",$22,";",$24,";",$25,";",$26,";",$27,";",$28,";",$29,";",$30,";",$31,";",$32,";",$33,";",$35,";",$36,";",$38,";",$39,";",$42,";",$47,";",$48,";",$49,";",$53,";",$54,";",$55,";","FILE"}' $1 | sed -e "s/ ; /;/g; s/; /;/g" | mclient -lsql -d demo -i -s "COPY INTO cdrwholesaler FROM STDIN USING DELIMITERS ';' NULL AS ''"
and this is how i call it :
find /usr/ftp -name '*.CDR' -exec /var/www/consolecdr/batch/test_monetdb/awk_csv_generator.sh {} \;
Is there anything im doing wrong ? I have the feeling there must be too many connections crashing the system .. coz after that i can't use mclient anymore, i have to restart the DB.
How many (sequential, not concurrent, right?) connections are we talking about, i.e., how many *.CDR files do you have? How many rows do these *.CDR files hold in total? Are all attributes of the cdrwholesaler table string attributes? ... "too many connections" *might* be the problem (though it should of course not) --- strange, though, that the merovingian log do not give any indication of whether, and if so why the server might have crashed ... Stefan
Please advise, Thanks SB
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On 13-09-2008 07:36:19 +0200, Stefan Manegold wrote:
Is there anything im doing wrong ? I have the feeling there must be too many connections crashing the system .. coz after that i can't use mclient anymore, i have to restart the DB.
How many (sequential, not concurrent, right?) connections are we talking about, i.e., how many *.CDR files do you have?
How many rows do these *.CDR files hold in total?
Are all attributes of the cdrwholesaler table string attributes?
... "too many connections" *might* be the problem (though it should of course not) --- strange, though, that the merovingian log do not give any indication of whether, and if so why the server might have crashed ...
It can't, because the server didn't crash. It indicates it is still alive. Most probably the server is confused by the amount of inserts, I've seen this a few times before. Somewhere something gets stuck, making the mserver refuse to operate. A restart of the server solves it, as the original reporter also indicates. Note from the logs that the server "demo" had one crash in the past, though.
Fabian Groffen wrote:
On 13-09-2008 07:36:19 +0200, Stefan Manegold wrote:
Is there anything im doing wrong ? I have the feeling there must be too many connections crashing the system .. coz after that i can't use mclient anymore, i have to restart the DB.
How many (sequential, not concurrent, right?) connections are we talking about, i.e., how many *.CDR files do you have?
How many rows do these *.CDR files hold in total?
Are all attributes of the cdrwholesaler table string attributes?
... "too many connections" *might* be the problem (though it should of course not) --- strange, though, that the merovingian log do not give any indication of whether, and if so why the server might have crashed ...
It can't, because the server didn't crash. It indicates it is still alive. Most probably the server is confused by the amount of inserts,
If the input data is not valid, such that the number of tuples are not correct, then indeed the server will at some point fail,.e.g waiting for tuples that never appeara.
I've seen this a few times before. Somewhere something gets stuck, making the mserver refuse to operate. A restart of the server solves it, as the original reporter also indicates.
Note from the logs that the server "demo" had one crash in the past, though.
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
participants (4)
-
Fabian Groffen -
Martin Kersten -
Stefan Manegold -
sylver_b