RAM: 32GB.
I am running an ETL that keeps sending Inserts/Updates/DDLs to the DB continuously(more details below).
The errors occured while the ETL was done with around 3.5 lakh inserts. The farm size is around 137MB.
The ETL sends batches of 100 inserts/1 update/1 DDL in a go with auto-commit=false. Initially inserts happened at around 1 insert batch per second.
When I checked after the table size was above 2 lakh records speed was 1 insert batch/10-15 seconds.
The client setup is as follows:
JDBC with c3p0 connection pooling parameters as:
min_size: 20,
max_size: 40,(pool size)
acquire_increment: 5,
idle_test_period: 60,
max_statements: 0,
timeout: 2000
I am also trying to achieve master-master replication here. I acquire a java lock per master before firing a query(to prevent commit failures due to optimistic locking). So we can assume that the inserts are pretty much funneled on the client itself.
I have never seen the memory usage exceed 3% after switching to Feb2013(No selects yet)
Following is the monetdb status of the failing master:
start count: 33
stop count: 7
crash count: 25
current uptime: 12m 6s
average uptime: 2h 20m 13s
maximum uptime: 4h 51m 24s
average of crashes in the last 10 start attempts: 0.50
average of crashes in the last 30 start attempts: 0.73
crash average: 0.00 0.50 0.73 (over 1, 15, 30 starts) in total 25 crashes
uptime stats (min/avg/max): 4s/2h/4h over 7 runs
As we use pooled connections we can assume that there have been around 40 connections with 3.5lakh/100/40 queries per connection.(at least theoretically)
NEW ISSUE:
After this incident I am facing a new issue - I am unable to connect to the database. The mclient just hangs after I enter my password and there is nothing being logged into merovingian.log. monetdbd stop fails to end the monetdbd process. Access comes back with luck after many force kills.
Thanks and Regards,
Tapomay.
________________________________
From: Fabian Groffen
Following is an excerpt from the merovingian logs:
2013-01-10 10:40:49 ERR msearch_stats_db[19210]: !mal_mapi.listen: expected filedescriptor, but received something else
2013-01-10 10:40:57 ERR merovingian[8996]: client error: client [XX.XX.XX.XX]:51573 sent challenge in incomplete block:
2013-01-10 10:45:29 ERR msearch_stats_db[19210]: mserver5: bat_storage.c:40: delta_bind_del: Assertion `b' failed.
2013-01-10 10:45:29 ERR msearch_stats_db[19210]: mserver5: bat_storage.c:40: delta_bind_del: Assertion `b' failed.
2013-01-10 23:48:18 ERR msearch_stats_db[31637]: !mal_mapi.listen: expected filedescriptor, but received something else
2013-01-10 23:48:19 ERR msearch_stats_db[31637]: !mal_mapi.listen: expected filedescriptor, but received something else
2013-01-10 23:48:32 ERR msearch_stats_db[31637]: !FATAL: BBPdir: subcommit attempted without backup BBP.dir.
2013-01-11 04:29:31 ERR msearch_stats_db[4480]: mserver5: gdk_batop.c:175: insert_string_bat: Assertion `v >= ((var_t) (((1<<10) * sizeof(stridx_t)) >> 0))' failed.
You have an interesting amount of errors. Can you describe what load you are imposing on the database, and the system? Something along the lines of connections per second, queries per connection, memory usage, load average, and start, stop and crashcounters of the database would be interesting. -- Fabian Groffen fabian@monetdb.org column-store pioneer http://www.monetdb.org/Home _______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list _______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list