[MonetDB-users] multi-users and memory problems

20 Dec 2008

      Dear developers,

I'm currently running the CVS head of Dec 11, 2008. I apologies in advance
for missing on details. If needed I'll provide more.

Thank you, and
lots of wonderful wishes for Xmas and for the New Year!!!
(this is the last email you'll get from me this year, I promise :))
l.

Observation 1:
When trying to run big numbers of clients simultaneously, the server
crushes. For me it crushed with 70 and 100 clients fired at the same time.

Observation 2:
After running one query in a multi-user scenario (N clients/threads
simultaneously) the memory footprint of mserver (% of the memory that the
server occupies) grew. After repeating the experiment M times the memory
grew each time with a constant. Running the same query sequentially the same
number of times leaves the footprint of mserver constant. Could it be a
faulty memory cleaning?

Query: q3.xq
let $col := fn:collection("MotiesTweedeKamer")
let $years := fn:distinct-values(
              for $date in $col//hiddendatum
              return fn:substring(fn:string($date),1,4))
for $y in $years
order by $y ascending
return <result year="{$y}" count="{
     count($col//document[fn:substring(fn:string(.//hiddendatum),1,4) = $y])
}"/>

N=50
M=1
$ perl runNclients.pl N=50
$ top
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

28563 lafanasi  20   0 6266m 5.1g 100m S    0 26.0  10:50.71
Mserver
M=2
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

28563 lafanasi  20   0 7546m 5.9g 100m S    0 30.1  21:42.67
Mserver
...
N=60
M=10
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

31182 lafanasi  20   0 17.3g 8.3g 100m S    0 42.3 263:45.37 Mserver
...
M=30
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

31182 lafanasi  20   0 22.7g 9.6g 100m S    0 48.8 395:08.87
Mserver

Sequential run:

N=1
M=50 times
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

28372 lafanasi  20   0  839m 430m 100m S    1  2.1   6:26.65
Mserver

M=100 times
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

28372 lafanasi  20   0  839m 430m 100m S    1  2.1   6:26.65
Mserver

Observation 3:
The times it takes to run 10 Tijah search queries simultaneously is the same
as the time it takes to run them sequentially. The percentage of CPU used in
both cases by mserver is also similar. Does tijah support multi-users?

Query: q2.xq
let $opt := <TijahOptions ft-index="polietiekedata" ir-model="NLLR"/>
let $c := collection("HAN")
let $qid :=  tijah:query-id($c, "//spreker[about(.,%KEYWORD%)]", $opt)
for $res in tijah:nodes($qid)
return <pair>{( string($res/@naam), tijah:score($qid, $res))}</pair>

20 users/threads
stopMserver,startMserver (to make sure that the server freed the memory)
$time Run20Threads(mclient -lx q2.xq)
0.134u 0.118s 0:20.01 1.1%    0+0k 0+8io 0pf+0w

sequential run, 20 times
stopMserver, startMserver
$ time for i in `seq 1 10`; do mclient -lx q2.xq; done
0.060u 0.067s 0:19.74 0.6%    0+0k 0+8io 0pf+0w

Observation 4:
The mserver memory footprint grows very fast when running Tijah queries.
When the footprint reaches 98% the query processing time gets really slow or
the server crushes.

Query: the same as above

Sequential run, M=90 times
$stopMserver, startMserver
$for i in `seq 1 10`; do mclient -lx q2.xq; done
$ top
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

24527 lafanasi  20   0 26.2g  19g  80m S   66 98.0   4:24.57 Mserver

Loredana Afanasiev

Stefan Manegold

tags

participants (2)