[MonetDB-users] Concurrent aggregation crashes MonetDB

6 Jan 2011

      Hi!

I'm running an evaluation test on the latest MonetDB/SQL stable version and
always crashes possibly due an out-of-memory problem.

My system: Windows 7 x86, 4GB memory, Core i7 920, data files are on an SSD
disk

MonetDB (default install):
# MonetDB server v5.22.1, based on kernel v1.40.1
# Release Oct2010
# Serving database 'demo', using 8 threads
# Compiled for i686-pc-win32/32bit with 32bit OIDs dynamically linked
# Found 3.186 GiB available main-memory.
# Copyright (c) 1993-July 2008 CWI.
# Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved
# Visit http://monetdb.cwi.nl/ for further information
#warning: please don't forget to set your vault key!
#(see C:\Program Files\MonetDB\MonetDB5\etc\monetdb5.conf)
# Listening for connection requests on mapi:monetdb://127.0.0.1:50000/
# MonetDB/SQL module v2.40.1 loaded

Using monetdb-1.17-jdbc driver.

The test involves a simple, two column table:

CREATE TABLE a (text VARCHAR(50), value INT)

Which is then filled with 10.000.000 records of random values. The data
files take up to 150MB disk space.

The test routine creates multiple threads which connect to the local demo
database, and simultaneously execute the same aggregation query. The test
simulates a concurrent analysis-data mining scenario running on the same
data.

void parallelTest(int threadCount) {
ExecutorService exec = Executors.newFixedThreadPool(threadCount);
 final CountDownLatch cnt = new CountDownLatch(threadCount);
 for (int i = 0; i < threadCount; i++) {
final int j = i;
exec.submit(new Runnable() {
@Override
public void run() {
try {
Class.forName("nl.cwi.monetdb.jdbc.MonetDriver");
Connection conn =
DriverManager.getConnection("jdbc:monetdb://localhost/demo", "monetdb",
"monetdb");
try {
conn.setAutoCommit(false);
 long t1 = System.currentTimeMillis();
Statement stmt = conn.createStatement();
try {
cnt.countDown();
cnt.await();
ResultSet rs = stmt.executeQuery("SELECT text, AVG(value) FROM a GROUP BY
text ");
while (rs.next()) {
rs.getString(1);
rs.getDouble(2);
}
System.out.printf("[%02d] Select time: %.2fs%n", j,
(System.currentTimeMillis() - t1) / 1000.0);
} finally {
stmt.close();
}
 } finally {
conn.close();
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
});
}
 exec.shutdown();
try {
exec.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
} catch (InterruptedException ex) {
 }
}

I noticed that if the threadCount >= 4 the MonetDB console starts to print
VirtualAlloc failed messages and sooner or later terminates.
The memory usage seems to jump by 600MB, e.g. with 3 threads it consumes 1.7
GB memory, but with 4 threads, it can't consume 2.4GB.

Replacing the query with a single result "SELECT MAX(VALUE) FROM a" causes
the same crash after 5 threads.

The same test running with the latest MySQL works perfectly, although
slowly. No crashes or memory issues.

Is this a memory leak in MonetDB or the result of its architecture, meaning
I should use a 64bit system with lots of RAM?

Regards,
David Karnok

[MonetDB-users] Concurrent aggregation crashes MonetDB

Dávid Karnok