I have a large
database that loads nicely on MonetDB from MySQL when it'd done with INSERTs
with one row at a time.
But when I group
them in sets of 300 or so, for any reason, i.e. INSERT INTO TABLE VALUES (FOO),
(BAR) ... ;
it develops a memory
leak which is not helped by setting gdk_mem_bigsize or
gdk_vm_minsize.
And when that makes
me abort the load, the result is that backend data are
corrupted.
I can start the
database server, but on the first connection attempt, this
happens:
[pid 18878] read(19,
"\0\0", 2) = 2
[pid 18878]
close(19)
= 0
[pid 18878] stat("bat/02/224.tail", {st_mode=S_IFREG|0600, st_size=254,
...}) = 0
[pid 18878] stat("bat/02/224.tail.new", 0x416d9b00) = -1 ENOENT (No
such file or directory)
[pid 18878] open("bat/02/224.tail", O_RDONLY) =
19
[pid 18878] read(19,
"\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1"..., 254) =
254
[pid 18878]
close(19)
= 0
[pid 18878] --- SIGBUS (Bus error) @ 0 (0) ---
Process 18878
detached
[pid 18877] +++ killed by SIGBUS +++
+++ killed by SIGBUS
+++
So, a couple
questions:
1. how hard would it
be to write a backend integritiy checker for the binary association tables?
2. what more can I
do to make sure all the changes in the backend are truly atomic
?
3. Are there good
prospects for this memory leak being traced and addressed?
4. Can the BAT file
format be changed to have more checks on its integrity? (checksums, sentinel
variables, etc)
5. Might it make
sense for the backend files to have a format relying on existing database
libraries? BDB?
Denham Quantitative
Management LP
200 Clarendon Street, 25th
Floor
Boston, MA 02216
T: 617-587-5812
M: 617-909-7786
omri.schwarz@denhamcapital.com
NOTICE: This e-mail message is intended for the named recipient(s) above and
may contain confidential information. If you are not the intended recipient, you
are hereby notified that any dissemination, distribution, or copying of this e-mail
and any attachment(s) is strictly prohibited. If you have received this e-mail in
error, please immediately notify the sender by replying to this e-mail and delete
the message and any attachment(s) from your system. NO REPRESENTATION
OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY
OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN.
Denham Capital Management(sm) is a trademark of Denham Capital Management LP.