I have a large database that loads nicely on MonetDB from MySQL when it'd done with INSERTs with one row at a time. But when I group them in sets of 300 or so, for any reason, i.e. INSERT INTO TABLE VALUES (FOO), (BAR) ... ; it develops a memory leak which is not helped by setting gdk_mem_bigsize or gdk_vm_minsize. And when that makes me abort the load, the result is that backend data are corrupted. I can start the database server, but on the first connection attempt, this happens: [pid 18878] read(19, "\0\0", 2) = 2 [pid 18878] close(19) = 0 [pid 18878] stat("bat/02/224.tail", {st_mode=S_IFREG|0600, st_size=254, ...}) = 0 [pid 18878] stat("bat/02/224.tail.new", 0x416d9b00) = -1 ENOENT (No such file or directory) [pid 18878] open("bat/02/224.tail", O_RDONLY) = 19 [pid 18878] read(19, "\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1"..., 254) = 254 [pid 18878] close(19) = 0 [pid 18878] --- SIGBUS (Bus error) @ 0 (0) --- Process 18878 detached [pid 18877] +++ killed by SIGBUS +++ +++ killed by SIGBUS +++ So, a couple questions: 1. how hard would it be to write a backend integritiy checker for the binary association tables? 2. what more can I do to make sure all the changes in the backend are truly atomic ? 3. Are there good prospects for this memory leak being traced and addressed? 4. Can the BAT file format be changed to have more checks on its integrity? (checksums, sentinel variables, etc) 5. Might it make sense for the backend files to have a format relying on existing database libraries? BDB? Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz@denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.