This appears to have fixed the data corruption issue. This seems like a major problem for MonetDB. Perhaps this configuration option should be added to the default config until a permanent fix can be applied. 73, Matthew W. Jones (KI4ZIB) http://matburt.net On Tue, Jul 28, 2009 at 2:22 PM, Matthew Jones wrote:

This was a COPY ... INTO

I'll give this setting a try and report back, though it seems like I tried this recently and still saw some corruption.

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Tue, Jul 28, 2009 at 1:30 PM, Allen Zhang wrote:

...

Hi, Matthew,

How are you importing your data? Copy or insert?

Because we had such issue before, but in our case, it’s extreme, it lost all the data we inserted when we restart the database.

The way to get around this is by setting our database into a single user mode, in the monetdb5.conf, add a line gdk_single_user=yes.

We managed keep the data in this way, so maybe you can give it a try.

Best Regards

Allen

------------------------------

*From:* Matthew Jones [mailto:mat@matburt.net] *Sent:* Monday, July 27, 2009 7:28 AM *To:* monetdb-users@lists.sourceforge.net *Subject:* [MonetDB-users] massive data loss on stop, start

I added this information as a comment to bug *2825243* but I'm not sure if it's the same so I thought I'd post it here also to see about getting some feedback:

I am experiencing widespread data corruption when restarting the May2009-SP1 MonetDB release also, but mine is much simpler.

On 64-bit ubuntu: Linux dsvm01 2.6.24-23-server #1 SMP Wed Apr 1 22:14:30 UTC 2009 x86_64 GNU/Linux

With a table defined as follows: CREATE TABLE "tablespace"."test_table" ( "col1" varchar(50), "col2" varchar(50), "col3" varchar(50), "col4" varchar(50), "col5" varchar(50), "col6" int, "col7" timestamp(7), "col8" int, "col9" int, "col10" int );

I imported 20998687 records into this table with all columns populated and ran a couple of queries to check the data:

select count(*) from test_table ; [ 20998687 ]

select count(*) from test_table where col1 = '' or col2 = '' or col3 = ''; [ 0 ]

select max(col7), min(col7) from test_table ; [ 2009-07-25 14:17:15.538000, 2009-07-24 15:29:12.524000 ]

During the time I loaded the data and ran these queries I did not restart the database, it had been up for about 3 days I then issued:

monetdb stop db

and

monetdb start db

and re-ran the queries:

select count(*) from raw_plays ; [ 20998687 ]

select count(*) from raw_plays where col1 = '' or col2 = '' or col3 = ''; [ 20852061 ]

select max(col7), min(col7) from test_table ; [ 2009-07-25 14:17:15.538000, -1-01-01 00:00:00.000000 ]

This is some serious data corruption, I've lost data in the vast majority of these records!

73, Matthew W. Jones (KI4ZIB) http://matburt.net

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users