[Monetdb-developers] Bug report - COPY RECORDS and encoded data
I was going crazy trying to figure out why my COPY RECORDS command wasn't working. Turns out it is caused from funny (encoded?) characters in the string for one of the fields. For instance one of the fields contained this "?a?as?e??, 18 ?a??? 2001 10:24:00 pµ" How did the data get this way? Well I exported it from SQL Server and I guess that db is set up to store encoded characters. When I did the export I told SQL Server to make the file ASCII since I figured that is what COPY RECORDS expects. So apparently SQL Server is writing the characters this way into the string. Then MonetDB (latest version) cannot handle this. So what happens is that the COPY x RECORDS command just returns immediately after I try this with this data in it. After that all my SQL statements in mclient return immediately. So I can say "select count(*) from foo;" and it just comes immediately back with the sql> prompt. So something with these funny characters sends mclient into a funk. I have to exit and restart it to get it going again. Any advice on how to handle this? Thank you.
Rt Ibmer wrote:
I was going crazy trying to figure out why my COPY RECORDS command wasn't working. Turns out it is caused from funny (encoded?) characters in the string for one of the fields.
For instance one of the fields contained this "?a?as?e??, 18 ?a??? 2001 10:24:00 pµ"
How did the data get this way? Well I exported it from SQL Server and I guess that db is set up to store encoded characters. When I did the export I told SQL Server to make the file ASCII since I figured that is what COPY RECORDS expects. So apparently SQL Server is writing the characters this way into the string. Then MonetDB (latest version) cannot handle this.
The details of SQL Server side can not be answered. MonetDB expects the control characters in the records to be encoded in the C-style, i.e \ddd. Using e.g. a null byte will terminate a line and will cause the parsing of the fields to go astray.
So what happens is that the COPY x RECORDS command just returns immediately after I try this with this data in it. After that all my SQL statements in mclient return immediately. So I can say "select count(*) from foo;" and it just comes immediately back with the sql> prompt. So something with these funny characters sends mclient into a funk. I have to exit and restart it to get it going again.
Any advice on how to handle this? Thank you.
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Sun, Jun 08, 2008 at 08:43:24AM +0200, Martin Kersten wrote: A fix for this bug was checked into the stable branch. Niels
Rt Ibmer wrote:
I was going crazy trying to figure out why my COPY RECORDS command wasn't working. Turns out it is caused from funny (encoded?) characters in the string for one of the fields.
For instance one of the fields contained this "?a?as?e??, 18 ?a??? 2001 10:24:00 pµ"
How did the data get this way? Well I exported it from SQL Server and I guess that db is set up to store encoded characters. When I did the export I told SQL Server to make the file ASCII since I figured that is what COPY RECORDS expects. So apparently SQL Server is writing the characters this way into the string. Then MonetDB (latest version) cannot handle this.
The details of SQL Server side can not be answered. MonetDB expects the control characters in the records to be encoded in the C-style, i.e \ddd. Using e.g. a null byte will terminate a line and will cause the parsing of the fields to go astray.
So what happens is that the COPY x RECORDS command just returns immediately after I try this with this data in it. After that all my SQL statements in mclient return immediately. So I can say "select count(*) from foo;" and it just comes immediately back with the sql> prompt. So something with these funny characters sends mclient into a funk. I have to exit and restart it to get it going again.
Any advice on how to handle this? Thank you.
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
participants (3)
-
Martin Kersten -
Niels Nes -
Rt Ibmer