[MonetDB-users] Question about used Encoding

Markus Gritsch

22 Nov 2007 22 Nov '07

4:59 p.m.

Hi, I would like to know how encoding related issues are handles in MonetDB. For example, when I run the following Python script # -*- coding: utf-8 -*- import MonetSQLdb x = MonetSQLdb.connect(host = 'localhost', user = 'voc', password = 'voc', lang = 'sql') c = x.cursor() c.execute('DROP TABLE test') c.execute('CREATE TABLE test(id int, name varchar(20))') c.execute('INSERT INTO test (id, name) VALUES (%s, %s)', (0, 'ä ö ü')) c.close() x.close() and then look at the inserted data using DbVisualizer I get Ã¤ Ã¶ Ã¼ in the 'name' column. When I try to edit the value in the DbVisualizer, I get an error message "unexpected end of input". (The SQL statements reads update voc.test set name = 'ö' where id = 0). Is this a JDBC driver problem? When I connect to an MySQL database with the MySQL JDBC driver included in DbVisualizer I can insert arbitrary characters, and they get properly displayed in the GUI. Kind regards, Markus

Show replies by date

Niels Nes

22 Nov 22 Nov

7:49 p.m.

On Thu, Nov 22, 2007 at 05:59:09PM +0100, Markus Gritsch wrote:

...

Hi,

I would like to know how encoding related issues are handles in MonetDB. For example, when I run the following Python script

# -*- coding: utf-8 -*-

import MonetSQLdb

x = MonetSQLdb.connect(host = 'localhost', user = 'voc', password = 'voc', lang = 'sql')

c = x.cursor()

c.execute('DROP TABLE test') c.execute('CREATE TABLE test(id int, name varchar(20))') c.execute('INSERT INTO test (id, name) VALUES (%s, %s)', (0, 'ä ö ü'))

c.close() x.close()

and then look at the inserted data using DbVisualizer I get

Ã¤ Ã¶ Ã¼ How does it look with mclient -lsql -s "select * from test;" ?

The strings should be cast to utf8 by the client side.

...

in the 'name' column. When I try to edit the value in the DbVisualizer, I get an error message

"unexpected end of input". (The SQL statements reads update voc.test set name = 'ö' where id = 0).

Is this a JDBC driver problem? When I connect to an MySQL database with the MySQL JDBC driver included in DbVisualizer I can insert arbitrary characters, and they get properly displayed in the GUI.

I'm not sure. Which jdbc driver version do you have? Niels

...

Kind regards, Markus

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

Fabian Groffen

7:52 p.m.

On 22-11-2007 20:49:03 +0100, Niels Nes wrote:

...

...
in the 'name' column. When I try to edit the value in the DbVisualizer, I get an error message

"unexpected end of input". (The SQL statements reads update voc.test set name = 'ö' where id = 0).

Is this a JDBC driver problem? When I connect to an MySQL database with the MySQL JDBC driver included in DbVisualizer I can insert arbitrary characters, and they get properly displayed in the GUI. I'm not sure. Which jdbc driver version do you have?

Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

Markus Gritsch

7:57 p.m.

On Nov 22, 2007 8:52 PM, Fabian Groffen wrote:

...

Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

Where does the debug information go to?

Fabian Groffen

7:59 p.m.

On 22-11-2007 20:57:14 +0100, Markus Gritsch wrote:

...

On Nov 22, 2007 8:52 PM, Fabian Groffen wrote:

...
Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

Where does the debug information go to?

Somewhere in the working directory in a file called monet_<numbers>.log

Markus Gritsch

8:20 p.m.

On Nov 22, 2007 8:52 PM, Fabian Groffen wrote:

...

Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

The attached logs show running this statement: UPDATE sys.test SET name = 'ä' WHERE id = 0 Markus

Fabian Groffen

8:29 p.m.

On 22-11-2007 21:20:50 +0100, Markus Gritsch wrote:

...

On Nov 22, 2007 8:52 PM, Fabian Groffen wrote:

...
Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

The attached logs show running this statement:

UPDATE sys.test SET name = 'ä' WHERE id = 0

Your logs actually show you don't even get to the tricky part: TD 1195762741893: write final block: 53 bytes TX 1195762741893: sPREPARE UPDATE sys.test SET name = '?' WHERE id = 0; RD 1195762741893: read final block: 25 bytes RX 1195762741893: !unexpected end of input The server aparently doesn't like this prepare statement. Niels, any idea on this?

Niels Nes

9:56 p.m.

On Thu, Nov 22, 2007 at 09:29:48PM +0100, Fabian Groffen wrote:

...

On 22-11-2007 21:20:50 +0100, Markus Gritsch wrote:

...
On Nov 22, 2007 8:52 PM, Fabian Groffen wrote:

...
Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

The attached logs show running this statement:

UPDATE sys.test SET name = 'ä' WHERE id = 0 ^

Your logs actually show you don't even get to the tricky part:

TD 1195762741893: write final block: 53 bytes TX 1195762741893: sPREPARE UPDATE sys.test SET name = '?' WHERE id = 0; fabian

This is from the jdbc logs, ie no iconv inbetween here. Niels

...

RD 1195762741893: read final block: 25 bytes RX 1195762741893: !unexpected end of input

The server aparently doesn't like this prepare statement.

Niels, any idea on this?

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Fabian Groffen

10:02 p.m.

On 22-11-2007 22:56:20 +0100, Niels Nes wrote:

...

On Thu, Nov 22, 2007 at 09:29:48PM +0100, Fabian Groffen wrote:

...
On 22-11-2007 21:20:50 +0100, Markus Gritsch wrote:

...
On Nov 22, 2007 8:52 PM, Fabian Groffen wrote:

...
Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

The attached logs show running this statement:

UPDATE sys.test SET name = 'ä' WHERE id = 0 ^

Your logs actually show you don't even get to the tricky part:

TD 1195762741893: write final block: 53 bytes TX 1195762741893: sPREPARE UPDATE sys.test SET name = '?' WHERE id = 0; fabian

This is from the jdbc logs, ie no iconv inbetween here.

mutt uses iconv to show the log, and I copied/pasted from Mutt's pager.

Niels Nes

10:08 p.m.

...

On 22-11-2007 22:56:20 +0100, Niels Nes wrote:

...
On Thu, Nov 22, 2007 at 09:29:48PM +0100, Fabian Groffen wrote:

...
On 22-11-2007 21:20:50 +0100, Markus Gritsch wrote:

...
On Nov 22, 2007 8:52 PM, Fabian Groffen wrote:

...
Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

The attached logs show running this statement:

UPDATE sys.test SET name = 'ä' WHERE id = 0 ^

Your logs actually show you don't even get to the tricky part:

TD 1195762741893: write final block: 53 bytes TX 1195762741893: sPREPARE UPDATE sys.test SET name = '?' WHERE id = 0; fabian

This is from the jdbc logs, ie no iconv inbetween here.

mutt uses iconv to show the log, and I copied/pasted from Mutt's pager. mutt had no problem with the a umlaut. So why would it using

On Thu, Nov 22, 2007 at 11:02:03PM +0100, Fabian Groffen wrote: the log. Niels

...

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Fabian Groffen

10:11 p.m.

On 22-11-2007 23:08:41 +0100, Niels Nes wrote:

...

...
On 22-11-2007 22:56:20 +0100, Niels Nes wrote:

...
On Thu, Nov 22, 2007 at 09:29:48PM +0100, Fabian Groffen wrote:

...
On 22-11-2007 21:20:50 +0100, Markus Gritsch wrote:

...
On Nov 22, 2007 8:52 PM, Fabian Groffen wrote:

...
Also, it would help us very much if you could generate a JDBC debug log when this happens. You can make such log by appending ?debug=true to the JDBC connection url.

The attached logs show running this statement:

UPDATE sys.test SET name = 'ä' WHERE id = 0 ^

Your logs actually show you don't even get to the tricky part:

TD 1195762741893: write final block: 53 bytes TX 1195762741893: sPREPARE UPDATE sys.test SET name = '?' WHERE id = 0; fabian

This is from the jdbc logs, ie no iconv inbetween here.

mutt uses iconv to show the log, and I copied/pasted from Mutt's pager. mutt had no problem with the a umlaut. So why would it using

On Thu, Nov 22, 2007 at 11:02:03PM +0100, Fabian Groffen wrote: the log.

Because the mail has: Content-Type: text/plain; charset="iso-8859-1" whereas the attachment is of type octet/application-stream, so Mutt has to guess, which it probably does wrong, using the default locale (UTF-8) in my case to view the file. libiconv inserts ? chars for things it cannot map.

Markus Gritsch

8:12 p.m.

On Nov 22, 2007 8:49 PM, Niels Nes wrote:

...

How does it look with mclient -lsql -s "select * from test;" ?

Alas, since I am on Windows XP, the cmd.exe is not UTF-8 aware, so I don't know if this helps much. However, I have made a screenshot which shows the output. It is attached as cmd.png.

...

I'm not sure. Which jdbc driver version do you have?

I use MonetDB5-SQL-Installer-i386-20071017.msi which contains monetdb-1.6-jdbc.jar. DbVisualizer is version 6.0.6. The attached logs were produced by connecting, running the query SELECT * FROM test and disconnecting. I hope this helps. Markus

Niels Nes

8:52 p.m.

On Thu, Nov 22, 2007 at 09:12:22PM +0100, Markus Gritsch wrote:

...

On Nov 22, 2007 8:49 PM, Niels Nes wrote:

...
How does it look with mclient -lsql -s "select * from test;" ?

Alas, since I am on Windows XP, the cmd.exe is not UTF-8 aware, so I don't know if this helps much. However, I have made a screenshot which shows the output. It is attached as cmd.png.

...
I'm not sure. Which jdbc driver version do you have?

I use MonetDB5-SQL-Installer-i386-20071017.msi which contains monetdb-1.6-jdbc.jar. DbVisualizer is version 6.0.6.

The attached logs were produced by connecting, running the query

SELECT * FROM test

and disconnecting. I hope this helps. The results seem indeed incorrect. Which would indicate that the insert allready didn't do the right job. So it atleast seems that the bug is in python monetdb interface not in jdbc. I'll have to look into it. Could you try the jdbc*jar from www.cwi.nl/~fabian/troep/, I hope its not giving the 'unexpected end of data' problem.

Niels

...

Markus

...

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Niels Nes

9:15 p.m.

On Thu, Nov 22, 2007 at 09:52:51PM +0100, Niels Nes wrote:

...

On Thu, Nov 22, 2007 at 09:12:22PM +0100, Markus Gritsch wrote:

...
On Nov 22, 2007 8:49 PM, Niels Nes wrote:

...
How does it look with mclient -lsql -s "select * from test;" ?

Alas, since I am on Windows XP, the cmd.exe is not UTF-8 aware, so I don't know if this helps much. However, I have made a screenshot which shows the output. It is attached as cmd.png.

...
I'm not sure. Which jdbc driver version do you have?

I use MonetDB5-SQL-Installer-i386-20071017.msi which contains monetdb-1.6-jdbc.jar. DbVisualizer is version 6.0.6.

The attached logs were produced by connecting, running the query

SELECT * FROM test

and disconnecting. I hope this helps. The results seem indeed incorrect. Which would indicate that the insert allready didn't do the right job. So it atleast seems that the bug is in python monetdb interface not in jdbc. I'll have to look into it. Could you try the jdbc*jar from www.cwi.nl/~fabian/troep/, I hope its not giving the 'unexpected end of data' problem.

Found the bug. In the file MonetSQLdb/converters.py, latin1 should be changed into UTF-8. Niels

...

Niels

...
Markus

...
------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

--

Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Markus Gritsch

9:16 p.m.

On Nov 22, 2007 9:52 PM, Niels Nes wrote:

...

The results seem indeed incorrect. Which would indicate that the insert allready didn't do the right job. So it atleast seems that the bug is in python monetdb interface not in jdbc. I'll have to look into it. Could you try the jdbc*jar from www.cwi.nl/~fabian/troep/, I hope its not giving the 'unexpected end of data' problem.

I tried http://homepages.cwi.nl/~fabian/troep/monetdb-1.6-jdbc.jar and get the same error. Logfiles are attached. Markus

Fabian Groffen

9:17 p.m.

On 22-11-2007 22:16:16 +0100, Markus Gritsch wrote:

...

On Nov 22, 2007 9:52 PM, Niels Nes wrote:

...
The results seem indeed incorrect. Which would indicate that the insert allready didn't do the right job. So it atleast seems that the bug is in python monetdb interface not in jdbc. I'll have to look into it. Could you try the jdbc*jar from www.cwi.nl/~fabian/troep/, I hope its not giving the 'unexpected end of data' problem.

I tried http://homepages.cwi.nl/~fabian/troep/monetdb-1.6-jdbc.jar and get the same error. Logfiles are attached.

Same problem as before: RD 1195766131798: inserting prompt TD 1195766131848: write final block: 53 bytes TX 1195766131848: sPREPARE INSERT INTO test (id, name) VALUES (0, '?'); RD 1195766131848: read final block: 25 bytes RX 1195766131848: !unexpected end of input You don't even get to the UTF-8 part.

Niels Nes

9:28 p.m.

On Thu, Nov 22, 2007 at 10:17:34PM +0100, Fabian Groffen wrote:

...

On 22-11-2007 22:16:16 +0100, Markus Gritsch wrote:

...
On Nov 22, 2007 9:52 PM, Niels Nes wrote:

...
The results seem indeed incorrect. Which would indicate that the insert allready didn't do the right job. So it atleast seems that the bug is in python monetdb interface not in jdbc. I'll have to look into it. Could you try the jdbc*jar from www.cwi.nl/~fabian/troep/, I hope its not giving the 'unexpected end of data' problem.

I tried http://homepages.cwi.nl/~fabian/troep/monetdb-1.6-jdbc.jar and get the same error. Logfiles are attached.

Same problem as before:

RD 1195766131798: inserting prompt TD 1195766131848: write final block: 53 bytes TX 1195766131848: sPREPARE INSERT INTO test (id, name) VALUES (0, '?'); RD 1195766131848: read final block: 25 bytes RX 1195766131848: !unexpected end of input

You don't even get to the UTF-8 part. THis is indeed a problem on the server side. Not sure what yet.

Niels

...

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Fabian Groffen

9:30 p.m.

On 22-11-2007 22:28:26 +0100, Niels Nes wrote:

...

...
...
I tried http://homepages.cwi.nl/~fabian/troep/monetdb-1.6-jdbc.jar and get the same error. Logfiles are attached.

Same problem as before:

RD 1195766131798: inserting prompt TD 1195766131848: write final block: 53 bytes TX 1195766131848: sPREPARE INSERT INTO test (id, name) VALUES (0, '?'); RD 1195766131848: read final block: 25 bytes RX 1195766131848: !unexpected end of input

You don't even get to the UTF-8 part. THis is indeed a problem on the server side. Not sure what yet.

I tried reproducing this on the release, but it works fine for me. Maybe a windows specific issue.

Niels Nes

9:32 p.m.

On Thu, Nov 22, 2007 at 10:28:26PM +0100, Niels Nes wrote:

...

On Thu, Nov 22, 2007 at 10:17:34PM +0100, Fabian Groffen wrote:

...
On 22-11-2007 22:16:16 +0100, Markus Gritsch wrote:

...
On Nov 22, 2007 9:52 PM, Niels Nes wrote:

...
The results seem indeed incorrect. Which would indicate that the insert allready didn't do the right job. So it atleast seems that the bug is in python monetdb interface not in jdbc. I'll have to look into it. Could you try the jdbc*jar from www.cwi.nl/~fabian/troep/, I hope its not giving the 'unexpected end of data' problem.

I tried http://homepages.cwi.nl/~fabian/troep/monetdb-1.6-jdbc.jar and get the same error. Logfiles are attached.

Same problem as before:

RD 1195766131798: inserting prompt TD 1195766131848: write final block: 53 bytes TX 1195766131848: sPREPARE INSERT INTO test (id, name) VALUES (0, '?'); RD 1195766131848: read final block: 25 bytes RX 1195766131848: !unexpected end of input

You don't even get to the UTF-8 part. THis is indeed a problem on the server side. Not sure what yet. Afcourse it strange to use prepare with out any parameters, ie '?' is not correct it should be ? I guess.

Niels

...

Niels

...
------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

--

Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Fabian Groffen

9:36 p.m.

On 22-11-2007 22:32:17 +0100, Niels Nes wrote:

...

...
...
...
I tried http://homepages.cwi.nl/~fabian/troep/monetdb-1.6-jdbc.jar and get the same error. Logfiles are attached.

Same problem as before:

RD 1195766131798: inserting prompt TD 1195766131848: write final block: 53 bytes TX 1195766131848: sPREPARE INSERT INTO test (id, name) VALUES (0, '?'); RD 1195766131848: read final block: 25 bytes RX 1195766131848: !unexpected end of input

You don't even get to the UTF-8 part. THis is indeed a problem on the server side. Not sure what yet. Afcourse it strange to use prepare with out any parameters, ie '?' is not correct it should be ? I guess.

Or we're looking at a rendering issue, where the ? is not a ? but the o-umlaut thing.

Fabian Groffen

9:42 p.m.

On 22-11-2007 22:36:22 +0100, Fabian Groffen wrote:

...

...
...
...
You don't even get to the UTF-8 part. THis is indeed a problem on the server side. Not sure what yet. Afcourse it strange to use prepare with out any parameters, ie '?' is not correct it should be ? I guess.

Or we're looking at a rendering issue, where the ? is not a ? but the o-umlaut thing.

I suspected libiconv inserting the ? for a non-printable char, but saving the file actually results in the same, unless also there iconv is being used inbetween... Not sure... my mutt may screw it up here when it decodes the attachment... Maybe someone else can try and see what char is in there in the original attachment?

Markus Gritsch

9:50 p.m.

On Nov 22, 2007 10:42 PM, Fabian Groffen wrote:

...

I suspected libiconv inserting the ? for a non-printable char, but saving the file actually results in the same, unless also there iconv is being used inbetween...

Not sure... my mutt may screw it up here when it decodes the attachment... Maybe someone else can try and see what char is in there in the original attachment?

I looked into the original file on my disk and it contains a question mark. Maybe the debug-logging does not use UTF-8 for writing the logfile? Markus

Fabian Groffen

9:54 p.m.

On 22-11-2007 22:50:36 +0100, Markus Gritsch wrote:

...

On Nov 22, 2007 10:42 PM, Fabian Groffen wrote:

...
I suspected libiconv inserting the ? for a non-printable char, but saving the file actually results in the same, unless also there iconv is being used inbetween...

Not sure... my mutt may screw it up here when it decodes the attachment... Maybe someone else can try and see what char is in there in the original attachment?

I looked into the original file on my disk and it contains a question mark. Maybe the debug-logging does not use UTF-8 for writing the logfile?

The logging writes whatever it has as bytes to disk. If it has an invalid char in UTF-8 representation it will dump the invalid bytes. Can you try the same query but with a normal character and check if it works?

Markus Gritsch

9:58 p.m.

On Nov 22, 2007 10:54 PM, Fabian Groffen wrote:

...

Can you try the same query but with a normal character and check if it works?

Of course I have already tried this :) It works fine, as you can see from the attached logfiles. Markus

Fabian Groffen

10:07 p.m.

On 22-11-2007 22:58:43 +0100, Markus Gritsch wrote:

...

On Nov 22, 2007 10:54 PM, Fabian Groffen wrote:

...
Can you try the same query but with a normal character and check if it works?

Of course I have already tried this :) It works fine, as you can see from the attached logfiles.

TX 1195768551757: sPREPARE UPDATE sys.test SET name = 'ok' WHERE id = 0; Ok, so that means the ? is actually the char that you try to insert, and there is a problem somewhere early in the stack to map this char into something useful, whereas it appears to be sent to the server, which just thinks it sees an EOF/invalid escape sequence. The fact that you have a ? on your disk (are you sure about that? How do you open up the file?) means that Java already couldn't map the character to something correct. It is a bug for us to make sure the server responds correctly. It is a bug for an yet to determine party why the LATIN-1 char you insert (most probably, as you're in the windows world) isn't mapped correctly into an utf-8 char. It seems like Java is to blame here, but on the other hand... we explicitly request UTF-8 streams, and Java internally is UTF-16, so it is hard to believe that Sun didn't test this on Windows...

Markus Gritsch

23 Nov 23 Nov

9:14 a.m.

On Nov 22, 2007 11:07 PM, Fabian Groffen wrote:

...

Ok, so that means the ? is actually the char that you try to insert, and there is a problem somewhere early in the stack to map this char into something useful, whereas it appears to be sent to the server, which just thinks it sees an EOF/invalid escape sequence.

The fact that you have a ? on your disk (are you sure about that? How do you open up the file?) means that Java already couldn't map the character to something correct.

Because xxd says it: 0000d90: 3035 3130 3635 3331 3a20 7350 5245 5041 05106531: sPREPA 0000da0: 5245 2049 4e53 4552 5420 494e 544f 2074 RE INSERT INTO t 0000db0: 6573 7420 5641 4c55 4553 2028 302c 2027 est VALUES (0, ' 0000dc0: 3f27 293b 0a52 4420 3131 3935 3830 3531 ?');.RD 11958051

...

It is a bug for us to make sure the server responds correctly.

It is a bug for an yet to determine party why the LATIN-1 char you insert (most probably, as you're in the windows world) isn't mapped

Windows does not exactly use latin-1, but windows-1252. However, I don't think the text I like to insert is using this encoding, since I use DbVisualizer, which has no problem inserting the Umlaut characters or an Euro sign into a MySQL database, which is configured to use UTF-8 as default encoding and into an SQLite database (using http://www.zentus.com/sqlitejdbc/). So to me it seems to be an issue of the MonetDB JDBC driver. The attached Emacs_MonetDB.png screenshot shows a correct behavior: A UTF-8 encoded sting is INSERTed, because the Python file is in UTF-8 encoding. The SELECT yields a string which is also encoded using UTF-8. This string is then decoded to a Unicode type by specifying, that the string is in 'utf-8' and the resulting Unicode type is encoded to 'windows-1252' so that is can easily be displayed in the Emacs *compilation* buffer. The attached DbVisualizer_MonetDB.png screenshot indicates, that something is not right with the MonetDB JDBC driver. For comparison the attachments Emacs_MySQL.png and DbVisualizer_MySQL.png show the same example using MySQL, and the attachments Emacs_SQLite.png and DbVisualizer_SQLite.png show the example using SQLite. No problem there. Markus

Markus Gritsch

4 Dec 4 Dec

1:36 p.m.

On 23/11/2007, Markus Gritsch wrote:

...

The attached DbVisualizer_MonetDB.png screenshot indicates, that something is not right with the MonetDB JDBC driver.

just for the record: http://homepages.cwi.nl/~fabian/troep/jdbcclient.jar from the 2007-12-04 fixes this. Thank you, Markus

Fabian Groffen

1:44 p.m.

On 04-12-2007 14:36:08 +0100, Markus Gritsch wrote:

...

On 23/11/2007, Markus Gritsch wrote:

...
The attached DbVisualizer_MonetDB.png screenshot indicates, that something is not right with the MonetDB JDBC driver.

just for the record: http://homepages.cwi.nl/~fabian/troep/jdbcclient.jar from the 2007-12-04 fixes this.

Thanks.

Niels Nes

22 Nov 22 Nov

9:50 p.m.

...

On 22-11-2007 22:36:22 +0100, Fabian Groffen wrote:

...
...
...
...
You don't even get to the UTF-8 part. THis is indeed a problem on the server side. Not sure what yet. Afcourse it strange to use prepare with out any parameters, ie '?' is not correct it should be ? I guess.

Or we're looking at a rendering issue, where the ? is not a ? but the o-umlaut thing.

I suspected libiconv inserting the ? for a non-printable char, but saving the file actually results in the same, unless also there iconv is being used inbetween...

Not sure... my mutt may screw it up here when it decodes the attachment... Maybe someone else can try and see what char is in there in the original attachment? On linux this isn't a problem. I could very well be that the mclient (mapi

On Thu, Nov 22, 2007 at 10:42:19PM +0100, Fabian Groffen wrote: lib) doesn't have iconv support on windows. But that still doesn't explain the jdbc / prepare problems which I also cannot reproduce sofar. Niels

...

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Niels Nes

9:53 p.m.

On Thu, Nov 22, 2007 at 10:50:57PM +0100, Niels Nes wrote:

...

...
On 22-11-2007 22:36:22 +0100, Fabian Groffen wrote:

...
...
...
...
You don't even get to the UTF-8 part. THis is indeed a problem on the server side. Not sure what yet. Afcourse it strange to use prepare with out any parameters, ie '?' is not correct it should be ? I guess.

Or we're looking at a rendering issue, where the ? is not a ? but the o-umlaut thing.

I suspected libiconv inserting the ? for a non-printable char, but saving the file actually results in the same, unless also there iconv is being used inbetween...

Not sure... my mutt may screw it up here when it decodes the attachment... Maybe someone else can try and see what char is in there in the original attachment? On linux this isn't a problem. I could very well be that the mclient (mapi

On Thu, Nov 22, 2007 at 10:42:19PM +0100, Fabian Groffen wrote: lib) doesn't have iconv support on windows. But that still doesn't explain the jdbc / prepare problems which I also cannot reproduce sofar.

I cannot reproduce the problem sofar. Niels

...

Niels

...
------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

--

Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

6233

Age (days ago)

6245

Last active (days ago)

List overview

Download

29 comments

3 participants

participants (3)

Fabian Groffen
Markus Gritsch
Niels Nes