python3 client bug+fix
Hi, Working with MonetDB I encountered a problem with queries containing utf-8 characters. Some queries errored. I traced the error back to mapi.py module for python3. The problem was that encode() was applied after splicing the packet. With encode, any utf-8 characters are expanded to multiple octets. For a given block that exceeded MAX_PACKAGE_LENGTH this resulted in the last few octets falling off a package. The patch below remedies this, please apply. Regards, Bas Bloemsaat --- /srv/src/monetdb/MonetDB-11.23.13/clients/python3/monetdb/mapi.py 2016-10-07 11:00:50.000000000 +0200 +++ /home/bas/.pyenv/versions/ent_venv1/lib/python3.5/site-packages/monetdb/mapi.py 2016-12-23 12:02:53.431357999 +0100 @@ -275,8 +275,9 @@ class Connection(object): def _putblock_inet(self, block): pos = 0 last = 0 + block = block.encode() while not last: - data = block[pos:pos + MAX_PACKAGE_LENGTH].encode() + data = block[pos:pos + MAX_PACKAGE_LENGTH] length = len(data) if length < MAX_PACKAGE_LENGTH: last = 1
Hello Bas, Thank you very much for the patch. I was investigating the same issue for the past couple of days. The pymonetdb module is maintained by Gijs Molenaar at Github: https://github.com/gijzelaerr/pymonetdb. You can open a pull request there, or I can do it for you if you prefer. Let me know. Best regards, Panos. Bas Bloemsaat @ 2016-12-23 11:25 GMT:
Hi,
Working with MonetDB I encountered a problem with queries containing utf-8 characters. Some queries errored. I traced the error back to mapi.py module for python3.
The problem was that encode() was applied after splicing the packet. With encode, any utf-8 characters are expanded to multiple octets. For a given block that exceeded MAX_PACKAGE_LENGTH this resulted in the last few octets falling off a package. The patch below remedies this, please apply.
Regards, Bas Bloemsaat
--- /srv/src/monetdb/MonetDB-11.23.13/clients/python3/monetdb/mapi.py 2016-10-07 11:00:50.000000000 +0200 +++ /home/bas/.pyenv/versions/ent_venv1/lib/python3.5/site-packages/monetdb/mapi.py 2016-12-23 12:02:53.431357999 +0100 @@ -275,8 +275,9 @@ class Connection(object): def _putblock_inet(self, block): pos = 0 last = 0 + block = block.encode() while not last: - data = block[pos:pos + MAX_PACKAGE_LENGTH].encode() + data = block[pos:pos + MAX_PACKAGE_LENGTH] length = len(data) if length < MAX_PACKAGE_LENGTH: last = 1 _______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
Pull request:
https://github.com/gijzelaerr/pymonetdb/pull/16
Thanks for pointing me to the repo!
Regards,
Bas
On Fri, Dec 23, 2016 at 1:24 PM, Panagiotis Koutsourakis
Hello Bas,
Thank you very much for the patch. I was investigating the same issue for the past couple of days. The pymonetdb module is maintained by Gijs Molenaar at Github: https://github.com/gijzelaerr/pymonetdb. You can open a pull request there, or I can do it for you if you prefer. Let me know.
Best regards, Panos.
Bas Bloemsaat @ 2016-12-23 11:25 GMT:
Hi,
Working with MonetDB I encountered a problem with queries containing utf-8 characters. Some queries errored. I traced the error back to mapi.py module for python3.
The problem was that encode() was applied after splicing the packet. With encode, any utf-8 characters are expanded to multiple octets. For a given block that exceeded MAX_PACKAGE_LENGTH this resulted in the last few octets falling off a package. The patch below remedies this, please apply.
Regards, Bas Bloemsaat
--- /srv/src/monetdb/MonetDB-11.23.13/clients/python3/monetdb/mapi.py 2016-10-07 11:00:50.000000000 +0200 +++ /home/bas/.pyenv/versions/ent_venv1/lib/python3.5/site-packages/monetdb/mapi.py 2016-12-23 12:02:53.431357999 +0100 @@ -275,8 +275,9 @@ class Connection(object): def _putblock_inet(self, block): pos = 0 last = 0 + block = block.encode() while not last: - data = block[pos:pos + MAX_PACKAGE_LENGTH].encode() + data = block[pos:pos + MAX_PACKAGE_LENGTH] length = len(data) if length < MAX_PACKAGE_LENGTH: last = 1 _______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
participants (2)
-
Bas Bloemsaat
-
Panagiotis Koutsourakis