[MonetDB-users] transaction concurrency
Hi, I'm using monetdb's SQL frontend through the python bindings (with minor patches to make it behave more like the DB-API specifies, and a patch to print error information because MonetSQLdb raises 'RuntimeError, None' when the database returns an error). I'm now running into yet another problem: I have an import process that imports large sets of records in a single transaction per dataset. This works fine (after fixing some problems in my code). However, when I perform other transactions during an "import" transactions the import tranaction fails hard even if those transactions only read from the database and only touch tables that aren't touched by the import script. I'm not at my development machine at the moment, but I'll post an example script and the exact error I'm getting when I'm back. Ronald P.S. This is using a recent build of a daily snapshot from the 2.14/4.14 branch.
Hi Ronald, On 01-12-2006 05:59:24 -0800, Ronald Oussoren wrote:
Hi,
I'm using monetdb's SQL frontend through the python bindings (with minor patches to make it behave more like the DB-API specifies, and a patch to print error information because MonetSQLdb raises 'RuntimeError, None' when the database returns an error).
Can you contribute this patch? If it's nice we might put it in before the release, because it sounds like a fix.
I'm now running into yet another problem: I have an import process that imports large sets of records in a single transaction per dataset. This works fine (after fixing some problems in my code). However, when I perform other transactions during an "import" transactions the import tranaction fails hard even if those transactions only read from the database and only touch tables that aren't touched by the import script.
This sounds like a problem with our FULLY_SERIALIZABLE transaction mode. Whether you like it or not, you probably (indirectly) touch tables that are in common with the running import, hence the last transaction to finish gets aborted.
I'm not at my development machine at the moment, but I'll post an example script and the exact error I'm getting when I'm back.
On Friday, December 01, 2006, at 03:04PM, "Fabian Groffen"
Hi Ronald,
On 01-12-2006 05:59:24 -0800, Ronald Oussoren wrote:
Hi,
I'm using monetdb's SQL frontend through the python bindings (with minor patches to make it behave more like the DB-API specifies, and a patch to print error information because MonetSQLdb raises 'RuntimeError, None' when the database returns an error).
Can you contribute this patch? If it's nice we might put it in before the release, because it sounds like a fix.
The DB-API changes should be easy enough to convert to a patch, I'm currently hotfixing to make it easier to convert to a new monet snapshot. The error-printing patch is a crude hack, I've changed the wrapper of map_query to print error information to stderr (basically using code ripped out of the C program MapiClient). I'll post the hotfix-code and patch once I get back to my developoment system (either tonight or over the weekend). With some luck I'll post a real patch soon after, because the build system for the python wrappers is currently broken (it installs the python extension in a subdirectory of site-packages and installs it as a versioned shared library). Ronald
On 1 Dec 2006, at 3:18 PM, Ronald Oussoren wrote:
On Friday, December 01, 2006, at 03:04PM, "Fabian Groffen"
wrote: Hi Ronald,
On 01-12-2006 05:59:24 -0800, Ronald Oussoren wrote:
Hi,
I'm using monetdb's SQL frontend through the python bindings (with minor patches to make it behave more like the DB-API specifies, and a patch to print error information because MonetSQLdb raises 'RuntimeError, None' when the database returns an error).
Can you contribute this patch? If it's nice we might put it in before the release, because it sounds like a fix.
The DB-API changes should be easy enough to convert to a patch, I'm currently hotfixing to make it easier to convert to a new monet snapshot.
The error-printing patch is a crude hack, I've changed the wrapper of map_query to print error information to stderr (basically using code ripped out of the C program MapiClient).
I've included a patch that does both the DB-API changes and the patch to _wrap_mapi_query. The latter patch is obviously incorrect (I've changed a generated file and the code now prints data to stderr), but I haven't had time yet to research a proper fix (that is, extract the error information from Python and raise an appropriate error there).
I'll post the hotfix-code and patch once I get back to my developoment system (either tonight or over the weekend). With some luck I'll post a real patch soon after, because the build system for the python wrappers is currently broken (it installs the python extension in a subdirectory of site-packages and installs it as a versioned shared library).
I haven't looked at this yet. The major problem here is that the python wrappers get build using automake instead of distutils.
Ronald
P.S. I haven't used the patched version of monet yet, I've reverted to postgres for now because the concurrency problem is a real showstopper for me.
participants (2)
-
Fabian Groffen
-
Ronald Oussoren