[MonetDB-users] Serious concurrency issues via python api

Matthew Jones

2 Sep 2009 2 Sep '09

7:48 p.m.

I have multiple processes reading and writing to the same table and I've noticed really persistent concurrency problems: File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 208, in execute self.__store_result(block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 520, in __store_result self.__exception_handler(InterfaceError, "Unknown state, %s" % block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 574, in __exception_handler raise exception_class(message) InterfaceError: Unknown state, &2 65541 -1 !ERROR: COMMIT: transaction is aborted because of concurency conflicts, will ROLLBACK instead I'm not sure what the problem is, but it exists in such a way that all operations that load or update information in MonetDB have to be put into a loop to continuously retry. I'm using autocommit, but I would expect MonetDB to handle concurrency issues in a decent way, like maybe blocking a query until the other query finished. I'm not clobbering data, most of these are just bulk load operations that insert new rows. Is there any way to address this issue without having to putting retry loops around all of the code I'm writing to write to the database? 73, Matthew W. Jones (KI4ZIB) http://matburt.net

Attachments:

attachment.html (text/html — 1.4 KB)

Show replies by date

Martin Kersten

2 Sep 2 Sep

9:02 p.m.

Dear Matthew, A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system. Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles. There are several application scenarios to deal with this issue, but all depend on the semantics required. regards, Martin Matthew Jones wrote:

...

I have multiple processes reading and writing to the same table and I've noticed really persistent concurrency problems:

File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 208, in execute self.__store_result(block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 520, in __store_result self.__exception_handler(InterfaceError, "Unknown state, %s" % block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 574, in __exception_handler raise exception_class(message) InterfaceError: Unknown state, &2 65541 -1 !ERROR: COMMIT: transaction is aborted because of concurency conflicts, will ROLLBACK instead

I'm not sure what the problem is, but it exists in such a way that all operations that load or update information in MonetDB have to be put into a loop to continuously retry.

I'm using autocommit, but I would expect MonetDB to handle concurrency issues in a decent way, like maybe blocking a query until the other query finished. I'm not clobbering data, most of these are just bulk load operations that insert new rows.

Is there any way to address this issue without having to putting retry loops around all of the code I'm writing to write to the database?

73, Matthew W. Jones (KI4ZIB) http://matburt.net

------------------------------------------------------------------------

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Matthew Jones

9:33 p.m.

There's no relational data in these tables, I guess I just need something internal to manage concurrency... like blocking table locks so I don't have to wait for a failure in order to manage concurrency on my own. I noticed in one of my operations which was a single update to a table that no other processes were using that it gave me that error, informed me of the rollback, but the changes still took effect.... which is alarming since I will now need to: 1) Try the operation 2) If it fails, check to see if the change actually took effect 3) if the changes didn't take effect re-try the operation This is disconcerting. 73, Matthew W. Jones (KI4ZIB) http://matburt.net On Wed, Sep 2, 2009 at 3:02 PM, Martin Kersten wrote:

...

Dear Matthew,

A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system.

Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles.

There are several application scenarios to deal with this issue, but all depend on the semantics required.

regards, Martin

Matthew Jones wrote:

...
I have multiple processes reading and writing to the same table and I've noticed really persistent concurrency problems:

File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 208, in execute self.__store_result(block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 520, in __store_result self.__exception_handler(InterfaceError, "Unknown state, %s" % block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 574, in __exception_handler raise exception_class(message) InterfaceError: Unknown state, &2 65541 -1 !ERROR: COMMIT: transaction is aborted because of concurency conflicts, will ROLLBACK instead

I'm not sure what the problem is, but it exists in such a way that all operations that load or update information in MonetDB have to be put into a loop to continuously retry.

I'm using autocommit, but I would expect MonetDB to handle concurrency issues in a decent way, like maybe blocking a query until the other query finished. I'm not clobbering data, most of these are just bulk load operations that insert new rows.

Is there any way to address this issue without having to putting retry loops around all of the code I'm writing to write to the database?

73, Matthew W. Jones (KI4ZIB) http://matburt.net

------------------------------------------------------------------------

------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Matthew Jones

9:38 p.m.

Curious, since I experienced this on a table that no one else was reading or writing to at the time. How does MonetDB determine that a concurrency violation has occurred? Does it look at table or record level read/write timestamps? 73, Matthew W. Jones (KI4ZIB) http://matburt.net On Wed, Sep 2, 2009 at 3:33 PM, Matthew Jones wrote:

...

There's no relational data in these tables, I guess I just need something internal to manage concurrency... like blocking table locks so I don't have to wait for a failure in order to manage concurrency on my own.

I noticed in one of my operations which was a single update to a table that no other processes were using that it gave me that error, informed me of the rollback, but the changes still took effect.... which is alarming since I will now need to:

1) Try the operation 2) If it fails, check to see if the change actually took effect 3) if the changes didn't take effect re-try the operation

This is disconcerting.

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:02 PM, Martin Kersten wrote:

...
Dear Matthew,

A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system.

Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles.

There are several application scenarios to deal with this issue, but all depend on the semantics required.

regards, Martin

Matthew Jones wrote:

...
I have multiple processes reading and writing to the same table and I've noticed really persistent concurrency problems:

File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 208, in execute self.__store_result(block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 520, in __store_result self.__exception_handler(InterfaceError, "Unknown state, %s" % block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 574, in __exception_handler raise exception_class(message) InterfaceError: Unknown state, &2 65541 -1 !ERROR: COMMIT: transaction is aborted because of concurency conflicts, will ROLLBACK instead

I'm not sure what the problem is, but it exists in such a way that all operations that load or update information in MonetDB have to be put into a loop to continuously retry.

I'm using autocommit, but I would expect MonetDB to handle concurrency issues in a decent way, like maybe blocking a query until the other query finished. I'm not clobbering data, most of these are just bulk load operations that insert new rows.

Is there any way to address this issue without having to putting retry loops around all of the code I'm writing to write to the database?

73, Matthew W. Jones (KI4ZIB) http://matburt.net

------------------------------------------------------------------------

------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Martin Kersten

9:41 p.m.

Matthew Jones wrote:

...

There's no relational data in these tables, I guess I just need something internal to manage concurrency... like blocking table locks so I don't have to wait for a failure in order to manage concurrency on my own. you either have to serialize the updates/access at your client interface, or hook into the MAL kernel to get access to the general lock/semaphore module.

...

I noticed in one of my operations which was a single update to a table that no other processes were using that it gave me that error, informed

Optimistic concurrency control should also ensure that non of the concurrent transactions reads stale data. This becomes tricky if you have multiple user sessions with both autocommit and compound transactions, because then the timespans to trigger a transaction failure are larger.

...

me of the rollback, but the changes still took effect.... which is alarming since I will now need to:

1) Try the operation 2) If it fails, check to see if the change actually took effect 3) if the changes didn't take effect re-try the operation

This is disconcerting.

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:02 PM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:

Dear Matthew,

A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system.

Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles.

There are several application scenarios to deal with this issue, but all depend on the semantics required.

regards, Martin

Matthew Jones wrote: > I have multiple processes reading and writing to the same table and I've > noticed really persistent concurrency problems: > > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 208, in execute > self.__store_result(block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 520, in __store_result > self.__exception_handler(InterfaceError, "Unknown state, %s" % block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 574, in __exception_handler > raise exception_class(message) > InterfaceError: Unknown state, &2 65541 -1 > !ERROR: COMMIT: transaction is aborted because of concurency conflicts, > will ROLLBACK instead > > I'm not sure what the problem is, but it exists in such a way that all > operations that load or update information in MonetDB have to be put > into a loop to continuously retry. > > I'm using autocommit, but I would expect MonetDB to handle concurrency > issues in a decent way, like maybe blocking a query until the other > query finished. I'm not clobbering data, most of these are just bulk > load operations that insert new rows. > > Is there any way to address this issue without having to putting retry > loops around all of the code I'm writing to write to the database? > > 73, > Matthew W. Jones (KI4ZIB) > http://matburt.net > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > > ------------------------------------------------------------------------ > > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Matthew Jones

10:03 p.m.

Are you saying that if someone reads data from a table while another transaction is bulk-loading data that that will cause a concurrency conflict and a forced rollback? I'm looking at the behavior of one of my long running batch loads and that looks to be the case... if so, that seems pretty extreme. I could understand protecting from simultaneous modification but it seems wrong to cause a write to fail for any read on a table. 73, Matthew W. Jones (KI4ZIB) http://matburt.net On Wed, Sep 2, 2009 at 3:41 PM, Martin Kersten wrote:

...

Matthew Jones wrote:

...
There's no relational data in these tables, I guess I just need something internal to manage concurrency... like blocking table locks so I don't have to wait for a failure in order to manage concurrency on my own. you either have to serialize the updates/access at your client interface, or hook into the MAL kernel to get access to the general lock/semaphore module.

...
I noticed in one of my operations which was a single update to a table that no other processes were using that it gave me that error, informed

Optimistic concurrency control should also ensure that non of the concurrent transactions reads stale data. This becomes tricky if you have multiple user sessions with both autocommit and compound transactions, because then the timespans to trigger a transaction failure are larger.

...
me of the rollback, but the changes still took effect.... which is alarming since I will now need to:

1) Try the operation 2) If it fails, check to see if the change actually took effect 3) if the changes didn't take effect re-try the operation

This is disconcerting.

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:02 PM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:

Dear Matthew,

A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system.

Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles.

There are several application scenarios to deal with this issue, but all depend on the semantics required.

regards, Martin

Matthew Jones wrote: > I have multiple processes reading and writing to the same table and I've > noticed really persistent concurrency problems: > > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 208, in execute > self.__store_result(block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 520, in __store_result > self.__exception_handler(InterfaceError, "Unknown state, %s" % block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 574, in __exception_handler > raise exception_class(message) > InterfaceError: Unknown state, &2 65541 -1 > !ERROR: COMMIT: transaction is aborted because of concurency conflicts, > will ROLLBACK instead > > I'm not sure what the problem is, but it exists in such a way that all > operations that load or update information in MonetDB have to be put > into a loop to continuously retry. > > I'm using autocommit, but I would expect MonetDB to handle concurrency > issues in a decent way, like maybe blocking a query until the other > query finished. I'm not clobbering data, most of these are just bulk > load operations that insert new rows. > > Is there any way to address this issue without having to putting retry > loops around all of the code I'm writing to write to the database? > > 73, > Matthew W. Jones (KI4ZIB) > http://matburt.net > > >

------------------------------------------------------------------------

...
> >

------------------------------------------------------------------------------

...
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new

with

...
> Crystal Reports now. http://p.sf.net/sfu/bobj-july > > >

------------------------------------------------------------------------

...
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------

------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Stefan Manegold

10:16 p.m.

On Wed, Sep 02, 2009 at 04:03:17PM -0400, Matthew Jones wrote:

...

Are you saying that if someone reads data from a table while another transaction is bulk-loading data that that will cause a concurrency conflict and a forced rollback?

I'm looking at the behavior of one of my long running batch loads and that looks to be the case... if so, that seems pretty extreme. I could understand protecting from simultaneous modification but it seems wrong to cause a write to fail for any read on a table.

writes conflict with reads, and reads with writes, just as writes conflict with writes; only multiple concurrent reads do not conlict with each other. Stefan

...

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:41 PM, Martin Kersten wrote:

...
Matthew Jones wrote:

...
There's no relational data in these tables, I guess I just need something internal to manage concurrency... like blocking table locks so I don't have to wait for a failure in order to manage concurrency on my own. you either have to serialize the updates/access at your client interface, or hook into the MAL kernel to get access to the general lock/semaphore module.

...
I noticed in one of my operations which was a single update to a table that no other processes were using that it gave me that error, informed

Optimistic concurrency control should also ensure that non of the concurrent transactions reads stale data. This becomes tricky if you have multiple user sessions with both autocommit and compound transactions, because then the timespans to trigger a transaction failure are larger.

...
me of the rollback, but the changes still took effect.... which is alarming since I will now need to:

1) Try the operation 2) If it fails, check to see if the change actually took effect 3) if the changes didn't take effect re-try the operation

This is disconcerting.

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:02 PM, Martin Kersten mailto:Martin.Kersten@cwi.nl> wrote:

Dear Matthew,

A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system.

Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles.

There are several application scenarios to deal with this issue, but all depend on the semantics required.

regards, Martin

Matthew Jones wrote: > I have multiple processes reading and writing to the same table and I've > noticed really persistent concurrency problems: > > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 208, in execute > self.__store_result(block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 520, in __store_result > self.__exception_handler(InterfaceError, "Unknown state, %s" % block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 574, in __exception_handler > raise exception_class(message) > InterfaceError: Unknown state, &2 65541 -1 > !ERROR: COMMIT: transaction is aborted because of concurency conflicts, > will ROLLBACK instead > > I'm not sure what the problem is, but it exists in such a way that all > operations that load or update information in MonetDB have to be put > into a loop to continuously retry. > > I'm using autocommit, but I would expect MonetDB to handle concurrency > issues in a decent way, like maybe blocking a query until the other > query finished. I'm not clobbering data, most of these are just bulk > load operations that insert new rows. > > Is there any way to address this issue without having to putting retry > loops around all of the code I'm writing to write to the database? > > 73, > Matthew W. Jones (KI4ZIB) > http://matburt.net > > >

------------------------------------------------------------------------

...
> >

------------------------------------------------------------------------------

...
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new

with

...
> Crystal Reports now. http://p.sf.net/sfu/bobj-july > > >

------------------------------------------------------------------------

...
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------

------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Matthew Jones

10:46 p.m.

If the write is Isolated from a read then should a read really be allowed to cause a concurrency failure on the write? The read only needs to be guaranteed to not see the table in a partial/incomplete state and the write's changes are isolated until it commits. A read conflicting with a write seems like something altogether different than a write conflicting with another write. Am I missing something here? 73, Matthew W. Jones (KI4ZIB) http://matburt.net On Wed, Sep 2, 2009 at 4:16 PM, Stefan Manegold wrote:

...

...
Are you saying that if someone reads data from a table while another transaction is bulk-loading data that that will cause a concurrency conflict and a forced rollback?

I'm looking at the behavior of one of my long running batch loads and

On Wed, Sep 02, 2009 at 04:03:17PM -0400, Matthew Jones wrote: that

...
looks to be the case... if so, that seems pretty extreme. I could understand protecting from simultaneous modification but it seems wrong to cause a write to fail for any read on a table.

writes conflict with reads, and reads with writes, just as writes conflict with writes; only multiple concurrent reads do not conlict with each other.

Stefan

...
73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:41 PM, Martin Kersten
...
Matthew Jones wrote:

...
There's no relational data in these tables, I guess I just need something internal to manage concurrency... like blocking table locks so I don't have to wait for a failure in order to manage concurrency on my own. you either have to serialize the updates/access at your client interface, or hook into the MAL kernel to get access to the general lock/semaphore module.

...
I noticed in one of my operations which was a single update to a

...
that no other processes were using that it gave me that error, informed Optimistic concurrency control should also ensure that non of the concurrent

table transactions reads stale data. This becomes tricky if you have multiple user sessions with both autocommit and compound transactions, because then the timespans to trigger a transaction failure are larger.

...
me of the rollback, but the changes still took effect.... which is alarming since I will now need to:

1) Try the operation 2) If it fails, check to see if the change actually took effect 3) if the changes didn't take effect re-try the operation

This is disconcerting.

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:02 PM, Martin Kersten < Martin.Kersten@cwi.nl mailto:Martin.Kersten@cwi.nl> wrote:

Dear Matthew,

A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system.

Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles.

There are several application scenarios to deal with this issue, but all depend on the semantics required.

regards, Martin

Matthew Jones wrote: > I have multiple processes reading and writing to the same table and I've > noticed really persistent concurrency problems: > > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 208, in execute > self.__store_result(block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 520, in __store_result > self.__exception_handler(InterfaceError, "Unknown state, %s" % block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 574, in __exception_handler > raise exception_class(message) > InterfaceError: Unknown state, &2 65541 -1 > !ERROR: COMMIT: transaction is aborted because of concurency conflicts, > will ROLLBACK instead > > I'm not sure what the problem is, but it exists in such a way that all > operations that load or update information in MonetDB have to be put > into a loop to continuously retry. > > I'm using autocommit, but I would expect MonetDB to handle concurrency > issues in a decent way, like maybe blocking a query until the other > query finished. I'm not clobbering data, most of these are just bulk > load operations that insert new rows. > > Is there any way to address this issue without having to putting retry > loops around all of the code I'm writing to write to the database? > > 73, > Matthew W. Jones (KI4ZIB) > http://matburt.net > > >

...
...
...
> >

...
...
...
and focus on > what you do best, core application coding. Discover what's new

with

...
> Crystal Reports now. http://p.sf.net/sfu/bobj-july > > >

...
...
...
> Let Crystal Reports handle the reporting - Free Crystal

Reports

...
2008 30-Day > trial. Simplify your report design, integration and deployment

...
...
...
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
...
Let Crystal Reports handle the reporting - Free Crystal Reports

2008

...
30-Day trial. Simplify your report design, integration and deployment -

and

...
focus on what you do best, core application coding. Discover what's new

with

...
Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
...

...
...
...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

...
...
...
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/http://www.cwi.nl/%7Emanegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Peter Boncz

11:06 p.m.

Hi, Snapshot isolation in principle only needs to abort write-write conflicts. However, MonetDB aims at serializabilty. Maybe we should consider offering SI as well? In addition to that, my advise has been been for some time to consider row-wise write-write conflict checking rather than full-table. Peter

...

If the write is Isolated from a read then should a read really be allowed to cause a concurrency failure on the write? The read only needs to be guaranteed to not see the table in a partial/incomplete state and the write's changes are isolated until it commits. A read conflicting with a write seems like something altogether different than a write conflicting with another write.

Am I missing something here?

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 4:16 PM, Stefan Manegold wrote:

...
...
Are you saying that if someone reads data from a table while another transaction is bulk-loading data that that will cause a concurrency conflict and a forced rollback?

I'm looking at the behavior of one of my long running batch loads and

On Wed, Sep 02, 2009 at 04:03:17PM -0400, Matthew Jones wrote: that

...
looks to be the case... if so, that seems pretty extreme. I could understand protecting from simultaneous modification but it seems wrong to cause a write to fail for any read on a table.

writes conflict with reads, and reads with writes, just as writes conflict with writes; only multiple concurrent reads do not conlict with each other.

Stefan

...
73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:41 PM, Martin Kersten
...
Matthew Jones wrote:

...
There's no relational data in these tables, I guess I just need something internal to manage concurrency... like blocking table locks so I don't have to wait for a failure in order to manage concurrency on my own. you either have to serialize the updates/access at your client interface, or hook into the MAL kernel to get access to the general lock/semaphore module.

...
I noticed in one of my operations which was a single update to a

...
that no other processes were using that it gave me that error, informed Optimistic concurrency control should also ensure that non of the concurrent

table transactions reads stale data. This becomes tricky if you have multiple user sessions with both autocommit and compound transactions, because then the timespans to trigger a transaction failure are larger.

...
me of the rollback, but the changes still took effect.... which is alarming since I will now need to:

1) Try the operation 2) If it fails, check to see if the change actually took effect 3) if the changes didn't take effect re-try the operation

This is disconcerting.

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:02 PM, Martin Kersten < Martin.Kersten@cwi.nl mailto:Martin.Kersten@cwi.nl> wrote:

Dear Matthew,

A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system.

Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles.

There are several application scenarios to deal with this issue, but all depend on the semantics required.

regards, Martin

Matthew Jones wrote: > I have multiple processes reading and writing to the same table and I've > noticed really persistent concurrency problems: > > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 208, in execute > self.__store_result(block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 520, in __store_result > self.__exception_handler(InterfaceError, "Unknown state, %s" % block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 574, in __exception_handler > raise exception_class(message) > InterfaceError: Unknown state, &2 65541 -1 > !ERROR: COMMIT: transaction is aborted because of concurency conflicts, > will ROLLBACK instead > > I'm not sure what the problem is, but it exists in such a way that all > operations that load or update information in MonetDB have to be put > into a loop to continuously retry. > > I'm using autocommit, but I would expect MonetDB to handle concurrency > issues in a decent way, like maybe blocking a query until the other > query finished. I'm not clobbering data, most of these are just bulk > load operations that insert new rows. > > Is there any way to address this issue without having to putting retry > loops around all of the code I'm writing to write to the database? > > 73, > Matthew W. Jones (KI4ZIB) > http://matburt.net > > >

...
...
...
> >

...
...
...
and focus on > what you do best, core application coding. Discover what's new

with

...
> Crystal Reports now. http://p.sf.net/sfu/bobj-july > > >

...
...
...
> Let Crystal Reports handle the reporting - Free Crystal

Reports

...
2008 30-Day > trial. Simplify your report design, integration and deployment

...
...
...
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
...
Let Crystal Reports handle the reporting - Free Crystal Reports

2008

...
30-Day trial. Simplify your report design, integration and deployment -

and

...
focus on what you do best, core application coding. Discover what's new

with

...
Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
...

...
...
...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

...
...
...
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/http://www.cwi.nl/%7Emanegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Niels Nes

3 Sep 3 Sep

8:36 a.m.

On Wed, Sep 02, 2009 at 11:06:12PM +0200, Peter Boncz wrote:

...

Hi,

Snapshot isolation in principle only needs to abort write-write conflicts.

However, MonetDB aims at serializabilty. Maybe we should consider offering SI Yes MonetDB has serializability. Still a single only read transaction shouldn't affect a writing transaction.

Niels

...

as well?

In addition to that, my advise has been been for some time to consider row-wise write-write conflict checking rather than full-table.

Peter

...
If the write is Isolated from a read then should a read really be allowed to cause a concurrency failure on the write? The read only needs to be guaranteed to not see the table in a partial/incomplete state and the write's changes are isolated until it commits. A read conflicting with a write seems like something altogether different than a write conflicting with another write.

Am I missing something here?

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 4:16 PM, Stefan Manegold wrote:

...
...
Are you saying that if someone reads data from a table while another transaction is bulk-loading data that that will cause a concurrency conflict and a forced rollback?

I'm looking at the behavior of one of my long running batch loads and

On Wed, Sep 02, 2009 at 04:03:17PM -0400, Matthew Jones wrote: that

...
looks to be the case... if so, that seems pretty extreme. I could understand protecting from simultaneous modification but it seems wrong to cause a write to fail for any read on a table.

writes conflict with reads, and reads with writes, just as writes conflict with writes; only multiple concurrent reads do not conlict with each other.

Stefan

...
73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:41 PM, Martin Kersten
...
Matthew Jones wrote:

...
There's no relational data in these tables, I guess I just need something internal to manage concurrency... like blocking table locks so I don't have to wait for a failure in order to manage concurrency on my own. you either have to serialize the updates/access at your client interface, or hook into the MAL kernel to get access to the general lock/semaphore module.

...
I noticed in one of my operations which was a single update to a

...
that no other processes were using that it gave me that error, informed Optimistic concurrency control should also ensure that non of the concurrent

table transactions reads stale data. This becomes tricky if you have multiple user sessions with both autocommit and compound transactions, because then the timespans to trigger a transaction failure are larger.

...
me of the rollback, but the changes still took effect.... which is alarming since I will now need to:

1) Try the operation 2) If it fails, check to see if the change actually took effect 3) if the changes didn't take effect re-try the operation

This is disconcerting.

73, Matthew W. Jones (KI4ZIB) http://matburt.net

On Wed, Sep 2, 2009 at 3:02 PM, Martin Kersten < Martin.Kersten@cwi.nl mailto:Martin.Kersten@cwi.nl> wrote:

Dear Matthew,

A quick initial reaction. MonetDB uses an optimistic concurrency control scheme, which is geared at high-read, low-write applications. If you hit the system with multiple concurrent writes on the same table, then indeed to secure acid properties, transactions are bound to fail. MonetDB indeed conforms to the semantics of a transaction safe system.

Blocking queries is equivalent to relation level locking. This is not the preferred way to go for the kernel, because someone will immediately ask for more granularities and als relaxation of the ACID principles.

There are several application scenarios to deal with this issue, but all depend on the semantics required.

regards, Martin

Matthew Jones wrote: > I have multiple processes reading and writing to the same table and I've > noticed really persistent concurrency problems: > > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 208, in execute > self.__store_result(block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 520, in __store_result > self.__exception_handler(InterfaceError, "Unknown state, %s" % block) > File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line > 574, in __exception_handler > raise exception_class(message) > InterfaceError: Unknown state, &2 65541 -1 > !ERROR: COMMIT: transaction is aborted because of concurency conflicts, > will ROLLBACK instead > > I'm not sure what the problem is, but it exists in such a way that all > operations that load or update information in MonetDB have to be put > into a loop to continuously retry. > > I'm using autocommit, but I would expect MonetDB to handle concurrency > issues in a decent way, like maybe blocking a query until the other > query finished. I'm not clobbering data, most of these are just bulk > load operations that insert new rows. > > Is there any way to address this issue without having to putting retry > loops around all of the code I'm writing to write to the database? > > 73, > Matthew W. Jones (KI4ZIB) > http://matburt.net > > >

...
...
...
> >

...
...
...
and focus on > what you do best, core application coding. Discover what's new

with

...
> Crystal Reports now. http://p.sf.net/sfu/bobj-july > > >

...
...
...
> Let Crystal Reports handle the reporting - Free Crystal

Reports

...
2008 30-Day > trial. Simplify your report design, integration and deployment

...
...
...
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
...
Let Crystal Reports handle the reporting - Free Crystal Reports

2008

...
30-Day trial. Simplify your report design, integration and deployment -

and

...
focus on what you do best, core application coding. Discover what's new

with

...
Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net mailto:MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
...

...
...
...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

...
...
...
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

...
------------------------------------------------------------------------------

...
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/http://www.cwi.nl/%7Emanegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

Gijs Molenaar

9 Sep 9 Sep

7:06 p.m.

Matthew Jones wrote:

...

I have multiple processes reading and writing to the same table and I've noticed really persistent concurrency problems:

File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 208, in execute self.__store_result(block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 520, in __store_result self.__exception_handler(InterfaceError, "Unknown state, %s" % block) File "/usr/lib/python2.5/site-packages/monetdb/sql/cursors.py", line 574, in __exception_handler raise exception_class(message) InterfaceError: Unknown state, &2 65541 -1 !ERROR: COMMIT: transaction is aborted because of concurency conflicts, will ROLLBACK instead

hi all, This is a bug in the python API also, it doesn't handle the error properly. The desired behavior would be that a DatabaseError exception is raised. Will fix asap. - gijs

...

I'm not sure what the problem is, but it exists in such a way that all operations that load or update information in MonetDB have to be put into a loop to continuously retry.

I'm using autocommit, but I would expect MonetDB to handle concurrency issues in a decent way, like maybe blocking a query until the other query finished. I'm not clobbering data, most of these are just bulk load operations that insert new rows.

Is there any way to address this issue without having to putting retry loops around all of the code I'm writing to write to the database?

73, Matthew W. Jones (KI4ZIB) http://matburt.net ------------------------------------------------------------------------

------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ------------------------------------------------------------------------

_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Gijs Molenaar http://gijs.pythonic.nl

5547

Age (days ago)

5554

Last active (days ago)

List overview

Download

10 comments

6 participants

participants (6)

Gijs Molenaar
Martin Kersten
Matthew Jones
Niels Nes
Peter Boncz
Stefan Manegold