[Monetdb-developers] Parallel insertions in monetdb table - some records do not insert
Hello Everyone! Brief Overview of the issue 1. Sample Data that erlang program tries to insert in monetdb {data, [{"a","string"}, {"b","string"}], %% a,b are column names [ [{"a","a1"}, {"b","b1"}], %% key, value pair of the data - {columnname, data} [{"a","a2"}, {"b","b2"}], [{"a","a3"}, {"b","b3"}], [{"a","a4"}, {"b","b4"}], [{"a","a5"}, {"b","b5"}], [{"a","a6"}, {"b","b6"}], [{"a","a7"}, {"b","b7"}], [{"a","a8"}, {"b","b8"}], [{"a","a9"}, {"b","b9"}], [{"a","a10"}, {"b","b10"}], [{"a","a11"}, {"b","b11"}] ], {"no_of_records","12"} } 2. no of threads/processes that would do parallel inserts would be determined by the thread size i.e how many of records the thread would insert, if say 5 records then 3 threads, if 3 records then 4 threads. 3. in order to connect to monetdb from erlang, the monetdb ODBC driver is specified in ODBCConfig utility of ubuntu, driver file is /usr/lib/libMonetODBC.so 4. threads are successfully created and the ODBC module quits successfully after inserting in the erlang program, but when we look in the monetdb table, some of the rows get missed, for example row from 1 to 11 get inserted without row 3 or 4 getting inserted. could this be a problem with the ODBC driver?, am I missing any configuration?, any help would be deeply appreciated. Thank you, Ram.
Hello, This 'issue' is raised several times,e.g. http://www.mail-archive.com/monetdb-developers@lists.sourceforge.net/msg0221... Google: monetdb optimistic concurrency for more references about how optimistic concurrency control can not support your intended OLTP behavior. You probably have to serialize the updates through a master or use independent servers. regards, Martin kss ram wrote:
Hello Everyone!
Brief Overview of the issue
1. Sample Data that erlang program tries to insert in monetdb
{data, [{"a","string"}, {"b","string"}], %% a,b are column names [ [{"a","a1"}, {"b","b1"}], %% key, value pair of the data - {columnname, data} [{"a","a2"}, {"b","b2"}], [{"a","a3"}, {"b","b3"}], [{"a","a4"}, {"b","b4"}], [{"a","a5"}, {"b","b5"}], [{"a","a6"}, {"b","b6"}], [{"a","a7"}, {"b","b7"}], [{"a","a8"}, {"b","b8"}], [{"a","a9"}, {"b","b9"}], [{"a","a10"}, {"b","b10"}], [{"a","a11"}, {"b","b11"}] ], {"no_of_records","12"} }
2. no of threads/processes that would do parallel inserts would be determined by the thread size i.e how many of records the thread would insert, if say 5 records then 3 threads, if 3 records then 4 threads.
3. in order to connect to monetdb from erlang, the monetdb ODBC driver is specified in ODBCConfig utility of ubuntu, driver file is /usr/lib/libMonetODBC.so
4. threads are successfully created and the ODBC module quits successfully after inserting in the erlang program, but when we look in the monetdb table, some of the rows get missed, for example row from 1 to 11 get inserted without row 3 or 4 getting inserted.
could this be a problem with the ODBC driver?, am I missing any configuration?, any help would be deeply appreciated.
Thank you,
Ram.
------------------------------------------------------------------------
------------------------------------------------------------------------------ ThinkGeek and WIRED's GeekDad team up for the Ultimate GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the lucky parental unit. See the prize list and enter to win: http://p.sf.net/sfu/thinkgeek-promo
------------------------------------------------------------------------
_______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Mon, Jun 14, 2010 at 8:46 PM, Martin Kersten
for more references about how optimistic concurrency control can not support your intended OLTP behavior. You probably have to serialize the updates through a master or use independent servers.
Martin, I am confused by this answer. I understand MonetDB is not optimized for such workloads, and optimistic concurrency will not perform well. But if all insert transactions have committed successfully (this is how I read Ram's report), then no data should be lost, no matter what. Jens -- Jens Teubner, jens.teubner@inf.ethz.ch
participants (3)
-
Jens Teubner -
kss ram -
Martin Kersten