[MonetDB-users] Repeatedly writing to MonetDB

29 Jul 2012

      We are encountering a frequent error when writing to MonetDb repeatedly 
using the Java MAPI socket. We are running the April 2012 release.

The error is:
[7/29/12 12:10:26 PM] Cory Isaacson: 
000!TypeException:user.s0_2[2]:'calc.str' undefined in: _3:any := 
calc.str(_4:str)
22000!TypeException:user.s0_2[3]:'calc.str' undefined in: _5:any := 
calc.str(_6:str)
22000!TypeException:user.s0_2[4]:'calc.str' undefined in: _7:any := 
calc.str(_8:str)
22000!TypeException:user.s0_2[5]:'calc.str' undefined in: _9:any := 
calc.str(_10:str)
22000!TypeException:user.s0_2[6]:'calc.str' undefined in: _11:any := 
calc.str(_12:str)

The only solution is to stop the farm and restart the farm/database.
We have a single Java process writing to the database using MAPI, in 
batches of 1000 rows at a time (or less), using the COPY INTO syntax 
based on the example Java loader.We have a single Java process writing 
to the database using MAPI, in batches of 1000 rows at a time (or less), 
using the COPY INTO syntax based on the example Java loader.

Here are the important snippets of the Java code. You can see that we 
are opening/closing a connection for each batch, and we have only a 
single thread performing this function, no other processes are writing 
to the database. We do have other connections reading from the database 
(but in the most recent case we only had idle connections). This is 
running on CentOS 5.4, with Java 1.6. There is no swap space configured 
on this server (we read that this may be a requirement). The data 
volumes are very small at this point, only a few million rows.

            String query = "COPY " + (dataRowList.size() + 10) + " 
RECORDS INTO " + tableName + " FROM STDIN USING DELIMITERS '\\t','\\n' 
NULL AS '\\\\N';";
            if(TRACE) logger.trace("commit: query : "  + query);

             // the leading 's' is essential, since it is a protocol
             // marker that should not be omitted, likewise the
             // trailing semicolon
             out.write('s');
             out.write(query);
             out.newLine();

             // Write all data rows.
             for(String dataRow : dataRowList) {
                 if(TRACE) logger.trace("commit: Writing data row: " + 
dataRow + " dataRow ends in newline: " + dataRow.endsWith("\n"));
                 // The dataRow is \n terminated, so no need to output 
newLine to output stream.
                 out.write(dataRow);
             }

             out.writeLine(""); // need this one for synchronisation 
over flush()
             error = in.waitForPrompt();
             if (error != null)
                 throw new Exception(error);
             out.writeLine(""); // server wants more, we're going to 
tell it, this is it
             error = in.waitForPrompt();
             if (error != null)
                 throw new Exception(error);
//            // disconnect from server
//            server.close();
         } catch (IOException e) {
             logger.error("Unable to connect", e);
             throw new RuntimeException("Unable to connect", e);
         } catch (Throwable th) {
             logger.error("Commit error", th);
...
                 throw new RuntimeException("Commit error: tableName: " 
+ tableName, th);
         } finally {
             try {
                 if(server != null) {
                     server.close();
                     server = null;
                 }
             } catch (Throwable th) {
                 logger.error("Error closing MonetDb server connection", 
th);
             }
         }

Thanks,
Jerry

[MonetDB-users] Repeatedly writing to MonetDB

Jerry Ramey