May 2005 - users-list

[MonetDB-users] RHEL 3 RPMs
by Henk van Lingen 28 Apr '06

28 Apr '06

Hi, Is there a special reason there are no i686 RPM's anymore like there were until 4.6.0_rc2? I think it's a bit weird there are RPM's for the hobby OS 'FedoraCore' but not for the much used server OS 'RedHat Enterprise 3 (and 4)'? The FedoraCore (which Core?) needs libstdc++.so.6 whereas RHEL 3 only has libstdc++.so.5 ... There also seems to be no SRPM's. Cheers, -- Henk van Lingen, Systems & Network Administrator (o- -+ Dept. of Computer Science, Utrecht University. /\ | phone: +31-30-2535278 v_/_ http://henk.vanlingen.net/ http://www.tuxtown.net/netiquette/

6 14

[MonetDB-users] monetdb failing tests on FreeBSD
by Joseph McDonald 09 Jun '05

09 Jun '05

Hi, I just installed MonetDB-4.8.0_rc0 on FreeBSD 5.4-STABLE running on an opteron. To get it to compile I added this to monetdb_config.h: #define __BSD_VISIBLE 1 and I changed /* #define _POSIX_C_SOURCE 200112L */ /* XXX JOE */ #define _POSIX_C_SOURCE 200112 because a freebsd system header already had _POSIX_C_SOURCE defined without the L. when running "gmake check" I get: !ERROR: Testing FAILED SIGNIFICANTLY ! 2 out of 69 tests could not be executed 2 out of 69 tests produced slightly different output 2 out of 69 tests produced SIGNIFICANTLY different output the output is at http://theoden.vpop.net/~joe/mtest/ Any idea what this means and why? I am brand new to monetdb. thanks, -joe

4 7

[MonetDB-users] Converting a unix timestamp (bigint) to timestamp/date?
by Arjen van der Meijden 24 May '05

24 May '05

Hi List, I'm trying to have an easy way to convert some fields that where initialy (in MySQL) stored as unsigned integers. I converted them using a dump/restore to MonetDB's bigint. Currently this works: CAST("1970-01-01 01:00:00" as timestamp) + (CAST(RecordTimestamp AS STRING) || 'LL') Appearantly bigint (long) is incompatible with sec_interval (long)? But afaik this should be possible in SQL as well: CAST("1970-01-01 01:00:00" as timestamp) + INTERVAL RecordTimestamp SECONDS And of course in MySQL there is FROM_UNIXTIME(RecordTimestamp), in PostgreSQL there is: timestamp 'epoch' + RecordTimestamp * interval '1 seconds' Converting it back is also quite a bit of work: SELECT ((CAST(current_timestamp as timestamp) - CAST("1970-01-01 00:00:00" as timestamp)) / 1000); That seems to work to get the current time's epoch... Why aren't timestamptz and timestamp compatible, btw? In MySQL there is UNIX_TIMESTAMP(RecordTimestamp), in PostgreSQL EXTRACT(epoch FROM current_timestamp)::bigint I was also looking for a way to format the date/timestamp output, is that possible? Casting it to a SQL-date is rather easy of course, but other formatting will very likely become a big mess if EXTRACT where to be used. Since my client-language is php, I either need preformatted output or a unix timestamp for ease of use. In SQL it is often better to work with dates/timestamps, rather than the not really correct unix timestamp arithmetics (add 86400 seconds to get the next day and so on). Did I overlook some easy to use extentions to SQL, or do I really need to use the constructs I already found? Best regards, Arjen van der Meijden

3 2

[MonetDB-users] Bug with average of decimals when an empty table is returned
by Arjen van der Meijden 24 May '05

24 May '05

Hi list, I noticed a weird result from AVG when doing one of my queries; -1.7976931348623158e+307 Appearantly it happens with this minimal testcase: monetdb-> create table emptyone (field decimal(3,1)); Operation successful monetdb-> select avg(field) from emptyone; +------------------------+ | scale_down_avg_field | +========================+ | -1.7976931348623158E307 | +------------------------+ 1 row The result should afaik be NULL, not this value. With a decimal(3) it works correctly by the way. The same behaviour is seen with numeric(3,1) vs numeric(3). Best regards, Arjen

2 4

[MonetDB-users] Re: MonetDB Questions
by Martin Kersten 17 May '05

17 May '05

Hello Mason, thank you for your interest in using MonetDB. Enclosed a quick response to your questions. If I miss something, someone from the monetdb-users list is likely to fill in the details. M Sharp wrote: > Hello Martin, > > We are interested in evaluating MonetDB for use with our product (a > potential customer of ours told us queries ran 5-6 times faster than on That;s what we know to be true for especially decision support applications. > PostgreSQL). Basically we would install MonetDB on several nodes, and > partition data amongst them, for data warehousing usage (more info about > our product can be found at http://www.extendb.com) > > I have looked through the documentation and website, but have some > questions I was hoping you could help answer (as the project lead). If > it is not too much trouble, I would greatly appreciate it if you could > take a few minutes to answer. > > 1. Size limitation. If I understand correctly, there is no size > limitation for tables or databases, just that a single column is limited > to 2 GB. Is that correct? Also, when do you believe this restriction > will be removed? The size limitation of 2 GB holds for 32-bit platforms, it is already removed for 64 bit machines. > > 1a. I see something in the config file about 3GB of virtual memory. If > the database (or table) is larger than that, does it negatively impact > performance? The performance degradation starts when MonetDB has to access BATs larger then the available memory, but won;t become slower than what you would expect in traditional database systems. The judicious use of the BATs mean that in practice of decision support you have a much lower chance hitting an IO bottleneck. > > 2. Multiple databases. I noticed in the config file that only one > database can be used at a time. Can we start mulitple MonetDB instances > to support multiple databases, just using a different port? There can be one Mserver per database directory, and ofcourse multiple Mserver instances per platform. > > 3. Backups. Is there any backup tool? Also, can you do incremental > backups? No, we are looking forward to people interesting in developing such supportive tools. > > 4. Is there point in time recovery? The SQL frontend ensures a log of all committed transactions to ensure stability. If you use the system at the MIL layer you have to schedule that work yourself. > > 5. Do you support savepoints (or subtranscations), not just > transactions? The SQL transaction semantics is based on the requirements in SQL99 > > 6. What kind of locking capability is offered; DB, table or row? Are > there different isolation levels? There are no separate isolation levels. The policy in MonetDB is to assume a cooperative environment, which means that (SQL) compilers have to explicity lock/release resources to support concurrent use from clients. Internally, the system is fully configured to safely run with tens of concurrent threads thereby protecting the integrity of the database structures (but not their content). > > 7. Is there a separate high-speed loader utility, or must INSERTs be > used? yes there is. > > 8. Are checkpoints used? And is it configurable? No particular support for taking a checkpoint > > 9. Are there utilities by which you can determine corruption? ... And > repair corruption? The additional tools have not been needed so far. The SQL log was sufficient to deal with the rare cases during development crashes. > > > > Thank you very much in advance for taking the time to answer these > questions. Also, please feel free to provide any other thoughts you have > regarding using MonetDB and handling large databases (~ 1 TB). It is the sizes we are after, but not on 32-bit platforms. > > Regards, > > Mason Sharp > ExtenDB >

1 0

[MonetDB-users] VOIDs and multiple BATs
by Edmund Dengler 09 May '05

09 May '05

Greetings! Just doing some more digging around MonetDB, and have a couple of questions. (1) If I am managing multiple BATs that are (supposedly) synced, how do I manage appends and the syncing? For example, I have a_p1 = BAT[void, int] and a_p2 = BAT[void, int] (a_p1 = <a> type things with property p1) (a_p2 = <a> type things with property p2) I wish to insert a new <a>=(1,2) thing. Do I do? bat("a_p1").append(1); bat("a_p2").append(2); How do I ensure that the "voids" are always synced? Is this just "assumed"? What if I fail in the middle of operations (such that a_p1 has an extra row) and stuck with dirty tables? Do I do a global "abort();" or a "clean();" to get back to a safe state? Will this reset the seqbase? (Is this documented someplace? I am trying to look at the architecture docs but cannot find anything that concretely answers similar questions.) (2) How does sync() work? Do I do a: sync(bat("a_p1"), bat("a_p2")); once and only once? After every update/append to the tables? There is the phrase "When two BATs effectively contain the same sequence of head elements, we call them 'synced'. This is implemented by storing a very large OID for each column. An update to the column destroys this OID.". This sounds as if any update/append breaks the correspondence, and also that it will limit the number of rows in the table (as an OID is a 32-bit number of 32-bit systems). If I have 10 properties I am tracking, would I pick one as the "primary" BAT, and sync() all others to it? Regards! Ed

3 5

[MonetDB-users] Counting open BATs
by Edmund Dengler 05 May '05

05 May '05

Greetings! In the architectural overview, I find the text: "Most memory resource problems in MonetDB are easily avoidable by minimizing your open BAT references (as discussed above) and making your access patterns cache friendly (see the radix module)." Is there anyplace that actually describes when a BAT is considered open? Is it a scope thing (BATs are closed as soon as they are out of scope)? Is a BAT open when I append/update? Does it close when I perform a commit? If not, is there a way to specify "I am done, clean it out"? I am noticing a large amount of open memory (approximately 500MB out of a 1GB system) being used by MonetDB. I have hand partitioned my BATs into groups using a naming convention, but I am wondering if during inserts I am somehow keeping open references to the BATs which may cause issues. Is there a way to find out where memory is being allocated? Which BATs are open? Also, I am setting some of the larger BATs to STORE_MMAP. I notice in the documentation that if I compress the heaps, then these must be set to STORE_MEM. Is there a reason that when uncompressing the heap, that the uncompressed version cannot be STORE_MMAP to avoid memory problems? Regards! Ed

2 3