[MonetDB-users] Repeated mmap/munmap calls
We are running the April '11 version of MonetDB on multi-core x86-64 hardware running Solaris 10 OS. We are running on a machine with 96GB of memory. The ulimit for memory shows 'unlimited'. We have an application that we believe should be appropriate for MonetDB fast response-time characteristics. We have a very simple schema that includes the following table: create table tuid ( TUID bigint not null, clientID int not null default 0, mafID bigint not null, addrMatch tinyint default 127, lastUsed timestamp(0) default now(), constraint tuid_pk primary key( TUID, clientID ), constraint client_fk1 foreign key ( clientID ) references client ( clientID ) ); The above table currently contains 13.65 million records ( mostly bulk loaded via 'copy into' ). The primary key was dropped and re-added via appropriate 'alter table' statements. There are other 5 other tables in the schema - all with less than 100 records. All 5 tables have fewer than 5 columns. The total size of the database directory in the dbfarm directory is 1.3 GB. Ultimately, the application could have between 500 million and 1 billion records in the above table. Except for one of the other tables, the number of records will be small ( less than 1000 ). We are running a Java application that uses the JDBC interface to query and update the table. Files of 1-3 million records are processed by the application in a given run. Less than 5% of the records require an update to above table ( no other tables are currently updated or inserted to ). When the application is processing these records, we run 'prstat' against 'mserver5' and see via the 'prstat' command the processing load appears to be distributed fairly evenly across many threads. However, occasionally at some point, one thread appears to be using most of the CPU. During these intervals, 'truss' shows 'mserver5' making repeated calls to mmap and munmap. (The calls to mmap have the same value for the mmap size parameter.) These intervals appear to last for many minutes. During these intervals, query response times slow to a crawl. As a result, the overall throughput of the application is disappointing. Given that we are actually accessing a fairly small number of columns all of which should be in memory ( we believe ), it would appear that MonetDB is not functioning properly. It is our belief that MonetDB should be a good fit for this particular application. I should note that we have 3 different MonetDB databases running on the machine ( thus requiring 3 instances of mserver5 ). However, two of the databases are idle when we run the above application. Given the amount of memory available the size of the database, the number of columns being queries ( and/or updated ), and the amount of memory on the machine, the behavior we are observing does not appear to be consistent with descriptions of MonetDB's expected behavior. We have 3 questions: * Is what we are observing concerning the frequency of mmap and munmap calls typical where the mmap size parameter has the same value? * Why would we be mapping columns into memory so frequently when we are not adding records? * Is it typical to have periods where one thread appears to be handling the entire load? * What should we look for concerning the tuning of the x86-64 platform or MonetDB to address these issues? Thanks Steve
participants (1)
-
Steve Elder