Alex Bo. wrote:
Hi, Hi happy newyear,
Does MonetDB split a single query plan over multiple CPU cores when its possible? (maybe the DataFlow optimizer?)
The dataflow optimizer is enabled by default for a long time on all platforms. The amount of effective parallelism depends on portions of the plan. You see the portions bracketed as 'barrier _230 := language.dataflow();... exit _230;' in the explain.
For example, the following query could be executed by all cores in parallel: SELECT field1,SUM(field2) FROM Table1 GROUP BY field1
There is not much parallelism here in the default optimizer setting. Aside from a grouping operation to compete with loading other columns concurrently. IO subsystem might get stressed.
When i tried to see if it does, i got some weird results: (I am running MonetDB on Windows, on a quad core machine)
When I set '--set gdk_nr_threads=1', queries never seem to use more than 25%. (seems logical since only one CPU core is being deployed)
When I set '--set gdk_nr_threads=2', queries use up to 50% CPU, but the query is ~15% slower. in wallclock time? Going parallel presupposes that threads do not compete for scarce resources. When I set '--set gdk_nr_threads=4', queries use up to 98% CPU, but the query is take twice as long as with 1 thread. That is indicative for a resource contention. The cause on Windows may stem from a variety of reasons. We have not investigated parallel
The accounting of resources is then different from Linux, where it would run between 0% and 400%. processing on that platform, because it is not the prime development platform.
So, my initial assumption that more CPU utilization (more cores over the same query) will result faster query results (where applicable) is wrong. Can it be something specific to the windows platform? or maybe single query
Most likely windows specific, as on Linux we see good behavior.
execution over multiple cores doesn't work without a specific optimizer flag enabled?... or maybe is it just a bug... hard to nail down such 'bugs', sometimes it is better to avoid their territory ;)
regards, martin
Thanks.