Re: [MonetDB-users] Recycler Optimizer

16 Mar 2011

      Hi Henry,

Thanks for your interest in using MonetDB and the recycler module.
The recycler is an experimental code, that was exhaustively tested with both TPCH and SkyServer databases during its development. 
However, we haven't used it ourselves in a production environment. Therefore, your experiences are very valuable for us.

I need to check your recycler problems with Oct2010-SP1 version. Our experiments were done with november 2009 pipeline extended with recycle.
It is possible that some interplay of recycler with new/modified optimisers (e.g. mitosis,mergetable) in the pipeline causes the problems you encountered.
I am surprised about your observation of lack of effect on views, since in the SkyServer tests there were a number of queries over views where the recycler was efficient. 

I will look at the problem using our databases. But it will be usefull to have some more details about your database. Would it be possible to provide me with table and view definitions, and the problematic queries. Ideally, some mini-subset of your database will also be useful.

With respect to the performance of queries over views in Oct2010, Martin and Stefan have more experiences and observations.

Regards,
Milena

----- Original Message -----
From: "Henry Addington" 
To: "Martin Kersten" Kersten@cwi.nl>, "Communication channel for MonetDB users" sourceforge.net>
Sent: Monday, March 7, 2011 3:41:23 AM
Subject: Re: [MonetDB-users] Recycler Optimizer

Hi Martin:

We tried the Recycle pipeline for Skyserver on the Oct2010-SP1 release. And,
we didn't see any problems, though the testing was fairly short..... One
thing we did notice is that queries against defined views do not seem to
benefit from the recycler pipeline at all. Running a simple query, select
distinct x from y, back to back seem to take just as long each time. So, I
don't think the Recycler pipeline for Skyserver is effective on views.

We set up a new Recycler pipeline using the Feb-2010 default pipeline by
adding "recycle" between "deadcode" and "reduce". Unfotunately, we were
getting erratic results after running a few queries. For example, we ran
some simple queries like select distinct x from y which returned couple
dozen rows. After running a few other queries, the same select distinct x
from y returned empty result set. We restarted the MonetDB server and got
expected results for a while. But, after running some more quries, the
simple select distinct x from y started to return null set. So, I think
there are some issues with using recycle optimizer with the Feb-2010
default pipeline, at least the way I have used it. Or, does the recycler
pipeline need to be set up with a different set of Feb-2010 optimizers than
what I have included?
One of the reasons that we are looking at using the recyler option is
because we saw performance degradation going from the Nov-2009SP2 to
Oct-2010SP1 for queries on views though we saw many improvements, in terms
of performance and resource use, from upgrading. Is this a bug or are
queries on views treated differently by design? Thanks!

Henry
On Thu, Mar 3, 2011 at 2:35 PM, Martin Kersten Kersten@cwi.nl>wrote:
...
Hi Henry
We would love to here experiences in using the recycler.
...
Hello:
I understand some of the optimizers/pipelines are experimental as
indicated in the monetdb5.conf file. Has anyone used the Recycle pipeline
for Skyserver? Would you mind sharing your experience, in terms of
...
used in a production environment? We are using the Oct2010-SP1 release
for a db that is read-only. I would guess if there are issues, I would think
regards, Martin Kersten
 On 3/3/11 11:00 PM, Henry Addington wrote:
performance improvement and whether it can bet
the problems would likely show up in read/write systems. Any feedback would
be appreciated....
...
It looks like the Recycle pipeline for Skyserver pipeline is based on the
Nov-2009 pipeline. Has anyone tried the recycler optimizer with the Feb2010
default pipeline? I was going to add "recycle" between "deadcode" and
"reduce" in the default pipeline and
see if it would work. Thanks.
Henry
------------------------------------------------------------------------------
...
What You Don't Know About Data Connectivity CAN Hurt You
This paper provides an overview of data connectivity, details
its effect on application quality, and explores various alternative
solutions. http://p.sf.net/sfu/progress-d2d
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------
What You Don't Know About Data Connectivity CAN Hurt You
This paper provides an overview of data connectivity, details
its effect on application quality, and explores various alternative
solutions. http://p.sf.net/sfu/progress-d2d
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------
What You Don't Know About Data Connectivity CAN Hurt You
This paper provides an overview of data connectivity, details
its effect on application quality, and explores various alternative
solutions. http://p.sf.net/sfu/progress-d2d
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users

Re: [MonetDB-users] Recycler Optimizer

Milena Ivanova