Hi Milena:
 
I apologize for the slow response.
 
I think you certainly are right. We tried the recycler pipeline in Oct-2010 release and that worked just fine. So, I do suspect that there may be issues in recycler interacting with the new optimizers in the Oct-2010 release.
 
One of the things that we did since my last email was to rewrite the view that was the source of very slow queries. And, what a difference that has made. We are getting over 90% reduction in query time. So, we stopped looking at using the recycler optimizer and implemented the updated view when we upgraded our MonetDB to 2010Oct-SP1 release.
 
Let me see if I can duplicate the issue that we saw in our test environment. We will need to put the old view back to check. And, I will get back to you. I am not sure if we can get the schema and the data to you easily though ..... We will figure something out to get you what you need. Thanks!
 
Henry
 

 
On Wed, Mar 16, 2011 at 7:40 AM, Milena Ivanova <Milena.Ivanova@cwi.nl> wrote:
Hi Henry,

Thanks for your interest in using MonetDB and the recycler module.
The recycler is an experimental code, that was exhaustively tested with both TPCH and SkyServer databases during its development.
However, we haven't used it ourselves in a production environment. Therefore, your experiences are very valuable for us.

I need to check your recycler problems with Oct2010-SP1 version. Our experiments were done with november 2009 pipeline extended with recycle.
It is possible that some interplay of recycler with new/modified optimisers (e.g. mitosis,mergetable) in the pipeline causes the problems you encountered.
I am surprised about your observation of lack of effect on views, since in the SkyServer tests there were a number of queries over views where the recycler was efficient.

I will look at the problem using our databases. But it will be usefull to have some more details about your database. Would it be possible to provide me with table and view definitions, and the problematic queries. Ideally, some mini-subset of your database will also be useful.

With respect to the performance of queries over views in Oct2010, Martin and Stefan have more experiences and observations.

Regards,
Milena

----- Original Message -----
From: "Henry Addington" <hs.addington@gmail.com>
To: "Martin Kersten" Kersten@cwi.nl>, "Communication channel for MonetDB users" sourceforge.net>
Sent: Monday, March 7, 2011 3:41:23 AM
Subject: Re: [MonetDB-users] Recycler Optimizer

Hi Martin:

We tried the Recycle pipeline for Skyserver on the Oct2010-SP1 release. And,
we didn't see any problems, though the testing was fairly short..... One
thing we did notice is that queries against defined views do not seem to
benefit from the recycler pipeline at all. Running a simple query, select
distinct x from y, back to back seem to take just as long each time. So, I
don't think the Recycler pipeline for Skyserver is effective on views.

We set up a new Recycler pipeline using the Feb-2010 default pipeline by
adding "recycle" between "deadcode" and "reduce". Unfotunately, we were
getting erratic results after running a few queries. For example, we ran
some simple queries like select distinct x from y which returned couple
dozen rows. After running a few other queries, the same select distinct x
from y returned empty result set. We restarted the MonetDB server and got
expected results for a while. But, after running some more quries, the
simple select distinct x from y started to return null set. So, I think
there are some issues with using recycle optimizer with the Feb-2010
default pipeline, at least the way I have used it. Or, does the recycler
pipeline need to be set up with a different set of Feb-2010 optimizers than
what I have included?
One of the reasons that we are looking at using the recyler option is
because we saw performance degradation going from the Nov-2009SP2 to
Oct-2010SP1 for queries on views though we saw many improvements, in terms
of performance and resource use, from upgrading. Is this a bug or are
queries on views treated differently by design? Thanks!

Henry
On Thu, Mar 3, 2011 at 2:35 PM, Martin Kersten Kersten@cwi.nl>wrote:

> Hi Henry
>
> We would love to here experiences in using the recycler.
>
> regards, Martin Kersten
>  On 3/3/11 11:00 PM, Henry Addington wrote:
> > Hello:
> > I understand some of the optimizers/pipelines are experimental as
> indicated in the monetdb5.conf file. Has anyone used the Recycle pipeline
> for Skyserver? Would you mind sharing your experience, in terms of
> performance improvement and whether it can bet
> > used in a production environment? We are using the Oct2010-SP1 release
> for a db that is read-only. I would guess if there are issues, I would think
> the problems would likely show up in read/write systems. Any feedback would
> be appreciated....
> > It looks like the Recycle pipeline for Skyserver pipeline is based on the
> Nov-2009 pipeline. Has anyone tried the recycler optimizer with the Feb2010
> default pipeline? I was going to add "recycle" between "deadcode" and
> "reduce" in the default pipeline and
> > see if it would work. Thanks.
> > Henry
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > What You Don't Know About Data Connectivity CAN Hurt You
> > This paper provides an overview of data connectivity, details
> > its effect on application quality, and explores various alternative
> > solutions. http://p.sf.net/sfu/progress-d2d
> >
> >
> >
> > _______________________________________________
> > MonetDB-users mailing list
> > MonetDB-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/monetdb-users
>
>
>
> ------------------------------------------------------------------------------
> What You Don't Know About Data Connectivity CAN Hurt You
> This paper provides an overview of data connectivity, details
> its effect on application quality, and explores various alternative
> solutions. http://p.sf.net/sfu/progress-d2d
> _______________________________________________
> MonetDB-users mailing list
> MonetDB-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-users
>

------------------------------------------------------------------------------
What You Don't Know About Data Connectivity CAN Hurt You
This paper provides an overview of data connectivity, details
its effect on application quality, and explores various alternative
solutions. http://p.sf.net/sfu/progress-d2d
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users