prevent mitosis from partitioning (filter) function arguments - users-list - monetdb.org

newer
Best way to force table into hot...

prevent mitosis from partitioning (filter) function arguments

older
Severe performance degradation

Roberto Cornacchia

9 Feb 2017 9 Feb '17

2:10 p.m.

I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are partitioned by mitosis. How can I prevent mitosis from partitioning its inputs? Roberto

Attachments:

attachment.html (text/html — 1.0 KB)

Reply

Sign in to reply online Use email software

Show replies by date

Roberto Cornacchia

9 Feb 9 Feb

3:09 p.m.

UPDATE: I solved this for now by hardcoding in opt_mitosis a bailout condition for that specific function. Wouldn't it be useful to use a function marker for this, along the lines of {inline|unsafe} markers? Something like: blocking command myfunction(...); and then disable mitosis for all blocking functions (i.e. functions that need to consume their whole input)? On the same topic, I also noticed that mitosis currently isn't disabled on unsafe functions. Is that correct? If a function is unsafe, calling it multiple times could be... .. well ... unsafe? ;-) Roberto On 9 February 2017 at 15:10, Roberto Cornacchia < roberto.cornacchia@gmail.com> wrote:

I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are partitioned by mitosis.

How can I prevent mitosis from partitioning its inputs?

Roberto

Reply

Sign in to reply online Use email software

Ying Zhang

10 Feb 10 Feb

3:52 a.m.

On 9 Feb 2017, at 16:09, Roberto Cornacchia wrote:

UPDATE:

I solved this for now by hardcoding in opt_mitosis a bailout condition for that specific function.

Wouldn't it be useful to use a function marker for this, along the lines of {inline|unsafe} markers?

Something like:

blocking command myfunction(...);

and then disable mitosis for all blocking functions (i.e. functions that need to consume their whole input)?

On the same topic, I also noticed that mitosis currently isn't disabled on unsafe functions. Is that correct? If a function is unsafe, calling it multiple times could be... .. well ... unsafe? ;-)

Hmmm, normally ‘unsafe’ is the keyword you need. Doesn’t it work for your function? I took a look at the impl. of quantile, because its input is surely not partitioned. but I can’t quickly find how that requirement is past to mitosis...

Roberto

On 9 February 2017 at 15:10, Roberto Cornacchia wrote: I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are partitioned by mitosis.

How can I prevent mitosis from partitioning its inputs?

Roberto

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Reply

Sign in to reply online Use email software

Roberto Cornacchia

7:45 a.m.

On 10 February 2017 at 04:52, Ying Zhang wrote:

...
On 9 Feb 2017, at 16:09, Roberto Cornacchia < roberto.cornacchia@gmail.com> wrote:

UPDATE:

I solved this for now by hardcoding in opt_mitosis a bailout condition for that specific function.

Wouldn't it be useful to use a function marker for this, along the lines of {inline|unsafe} markers?

Something like:

blocking command myfunction(...);

and then disable mitosis for all blocking functions (i.e. functions that need to consume their whole input)?

On the same topic, I also noticed that mitosis currently isn't disabled on unsafe functions. Is that correct? If a function is unsafe, calling it multiple times could be... .. well ... unsafe? ;-)

Hmmm, normally ‘unsafe’ is the keyword you need.

In my view, yes and no. Yes, 'unsafe' should be a trigger for mitosis, bur apparently it isn't. No, it isn't what I'm looking for, because it has much stronger consequences. If I'm not mistaken, unsafe means "has side effects", which implies: 1) not thread-safe 2) needs all input at once 3) its result cannot be reused in the plan. The third is a show-stopper for me. That is why is was suggesting one more keyword, which would have only the first two effects from the list above.

Doesn’t it work for your function?

No, 'unsafe' doesn't affect my function, i.e. it still partitions the input and calls it multiple times. I did not check whether it at least serializes its execution, which would at least cover point 1 above. I took a look at the impl. of quantile, because its input is surely not

partitioned. but I can’t quickly find how that requirement is past to mitosis...

...
Roberto

On 9 February 2017 at 15:10, Roberto Cornacchia <

roberto.cornacchia@gmail.com> wrote:

...
I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are partitioned by mitosis.

How can I prevent mitosis from partitioning its inputs?

Roberto

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Reply

Sign in to reply online Use email software

Roberto Cornacchia

9:25 a.m.

About quantile, it is excluded in opt_mitosis by this: if (p->argc > 2 && getModuleId(p) == aggrRef && getFunctionId(p) != subcountRef && getFunctionId(p) != subminRef && getFunctionId(p) != submaxRef && getFunctionId(p) != subavgRef && getFunctionId(p) != subsumRef && getFunctionId(p) != subprodRef) return 0; It excludes all aggregates except those listed. For my function, I did something similar here. On 10 February 2017 at 08:45, Roberto Cornacchia < roberto.cornacchia@gmail.com> wrote:

On 10 February 2017 at 04:52, Ying Zhang wrote:

...
...
On 9 Feb 2017, at 16:09, Roberto Cornacchia < roberto.cornacchia@gmail.com> wrote:

UPDATE:

I solved this for now by hardcoding in opt_mitosis a bailout condition for that specific function.

Wouldn't it be useful to use a function marker for this, along the lines of {inline|unsafe} markers?

Something like:

blocking command myfunction(...);

and then disable mitosis for all blocking functions (i.e. functions that need to consume their whole input)?

On the same topic, I also noticed that mitosis currently isn't disabled on unsafe functions. Is that correct? If a function is unsafe, calling it multiple times could be... .. well ... unsafe? ;-)

Hmmm, normally ‘unsafe’ is the keyword you need.

In my view, yes and no.

Yes, 'unsafe' should be a trigger for mitosis, bur apparently it isn't.

No, it isn't what I'm looking for, because it has much stronger consequences. If I'm not mistaken, unsafe means "has side effects", which implies: 1) not thread-safe 2) needs all input at once 3) its result cannot be reused in the plan.

The third is a show-stopper for me. That is why is was suggesting one more keyword, which would have only the first two effects from the list above.

...
Doesn’t it work for your function?

No, 'unsafe' doesn't affect my function, i.e. it still partitions the input and calls it multiple times. I did not check whether it at least serializes its execution, which would at least cover point 1 above.

I took a look at the impl. of quantile, because its input is surely not

...
partitioned. but I can’t quickly find how that requirement is past to mitosis...

...
Roberto

On 9 February 2017 at 15:10, Roberto Cornacchia <

...
I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are

roberto.cornacchia@gmail.com> wrote: partitioned

...
by mitosis.

How can I prevent mitosis from partitioning its inputs?

Roberto

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Reply

Sign in to reply online Use email software

Ying Zhang

12 Feb 12 Feb

7:01 a.m.

On 10 Feb 2017, at 10:25, Roberto Cornacchia wrote:

About quantile, it is excluded in opt_mitosis by this:

if (p->argc > 2 && getModuleId(p) == aggrRef &&

getFunctionId(p) != subcountRef && getFunctionId(p) != subminRef && getFunctionId(p) != submaxRef && getFunctionId(p) != subavgRef && getFunctionId(p) != subsumRef && getFunctionId(p) != subprodRef)

return 0;

It excludes all aggregates except those listed. For my function, I did something similar here.

Hmm, this is not the cleanest / most elegant solution. For UDFs, a user won’t want to “mess” with the MonetDB internal code. Since the third behaviour of ‘unsafe’ is a particular show stopper for you, I’m afraid MonetDB doesn’t have the exact feature you’re looking for. Fire a feature request?

On 10 February 2017 at 08:45, Roberto Cornacchia wrote: On 10 February 2017 at 04:52, Ying Zhang wrote:

...
On 9 Feb 2017, at 16:09, Roberto Cornacchia wrote:

UPDATE:

I solved this for now by hardcoding in opt_mitosis a bailout condition for that specific function.

Wouldn't it be useful to use a function marker for this, along the lines of {inline|unsafe} markers?

Something like:

blocking command myfunction(...);

and then disable mitosis for all blocking functions (i.e. functions that need to consume their whole input)?

On the same topic, I also noticed that mitosis currently isn't disabled on unsafe functions. Is that correct? If a function is unsafe, calling it multiple times could be... .. well ... unsafe? ;-)

Hmmm, normally ‘unsafe’ is the keyword you need.

In my view, yes and no.

Yes, 'unsafe' should be a trigger for mitosis, bur apparently it isn't.

No, it isn't what I'm looking for, because it has much stronger consequences. If I'm not mistaken, unsafe means "has side effects", which implies: 1) not thread-safe 2) needs all input at once 3) its result cannot be reused in the plan.

The third is a show-stopper for me. That is why is was suggesting one more keyword, which would have only the first two effects from the list above.

Doesn’t it work for your function?

No, 'unsafe' doesn't affect my function, i.e. it still partitions the input and calls it multiple times. I did not check whether it at least serializes its execution, which would at least cover point 1 above.

I took a look at the impl. of quantile, because its input is surely not partitioned. but I can’t quickly find how that requirement is past to mitosis...

...
Roberto

On 9 February 2017 at 15:10, Roberto Cornacchia wrote: I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are partitioned by mitosis.

How can I prevent mitosis from partitioning its inputs?

Roberto

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Reply

Sign in to reply online Use email software

Roberto Cornacchia

11:40 a.m.

Hi Jennie, thanks for the feedback. I will file a feature request indeed About 'unsafe', is there a bug here? I.e. should mitosis be disabled on functions marked as unsafe, as I would expect? On 12 Feb 2017 8:02 am, "Ying Zhang" wrote:

...
On 10 Feb 2017, at 10:25, Roberto Cornacchia < roberto.cornacchia@gmail.com> wrote:

About quantile, it is excluded in opt_mitosis by this:

if (p->argc > 2 && getModuleId(p) == aggrRef &&

getFunctionId(p) != subcountRef && getFunctionId(p) != subminRef && getFunctionId(p) != submaxRef && getFunctionId(p) != subavgRef && getFunctionId(p) != subsumRef && getFunctionId(p) != subprodRef)

return 0;

It excludes all aggregates except those listed. For my function, I did something similar here.

Hmm, this is not the cleanest / most elegant solution. For UDFs, a user won’t want to “mess” with the MonetDB internal code.

Since the third behaviour of ‘unsafe’ is a particular show stopper for you, I’m afraid MonetDB doesn’t have the exact feature you’re looking for. Fire a feature request?

...
On 10 February 2017 at 08:45, Roberto Cornacchia <

...
On 10 February 2017 at 04:52, Ying Zhang wrote:

...
On 9 Feb 2017, at 16:09, Roberto Cornacchia < roberto.cornacchia@gmail.com> wrote:

UPDATE:

I solved this for now by hardcoding in opt_mitosis a bailout condition for that specific function.

Wouldn't it be useful to use a function marker for this, along the

...
...
Something like:

blocking command myfunction(...);

and then disable mitosis for all blocking functions (i.e. functions

...
...
On the same topic, I also noticed that mitosis currently isn't

disabled on unsafe functions. Is that correct? If a function is unsafe, calling it multiple times could be... .. well ... unsafe? ;-)

Hmmm, normally ‘unsafe’ is the keyword you need.

In my view, yes and no.

Yes, 'unsafe' should be a trigger for mitosis, bur apparently it isn't.

No, it isn't what I'm looking for, because it has much stronger consequences. If I'm not mistaken, unsafe means "has side effects", which implies: 1) not thread-safe 2) needs all input at once 3) its result cannot be reused in the plan.

The third is a show-stopper for me. That is why is was suggesting one more keyword, which would have only the first two effects from the list above.

Doesn’t it work for your function?

No, 'unsafe' doesn't affect my function, i.e. it still partitions the input and calls it multiple times. I did not check whether it at least serializes its execution, which would at least cover point 1 above.

I took a look at the impl. of quantile, because its input is surely not

...
but I can’t quickly find how that requirement is past to mitosis...

...
Roberto

On 9 February 2017 at 15:10, Roberto Cornacchia <

roberto.cornacchia@gmail.com> wrote:

...
I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are

roberto.cornacchia@gmail.com> wrote: lines of {inline|unsafe} markers? that need to consume their whole input)? partitioned. partitioned

...
...
by mitosis.

How can I prevent mitosis from partitioning its inputs?

Roberto

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

Reply

Sign in to reply online Use email software

2876

Age (days ago)

2879

Last active (days ago)

Download

6 comments

2 participants

tags

participants (2)

Roberto Cornacchia
Ying Zhang