[Monetdb-developers] GDKupgradevarheap allocate/copy inconsistency
Dear MonetDB developers, please see the newly created tracker item [1]. I haven't seen relevant design documentation which would have allowed me to deduce if BATs from VIEWcreate_ are actually supposed to end up in GDKupgradevarheap, as it happened to me. Still, I could imagine the suggested fix to be a step in the right direction as it helps modularity by making the function working with less preassumptions on the inputs. However, if it fits into the global design, the capacity argument can (should?) be removed from the function signature. Best regards, Isidor Zeuner [1] https://sourceforge.net/tracker/?func=detail&aid=2936616&group_id=56967&atid=482468
On 2010-01-21 23:06, Isidor Zeuner wrote:
Dear MonetDB developers,
please see the newly created tracker item [1].
I haven't seen relevant design documentation which would have allowed me to deduce if BATs from VIEWcreate_ are actually supposed to end up in GDKupgradevarheap, as it happened to me. Still, I could imagine the suggested fix to be a step in the right direction as it helps modularity by making the function working with less preassumptions on the inputs. However, if it fits into the global design, the capacity argument can (should?) be removed from the function signature.
Best regards,
Isidor Zeuner
[1] https://sourceforge.net/tracker/?func=detail&aid=2936616&group_id=56967&atid=482468
I saw the tracker item, and was wondering about it. What did you do to get into the situation? In other words, how can I reproduce? I don't think BATs created in VIEWcreate_ should end up in GDKupgradevarheap. Those BATs are views, and I would think views are read-only, and read-only BATs should not end up here. The BAT capacity is the number of elements that can be stored in the BAT, i.e. it is exactly the size of the heap (apart from the multiplication with the width of the elements). However, this may break down in the case of views. When I wrote the code, I wasn't thinking of views, since I didn't expect them to end up here, and I still think they shouldn't. Having said this, it is probably possible, perhaps even equivalent, to use the size as you suggested instead of the capacity -- as long as views are left out of the equation. So I would really like to see how views end up being upgraded. -- Sjoerd Mullender
[1] https://sourceforge.net/tracker/?func=detail&aid=2936616&group_id=56967&atid=482468
I saw the tracker item, and was wondering about it.
What did you do to get into the situation? In other words, how can I reproduce?
Unfortunately, I don't have an isolated failing case yet. I am continuously sending queries (mostly SELECT, INSERT and DELETE) using MonetDB-SQL, many of which are grouped inside large transactions. After initializing the database, it takes a while for the issue to occur, and I'm not yet sure what triggers it. I am currently adding instrumentation code around the code communicating with Mapi, which will hopefully get me some reproducible sessions for this and the other issue I am seeing
I don't think BATs created in VIEWcreate_ should end up in GDKupgradevarheap. Those BATs are views, and I would think views are read-only, and read-only BATs should not end up here.
Yes, intuitively it seemed a bit unusual when I thought about it, but then again I am not much used to the MonetDB internals yet. I had found GDKupgradevarheap being supplied with a capacity less than heap.size >> shift (it was capacity == 256, heap.size == 512 and shift == 0), and was wondering where it came from. So I added assert statements to all locations modifying capacity, shift or the heap allocation, which errored out in VIEWcreate_. It happened about the same time where I would have expected the GDKupgradevarheap issue to occur, so I found it most likely to be related. Of course the problematic BAT might still come from another source, so if you say views won't reach GDKupgradevarheap, I'll probably go for another run without the assertions in VIEWcreate_. On the other hand, will it suffice to put assert(!isVIEW(b)) statements in front of the GDKupgradevarheap calls in the code to verify?
The BAT capacity is the number of elements that can be stored in the BAT, i.e. it is exactly the size of the heap (apart from the multiplication with the width of the elements). However, this may break down in the case of views. When I wrote the code, I wasn't thinking of views, since I didn't expect them to end up here, and I still think they shouldn't.
Having said this, it is probably possible, perhaps even equivalent, to use the size as you suggested instead of the capacity -- as long as views are left out of the equation.
So, as views aren't supposed to get there anyway, capacity is redundant in that function, right? Best regards, Isidor Zeuner
[1] https://sourceforge.net/tracker/?func=detail&aid=2936616&group_id=56967&atid=482468 I saw the tracker item, and was wondering about it.
What did you do to get into the situation? In other words, how can I reproduce?
Unfortunately, I don't have an isolated failing case yet. I am continuously sending queries (mostly SELECT, INSERT and DELETE) using MonetDB-SQL, many of which are grouped inside large transactions. After initializing the database, it takes a while for the issue to occur, and I'm not yet sure what triggers it.
I am currently adding instrumentation code around the code communicating with Mapi, which will hopefully get me some reproducible sessions for this and the other issue I am seeing FYI. The stethoscope program can be attached to any running mserver to extract information about the instructions being executed and
Isidor Zeuner wrote: the necessary performance indicators.
I don't think BATs created in VIEWcreate_ should end up in GDKupgradevarheap. Those BATs are views, and I would think views are read-only, and read-only BATs should not end up here.
Yes, intuitively it seemed a bit unusual when I thought about it, but then again I am not much used to the MonetDB internals yet.
I had found GDKupgradevarheap being supplied with a capacity less than heap.size >> shift (it was capacity == 256, heap.size == 512 and shift == 0), and was wondering where it came from. So I added assert statements to all locations modifying capacity, shift or the heap allocation, which errored out in VIEWcreate_. It happened about the same time where I would have expected the GDKupgradevarheap issue to occur, so I found it most likely to be related.
Of course the problematic BAT might still come from another source, so if you say views won't reach GDKupgradevarheap, I'll probably go for another run without the assertions in VIEWcreate_. On the other hand, will it suffice to put assert(!isVIEW(b)) statements in front of the GDKupgradevarheap calls in the code to verify?
The BAT capacity is the number of elements that can be stored in the BAT, i.e. it is exactly the size of the heap (apart from the multiplication with the width of the elements). However, this may break down in the case of views. When I wrote the code, I wasn't thinking of views, since I didn't expect them to end up here, and I still think they shouldn't.
Having said this, it is probably possible, perhaps even equivalent, to use the size as you suggested instead of the capacity -- as long as views are left out of the equation.
So, as views aren't supposed to get there anyway, capacity is redundant in that function, right?
Best regards,
Isidor Zeuner
------------------------------------------------------------------------------ Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
I am currently adding instrumentation code around the code communicating with Mapi, which will hopefully get me some reproducible sessions for this and the other issue I am seeing FYI. The stethoscope program can be attached to any running mserver to extract information about the instructions being executed and the necessary performance indicators.
Thanks for the pointer. I think in this case it will be more convenient to work in application space to isolate failing samples. For stethoscope, you might want to commit this patch: [1] (stethoscope would not allow me to specify a server host to attach to). Best regards, Isidor Zeuner [1] https://sourceforge.net/tracker/?func=detail&aid=2938069&group_id=56967&atid=482468
On 2010-01-22 16:15, Isidor Zeuner wrote:
[1] https://sourceforge.net/tracker/?func=detail&aid=2936616&group_id=56967&atid=482468
I saw the tracker item, and was wondering about it.
What did you do to get into the situation? In other words, how can I reproduce?
Unfortunately, I don't have an isolated failing case yet. I am continuously sending queries (mostly SELECT, INSERT and DELETE) using MonetDB-SQL, many of which are grouped inside large transactions. After initializing the database, it takes a while for the issue to occur, and I'm not yet sure what triggers it.
I am currently adding instrumentation code around the code communicating with Mapi, which will hopefully get me some reproducible sessions for this and the other issue I am seeing
I don't think BATs created in VIEWcreate_ should end up in GDKupgradevarheap. Those BATs are views, and I would think views are read-only, and read-only BATs should not end up here.
Yes, intuitively it seemed a bit unusual when I thought about it, but then again I am not much used to the MonetDB internals yet.
I had found GDKupgradevarheap being supplied with a capacity less than heap.size >> shift (it was capacity == 256, heap.size == 512 and shift == 0), and was wondering where it came from. So I added assert statements to all locations modifying capacity, shift or the heap allocation, which errored out in VIEWcreate_. It happened about the same time where I would have expected the GDKupgradevarheap issue to occur, so I found it most likely to be related.
Of course the problematic BAT might still come from another source, so if you say views won't reach GDKupgradevarheap, I'll probably go for another run without the assertions in VIEWcreate_. On the other hand, will it suffice to put assert(!isVIEW(b)) statements in front of the GDKupgradevarheap calls in the code to verify?
If you get to this situation again, and you happen to catch it in the debugger (inside GDKupgradevarheap), perhaps you can look at the c->parentid value. I expect it to be 0, meaning the column is not shared with another BAT (i.e. not a view). I'll try an assert to that effect and see what that gives in our tests.
The BAT capacity is the number of elements that can be stored in the BAT, i.e. it is exactly the size of the heap (apart from the multiplication with the width of the elements). However, this may break down in the case of views. When I wrote the code, I wasn't thinking of views, since I didn't expect them to end up here, and I still think they shouldn't.
Having said this, it is probably possible, perhaps even equivalent, to use the size as you suggested instead of the capacity -- as long as views are left out of the equation.
So, as views aren't supposed to get there anyway, capacity is redundant in that function, right?
I think you're right that the capacity argument is redundant. I have a patch ready for that. It's slightly different from your patch, but the effect is the same. -- Sjoerd Mullender
I don't think BATs created in VIEWcreate_ should end up in GDKupgradevarheap. Those BATs are views, and I would think views are read-only, and read-only BATs should not end up here.
Yes, intuitively it seemed a bit unusual when I thought about it, but then again I am not much used to the MonetDB internals yet.
I had found GDKupgradevarheap being supplied with a capacity less than heap.size >> shift (it was capacity == 256, heap.size == 512 and shift == 0), and was wondering where it came from. So I added assert statements to all locations modifying capacity, shift or the heap allocation, which errored out in VIEWcreate_. It happened about the same time where I would have expected the GDKupgradevarheap issue to occur, so I found it most likely to be related.
Of course the problematic BAT might still come from another source, so if you say views won't reach GDKupgradevarheap, I'll probably go for another run without the assertions in VIEWcreate_. On the other hand, will it suffice to put assert(!isVIEW(b)) statements in front of the GDKupgradevarheap calls in the code to verify?
If you get to this situation again, and you happen to catch it in the debugger (inside GDKupgradevarheap), perhaps you can look at the c->parentid value. I expect it to be 0, meaning the column is not shared with another BAT (i.e. not a view). I'll try an assert to that effect and see what that gives in our tests.
My current setup makes it a bit impractical to attach a debugger, but I'm investigating further using asserts. I had an "assert(max_capacity == capacity)" to find out when it happens again, and now I have an "assert(c->parentid == 0 || capacity == max_capacity)" right in front of it, so I can inform you if the sizes are indeed mismatching with c->parentid being non-zero at the same time. Right now, my search for clues let me stumble on yet another issue for which I'm not sure if it's related: [1]. At least it always corrupts my data before I get to the point of investigating the heap upgrading issue. Best regards, Isidor Zeuner [1] http://sourceforge.net/tracker/?func=detail&aid=2938065&group_id=56967&atid=482468
participants (3)
-
Isidor Zeuner -
Martin Kersten -
Sjoerd Mullender