Hello,
Thank you for all the replies and information!
The version is 11.23, plus I'm using the cloud platform with 4cpu-16ram-hpc.
The number of rows was just 1K for 15K attributes in a table.
I understand that MonetDB was made with no intention to create and process
tables with that many columns. But I'm interested in matrix operations
integration, so I would like to know what is the main bottleneck to process
such data (matrix-like sized) in the system.
Best regards,
Oksana
On Tue, 9 Oct 2018 at 21:02, Eyal Rozenberg
I'll add two observations to what Zhang Ying said.
1. Internally in MonetDB, the index of a column (or a BAT) in MonetDB's BBP (BAT Buffer Pool) is a signed 32-bit integer (on most platforms), so there isn't a theoretical problem having more than 15,000 columns; but MonetDB was not designed with that many tables in mind. The "long dimension" is expected to be the length of columns.
2. When you have as many as 15,000 columns, you're not simply finding more attributes of the same items. So, for example, a person has a first name, last name, year of birth, etc. etc. - but after several dozens of these you're probably adding columns such as "data relevant to this person at point X", "data relevant to this person at point Y" and so on.
In your case, I'm guessing you have a lot of points and hence a lot of columns with data for your different items. What you should consider doing instead is having an extra "coordinate" column. Thus, in the example I gave, I would drop most of the columns, in favor of a second table named "person_data". This table's columns would be "person_id" (or some other key columns), "sample_point" and "value". Thus instead of M columns and N records you would have N x M records with 3 columns.
Eyal
Hai Oksana,
A.f.a.I.k., there is no explicit limit on the #columns you can put in a relation (did you mean “table”?). The reason that mserver5 stopped responding is probably because it is busy with I/O.
A long time ago, I once tried to create 1M tables, each with 3 columns. After about a day or something, MonetDB has created ~400K tables, and I could hear my HDD being constantly busy. MonetDB was still working, but
On 09/10/2018 16:50, Ying Zhang wrote: that was already a sufficient evidence that that idea wasn’t going to work.
So the question is actually what do you want to achieve?
Regards, Jennie
On 9 Oct 2018, at 13:43, Oxana Dolmatova
Hello,
Could you tell the limitation for number of columns in a relation and
wrote: the main reason for that limitation. So far I could create relations with up to 15000 columns. When the number of columns is bigger mserver5 stops responding.
Thank you and best regards, Oksana Dolmatova _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- Best regards, Oksana Dolmatova