(1) This problem pertains to the size of the data in question. I've tested insert 17K rows into table_1 and 13K rows into table_2 then the combined table gives the correct 30K rows (2) The amount of data is over 45GB, but very simple 10 columns of doubles. Nothing fancy, no characters or funny code. Just plain numbers (3) Is it necessary to have the exactly numbers? (4) If not, then any 10 column deep data of doubles would suffice. (5) How to replicate the problem (i) generate 600 million rows of doubles, 10 column deep (ii) insert 300 million into one table (iii) insert 300 million into another (iv) combined the two tables (v) see if the combined table has all the data from the two smaller tables I was wondering if someone has successfully created a SINGLE table with
600million-by-10; Maybe the bug has something to do with not able to address all the data in a large table? We have 6 billion cells...
My MonetDB is complied in 64bit.
Thanks.
On Mon, Mar 23, 2009 at 10:26 AM, Niels Nes
On Mon, Mar 23, 2009 at 09:28:24AM -0700, Yue Sheng wrote:
Hi, then I'm totally lost.
That was my last resort in creating a big table.
So:
(1) I can't create a 600-million table using COPY (because it hangs)
(2) I can't create two 300-million tables and combined them into one (bug) Since we don't have your data. And your probably not able to supply us with it, it is probably better that we look for some 'similar' generated data set which could be the basis to solve this bug(s). So could you make an anonimized schema (create table statement(s) etc) and possibly some data generator which give the before mentioned problems.
Niels
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support
600-million-by-10 records in a SINGLE table?
Thanks. On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes <[1]Niels.Nes@cwi.nl> wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote: table_1 and
The way you explain it, it seems a bug. Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng
<[1][2]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.
(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query
> On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2][3]lsidir@gmail.com>
wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefteris
On Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng
two tables: table_1 and table_2 both have exactly the same schema. table_1 has 2.15MM rows of data, table_2 has 3.35MM row of data. No indexing created a new table: table_combined using same schema as
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 +
<[3][4]yuesheng8@gmail.com> wrote: table_1 table_2 )
So it's there but "just not seeing it"
>
------------------------------------------------------------
------------------ > Apps built with the Adobe(R) Flex(R) framework and
Flex
Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and
step-through
debugging. > Download the free 60 day trial.
> [4][5]http://p.sf.net/sfu/www-adobe-com > > _______________________________________________ > > MonetDB-users mailing list > > [5][6]MonetDB-users@lists.sourceforge.net > > > [6][7]https://lists.sourceforge.net/lists/listinfo/monetdb-u ser > s > > > > > ------------------------------------------------------------ > ------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform
capabilities.
Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and
step-through
debugging. Download the free 60 day trial.
> [7][8]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [8][9]MonetDB-users@lists.sourceforge.net > [9][10]https://lists.sourceforge.net/lists/listinfo/monetdb- user > s > > References > > 1. mailto:[11]yuesheng8@gmail.com > 2. mailto:[12]lsidir@gmail.com > 3. mailto:[13]yuesheng8@gmail.com
> 5. mailto:[15]MonetDB-users@lists.sourceforge.net > 6. [16]https://lists.sourceforge.net/lists/listinfo/monetdb-use rs
> 8. mailto:[18]MonetDB-users@lists.sourceforge.net > 9. [19]https://lists.sourceforge.net/lists/listinfo/monetdb-use rs
--------------------------------------------------------------- ---------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [20]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [21]MonetDB-users@lists.sourceforge.net
[22]https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: [23]http://www.cwi.nl/~niels e-mail: [24]Niels.Nes@cwi.nl
--------------------------------------------------------------- --------------- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [25]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [26]MonetDB-users@lists.sourceforge.net [27]https://lists.sourceforge.net/lists/listinfo/monetdb-users
References
1. mailto:Niels.Nes@cwi.nl 2. mailto:yuesheng8@gmail.com 3. mailto:lsidir@gmail.com 4. mailto:yuesheng8@gmail.com 5. http://p.sf.net/sfu/www-adobe-com 6. mailto:MonetDB-users@lists.sourceforge.net 7. https://lists.sourceforge.net/lists/listinfo/monetdb-user 8. http://p.sf.net/sfu/www-adobe-com 9. mailto:MonetDB-users@lists.sourceforge.net 10. https://lists.sourceforge.net/lists/listinfo/monetdb-user 11. mailto:yuesheng8@gmail.com 12. mailto:lsidir@gmail.com 13. mailto:yuesheng8@gmail.com 14. http://p.sf.net/sfu/www-adobe-com 15. mailto:MonetDB-users@lists.sourceforge.net 16. https://lists.sourceforge.net/lists/listinfo/monetdb-users 17. http://p.sf.net/sfu/www-adobe-com 18. mailto:MonetDB-users@lists.sourceforge.net 19. https://lists.sourceforge.net/lists/listinfo/monetdb-users 20. http://p.sf.net/sfu/www-adobe-com 21. mailto:MonetDB-users@lists.sourceforge.net 22. https://lists.sourceforge.net/lists/listinfo/monetdb-users 23. http://www.cwi.nl/~niels 24. mailto:Niels.Nes@cwi.nl 25. http://p.sf.net/sfu/www-adobe-com 26. mailto:MonetDB-users@lists.sourceforge.net 27. https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users