On Mon, Mar 23, 2009 at 09:28:24AM -0700, Yue Sheng wrote:
> Hi, then I'm totally lost.
>
> That was my last resort in creating a big table.
>
> So:
>
> (1) I can't create a 600-million table using COPY (because it
> hangs)
>
> (2) I can't create two 300-million tables and combined them
> into one (bug)
Since we don't have your data. And your probably not able to supply us
with it, it is probably better that we look for some 'similar' generated
data set which could be the basis to solve this bug(s). So could you
make an anonimized schema (create table statement(s) etc) and possibly
some data generator which give the before mentioned problems.
Niels
>
> Have I missed other ways of doing this?
>
> Just wanted to make sure, the MonetDB can support
> >600-million-by-10 records in a SINGLE table?
>
> Thanks.
> On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes
> <[1]
Niels.Nes@cwi.nl> wrote:
>
> On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
> > Tried the whole process. Still the same problem
> (table_combined
> > has the same scheme as table_1):
> >
> > insert into table_combined select * from table_1;
> > insert into table_combined select * from table_2;
> > select count(*) from table_combined;
> > this gives ONLY the number of records in table_1, but
> > table_combined as roughly same size as the other two
> combined.
> > NONE OF THE TABLE HAS ANY INDEXING
> > Could someone please confirm if this is a bug? Both
> table_1 and
>
> The way you explain it, it seems a bug.
> Niels
>
> > table_2 has over 200million rows of data.
> >
> > On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng
>
> > <[1][2]
yuesheng8@gmail.com> wrote:
> >
> > (1) There's no index or key or anything on the table.
> >
> > (2) The combined table is the size of table_1 + table_2.
> But
> > only table_1 shows up in query
>
> > On Sat, Mar 21, 2009 at 1:51 PM, Lefteris
>
> > wrote:
> >
> > Hi Yue,
> > is there any chance that you have set uniqueness in the
> > table specs
> > and thus not inserting duplicate tuples from table_1 and
> > table_2? If
> > not, then I would guess that this is a bug and it needs
> > further
> > investigation.
> > lefteris
> >
> > On Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng
>
> > <[3][4]
yuesheng8@gmail.com> wrote:
> > > two tables: table_1 and table_2 both have exactly the
> same
> > schema. table_1
> > > has 2.15MM rows of data, table_2 has 3.35MM row of data.
> No
> > indexing
> > > created a new table: table_combined using same schema as
> > table_1
> > > insert into table_combined select * from table_1;
> > > (30 minutes later)
> > > insert into table_combined select * from table_2;
> > >
> > > (40 minutes later)
> > > select count(*) * 1 from table_combined;
> > > this give ONLY 2.15MM rows with same rows as table_1
> > > What happened to table_2?!?
> > > Yes, I've check dfarm size. It equals 2*(table_1 +
> table_2 )
> > > So it's there but "just not seeing it"
> > >
> > >
> > >
> > >
> > >
> >
> > >
> >
> ------------------------------------------------------------
> > ------------------
> > > Apps built with the Adobe(R) Flex(R) framework and
> Flex
> > Builder(TM) are
> > > powering Web 2.0 with engaging, cross-platform
> > capabilities. Quickly and
> > > easily build your RIAs with Flex Builder, the
> > Eclipse(TM)based development
> > > software that enables intelligent coding and
> step-through
> > debugging.
> > > Download the free 60 day trial.
>
> > > _______________________________________________
> > > MonetDB-users mailing list
> > > [5][6]MonetDB-users@lists.sourceforge.net
> > >
> >
> [6][7]https://lists.sourceforge.net/lists/listinfo/monetdb-u
> ser
> > s
> > >
> > >
> >
> ------------------------------------------------------------
> > ------------------
>
> > Apps built with the Adobe(R) Flex(R) framework and Flex
> > Builder(TM) are
> > powering Web 2.0 with engaging, cross-platform
> capabilities.
> > Quickly and
> > easily build your RIAs with Flex Builder, the
> > Eclipse(TM)based development
> > software that enables intelligent coding and
> step-through
> > debugging.
> > Download the free 60 day trial.
>
> > [7][8]http://p.sf.net/sfu/www-adobe-com
> > _______________________________________________
> > MonetDB-users mailing list
> > [8][9]MonetDB-users@lists.sourceforge.net
> >
> [9][10]https://lists.sourceforge.net/lists/listinfo/monetdb-
> user
> > s
> >
> > References
> >
> > 1. mailto:[11]yuesheng8@gmail.com
> > 2. mailto:[12]lsidir@gmail.com
> > 3. mailto:[13]yuesheng8@gmail.com
>
> > 4. [14]http://p.sf.net/sfu/www-adobe-com
>
> > 5. mailto:[15]MonetDB-users@lists.sourceforge.net
> > 6.
> [16]https://lists.sourceforge.net/lists/listinfo/monetdb-use
> rs
>
> > 7. [17]http://p.sf.net/sfu/www-adobe-com
>
> > 8. mailto:[18]MonetDB-users@lists.sourceforge.net
> > 9.
> [19]https://lists.sourceforge.net/lists/listinfo/monetdb-use
> rs
>
> >
> ---------------------------------------------------------------
> ---------------
> > Apps built with the Adobe(R) Flex(R) framework and Flex
> Builder(TM) are
> > powering Web 2.0 with engaging, cross-platform capabilities.
> Quickly and
> > easily build your RIAs with Flex Builder, the
> Eclipse(TM)based development
> > software that enables intelligent coding and step-through
> debugging.
> > Download the free 60 day trial.
> [20]http://p.sf.net/sfu/www-adobe-com
> > _______________________________________________
> > MonetDB-users mailing list
> > [21]MonetDB-users@lists.sourceforge.net
> >
> [22]https://lists.sourceforge.net/lists/listinfo/monetdb-users
>
> --
> Niels Nes, Centre for Mathematics and Computer Science (CWI)
> Science Park 123, 1098 XG Amsterdam, The Netherlands
> room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312
> url: [23]http://www.cwi.nl/~niels e-mail:
> [24]Niels.Nes@cwi.nl
>
> ---------------------------------------------------------------
> ---------------
> Apps built with the Adobe(R) Flex(R) framework and Flex
> Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities.
> Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based
> development
> software that enables intelligent coding and step-through
> debugging.
> Download the free 60 day trial.
> [25]http://p.sf.net/sfu/www-adobe-com
> _______________________________________________
> MonetDB-users mailing list
> [26]MonetDB-users@lists.sourceforge.net
> [27]https://lists.sourceforge.net/lists/listinfo/monetdb-users
>
> References
>
> 1. mailto:Niels.Nes@cwi.nl
> 7. https://lists.sourceforge.net/lists/listinfo/monetdb-user
> 10. https://lists.sourceforge.net/lists/listinfo/monetdb-user
> 11. mailto:yuesheng8@gmail.com
> 12. mailto:lsidir@gmail.com
> 13. mailto:yuesheng8@gmail.com
> 14. http://p.sf.net/sfu/www-adobe-com
> 15. mailto:MonetDB-users@lists.sourceforge.net
> 16. https://lists.sourceforge.net/lists/listinfo/monetdb-users
> 17. http://p.sf.net/sfu/www-adobe-com
> 18. mailto:MonetDB-users@lists.sourceforge.net
> 19. https://lists.sourceforge.net/lists/listinfo/monetdb-users
> 20. http://p.sf.net/sfu/www-adobe-com
> 21. mailto:MonetDB-users@lists.sourceforge.net
> 22. https://lists.sourceforge.net/lists/listinfo/monetdb-users
> 23. http://www.cwi.nl/~niels
> 24. mailto:Niels.Nes@cwi.nl
> 25. http://p.sf.net/sfu/www-adobe-com
> 26. mailto:MonetDB-users@lists.sourceforge.net
> 27. https://lists.sourceforge.net/lists/listinfo/monetdb-users