failed to import record too long: Loading 1.5 Terabyte table
Hello, I'm trying to load a table with 1.5 Terabytes of data, the command I'm using COPY INTO lineitem FROM '/vol1/tpch-dbgen/2000/lineitem.tbl' USING DELIMITERS '|', ' ' LOCKED; The physical memory I'm using is 256G, the swap is 2.2 terabytes, the dbfarm is located within area has 4 Terabytes of free storage. The message I'm getting: "Failed to import table 'lineitem', record too long" So what could be a possible solution to this problem, Thank you a lot in advance, Mahmoud Mohsen
Dear Mahmoud, how do the first two lines of your lineitem.tbl look like? If the first line is not actual data but headers, you'd need to explicitly skip that using the OFFSET feature of COPY INTO. Why we (nowadays?) get this possibly misleading error message, I don't know ... (I assume you're using the latest Jul2017-SP1 release, aren't you?) Best, Stefan ----- On Aug 18, 2017, at 11:54 AM, Mahmoud Mohsen mmohsen35@gmail.com wrote:
Hello,
I'm trying to load a table with 1.5 Terabytes of data, the command I'm using
COPY INTO lineitem FROM '/vol1/tpch-dbgen/2000/lineitem.tbl' USING DELIMITERS '|', ' ' LOCKED;
The physical memory I'm using is 256G, the swap is 2.2 terabytes, the dbfarm is located within area has 4 Terabytes of free storage.
The message I'm getting:
"Failed to import table 'lineitem', record too long"
So what could be a possible solution to this problem,
Thank you a lot in advance,
Mahmoud Mohsen
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Dear Stefan mailto:users-list%40monetdb.org?Subject=Re:%20Re%3A%20failed%20to%20import%20record%20too%20long%3A%20Loading%201.5%20Terabyte%20table&In-Reply-To=%3C638412639.7722845.1503059978768.JavaMail.zimbra%40cwi.nl%3E, Thank you very much for your quick reply, so this is the first line of the file "lineitem.tbl", so no need to use an offset I think 1|310378689|15378705|1|17|29786.89|0.04|0.02|N|O|1996-03-13|1996-02-12|1996-03-22|DELIVER IN PERSON|TRUCK|egular courts above the| as for the version, I am using: MonetDB Database Server Toolkit v1.1 (Dec2016-SP5). So can this may be the cause of the error... an old version? I am looking forward hearing from you, Best regards, Mahmoud Mohsen
I haven't seen this specific problem, but I have had problems importing large tables with other errors. Would it be possible for MonetDB to return a more informative error message? Maybe the line number of the offending line?? -----Original Message----- From: users-list [mailto:users-list-bounces+david.b.anderson=citi.com@monetdb.org] On Behalf Of Mahmoud Mohsen Sent: Friday, August 18, 2017 5:55 AM To: users-list@monetdb.org Cc: gabrielcampero@gmail.com Subject: failed to import record too long: Loading 1.5 Terabyte table Hello, I'm trying to load a table with 1.5 Terabytes of data, the command I'm using COPY INTO lineitem FROM '/vol1/tpch-dbgen/2000/lineitem.tbl' USING DELIMITERS '|', ' ' LOCKED; The physical memory I'm using is 256G, the swap is 2.2 terabytes, the dbfarm is located within area has 4 Terabytes of free storage. The message I'm getting: "Failed to import table 'lineitem', record too long" So what could be a possible solution to this problem, Thank you a lot in advance, Mahmoud Mohsen _______________________________________________ users-list mailing list users-list@monetdb.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailman_listinfo_users-2Dlist&d=DwICAg&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=8_GYjk1edsyLJlNaxMxYxBJsviF3JXYvwDK42uy5KWU&m=CkcfW7oowAoSH8VpmERxM0Kf4goRTtDOAFCfgtvYrqg&s=AU5JrBUICoGCnVB6DvtPRgs_NmlIc3ze0bED_JJa4Sk&e=
participants (3)
-
Anderson, David B -
Mahmoud Mohsen -
Stefan Manegold