GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
The free disk space is 800G. And the query also fail when I run query right after a restart of the MonetDB server. And I'm not very sure what the mserver5 size means. When I run the query, the log file provide some information below, is that your mentioned? #GDKmremap(2465464320) fails, try to free up space [memory in use=149750256,virtual memory in use=4497801712] Also, I'm sorry that the information(query statement and result) I put in is incomplete by my negligence. I copy the query statement and result and error message again as followed: result and error message: GDK reported error. MT_mremap: mremap(0x7ffc80000000,2054553600,2465464320) failed !OS: Cannot allocate memory HEAPextend: failed to extend to 2465464320 for 55/5577.tail: GDKmremap() failed +-------------+--------------+-------+ | c_last_name | c_first_name | sales | +=============+==============+=======+ +-------------+--------------+-------+ 0 tuples query statement: with frequent_ss_items as ( select substr(i_item_desc,1,30) itemdesc ,i_item_sk item_sk ,d_date solddate ,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999+1,1999+2,1999+3) group by itemdesc ,i_item_sk ,d_date having count(*) >4 ), max_store_sales as ( select max(csales) tpcds_cmax from ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk ) x ), best_ss_customer as ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * ( select * from max_store_sales ) ) select sum(sales) from ( select cs_quantity*cs_list_price sales from catalog_sales ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) union all select ws_quantity*ws_list_price sales from web_sales ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) y limit 100; with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999 + 1,1999 + 2,1999 + 3) group by itemdesc,i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk) x), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select * from max_store_sales)) select c_last_name,c_first_name,sales from (select c_last_name,c_first_name,sum(cs_quantity*cs_list_price) sales from catalog_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) and cs_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name union all select c_last_name,c_first_name,sum(ws_quantity*ws_list_price) sales from web_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer) and ws_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name) x order by c_last_name,c_first_name,sales limit 100; Thanks! Regards, Rancho
Hi Rancho, thanks for the info! With the size of mserver5, I was aiming at the virtual and memory-resident sizes as reported by, e.g., top, but the numbers for the log are also fine. Given the 8GB RAM of your machine and the 800 GB free disk space, I honestly have no idea why the mremap of less than 2.5 GB files with a mserver5 that is less than 4.5 GB in total size (of which less than 1.5 GB in memory) --- all of this assuming that you indeed run on a 64-bit Ubuntu with a 64-bit MonetDB. Just in case, maybe you can also share the output of `mserver5 --version` ? Do you happen to have any OS limits activated to restrict the size of processes, e.g., via ulimit ? While we haven't experimented with TPC-DS, yet, we regularly run TPC-H up to scale factor 100 (100 GB data) on a 8 GB RAM machine without any problems. Could you possibly share your scripts to run TPC-DS with MonetDB, i.e., in particular your scripts to generate and load the data, and possible as well as the (MonetDB-)SQL code of the (remaining) queries? Then we could ("easily") try to reproduce the problem, in order to analyze and fix it. Thanks! Best, Stefan ----- On Mar 15, 2018, at 10:22 AM, RanchoYuan yuanshijia@ww-it.cn wrote:
The free disk space is 800G. And the query also fail when I run query right after a restart of the MonetDB server. And I'm not very sure what the mserver5 size means. When I run the query, the log file provide some information below, is that your mentioned? #GDKmremap(2465464320) fails, try to free up space [memory in use=149750256,virtual memory in use=4497801712]
Also, I'm sorry that the information(query statement and result) I put in is incomplete by my negligence. I copy the query statement and result and error message again as followed:
result and error message:
GDK reported error. MT_mremap: mremap(0x7ffc80000000,2054553600,2465464320) failed !OS: Cannot allocate memory HEAPextend: failed to extend to 2465464320 for 55/5577.tail: GDKmremap() failed +-------------+--------------+-------+ | c_last_name | c_first_name | sales | +=============+==============+=======+ +-------------+--------------+-------+ 0 tuples
query statement:
with frequent_ss_items as ( select substr(i_item_desc,1,30) itemdesc ,i_item_sk item_sk ,d_date solddate ,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999+1,1999+2,1999+3) group by itemdesc ,i_item_sk ,d_date having count(*) >4 ), max_store_sales as ( select max(csales) tpcds_cmax from ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk ) x ), best_ss_customer as ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * ( select * from max_store_sales ) ) select sum(sales) from ( select cs_quantity*cs_list_price sales from catalog_sales ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) union all select ws_quantity*ws_list_price sales from web_sales ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) y limit 100; with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999 + 1,1999 + 2,1999 + 3) group by itemdesc,i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk) x), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select * from max_store_sales)) select c_last_name,c_first_name,sales from (select c_last_name,c_first_name,sum(cs_quantity*cs_list_price) sales from catalog_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) and cs_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name union all select c_last_name,c_first_name,sum(ws_quantity*ws_list_price) sales from web_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer) and ws_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name) x order by c_last_name,c_first_name,sales limit 100;
Thanks! Regards,
Rancho
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
participants (2)
-
RanchoYuan
-
Stefan Manegold