- users-list - monetdb.org

Memory management, 8 years later
by Roberto Cornacchia 28 Jun '18

28 Jun '18

Hi all, 8 yeas after this interesting thread about using ulimit to limit the data segment size ( https://www.monetdb.org/pipermail/users-list/2010-March/004007.html) a few things have changed and I find myself a bit lost again about memory management. At some point (can't remember when) mserver5 stopped respecting ulimit. No matter how I change the limits, it always sees all available physical ram. Does any other mechanism exist? The issue I'm facing all the time, and for which I find very little documentation, is: - multiple dbfarms, possibly tens of mserver5 instances, running on the same (virtual) machine, plus other applications - Not necessarily, but most times each dmfarm is served by a Docker container - CPU and memory resources need to be share fairly (ideally, controlling priorities) - No mserver5 should ever be killed because of memory issues Now, especially when using Docker, there are means to control the maximum memory size allowed to each process. However, the resolution in case of overusage is to kill the process. That is of course not a solution. What I'm looking for is: can I set a cap to resources *at process level*, in a way that mserver5 will just *see* those, make the best out of them, and never try to use more? (yes, one way would be a VM, but far from ideal for me) Does anyone have experience with / tips about multi-dbfarm environments? Thanks! Roberto

2 2

Troublesome Query
by Doherty, Adam 23 Jun '18

23 Jun '18

Hi all: I'm trying to figure how to calculate the time delta between two events. My events have a timestamp, event source, event type, and an event state and event id. So an event looks like - event_date_time event_source event_type event_state event_id 2018-06-22 09:05:07.000000 MODULE123 TYPEABC ACK 24567890 2018-06-22 11:32:14.000000 MODULE123 TYPEABC UNACK 24812266 This makes up a unique "event", start to finish. However I have hundreds or thousands of entries between and I want be able to group them together in their unique pairings as above. I'm not sure how to wrap my head around this one. Adam Doherty

2 1

Performance issues when streaming records from table.
by Manuel Pagliai 20 Jun '18

20 Jun '18

We are interested in adding support to import records from Monetdb tables in the ETL we are developing. We have performance issues when executing queries like: select col0, col1, ... coln from <table> on tables with a large number of records. What I am concerned about is the time spent upfront, before the first record is retrieved: once records become available we can start processing them. For example, on my machine, executing a executing a select * from <table> query on a 68 million records table takes around 24 seconds for the query to execute, before it is actually possible to start retrieving records from the server response. Queries are submitted via the JDBC driver. Is there a more efficient way of streaming records from a table ? Thanks, Best regards, *Manuel Pagliai* Visokio Ltd www.visokio.com +44 20 7351 4646

1 0

Announcement: New Mar2018-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 20 Jun '18

20 Jun '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Mar2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Mar 2018-SP1 bugfix release (11.29.7) Build Environment * On Linux and Unix, try not to link to libraries multiple times: that only causes the code (and worse, the variables in those libraries) to be included multiple times. On Windows, we do need to link to libraries multiple times (once for each DLL we create if the DLL references anything from the library) but that doesn't cause the linked library to be loaded multiple times. SQL Frontend * Corrected the definition of view: sys.ids. Bug Fixes * 6562: Sqlitelogictest crash on group by query with not in operator * 6565: Sqlitelogictest crash on complex select query with coalesce call * 6566: Sqlitelogictest unavailable calc.- MAL operations * 6568: Sqlitelogictest crash on complex case query * 6569: Sqlitelogictest select query with not between cause with wrong results * 6570: Sqlitelogictest select coalesce undefined calc * 6572: ordered index Error in optimizer garbageCollector * 6573: Sqlitelogictest crash on complex select query with case statement * 6574: server crashed could not find ordered index while creating a table * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6577: creating temp table kills performance of the original query * 6578: One two-tuple insert gives different results than two single inserts * 6581: Join condition errors. * 6583: Fixed size string concatenation with integer results in fixed size string of size 0 * 6584: SELECT FROM REMOTE TABLE WHERE IS NOT NULL produces wrong result * 6585: Nested Merge tables cause an infinite loop in rel_optimizer * 6587: Sqlitelogictest crash on complex case statement * 6589: Sqlitelogictest crash on complex on complex expression * 6594: Sqlitelogictest crash on complex case statement * 6595: Remote decimal division triggers assertion / returns wrong answer * 6598: Python 3.4 not supported (due to usage of Py_DecodeLocale) * 6600: Sqlitelogictest queries fail to execute * 6601: "where is null" clause on remote table causes problem with next query * 6602: Sqlitelogictest wrong results in IN query * 6603: Sqlitelogictest: Aggregation query with distinct clause produces duplicated rows

1 0

pymonetdb or Python UDF
by Doherty, Adam 17 Jun '18

17 Jun '18

Hello all: This is more of curiosity but I have data in my events table currently trapped in a nvarchar field. I am wondering if I can build a table and do execute a select into with a python UDF pulling data from the existing fields plus the values of my nvarchar field. My string data has the following format - "key1=value, key2=val, key3=value"... I want to break out the data first by splitting on the , and then the = Keys would be the columns in the table. I can do this with a python script and pymonetdb but a UDF would be faster. Adam Doherty Process Control Systems Administrator Kearl Technical, Imperial 505 Quarry Park Blvd SE W3B.224 Calgary, AB T2C 5N1 adam.doherty(a)esso.ca<mailto:adam.doherty@esso.ca> P 587.476.4640<tel:+15874764640> M 403.869.1636<tel:+14038691636> imperialoil.ca<http://www.imperialoil.ca/Canada-English/default.aspx> | Twitter<http://www.twitter.com/imperialoil> | YouTube<http://www.youtube.com/imperialoil>

3 3

joinable generate_series with arguments from a table
by Stefan de Konink 13 Jun '18

13 Jun '18

Goodmorning, At this moment there is no functional compatibility between the PostgreSQL and MonetDB with respect to the generate_series function. The range in PostgreSQL is inclusive, in MonetDB exclusive. More importantly generate_series can't be used with argument that is joined from a different table, MonetDB just seems to support a subquery, hence joining that back without a primary is impossible. PostgreSQL; select generate_series(a, b), a from (select 1 as a, 2 as b) as a; generate_series | a -----------------+--- 1 | 1 2 | 1 (2 rows) MonetDB; select * from generate_series((select 1 as a, 2 as b)); +-------+ | value | +=======+ | 1 | +-------+ If I would like to functionally achieve the following; I have list that states the speeds over partial segments, identified by linear referencing. From 10 meter to 12 meter, the distance was 2 meter and speed was 2s. 10 2 2 I would like to expand these values to segments of one meter, hence end up with a table that would state: 10 2 2 11 2 2 12 2 2 Now I am aware of the Python bindings and I could achieve what I wanted by the function below, but I would find generate_series still very practical. DROP FUNCTION expand_per_meter; CREATE FUNCTION expand_per_meter(trip_hash BIGINT, userstopcode STRING, distancesincelastuserstop INTEGER, s INTEGER, t INTEGER) RETURNS TABLE(trip_hash BIGINT, userstopcode STRING, distancesincelastuserstop INTEGER, s INTEGER, t INTEGER) LANGUAGE PYTHON { result = dict() result['trip_hash'] = [] result['userstopcode'] = [] result['distancesincelastuserstop'] = [] result['s'] = [] result['t'] = [] for i in range(0, len(trip_hash)): expanded = range(distancesincelastuserstop[i], distancesincelastuserstop[i] + s[i] + 1) expand = len(expanded) result['trip_hash'] += [trip_hash[i]] * expand result['userstopcode'] += [userstopcode[i]] * expand result['distancesincelastuserstop'] += expanded result['s'] += [s[i]] * expand result['t'] += [t[i]] * expand return result }; -- Stefan

4 4

Re: Combine a table-returning function with the source table
by Mark Clements 13 Jun '18

13 Jun '18

Stefan, This is a good question. For the specific example, you are quite correct. However, the motivation for the question came from using the MonetDB-rmath extension and embedded R: often it is more efficient to calculate a number of values (e.g. the confidence intervals and p-values) together and then return all of the values as a table. This probably also reflects that I am a statistician and an old R user. The recent article by Raasveldt suggested that analysis objects from Python could be stored natively rather than being serialised BLOBs. This could be further extended to storing R objects natively. Members of the objects could then be extracted as they are needed. Admittedly, this would lose many relational advantages. My suggestion is halfway, where R and Python can be used for analytics, with simple relational storage of the results. Actually, is there any way to optimise a lateral call using BATs? -- Mark On 13 Jun 2018 17:43, Stefan Manegold <Stefan.Manegold(a)cwi.nl> wrote: Hi Mark, just out of curiosity, (in the given case/example) what's wrong with a simple select *, mean-1.96*sd as lci, mean+1.96*sd as uci from test; or create or replace function ci(se double) returns double begin return 1.96*se; end; select *, mean-ci(sd) as lci, mean+ci(sd) as uci from test; in case you really want/need to avoid the "redundant" occurrence or "1.96" ... Thanks! Best, Stefan ----- On Jun 13, 2018, at 5:30 PM, Mark Clements mark.clements(a)ki.se wrote: > Jennie, > > Thank you for replying to this inquiry - it's appreciated. > > Your solution gives the right result - but it requires changing the > function call and will not generalise easily to other column types or > other number of covariates. If we had another table such as: > > drop table test; > create table test as select 1 as id, 2 as sex, cast(3 as double) as > mean, cast(4.0 as double) as sd; > -- and repeating the function > create or replace function cis(mu double, se double) returns table(lci > double, uci double) > begin return select mu-1.96*se, mu+1.96*se; > end; > > we can get all of the test table combined with the confidence interval by: > > -- row_number() > select * from > (select *, row_number() over() as rid from test) as t1 > natural join > (select *, row_number() over() as rid from cis((select mean,sd from > test))) as t2; > -- or using lateral > select * from test, lateral cis(test.mean,test.sd); > > The row_number solution is fast, and the lateral solution is simple. I > was hoping for: > > select *, cis(mean,test).* from test; > > Again, thank you for your help. > > -- Mark > > On 06/08/2018 01:53 PM, Ying Zhang wrote: >> Hai Mark, >> >> I’m just wonder if this gives you wat you want: >> >> create function cis(mu double, se double) >> returns table(mu_out double, lci double, uci double) >> begin >> return select mu, mu-1.96*se, mu+1.96*se; >> end; >> >> Regards, >> Jennie > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/<http://www.CWI.nl/~manegold/> | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list(a)monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

1 0

Combine a table-returning function with the source table
by Mark Clements 13 Jun '18

13 Jun '18

For using the MonetDB-rmath extension, I was interested in using table-returning functions to return multiple values. Is there a fast and elegant way to combine the results with the source table in SQL? To make this concrete, consider some test data and a function to calculate a 95% confidence interval from a mean and standard error: drop table test; create table test as select cast(value as double) as value from generate_series(0,5); drop function cis(double,double); create function cis(mu double, se double) returns table(lci double, uci double) begin return select mu-1.96*se, mu+1.96*se; end; We can then call the table-returning function: select * from cis((select value, 1.0 from test)); If we want to re-join the results with the source table, we could use row_number() or use lateral: -- Using row_number: select * from (select *, row_number() over() as rid from test) as t1 natural join (select *, row_number() over() as rid from cis((select value,1.0 from test))) as t2; -- Using lateral: select * from test, lateral cis(test.value, cast(1.0 as double)) as t2; I believe that the latter will lead to a slow loop, even when BAT functions are available. The table-returning function can be used in the select statement only for a scalar expression: select cis(0,1).*; -- ok select cis(0,1).* from test; -- Error: no such operator 'cis' select cis(value,1).* from test; -- Error: identifier 'value' unknown Again, is there a fast and elegant way to combine the source table with a table-returning function in SQL? Kindly, Mark.

3 4

Temporal Function Issues
by Doherty, Adam 13 Jun '18

13 Jun '18

Hello all: I am trying to find the most efficient way to filter results for those matching today only. My events table has 550 million rows and I want a view as follows- CREATE VIEW events_today AS SELECT * FROM events WHERE str_to_date(timestamp_to_str(event_timestamp, '%Y-%m-%d'), '%Y-%m-%d') = CURRENT_DATE which should give me all events with today's date. Running the SELECT by itself also never returns. However, the query just hangs without ever returning. I'm running 11.29.3 (Mar2018) on CentOS 7 with 128GB and 16 Cores. Adam

2 2

High Memory Usage - Sever Not Releasing Resources?
by Doherty, Adam 08 Jun '18

08 Jun '18

Hi all: I ran into an issue yesterday when I ran a simple group count query - The query took 7 minutes, 13 seconds to run. Query was as follows - SELECT zone, count(*) FROM chronicle.events WHERE event_type = 'mb' GROUP BY zone This table, which a merge table currently has 546 million events. The merge table is made up of currently two merge table each representing a year (events_2018, and events_2017). Those merge tables are each made up of four tables, events_year_zone_(1-4) I have a scheduled task which inserts events into the relevant tables every 5 minutes from 5 Microsoft SQL Servers. Roughly 12,000 events broken across 5 inserts (one for each server). The import process is handled with a C# Windows service that utilizes ODBC to talk to MonetDB. The server is currently MonetDB v11.27.11 (July2017-SP3) running on VMware 6.5 in a CentOS 7 VM with 64GB of RAM, and 16 CPU cores. When I ran into this issue yesterday, the mserver5 process was utilizing 90% of available RAM. This was making my insertion task take longer and longer each time, and simple queries like the one above took minutes. Logged into the server via SSH, the input lag as a result of the almost total resource usage caused me to reboot the server. I restarted the VM, and the same query took 20 seconds. I'd like to sort where this issue might be coming from, but I'm not sure where to look. This server is currently part of a proof-of-concept project that will soon be going to production. Any guidance is greatly appreciated. Adam

2 2