- users-list - monetdb.org

Fwd: Re: Provide --password option for msqldump tool
by Sherzod Mutalov 16 Jul '15

16 Jul '15

I added "password" option to mclient.c and msqldump.c, you can get changed sources from: https://gist.github.com/shmutalov/d3695ad67ad5195eeee6 I know, it is dangerous, but we need this option in our application. If you want I can provide mclient binaryies for Win7 x64.

1 0

Provide --password option for msqldump tool
by Aravind S Reddy 16 Jul '15

16 Jul '15

Hello Team, We were trying to dump multiple tables through 'msqldump' tool, and restoring it in different monetdb instance in a programatic way. But the difficulty here is, I do not see an option to pass the --password to 'msqldump' tool, where as it prompts for the password and accepts input from STDIN. Due to this, we are not able achieve our requirement programatically. And it would be good if -t(--table) option accepts multiple table names, through which we can dump the selected tables in a single shot. Regards, Aravind.S

4 5

Cluster advantages MonetDB
by Luciano Sasso 15 Jul '15

15 Jul '15

Hi, What the cluster Advantages MonetDB, If a query in shard tables in a cluster can not be ordered, what the advantage of using a MonetDB cluster? besides the replication in case of failure. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. http://www.avast.com

5 5

Job openings at MonetDB Solutions
by Ying Zhang 14 Jul '15

14 Jul '15

Dear all, MonetDB Solutions, the supporting company of MonetDB, is looking for core database developers. If you’re interested, please see the job description here: https://www.monetdbsolutions.com/company/careers Regards, Jennie

1 0

Re: User for multiple schemas
by Ying Zhang 14 Jul '15

14 Jul '15

Hai Brian, Glad to know it helps. We’ve found the cause of the problem I mentioned. It’s not random. It has something to do with changing the ROLE of a USER, and then the privileges of that ROLE seem not properly handled. We’re working on improving the documentation, extending the test suite, and fixing bugs concerning privileges. Regards, Jennie > On Jul 14, 2015, at 12:10, Brian Hood <brianh6854(a)googlemail.com> wrote: > > Hi Jennie, > > That was actually what i was looking for thanks for that i didn't realize that you could have more than one create schema schema_name AUTHORIZATION per_user > > I'll also see if i encounter the bug and if i can find anything reproducible. > > Regards, > > Brian Hood > > > On Tue, Jul 14, 2015 at 10:03 AM, Ying Zhang <Y.Zhang(a)cwi.nl> wrote: > Hello Brian, > > What authorisation are you looking for? MonetDB does support authorise one user as the owner of multiple schema’s, e.g.: > > create user brian with password ’brian123' name ’brian' schema sys; > create schema schema1 AUTHORIZATION brian; > create schema schema2 AUTHORIZATION brian; > > Not sure if this is what you’re looking for. > > However, we have noticed a problem (in the current default branch, probably also exists in Jul2015) in handling the privileges. After a user has been authorised as the owner of a schema, the user can CREATE, INSERT, UPDATE, … tables, however, *sometimes* the user can’t SELECT from his/her own tables. This seems to happen at random, which makes it extremely difficult to debug. However, restarting the MonetDB server seems to temporarily resolve the problem. > > We’re working on improving handling privileges in MonetDB. If you notice any errors and missing features, please let us know. > > Regards, > > Jennie > > > On Jun 23, 2015, at 08:05, Stefan Manegold <Stefan.Manegold(a)cwi.nl> wrote: > > > > Hi Brian, > > > > If i'm not mistaken and recall correctly, access rights / authorization are per table, not per schema - at least in MonetDB; hence, you'd have to grant access for each table in other than the user's schema separately. > > > > Best, > > Stefan > > > > > > On June 22, 2015 8:45:08 PM CEST, Brian Hood <brianh6854(a)googlemail.com> wrote: > > Hi All, > > > > There is something I have pondered on is there a way to have a username that can be authorized for multiple schemas under your dbfarm. > > > > If not what is the reason ? > > > > I haven't seen a way to do this so far. > > > > Regards, > > > > Brian Hood > > > > > > > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > > > > -- > > | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | > > | www.CWI.nl/~manegold/ | Science Park 123 (L321) | > > | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > >

1 0

User for multiple schemas
by Brian Hood 14 Jul '15

14 Jul '15

Hi All, There is something I have pondered on is there a way to have a username that can be authorized for multiple schemas under your dbfarm. If not what is the reason ? I haven't seen a way to do this so far. Regards, Brian Hood

3 2

How do you generate documentation from MonetDB source files?
by Jon Mulieri 12 Jul '15

12 Jul '15

I have long appreciated the wealth of information in the comments of source files throughout the code base. It is apparently formatted for some documentation extraction. Can someone provide instructions for generating the documentation from the source? Thanks! Jon

3 2

Reproducing EmbeddedR results from MonetDB.org site
by George, Glover E ERDC-RDE-ITL-MS 10 Jul '15

10 Jul '15

Hi all, I’ve recently been profiling several techniques for a workflow that we’ve been trying to improve here at USACE. Originally we used python scripts with sqlite, but we ran into scalability problems on large data sets. This led us to MonetDB, with the promise of columnar-based analysis and the hope of both parallel query’s “under-the-hood” and possibly a distributed workflow across an HPC system. Looking at my profiling results, this has led me to a number of questions that hopefully you all can help us with. I have a single table from the TPC-H benchmark – lineitem – populated with 360million entries: "l_orderkey" INTEGER NOT NULL, "l_partkey" INTEGER NOT NULL, "l_suppkey" INTEGER NOT NULL, "l_linenumber" INTEGER NOT NULL, "l_quantity" DECIMAL(15,2) NOT NULL, "l_extendedprice" DECIMAL(15,2) NOT NULL, "l_discount" DECIMAL(15,2) NOT NULL, "l_tax" DECIMAL(15,2) NOT NULL, "l_returnflag" CHAR(1) NOT NULL, "l_linestatus" CHAR(1) NOT NULL, "l_shipdate" DATE NOT NULL, "l_commitdate" DATE NOT NULL, "l_receiptdate" DATE NOT NULL, "l_shipinstruct" CHAR(25) NOT NULL, "l_shipmode" CHAR(10) NOT NULL, "l_comment" VARCHAR(44) NOT NULL, System Setup: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz - 16 cores (HT off) / 256G RAM MonetDB version 11.19.15 R version 3.2.0 1. I have followed the instructions here: https://www.monetdb.org/content/embedded-r-monetdb and tried to reproduce these results. I only attempt to reproduce the results from R (reading the data from CSV since – as noted in a previous email to this list – R fails to read in the 360 million rows using MonetDB.R but works fine reading the same amount from CSV), MonetDB, as well as MonetDB with Embedded R. - I compile both R and MonetDB from source. When building R, I include the option to build the R shared library (libR.so) and when compiling MonetDB I include the option for embedded R. - When creating the MonetDB database, I set the option for embedr=true. - The R function from the above URL is used, as well as the same SQL query – and it works…. However, the performance is much worse than using R alone. I know I must be overlooking something. Please see the image at https://goo.gl/wUvB2J (png on Google Drive). The X-axis is the number of rows, and the Y axis is the time in seconds. As you can see, embedded R is unexpectedly much worse than the other two. 2. While running the quantile function, only one core is active – whether embeddedR or just MonetDB’s. 3. When running the following query (after altering the table to include a new column z, of course): sql>update lineitem set z=l_extendedprice / l_quantity; 360011594 affected rows (3m 49s) … Multiple cores are active for the first 10 seconds, then it resorts to single core. A. What am I not understanding about MonetDB’s ability to use multiple cores? The only time I really see it use multiple cores seems to be when doing “copy into”. B. Can someone provide assistance in trying to replicate the Embedded R results? Should it be running in parallel? Or might something in my setup be configured incorrectly. Cheers!

3 3

(no subject)
by musa 10 Jul '15

10 Jul '15

Can someone please provide a minimal example (how to start the server) for running the embedded R feature in windows? Regards, mpa

3 2

RE: Concurrency test causes crash of MonetDB 5 server on Windows
by michael.witt＠uniserv.com 08 Jul '15

08 Jul '15

Hello, Guillaume: Thanks for the tip avoiding concurrency conflicts. We already considered that and changed to code to avoid such conflicts. In the meanwhile I tested both on Windows and Linux machines and now it is clear that it is a problem specific to the Windows builds of MonetDB. The mserver5 on Windows starts with 2 threads serving requests as stated in the console after starting... # Serving database 'uniserv', using 2 threads Then I found a statement about the thread implementation on Windows by Stefan Mangegold: https://www.monetdb.org/pipermail/developers-list/2010-July/003492.html "However, we do not know how well the multi-threading works on Windows. We rely on pthreads, and their implementation on Windows might not be as efficient and/or stable as on Unix-like systems ..." This is absolutely true! My tests show that mserver5 started with 2 threads always crashes after some time firing concurrent read requests. By the way, the windows machine has 2 physical and 2 logical cores. But if I do the same on a Linux machine (also with 2 threads) Mserver5 runs stable. What is the solution to stabilize Mserver5 on Windows? Set the number of threads to 1 by setting the start option: --set gdk_nr_threads=1 In the M5server.bat script. After setting that, Mserver5 run stable on Windows, too. Does anyone know better solutions? Kind regards, Michael -----Original Message----- From: users-list [mailto:users-list-bounces+michael.witt=uniserv.com@monetdb.org] On Behalf Of Guillaume Bassot Sent: Dienstag, 30. Juni 2015 10:26 To: Communication channel for MonetDB users Subject: Re: Concurrency test causes crash of MonetDB 5 server Hello Michael, From my experience, all "write" request executed concurrently always end with concurrency conflict. I'm talking of heavey request during several minutes, so it's easy to get concurrent access. Since i only use monetdb with a single process, stability was greatly improved. Regards Guillaume BASSOT Web Developer +33 (0)1 55 48 02 82 gbassot(a)inbox.fr <mailto:gbassot@inbox.fr> 30, rue Vincent Moris 92240 Malakoff - France Fax : +33 (0)1 55 48 02 79 2015-06-30 10:09 GMT+02:00 <michael.witt(a)uniserv.com <mailto:michael.witt@uniserv.com> >: Hi all, It’s a pity that no one answered the question I asked a few weeks ago about crashes of MonetDB 5 server under windows 64 Bit. So I think I offer more information about a scenario where the databases crashes. Hopefully someone could give me a hint how to improve stability of MonetDB under Windows 64 Bit with concurrent client connections. We are using Monet DB for a couple of months and facing some graved issues with concurrent queries. The problem is, when we query the database with two or more clients in a parallel, the database crashes! These are the facts: MonetDB server version: v11.19.9 ================================ # MonetDB 5 server v11.19.9 "Oct2014-SP2" # Serving database 'uniserv', using 2 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 8.000 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ <http://www.monetdb.org/> for further information # Listening for connection requests on mapi:monetdb://QADQSHWIN:50000/ # MonetDB/SQL module loaded Server crashes also occur on versions v11.19.8 and v11.19.7. Operation system: ================= Microsoft Windows Server 2008 R2 Datacenter Version 6.1.7601 Service Pack 1 Build 7601 hosted on VMWare Virtual Platform x64-based PC 2x Processor Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 2800 Mhz, 1 Core(s), 1 Logical Processor(s) Installed Physical Memory (RAM) 8,00 GB Available Physical Memory 5,25 GB Total Virtual Memory 16,0 GB Server installation folder: =========================== E:\Program Files\MonetDB\MonetDB5 Server startup script M5server.bat: =================================== @REM Copyright August 2008-2014 MonetDB B.V. @REM All Rights Reserved. @echo off setlocal rem figure out the folder name set MONETDB=%~dp0 rem remove the final backslash from the path set MONETDB=%MONETDB:~0,-1% rem extend the search path with our EXE and DLL folders set PATH=%MONETDB%\bin;%MONETDB%\lib;%MONETDB%\lib\MonetDB5;%PATH% rem prepare the arguments to mserver5 to tell it where to put the dbfarm REM Uniserv: Don't use APPDATA, so set APPDATA to empty String. set APPDATA= if "%APPDATA%" == "" goto usevar rem if the APPDATA variable does exist, put the database there set MONETDBDIR=%APPDATA%\MonetDB5 set MONETDBFARM="--dbpath=%MONETDBDIR%\dbfarm\demo" goto skipusevar :usevar rem if the APPDATA variable does not exist, put the database in the rem installation folder (i.e. default location, so no command line argument) REM Uniserv: Specify the DB installation folder rem set MONETDBDIR=%MONETDB%\var\MonetDB5 set MONETDBDIR=%MONETDB%\db set MONETDBFARM="--dbpath=%MONETDBDIR%\dbfarm\uniserv" :skipusevar rem the SQL log directory used to be in %MONETDBDIR%, but we now rem prefer it inside the dbfarm, so move it there if not exist "%MONETDBDIR%\sql_logs" goto skipmove for /d %%i in ("%MONETDBDIR%"\sql_logs\*) do move "%%i" "%MONETDBDIR%\dbfarm"\%%~ni\sql_logs rmdir "%MONETDBDIR%\sql_logs" :skipmove rem start the real server "%MONETDB%\bin\mserver5.exe" --set "prefix=%MONETDB%" --set mapi_open=true --set mapi_port=50000 --set max_clients=1000 --set "exec_prefix=%MONETDB%" %MONETDBFARM% %* if ERRORLEVEL 1 pause endlocal MonetDB JDBC Client: ================ We query the database via the latest MonetDB JDBC driver loaded from the central Maven repository: <groupId>monetdb</groupId> <artifactId>monetdb-jdbc</artifactId> <version>2.14</version> Database tables: ================ We create tables called "rule_result_<n>" CREATE TABLE "dq_scorecard"."rule_results_1" ("id" BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT, "record_id" VARCHAR(1024), "rule_1" INT, "rule_2" INT, ...); Each table has about 50+ INT columns. Query scenarios: ================ We have concurrent read access via JDBC driver and Spring Data Access Objects, the query look like that: *Client 1: 50 aggregate queries on 50 different columns on one database table like that: SELECT AVG("rule_n") FROM "dq_scorecard"."rule_results_1" WHERE "rule_n" >= 0 *Client 2: 50 aggregate queries on 50 different columns on another database table like that: SELECT AVG("rule_n") FROM "dq_scorecard"."rule_results_2" WHERE "rule_n" >= 0 *Client 3: 50 aggregate queries on 50 different columns on another database table like that: SELECT AVG("rule_n") FROM "dq_scorecard"."rule_results_3" WHERE "rule_n" >= 0 All clients run concurrently, at the moment we don't use a connection pool and create a connection for every query. During concurrent access the M5server crashes. The server crashes when each table has 100 records! Our production tables can have up to 50. 000. 000 of records! Kind regards, Michael From: users-list [mailto:users-list-bounces+michael.witt <mailto:users-list-bounces%2Bmichael.witt> =uniserv.com(a)monetdb.org <mailto:uniserv.com@monetdb.org> ] On Behalf Of michael.witt(a)uniserv.com <mailto:michael.witt@uniserv.com> Sent: Donnerstag, 28. Mai 2015 14:32 To: users-list(a)monetdb.org <mailto:users-list@monetdb.org> Subject: Concurrency test causes crash of MonetDB 5 server Hi, we experience a crash of MonetDB 5 server during tests of a multi-threaded client access via JDBC-Driver and MapiSocket: The application is already optimized to avoid “concurrency conflicts” because of the “Optimistic Concurrency Control” concept. The test simulates client access via two threads. All update operations (INSERT, UPDATE, DELETE, DROP TABLE, CREATE TABLE) are done in serial. All read operations are done in parallel to other read operations or to update operations. The test causes a crash of mserver5.exe after a few minutes. If the test uses just one thread, all read and update operations are done in serial and the server does not crash. What is the rule of thumb to avoid crashes when using several client threads? Crash details: Problem signature: Problem Event Name: APPCRASH Application Name: mserver5.exe Application Version: 0.0.0.0 Application Timestamp: 55389f58 Fault Module Name: StackHash_8ec6 Fault Module Version: 6.1.7601.18247 Fault Module Timestamp: 521eaf24 Exception Code: c0000374 Exception Offset: 00000000000c4102 OS Version: 6.1.7601.2.1.0.144.8 Locale ID: 1031 Additional Information 1: 8ec6 Additional Information 2: 8ec62499039d1d8234b025a62c725eb7 Additional Information 3: f8f3 Additional Information 4: f8f3e520847c0684b15d4080171779ef # MonetDB 5 server v11.19.9 "Oct2014-SP2" # Serving database 'uniserv', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 10.000 GiB available main-memory. .. # MonetDB/SQL module loaded Michael Witt _______________________________________________ users-list mailing list users-list(a)monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list

2 1