Re: Creation of a remote table overhead

Hi Yannis, one more remark. Looking at your original message I get the impression that you're using the remote tables on a one-off basis. So, you're executing a particular distributed query, your Python client decides to move some data, it creates remote tables, accesses them and tears them down again. However, the remote tables were originally conceived more as a permanent thing, something you set up as part of your schema. Would it be possible to rework the Python client to create the remote tables at database initialization time or at startup time? That would sidestep the issue entirely. Joeri On 7/10/20 4:39 PM, Panagiotis Koutsourakis wrote:

Hi Yannis,

The reasoning behind the authentication for remote tables is explained here: https://www.monetdb.org/blog/remote-tables-auth

The authentication scheme is using the same machinery that the server is using to authenticate any client. I don't think that using a different authentication scheme for remote tables is feasible right now. The idea is that the credentials you specify for a remote table need to be stored safely in the local authentication vault so that they can be retrieved and used when a connection needs to be established. This is more or less the same process as creating a new user.

What happens authentication-wise when you create a remote table is the following: - Check if the specified table already has an entry in the vault. If you find one delete it since this can only happen if an earlier transaction that created a table with the same name in the same schema did not commit. - Check if the user specified an already encrypted (hashed) password. If not, hash it. - Perform some simple checks that the password we are storing is indeed encrypted. - Create an entry in the credentials vault. - Commit the changes to disk.

You can try to avoid the hashing step which might be expensive, by providing a pre-computed hash of the remote password as detailed in the 3rd or 5th example in the above page. If you compiled MonetDB with a reasonably recent openssl library, or you are using the pre-compiled binaries we supply, the hash function used should be SHA512. Unfortunately the rest of the steps cannot be avoided.

Hope this helps.

Best regards, Panos.

On 7/10/20 5:22 PM, Ioannis Foufoulas wrote:

...

Probably not, but I ‘m trying to figure out why in versions prior to Aug.2018 I ‘m getting this:

sql>CREATE REMOTE TABLE rmt_tbl (c1 int) ON 'mapi:monetdb://127.0.0.1:50000/voc3/sys/data'; operation successful sql:0.000 opt:0.359 run:0.473 clk:4.634 ms

While in versions after Aug. 2018 I have: sql>CREATE REMOTE TABLE rmt_tbl (c1 int) ON 'mapi:monetdb://127.0.0.1:50000/voc3/sys/data'; operation successful sql:0.000 opt:0.068 run:0.057 clk:87.766 ms

...

On 10 Jul 2020, at 4:51 PM, Ying Zhang wrote:

...

On 10 Jul 2020, at 15:26, Ioannis Foufoulas mailto:johnfouf@di.uoa.gr> wrote:

Hi, I am trying to develop a Python client which sends queries to different Monetdb servers and transfers the results between them using the remote tables functionalities. I’ve seen than my main overhead is execution of CREATE REMOTE TABLE statement. It requires 70-90msec so in cases with many transfers and fast and small queries this makes a difference. After looking at some older versions I’ve seen that the creation of a remote table was very fast before August 2018 (this is when authentication added). Is there any hint on this? Could the. authentication strategy change in someway so that it runs faster or is it possible to select among different authentication methods for the remote tables? Not sure if that’s the cause of the overhead, but better authentication was indeed added since Aug2018 (https://www.monetdb.org/blog/remote-tables-auth https://www.monetdb.org/blog/remote-tables-auth)

Jennie

...

Best, Yannis

_______________________________________________ users-list mailing list users-list@monetdb.org mailto:users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

---

users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list