RE: R UDFs

5 Jul 2016

      ...
4.   unique = numpy.unique(aggr_group)
Mark,

Thanks for the pointers. I was able to build MonetDB with integrated Python, but I haven't been able to get aggregate Python functions to execute in parallel. I was hoping that when I used a Python aggregate function in conjunction with a group by clause, multiple Python interpeters would be launched in parallel to work on subsets of the hidden parameter aggr_group, but it appears that the computation for all elements of aggr_group occurs within a single thread. Is this the expected behavior? 

I tested using a modification of one of your aggregate examples, where I also had "--set gdk_nr_threads 4 --forcemito" passed into the mserver5 command line arguments.

CREATE AGGREGATE python_pid(val INTEGER) 
RETURNS INTEGER 
LANGUAGE PYTHON {
    import os
    import time
    pid = os.getpid()
    time.sleep(10)
    unique = numpy.unique(aggr_group)
    x = numpy.zeros(shape=(unique.size))
    for i in range(0, unique.size):
        x[i] = pid
    return(x)
};

This always returns the same PID, and I only see one thread running in top.

If I try to use the PYTHON_MAP API,

CREATE AGGREGATE python_map_pid(val INTEGER) 
RETURNS INTEGER 
LANGUAGE PYTHON_MAP {
    import os
    pid = os.getpid()
    unique = numpy.unique(aggr_group)
    x = numpy.zeros(shape=(unique.size))
    for i in range(0, unique.size):
        x[i] = pid
    return(x)
};

I get the following error: 

sql>select fctr, python_map_pid(value) from mini group by fctr;
MALException:pyapi.eval:Python exception
  2.   import os
  3.   pid = os.getpid()
  5.   x = numpy.zeros(shape=(unique.size))
  6.   for i in range(0, unique.size):
global name 'aggr_group' is not defined

So it looks like PYTHON_MAP isn't supposed to be used with aggregate functions? 

Thanks,
Dave

-----Original Message-----
From: users-list [mailto:users-list-bounces+david.b.anderson=citi.com@monetdb.org] On Behalf Of Mark Raasveldt
Sent: Tuesday, June 28, 2016 5:35 PM
To: users-list
Subject: Re: R UDFs

Hey Anderson,

That should already be possible. The configure script performs the following steps:

- Get Python link flags by calling 'python-config --ldflags'
- Get Python include directory by calling 'python -c "from distutils.sysconfig import get_python_inc; get_python_inc();"'
- Get the NumPy include path by calling 'python -c "import numpy; numpy.get_include()"'

It is possible that the wrong python-config or the wrong python is being used in these steps if you are not using the system installation of Python. You can manually specify 'python-config' using --with-pyconfig="/path/to/python-config", and specify Python using --with-python2="/path/to/python2". 

Hope that helps.

Mark

----- Original Message -----
From: "Anderson, David B" 
To: "users-list" 
Sent: Tuesday, June 28, 2016 10:28:50 PM
Subject: RE: R UDFs

Mark,

It looks like you are expecting numpy to be installed in the standard system directories. Are there plans to allow it to be installed in a non-standard location? Our build environment is atypical/old and I need to build many libraries from source and install them in non-system directories. Trying to not hack the configure scripts.

Thanks,
Dave

-----Original Message-----
From: users-list [mailto:users-list-bounces+david.b.anderson=citi.com@monetdb.org] On Behalf Of Mark Raasveldt
Sent: Saturday, June 18, 2016 4:49 AM
To: users-list
Subject: Re: R UDFs

Hey Dave,

I have only tested it with Python 2.7, but I see no reason why it won't work with Python 2.6.

Regards,

Mark

----- Original Message -----
From: "Anderson, David B" 
To: "users-list" 
Sent: Friday, June 17, 2016 6:07:25 PM
Subject: RE: R UDFs

Mark,

Can I use Python 2.6, or does it have to be 2.7?

Dave

-----Original Message-----
From: users-list [mailto:users-list-bounces+david.b.anderson=citi.com@monetdb.org] On Behalf Of Mark Raasveldt
Sent: Friday, June 17, 2016 9:32 AM
To: users-list
Subject: Re: R UDFs

Hey Dave,

In the 'default' branch only Python 2.7 UDFs are supported, Python 3 UDFs are currently in the works so you can choose which version you want to use. If you want to use Python 3 UDFs right now you can use the 'python3udf' branch instead.

Regards,

Mark

----- Original Message -----
From: "Anderson, David B" 
To: "users-list" 
Sent: Thursday, June 16, 2016 6:05:30 PM
Subject: RE: R UDFs

I just read your Embedded Python/NumPy blog post. Excellent!

Is there a preferred/required version of Python? I was looking at the config log and I see it checking for Python2 & Python3. 

-----Original Message-----
From: Anderson, David B [ICG-MKTS] 
Sent: Thursday, June 16, 2016 11:34 AM
To: users-list
Subject: RE: R UDFs

Mark,

Thanks for the info. To confirm, If I write my UDFs in Python, they will be executed in parallel (if required)?

Thanks,
Dave

-----Original Message-----
From: users-list [mailto:users-list-bounces+david.b.anderson=citi.com@monetdb.org] On Behalf Of Mark Raasveldt
Sent: Thursday, June 16, 2016 8:05 AM
To: users-list
Subject: Re: R UDFs

Hey Dave,

There's only a single R interpreter running. When an R function is called a lock is acquired that prevents other R functions from running at the same time. There has been some work to running multiple interpreters in parallel in separate processes for Python UDFs, but this has not been ported to R UDFs yet. 

Regards,

Mark

----- Original Message -----
From: "Anderson, David B" 
To: "users-list" 
Sent: Thursday, June 16, 2016 1:30:28 PM
Subject: R UDFs

All,

Quick question, how many R interpreters are running within MonetDB if you are using R UDFs? Is there a single shared interpreter or are there multiple copies running?

Thanks,
Dave

_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list