Re: MonetDB: Jul2017 - Initialize sys.argv, because tensorflow requi...
Hello developers, Taking the opportunity from this commit, I wanted to initiate a discussion with respect the executable names inside the code. A few weeks ago I faced a problem when using the configure options --program-prefix and --program-suffix, that change the executable names. The problem was that the name mserver5 was hard coded and the daemon could not find the actual executable. I see two possible solutions: 1. Implement a mechanism that make a note in the monetdb_config.h 2. Implement a run time mechanism to compute the relevant file names. Solution #1 is probably a bit more difficult because the prefix/suffix transformation of the file names happens during the installation. Solution #2 is already partially implemented in the daemon (see the function get_bin_path in the file mutils.c), but will probably need to be generalized somehow. Comments? Ideas? Best, Panos. Mark Raasveldt @ 2017-06-15 13:22 GMT:
Changeset: 44e0bc7dab59 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=44e0bc7dab59 Modified Files: sql/backends/monet5/UDF/pyapi/pyapi.c Branch: Jul2017 Log Message:
Initialize sys.argv, because tensorflow requires this.
diffs (20 lines):
diff --git a/sql/backends/monet5/UDF/pyapi/pyapi.c b/sql/backends/monet5/UDF/pyapi/pyapi.c --- a/sql/backends/monet5/UDF/pyapi/pyapi.c +++ b/sql/backends/monet5/UDF/pyapi/pyapi.c @@ -1425,9 +1425,16 @@ PYFUNCNAME(PyAPIprelude)(void *ret) { MT_lock_init(&queryLock, "query_lock"); MT_lock_set(&pyapiLock); if (!pyapiInitialized) { +#ifdef IS_PY3K + wchar_t* program = Py_DecodeLocale("mserver5", NULL); + wchar_t* argv[] = { program }; +#else + char* argv[] = {"mserver5"}; +#endif str msg = MAL_SUCCEED; PyObject *tmp; Py_Initialize(); + PySys_SetArgvEx(1, argv, 0); _import_array(); msg = _connection_init(); if (msg != MAL_SUCCEED) { _______________________________________________ checkin-list mailing list checkin-list@monetdb.org https://www.monetdb.org/mailman/listinfo/checkin-list
Do you only want flexibility in the naming of mserver5, or of other binaries as well? Applications with a single binary, should typically be flexible enough to allow for users just renaming them willy-nilly, and get their own names from argv[0] and path to the binary from the CWD environment variable plus argv[0] (or by resolving the link /proc/self/exe on Linux). But when you have multiple binaries that can't work - which makes you wonder if changing binary names at all is such a good idea to begin with. Eyal On 06/16/2017 12:44 PM, Panagiotis Koutsourakis wrote:
Hello developers,
Taking the opportunity from this commit, I wanted to initiate a discussion with respect the executable names inside the code. A few weeks ago I faced a problem when using the configure options --program-prefix and --program-suffix, that change the executable names. The problem was that the name mserver5 was hard coded and the daemon could not find the actual executable.
I see two possible solutions: 1. Implement a mechanism that make a note in the monetdb_config.h 2. Implement a run time mechanism to compute the relevant file names.
Solution #1 is probably a bit more difficult because the prefix/suffix transformation of the file names happens during the installation.
Solution #2 is already partially implemented in the daemon (see the function get_bin_path in the file mutils.c), but will probably need to be generalized somehow.
Comments? Ideas?
Best, Panos.
Mark Raasveldt @ 2017-06-15 13:22 GMT:
Changeset: 44e0bc7dab59 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=44e0bc7dab59 Modified Files: sql/backends/monet5/UDF/pyapi/pyapi.c Branch: Jul2017 Log Message:
Initialize sys.argv, because tensorflow requires this.
diffs (20 lines):
diff --git a/sql/backends/monet5/UDF/pyapi/pyapi.c b/sql/backends/monet5/UDF/pyapi/pyapi.c --- a/sql/backends/monet5/UDF/pyapi/pyapi.c +++ b/sql/backends/monet5/UDF/pyapi/pyapi.c @@ -1425,9 +1425,16 @@ PYFUNCNAME(PyAPIprelude)(void *ret) { MT_lock_init(&queryLock, "query_lock"); MT_lock_set(&pyapiLock); if (!pyapiInitialized) { +#ifdef IS_PY3K + wchar_t* program = Py_DecodeLocale("mserver5", NULL); + wchar_t* argv[] = { program }; +#else + char* argv[] = {"mserver5"}; +#endif str msg = MAL_SUCCEED; PyObject *tmp; Py_Initialize(); + PySys_SetArgvEx(1, argv, 0); _import_array(); msg = _connection_init(); if (msg != MAL_SUCCEED) { _______________________________________________ checkin-list mailing list checkin-list@monetdb.org https://www.monetdb.org/mailman/listinfo/checkin-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
Dear all, IMHO, each individual binary should indeed be robust again manual renaming, i.e., by itself work properly no matter its own name. Also, if we do not disable configure options such as --program-prefix or --program-suffix, we should properly support them, i.e., with that very build, binaries should not only be named but als called using the name as set/modified by these options. I have no preference as to how this can/should be implemented, but it should either work, or we should disable these configure options. Best, Stefan ----- On Jun 16, 2017, at 2:24 PM, Eyal Rozenberg E.Rozenberg@cwi.nl wrote:
Do you only want flexibility in the naming of mserver5, or of other binaries as well?
Applications with a single binary, should typically be flexible enough to allow for users just renaming them willy-nilly, and get their own names from argv[0] and path to the binary from the CWD environment variable plus argv[0] (or by resolving the link /proc/self/exe on Linux). But when you have multiple binaries that can't work - which makes you wonder if changing binary names at all is such a good idea to begin with.
Eyal
On 06/16/2017 12:44 PM, Panagiotis Koutsourakis wrote:
Hello developers,
Taking the opportunity from this commit, I wanted to initiate a discussion with respect the executable names inside the code. A few weeks ago I faced a problem when using the configure options --program-prefix and --program-suffix, that change the executable names. The problem was that the name mserver5 was hard coded and the daemon could not find the actual executable.
I see two possible solutions: 1. Implement a mechanism that make a note in the monetdb_config.h 2. Implement a run time mechanism to compute the relevant file names.
Solution #1 is probably a bit more difficult because the prefix/suffix transformation of the file names happens during the installation.
Solution #2 is already partially implemented in the daemon (see the function get_bin_path in the file mutils.c), but will probably need to be generalized somehow.
Comments? Ideas?
Best, Panos.
Mark Raasveldt @ 2017-06-15 13:22 GMT:
Changeset: 44e0bc7dab59 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=44e0bc7dab59 Modified Files: sql/backends/monet5/UDF/pyapi/pyapi.c Branch: Jul2017 Log Message:
Initialize sys.argv, because tensorflow requires this.
diffs (20 lines):
diff --git a/sql/backends/monet5/UDF/pyapi/pyapi.c b/sql/backends/monet5/UDF/pyapi/pyapi.c --- a/sql/backends/monet5/UDF/pyapi/pyapi.c +++ b/sql/backends/monet5/UDF/pyapi/pyapi.c @@ -1425,9 +1425,16 @@ PYFUNCNAME(PyAPIprelude)(void *ret) { MT_lock_init(&queryLock, "query_lock"); MT_lock_set(&pyapiLock); if (!pyapiInitialized) { +#ifdef IS_PY3K + wchar_t* program = Py_DecodeLocale("mserver5", NULL); + wchar_t* argv[] = { program }; +#else + char* argv[] = {"mserver5"}; +#endif str msg = MAL_SUCCEED; PyObject *tmp; Py_Initialize(); + PySys_SetArgvEx(1, argv, 0); _import_array(); msg = _connection_init(); if (msg != MAL_SUCCEED) { _______________________________________________ checkin-list mailing list checkin-list@monetdb.org https://www.monetdb.org/mailman/listinfo/checkin-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
On 16/06/17 14:58, Stefan Manegold wrote:
Dear all,
IMHO, each individual binary should indeed be robust again manual renaming, i.e., by itself work properly no matter its own name.
Also, if we do not disable configure options such as --program-prefix or --program-suffix, we should properly support them, i.e., with that very build, binaries should not only be named but als called using the name as set/modified by these options. I have no preference as to how this can/should be implemented, but it should either work, or we should disable these configure options.
We should probably have a configure option in monetdbd's configuration file that says which program to call when it wants to start an mserver. The default can then be argv[0] of monetdbd with monetdbd changed to mserver5 (as is done now).
Best, Stefan
----- On Jun 16, 2017, at 2:24 PM, Eyal Rozenberg E.Rozenberg@cwi.nl wrote:
Do you only want flexibility in the naming of mserver5, or of other binaries as well?
Applications with a single binary, should typically be flexible enough to allow for users just renaming them willy-nilly, and get their own names from argv[0] and path to the binary from the CWD environment variable plus argv[0] (or by resolving the link /proc/self/exe on Linux). But when you have multiple binaries that can't work - which makes you wonder if changing binary names at all is such a good idea to begin with.
Eyal
On 06/16/2017 12:44 PM, Panagiotis Koutsourakis wrote:
Hello developers,
Taking the opportunity from this commit, I wanted to initiate a discussion with respect the executable names inside the code. A few weeks ago I faced a problem when using the configure options --program-prefix and --program-suffix, that change the executable names. The problem was that the name mserver5 was hard coded and the daemon could not find the actual executable.
I see two possible solutions: 1. Implement a mechanism that make a note in the monetdb_config.h 2. Implement a run time mechanism to compute the relevant file names.
Solution #1 is probably a bit more difficult because the prefix/suffix transformation of the file names happens during the installation.
Solution #2 is already partially implemented in the daemon (see the function get_bin_path in the file mutils.c), but will probably need to be generalized somehow.
Comments? Ideas?
Best, Panos.
Mark Raasveldt @ 2017-06-15 13:22 GMT:
Changeset: 44e0bc7dab59 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=44e0bc7dab59 Modified Files: sql/backends/monet5/UDF/pyapi/pyapi.c Branch: Jul2017 Log Message:
Initialize sys.argv, because tensorflow requires this.
diffs (20 lines):
diff --git a/sql/backends/monet5/UDF/pyapi/pyapi.c b/sql/backends/monet5/UDF/pyapi/pyapi.c --- a/sql/backends/monet5/UDF/pyapi/pyapi.c +++ b/sql/backends/monet5/UDF/pyapi/pyapi.c @@ -1425,9 +1425,16 @@ PYFUNCNAME(PyAPIprelude)(void *ret) { MT_lock_init(&queryLock, "query_lock"); MT_lock_set(&pyapiLock); if (!pyapiInitialized) { +#ifdef IS_PY3K + wchar_t* program = Py_DecodeLocale("mserver5", NULL); + wchar_t* argv[] = { program }; +#else + char* argv[] = {"mserver5"}; +#endif str msg = MAL_SUCCEED; PyObject *tmp; Py_Initialize(); + PySys_SetArgvEx(1, argv, 0); _import_array(); msg = _connection_init(); if (msg != MAL_SUCCEED) { _______________________________________________ checkin-list mailing list checkin-list@monetdb.org https://www.monetdb.org/mailman/listinfo/checkin-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
-- Sjoerd Mullender
Speaking of configure options, if I build and install MonetDB with a --bin-dir outside the regular --prefix, mserver can't find its scripts directory. The result is that it does not start listening on its requested mapi port, without a clear indication in the logs as to what's going wrong. Maybe we should remove the --bin-dir option altogether because if we want to allow it, figuring out where to find supporting files is much harder. Looking in `dirname "$0"`/../lib is a little inflexible but at least it's clear what's going on. By the way, i can image there is or has been a reason for not logging anything when sql.mal cannot be found but in this scenario it was inconvenient. Would logging a warning clutter the log file with useless warnings? I'll file a bugzilla issue when I have a better idea of what the desired behaviour actually is. What do you think? Joeri On Mon, Jun 19, 2017 at 09:41:11AM +0200, Sjoerd Mullender wrote:
On 16/06/17 14:58, Stefan Manegold wrote:
Dear all,
IMHO, each individual binary should indeed be robust again manual renaming, i.e., by itself work properly no matter its own name.
Also, if we do not disable configure options such as --program-prefix or --program-suffix, we should properly support them, i.e., with that very build, binaries should not only be named but als called using the name as set/modified by these options. I have no preference as to how this can/should be implemented, but it should either work, or we should disable these configure options.
We should probably have a configure option in monetdbd's configuration file that says which program to call when it wants to start an mserver. The default can then be argv[0] of monetdbd with monetdbd changed to mserver5 (as is done now).
Best, Stefan
----- On Jun 16, 2017, at 2:24 PM, Eyal Rozenberg E.Rozenberg@cwi.nl wrote:
Do you only want flexibility in the naming of mserver5, or of other binaries as well?
Applications with a single binary, should typically be flexible enough to allow for users just renaming them willy-nilly, and get their own names from argv[0] and path to the binary from the CWD environment variable plus argv[0] (or by resolving the link /proc/self/exe on Linux). But when you have multiple binaries that can't work - which makes you wonder if changing binary names at all is such a good idea to begin with.
Eyal
On 06/16/2017 12:44 PM, Panagiotis Koutsourakis wrote:
Hello developers,
Taking the opportunity from this commit, I wanted to initiate a discussion with respect the executable names inside the code. A few weeks ago I faced a problem when using the configure options --program-prefix and --program-suffix, that change the executable names. The problem was that the name mserver5 was hard coded and the daemon could not find the actual executable.
I see two possible solutions: 1. Implement a mechanism that make a note in the monetdb_config.h 2. Implement a run time mechanism to compute the relevant file names.
Solution #1 is probably a bit more difficult because the prefix/suffix transformation of the file names happens during the installation.
Solution #2 is already partially implemented in the daemon (see the function get_bin_path in the file mutils.c), but will probably need to be generalized somehow.
Comments? Ideas?
Best, Panos.
Mark Raasveldt @ 2017-06-15 13:22 GMT:
Changeset: 44e0bc7dab59 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=44e0bc7dab59 Modified Files: sql/backends/monet5/UDF/pyapi/pyapi.c Branch: Jul2017 Log Message:
Initialize sys.argv, because tensorflow requires this.
diffs (20 lines):
diff --git a/sql/backends/monet5/UDF/pyapi/pyapi.c b/sql/backends/monet5/UDF/pyapi/pyapi.c --- a/sql/backends/monet5/UDF/pyapi/pyapi.c +++ b/sql/backends/monet5/UDF/pyapi/pyapi.c @@ -1425,9 +1425,16 @@ PYFUNCNAME(PyAPIprelude)(void *ret) { MT_lock_init(&queryLock, "query_lock"); MT_lock_set(&pyapiLock); if (!pyapiInitialized) { +#ifdef IS_PY3K + wchar_t* program = Py_DecodeLocale("mserver5", NULL); + wchar_t* argv[] = { program }; +#else + char* argv[] = {"mserver5"}; +#endif str msg = MAL_SUCCEED; PyObject *tmp; Py_Initialize(); + PySys_SetArgvEx(1, argv, 0); _import_array(); msg = _connection_init(); if (msg != MAL_SUCCEED) { _______________________________________________ checkin-list mailing list checkin-list@monetdb.org https://www.monetdb.org/mailman/listinfo/checkin-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
-- Sjoerd Mullender
_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list
participants (5)
-
Eyal Rozenberg -
Joeri van Ruth -
Panagiotis Koutsourakis -
Sjoerd Mullender -
Stefan Manegold