[MonetDB-users] Comparing TPC-H benchmark results?
Hi, The TPC-H benchmark (http://www.tpc.org/tpch/default.asp) is part of the MonetDB SQL source and the benchmark is run during weekends, as I can see on the TestWeb ( http://monetdb.cwi.nl/testing/projects/monetdb/Current/sql/.mTests03/times.h...). How should I interpret the listed results? According to TPC, results should be stated as a "QphH@Size" value, where "QphH" is the number of queries per hour for the H benchmark, and "Size" is a "scaling factor". On the TestWeb, the results are stated as seconds of execution time. Is it correct to compute the QphH value as follows: Queries 01-22 took 1.985 seconds, so QphH is (3600 / 1.985) * 22 = 39.899 ? What scaling factor is used in the benchmark runs; is this 1? Is it possible to let the TestWeb run with scaling factor 100? Because that's the smallest Size for which results are published on the TPC web site. Did anyone yet try to determine the Price of the test platforms, following the TPC Pricing Spec? TPC-H results normally include a Price/Performance metric in the form of $/QphH@Size, and I guess MonetDB could score particularly well on this metric. Regards, Wouter Scherphof. Utrecht Professional School, The Netherlands.
Hi, The TPC-H benchmark (http://www.tpc.org/tpch/default.asp) is part of the MonetDB SQL source and the benchmark is run during weekends, as I can see on the TestWeb (http://monetdb.cwi.nl/testing/projects/monetdb/Current/sql/.mTests03/times.h...). How should I interpret the listed results? According to TPC, results should be stated as a "QphH@Size" value, where "QphH" is the number of queries per hour for the H benchmark, and "Size" is a "scaling factor". On the TestWeb, the results are stated as seconds of execution time. Is it correct to compute the QphH value as follows: Queries 01-22 took 1.985 seconds, so QphH is (3600 / 1.985) * 22 = 39.899 ? What scaling factor is used in the benchmark runs; is this 1? Is it possible to let the TestWeb run with scaling factor 100? Because that's the smallest Size for which results are published on the TPC web site. Did anyone yet try to determine the Price of the test platforms, following the TPC Pricing Spec? TPC-H results normally include a Price/Performance metric in the form of $/QphH@Size, and I guess MonetDB could score particularly well on this metric. Regards, Wouter Scherphof. Utrecht Professional School, The Netherlands.
Hi, No need to resubmit your question. It was received, but I'm sorry that we've been too busy to answer this question. On 15-09-2006 21:25:15 +0200, Wouter Scherphof wrote:
Hi,
The TPC-H benchmark (http://www.tpc.org/tpch/default.asp) is part of the MonetDB SQL source and the benchmark is run during weekends, as I can see on the TestWeb (http://monetdb.cwi.nl/testing/projects/monetdb/Current/sql/.mTests03/times.h...). How should I interpret the listed results? According to TPC, results should be stated as a "QphH@Size" value, where "QphH" is the number of queries per hour for the H benchmark, and "Size" is a "scaling factor". On the TestWeb, the results are stated as seconds of execution time.
The testweb tpc-h is very very very small, to keep both testweb small in diskspace, and to keep the test time reasonable on our slower testing machines.
Is it correct to compute the QphH value as follows: Queries 01-22 took 1.985 seconds, so QphH is (3600 / 1.985) * 22 = 39.899 ?
Well, the testweb framework does way more than just executing the test, so the performance time of the test might not be accurate. Also, you probably want to test with a higher workload, to get a more accurate result.
What scaling factor is used in the benchmark runs; is this 1? Is it possible to let the TestWeb run with scaling factor 100? Because that's the smallest Size for which results are published on the TPC web site.
It's very small. I don't recall how small, but very very small.
Did anyone yet try to determine the Price of the test platforms, following the TPC Pricing Spec? TPC-H results normally include a Price/Performance metric in the form of $/QphH@Size, and I guess MonetDB could score particularly well on this metric.
We never really did this. But, feel free to try and setup such test, we are very interested in the outcomes, of course. I hope this answers your questions. Regards
If I a remember correctly the scale factor is 0.01 in the testweb. Beware that some queries in TPC-H require a lot of intermediate space and the plans produced for MIL are not optimized. (It is the target for the MonetDB V5 release) Actually, we would be interested in experiences in the upcoming TPC-DS, which is more modern and not well explored by any system. Fabian Groffen wrote:
Hi,
No need to resubmit your question. It was received, but I'm sorry that we've been too busy to answer this question.
On 15-09-2006 21:25:15 +0200, Wouter Scherphof wrote:
Hi,
The TPC-H benchmark (http://www.tpc.org/tpch/default.asp) is part of the MonetDB SQL source and the benchmark is run during weekends, as I can see on the TestWeb (http://monetdb.cwi.nl/testing/projects/monetdb/Current/sql/.mTests03/times.h...). How should I interpret the listed results? According to TPC, results should be stated as a "QphH@Size" value, where "QphH" is the number of queries per hour for the H benchmark, and "Size" is a "scaling factor". On the TestWeb, the results are stated as seconds of execution time.
The testweb tpc-h is very very very small, to keep both testweb small in diskspace, and to keep the test time reasonable on our slower testing machines.
Is it correct to compute the QphH value as follows: Queries 01-22 took 1.985 seconds, so QphH is (3600 / 1.985) * 22 = 39.899 ?
Well, the testweb framework does way more than just executing the test, so the performance time of the test might not be accurate. Also, you probably want to test with a higher workload, to get a more accurate result.
What scaling factor is used in the benchmark runs; is this 1? Is it possible to let the TestWeb run with scaling factor 100? Because that's the smallest Size for which results are published on the TPC web site.
It's very small. I don't recall how small, but very very small.
Did anyone yet try to determine the Price of the test platforms, following the TPC Pricing Spec? TPC-H results normally include a Price/Performance metric in the form of $/QphH@Size, and I guess MonetDB could score particularly well on this metric.
We never really did this. But, feel free to try and setup such test, we are very interested in the outcomes, of course.
I hope this answers your questions.
Regards
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (3)
-
Fabian Groffen -
Martin Kersten -
Wouter Scherphof