I have tables with around 12 million rows and large primary keys. I
find the performance for various queries to be rather strange.
* Why is Q2 (see below) so much slower than Q1? It is actually faster
for me to use Q1 and do the rest of the matching myself.
* Why is Q3 so immensely much faster than Q1? I understand that the
first part of the primary key is selective in Q3. Do I understand right
if Q1 scans until "type" matches, then scans until "name" matches, then
scans until "value" matches? No indexes are put on disk, right? Then
the matching for Q3 should be scan until "value" matches, which
shouldn't be more than at most three times faster, right (if three
columns are read instead of one for Q1)? I do Q1 twice to look for cash
effects, but none are seen... Could some columns from the table "value"
be cached and kept, without giving room for those from "type_name_value"?
* Q4 has the same performance as Q1, as expected.
* Why isn't Q5 faster than Q6? 82% of the rows have type=3. Fewer
should be scanned to find that "!!foo" doesn't exist than for "~~foo",
as "!" has a low ASCII value, while "~" is high. (See COUNT queries at
the bottom.)
Klem fra Nils
Q1: SELECT * FROM type_name_value WHERE type = 3 AND name = 'author' AND
value = 'Jurgen Annevelink'
Q2: SELECT * FROM type_name_value WHERE type = 3 AND name = 'author' AND
value = 'Jurgen Annevelink' AND doc = 1 AND nenc = '1.1.1.1' AND pos = 4
Q3: SELECT * FROM value WHERE value = 'Jurgen Annevelink' AND type = 3
AND name = 'author'
Q4: SELECT * FROM type_name_value WHERE type = 3 AND name = 'author' AND
value = '__bar'
Q5: SELECT * FROM type_name_value WHERE type = 3 AND name = '!!foo' AND
value = '__bar'
Q6: SELECT * FROM type_name_value WHERE type = 3 AND name = '~~foo' AND
value = '__bar'
Q7: SELECT * FROM type_name_value WHERE type = 3000 AND name = '__foo'
AND value = '__bar'
See runs below.
Number of rows
SELECT COUNT(type) FROM type_name_value
% nilsgri.type_name_value # table_name
% count_no_nil_type # name
% int # type
% 8 # length
[ 12165109 ]
Query time: 0.066192
SELECT COUNT(type) FROM value
% nilsgri.value # table_name
% count_no_nil_type # name
% int # type
% 8 # length
[ 12165109 ]
Query time: 0.058111
The PRIMARY KEY of type_name_value is (type,name,value,doc,nenc,pos)
The PRIMARY KEY of value is (value,doc,nenc,pos)
Q1: Search 3 first, match 3 first.
SELECT * FROM type_name_value WHERE type = 3 AND name = 'author' AND
value = 'Jurgen Annevelink'
% nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value # table_name
% type, name, value, doc, nenc, pos, pathid, complete # name
% int, varchar, clob, int, varchar, int, int, boolean # type
% 1, 6, 17, 1, 12, 8, 3, 5 # length
[ 3, "author", "Jurgen Annevelink", 1, "1.1.1.1", 4, 5, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.23579.2.1", 724888, 56, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.2487.3.1", 81996, 5, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.42.2.1", 1071, 5, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.533986.2.1", 17294725, 56, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.576139.3.1", 18745432, 56, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.944085.1.1", 30104242, 107, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.99173.2.1", 3191470, 56, true ]
Query time: 1.643438
Q1: Repeat to check for disk cache effects.
SELECT * FROM type_name_value WHERE type = 3 AND name = 'author' AND
value = 'Jurgen Annevelink'
% nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value # table_name
% type, name, value, doc, nenc, pos, pathid, complete # name
% int, varchar, clob, int, varchar, int, int, boolean # type
% 1, 6, 17, 1, 12, 8, 3, 5 # length
[ 3, "author", "Jurgen Annevelink", 1, "1.1.1.1", 4, 5, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.23579.2.1", 724888, 56, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.2487.3.1", 81996, 5, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.42.2.1", 1071, 5, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.533986.2.1", 17294725, 56, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.576139.3.1", 18745432, 56, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.944085.1.1", 30104242, 107, true ]
[ 3, "author", "Jurgen Annevelink", 1, "1.99173.2.1", 3191470, 56, true ]
Query time: 1.615900
Q2: Search entire, match entire.
SELECT * FROM type_name_value WHERE type = 3 AND name = 'author' AND
value = 'Jurgen Annevelink' AND doc = 1 AND nenc = '1.1.1.1' AND pos = 4
% nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value # table_name
% type, name, value, doc, nenc, pos, pathid, complete # name
% int, varchar, clob, int, varchar, int, int, boolean # type
% 1, 6, 17, 1, 7, 1, 1, 5 # length
[ 3, "author", "Jurgen Annevelink", 1, "1.1.1.1", 4, 5, true ]
Query time: 11.073827
Q3: Use table value instead.
SELECT * FROM value WHERE value = 'Jurgen Annevelink' AND type = 3 AND
name = 'author'
% nilsgri.value, nilsgri.value, nilsgri.value, nilsgri.value,
nilsgri.value, nilsgri.value, nilsgri.value, nilsgri.value # table_name
% value, doc, nenc, pos, type, name, pathid, complete # name
% clob, int, varchar, int, int, varchar, int, boolean # type
% 17, 1, 12, 8, 1, 6, 3, 5 # length
[ "Jurgen Annevelink", 1, "1.1.1.1", 4, 3, "author", 5, true ]
[ "Jurgen Annevelink", 1, "1.23579.2.1", 724888, 3, "author", 56, true ]
[ "Jurgen Annevelink", 1, "1.2487.3.1", 81996, 3, "author", 5, true ]
[ "Jurgen Annevelink", 1, "1.42.2.1", 1071, 3, "author", 5, true ]
[ "Jurgen Annevelink", 1, "1.533986.2.1", 17294725, 3, "author", 56, true ]
[ "Jurgen Annevelink", 1, "1.576139.3.1", 18745432, 3, "author", 56, true ]
[ "Jurgen Annevelink", 1, "1.944085.1.1", 30104242, 3, "author", 107, true ]
[ "Jurgen Annevelink", 1, "1.99173.2.1", 3191470, 3, "author", 56, true ]
Query time: 0.000477
Q4: Search 3 first, match 2 first.
SELECT * FROM type_name_value WHERE type = 3 AND name = 'author' AND
value = '__bar'
% nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value # table_name
% type, name, value, doc, nenc, pos, pathid, complete # name
% int, varchar, clob, int, varchar, int, int, boolean # type
% 1, 0, 0, 1, 0, 1, 1, 5 # length
Query time: 1.617259
Q5: Search 3 first, match 1 first. '!' early in alphabet.
SELECT * FROM type_name_value WHERE type = 3 AND name = '!!foo' AND
value = '__bar'
% nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value # table_name
% type, name, value, doc, nenc, pos, pathid, complete # name
% int, varchar, clob, int, varchar, int, int, boolean # type
% 1, 0, 0, 1, 0, 1, 1, 5 # length
Query time: 0.530727
Q6: Search 3 first, match 1 first. '~' late in alphabet.
SELECT * FROM type_name_value WHERE type = 3 AND name = '~~foo' AND
value = '__bar'
% nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value # table_name
% type, name, value, doc, nenc, pos, pathid, complete # name
% int, varchar, clob, int, varchar, int, int, boolean # type
% 1, 0, 0, 1, 0, 1, 1, 5 # length
Query time: 0.534592
Q7: Search 3 first, match 0 first.
SELECT * FROM type_name_value WHERE type = 3000 AND name = '__foo' AND
value = '__bar'
% nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value,
nilsgri.type_name_value, nilsgri.type_name_value # table_name
% type, name, value, doc, nenc, pos, pathid, complete # name
% int, varchar, clob, int, varchar, int, int, boolean # type
% 1, 0, 0, 1, 0, 1, 1, 5 # length
Query time: 0.000407
Number of rows with type and name mathcing.
SELECT COUNT(type) FROM type_name_value WHERE type = 3 AND name = 'author'
% nilsgri.type_name_value # table_name
% count_no_nil_type # name
% int # type
% 7 # length
[ 2730959 ]
Query time: 0.683863
Number of rows with type mathcing.
SELECT COUNT(type) FROM type_name_value WHERE type = 3
% nilsgri.type_name_value # table_name
% count_no_nil_type # name
% int # type
% 8 # length
[ 10020569 ]
Query time: 0.032573
SELECT COUNT(type) FROM type_name_value WHERE type = 3 AND name < '!!foo'
% nilsgri.type_name_value # table_name
% count_no_nil_type # name
% int # type
% 1 # length
[ 0 ]
Query time: 0.161186
SELECT COUNT(type) FROM type_name_value WHERE type = 3 AND name < '~~foo'
% nilsgri.type_name_value # table_name
% count_no_nil_type # name
% int # type
% 8 # length
[ 10020569 ]
Query time: 0.341732
Number of rows with type mathcing.
real 0m18.372s
user 0m0.005s
sys 0m0.002s