|
[Asrg] How well do DNSxL lookups cache ?
2011-01-04 21:57:09
Over on the Spamassassin users' list, we've been having a spirited debate
about the design of DNSBLs and WLs for IPv6, starting with my draft.
One of the key questions is how DNSxL lookups will play with DNS caches
and servers, which of course depends on the pattern of IP addresses that
send mail. David Skoll collected a few stats from his servers, which
suggested that for low volume mail servers, only about 20% of lookups are
answered from the cache, and for a largish server handling 1M
connections/day, it was only about 2/3.
That seemed hard to believe, so I took a week of logs from my mail server,
did a similar analysis, and by golly, he's right. The analysis takes
timestamps and IP addresses, and computes how many DNSxL queries would be
answered from a DNS cache under various scenarios. Each group shows the
number of cache hits and total queries by 24 hour period. (The first and
last are short, since my logs didn't start and end at midnight.) Then it
shows the number of unique IPs queried, and a list showing cache
effectiveness, with the number of times each cache entry was used, e.g.,
in the first dataset, there was one cache entry used 514 times, down to
536,009 entries used only once.
To see how sensitive these answers were to DNSxL design, I tried it with a
15 minute and 60 minute TTL, and I tried with /24 rather than
/32 granularity. Neither made much difference: the cache effectiveness
rarely got above 20% and the vast majority of cache entries were never
reused.
So now I'm scratching my head. If these numbers are typical, I don't
understand why DNSBL servers don't all fall over now.
My analyses use a small perl script which you can have if you have logs
you'd like to try it on. All it needs for input is a series of IP
addresses and timestamps. It extracts them from the peculiar log format
that mailfront and daemontools produce, but adjusting the input format is
easy. I'll see if I can get some data from larger mail systems and see
how different the answers are.
Regards,
John Levine, johnl(_at_)iecc(_dot_)com, Primary Perpetrator of "The Internet for
Dummies",
Please consider the environment before reading this e-mail. http://jl.ly
--- 15 minute TTL ---
7078 of 56237 ( 12 %) Sat Dec 25 00:00:00 2010
6146 of 45213 ( 13 %) Sun Dec 26 00:00:00 2010
9398 of 67167 ( 13 %) Mon Dec 27 00:00:00 2010
12155 of 76827 ( 15 %) Tue Dec 28 00:00:00 2010
10845 of 83165 ( 13 %) Wed Dec 29 00:00:00 2010
8781 of 73894 ( 11 %) Thu Dec 30 00:00:00 2010
10240 of 59080 ( 17 %) Fri Dec 31 00:00:00 2010
7753 of 51736 ( 14 %) Sat Jan 1 00:00:00 2011
7918 of 53362 ( 14 %) Sun Jan 2 00:00:00 2011
8440 of 63016 ( 13 %) Mon Jan 3 00:00:00 2011
9022 of 56342 ( 16 %) Tue Jan 4 00:00:00 2011
Total IPs = 358782
hits count
514 1
423 1
420 1
415 1
403 1
378 1
373 1
369 1
368 1
366 2
309 1
283 1
280 1
269 1
267 1
266 1
265 1
257 2
251 1
247 1
246 1
244 1
239 1
223 1
220 1
218 1
208 1
203 1
199 1
197 1
195 1
193 1
181 2
175 1
173 1
168 1
166 1
162 1
160 1
159 2
157 1
156 1
149 1
148 1
145 1
141 1
135 1
134 1
133 2
132 1
131 1
127 1
124 3
122 1
120 1
119 1
118 1
115 1
113 1
111 1
110 1
106 1
105 1
104 1
103 1
100 12
99 4
98 4
97 2
95 3
94 1
93 2
92 2
91 1
90 1
89 1
87 1
86 2
81 1
79 2
77 1
75 1
73 1
72 1
68 4
67 1
66 2
65 1
64 1
62 2
61 1
59 5
58 4
57 2
56 5
54 2
53 3
52 2
51 3
50 7
48 1
46 1
45 3
43 1
42 3
41 3
40 1
39 1
38 2
37 2
36 3
32 1
31 4
30 9
29 9
28 8
27 6
26 2
25 5
24 10
23 6
22 7
21 5
20 53
19 9
18 5
17 10
16 17
15 19
14 35
13 26
12 49
11 60
10 69
9 113
8 227
7 185
6 411
5 762
4 1918
3 6599
2 41434
1 536009
--- 60 minute TTL ---
9652 of 56237 ( 17 %) Sat Dec 25 00:00:00 2010
8364 of 45213 ( 18 %) Sun Dec 26 00:00:00 2010
13141 of 67167 ( 19 %) Mon Dec 27 00:00:00 2010
15976 of 76827 ( 20 %) Tue Dec 28 00:00:00 2010
15708 of 83165 ( 18 %) Wed Dec 29 00:00:00 2010
12886 of 73894 ( 17 %) Thu Dec 30 00:00:00 2010
13266 of 59080 ( 22 %) Fri Dec 31 00:00:00 2010
10442 of 51736 ( 20 %) Sat Jan 1 00:00:00 2011
10391 of 53362 ( 19 %) Sun Jan 2 00:00:00 2011
11559 of 63016 ( 18 %) Mon Jan 3 00:00:00 2011
11876 of 56342 ( 21 %) Tue Jan 4 00:00:00 2011
Total IPs = 358782
hits count
514 1
423 1
420 1
415 1
411 1
403 1
378 1
369 1
368 1
366 2
309 1
283 1
280 1
270 1
269 2
267 1
266 1
265 1
259 1
257 2
251 1
248 1
246 1
244 1
239 1
223 1
220 1
219 1
208 1
203 1
199 1
197 1
195 1
193 1
186 1
181 2
175 1
173 1
168 1
162 1
160 1
159 2
157 1
154 1
149 1
148 1
145 1
141 1
135 1
134 1
133 2
132 1
131 1
124 3
122 1
120 1
119 1
117 1
115 1
111 1
110 1
106 1
104 1
103 1
100 12
99 4
98 4
97 2
95 2
94 1
93 1
92 2
91 1
90 1
89 1
87 1
86 2
85 1
83 1
81 1
79 2
77 1
75 1
74 1
73 1
72 1
69 1
68 3
67 1
66 2
65 2
64 1
62 2
61 2
59 2
58 4
57 2
56 6
54 2
53 2
52 2
51 3
50 7
48 1
47 1
46 2
45 4
43 1
42 3
41 2
40 2
39 2
38 1
37 3
36 4
35 2
34 1
33 2
32 2
31 2
30 8
29 10
28 8
27 3
26 3
25 6
24 9
23 8
22 9
21 12
20 60
19 8
18 13
17 19
16 20
15 31
14 42
13 37
12 58
11 65
10 89
9 149
8 302
7 311
6 656
5 1367
4 3209
3 10400
2 58038
1 477633
--- 15 minute TTL, combine /24s ---
7828 of 56237 ( 13 %) Sat Dec 25 00:00:00 2010
6837 of 45213 ( 15 %) Sun Dec 26 00:00:00 2010
10614 of 67167 ( 15 %) Mon Dec 27 00:00:00 2010
13518 of 76827 ( 17 %) Tue Dec 28 00:00:00 2010
12430 of 83165 ( 14 %) Wed Dec 29 00:00:00 2010
10142 of 73894 ( 13 %) Thu Dec 30 00:00:00 2010
11045 of 59080 ( 18 %) Fri Dec 31 00:00:00 2010
8433 of 51736 ( 16 %) Sat Jan 1 00:00:00 2011
8712 of 53362 ( 16 %) Sun Jan 2 00:00:00 2011
9342 of 63016 ( 14 %) Mon Jan 3 00:00:00 2011
9830 of 56342 ( 17 %) Tue Jan 4 00:00:00 2011
Total IPs = 186854
hits count
514 1
423 1
420 1
415 1
403 1
378 1
373 1
369 1
368 1
366 2
309 1
283 1
280 1
269 1
267 1
266 1
265 1
257 2
251 1
247 1
246 1
244 1
239 1
223 1
220 1
218 1
208 1
203 1
199 1
197 1
195 1
193 1
181 2
175 1
173 1
168 1
166 1
162 1
160 1
159 2
157 1
156 1
149 1
148 1
145 1
141 1
135 1
134 1
133 2
132 1
131 1
127 1
124 3
122 1
120 1
119 1
118 1
115 1
113 1
111 1
110 1
106 1
105 1
104 1
103 1
100 12
99 4
98 4
97 2
95 3
94 1
93 2
92 3
90 1
89 1
87 1
86 2
81 1
79 2
77 1
75 1
73 1
72 1
68 4
67 1
66 2
65 1
64 1
62 2
61 1
59 5
58 4
57 2
56 5
54 2
53 4
52 1
51 3
50 7
48 1
46 1
45 3
43 1
42 4
41 3
40 1
39 1
38 2
37 2
36 3
35 1
32 3
31 3
30 9
29 9
28 10
27 6
26 5
25 3
24 10
23 7
22 8
21 7
20 55
19 12
18 13
17 17
16 21
15 27
14 43
13 35
12 66
11 86
10 93
9 165
8 288
7 264
6 549
5 954
4 2253
3 7508
2 45131
1 519465
--- 60 minute TTL, combine /24s ---
11210 of 56237 ( 19 %) Sat Dec 25 00:00:00 2010
9595 of 45213 ( 21 %) Sun Dec 26 00:00:00 2010
15383 of 67167 ( 22 %) Mon Dec 27 00:00:00 2010
18675 of 76827 ( 24 %) Tue Dec 28 00:00:00 2010
18714 of 83165 ( 22 %) Wed Dec 29 00:00:00 2010
15380 of 73894 ( 20 %) Thu Dec 30 00:00:00 2010
14832 of 59080 ( 25 %) Fri Dec 31 00:00:00 2010
11827 of 51736 ( 22 %) Sat Jan 1 00:00:00 2011
11987 of 53362 ( 22 %) Sun Jan 2 00:00:00 2011
13420 of 63016 ( 21 %) Mon Jan 3 00:00:00 2011
13543 of 56342 ( 24 %) Tue Jan 4 00:00:00 2011
Total IPs = 186854
hits count
514 1
423 1
420 1
415 1
411 1
403 1
378 1
370 1
368 1
366 2
309 1
283 1
280 1
270 1
269 2
267 1
266 1
265 1
259 1
257 2
251 1
248 1
246 1
244 1
239 1
223 1
220 1
219 1
208 1
203 1
199 1
197 1
195 1
193 1
186 1
181 2
175 1
173 1
168 1
162 1
160 1
159 2
157 1
154 1
149 1
148 1
145 1
141 1
135 1
134 1
133 2
132 1
131 1
125 1
124 3
121 1
120 1
119 1
115 1
111 1
110 1
106 1
104 1
103 1
100 12
99 4
98 4
97 2
95 2
94 1
93 1
92 3
90 1
89 1
87 2
86 2
85 1
81 1
79 2
77 1
75 1
74 1
73 1
72 1
69 1
68 3
67 2
66 2
65 2
64 1
62 2
61 2
59 2
58 4
57 2
56 5
54 3
53 3
52 1
51 3
50 7
48 2
47 1
46 3
45 4
44 1
43 1
42 4
41 3
40 6
39 2
38 3
37 4
36 6
35 5
34 2
33 1
32 5
31 7
30 10
29 19
28 9
27 11
26 10
25 9
24 12
23 13
22 17
21 19
20 70
19 16
18 24
17 36
16 40
15 47
14 63
13 62
12 79
11 84
10 125
9 203
8 346
7 394
6 859
5 1666
4 4022
3 12676
2 64013
1 446298
_______________________________________________
Asrg mailing list
Asrg(_at_)irtf(_dot_)org
http://www.irtf.org/mailman/listinfo/asrg
| <Prev in Thread] |
Current Thread |
[Next in Thread>
|
- [Asrg] How well do DNSxL lookups cache ?,
John R. Levine <=
|
|
|