Quoting a research paper from Gordon Cormack [1] :
"When explicitly asked to classify messages, human subjects have been
reported to exhibit error rates of 3%-7% [30, 16]. Tacitly derived
labels, such as those obtained from a “report spam” button, where it
is assumed that unreported messages are ham, may have even higher rates."
The references 30 and 16 in this paper comes from experiments with real
data and real subjects (SpamOrHam - Graham-Cumming and Hotmail -
Yih/Kolcz)
These error rates are most of the time bigger than what can be achieved
by spam filters. So it's probably a bad idea to consider that user
feedback is reliable. User interface shall be as simple as possible.
But wait, we're talking about messages that the spam filter hasn't
rejected. Additional data *has* to be useful.
The false positive rates are only a problem if the admin stupid enough to
consider a single report as definitive. If you deliver a message to 100
users, and three report it as spam, then you probably take no action. If 20
report it as spam, then you need to take a closer look.
I certainly don't think a 7% error rate is enough to determine that users
should not be given the opportunity to distinguish between unwanted mail
and reportable junk.
You can also combine reporting rates with your bayesian content analyser or
spamassassin score, or with your reputational score for the sender domain,
etc.
[1] Cormack, G. V. and Kolcz, A. 2009. Spam filter evaluation with
imprecise ground truth. In Proceedings of the 32nd international ACM
SIGIR Conference on Research and Development in information Retrieval
(Boston, MA, USA, July 19 - 23, 2009). SIGIR '09. ACM, New York, NY,
604-611. http://plg.uwaterloo.ca/~gvcormac/cormacksigir09-spam.pdf
JM
for folks who do serious UX work, they do not guarantee that their
suggestions are right, merely that they are worth testing.
d/
-- ---------------------------------------------------------------
Jose Marcio MARTINS DA CRUZ http://j-chkmail.ensmp.fr
Ecole des Mines de Paris
60, bd Saint Michel 75272 - PARIS CEDEX 06
mailto:Jose-Marcio(_dot_)Martins(_at_)mines-paristech(_dot_)fr
_______________________________________________
Asrg mailing list
Asrg(_at_)irtf(_dot_)org
http://www.irtf.org/mailman/listinfo/asrg
--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/
_______________________________________________
Asrg mailing list
Asrg(_at_)irtf(_dot_)org
http://www.irtf.org/mailman/listinfo/asrg