Bennett Haselton, bennett@peacefire.org
10/23/2000
Using "zone files" from Network Solutions (which list all .com domains in existence), we obtained a list of the first 1,000 active .com domains on the Internet as of June 14, 2000. We tested this list of 1,000 domains using five popular blocking programs: Cyber Patrol, SurfWatch, Bess, AOL Parental Controls, and SafeServer, to see how many sites each program blocked as "pornography", and of those sites, how many were actually pornographic.
Summary:
|
Contents:
Introduction
Interpreting the results
Reports on each individual program
Limitations of the experiment
Conclusion
For each program, the error rate was computed as:
(number of non-pornographic sites blocked)/(total number of sites
blocked)
Average error rates |
|
| Cyber Patrol | 17/21 = 81% error rate |
| SurfWatch | 42/51 = 82% error rate |
| Bess | 7/26 = 27% error rate |
| AOL Parental Controls ("mature teen" setting) | 1/5 = 20% error rate |
| SafeServer | 10/29 = 34% error rate |
Our first test was done with SurfWatch in August, and our results were submitted to the COPA Commission as part of Peacefire's testimony on blocking software. For SurfWatch, we tested the first 1,000 pingable .com domains and found 51 sites blocked, 42 of which were errors.
Because we published this report in August, there was a risk that we would be biasing our experiment; a censorware company could hear about our SurfWatch experiment and then go through the first 1,000 ".com" domains to make sure their software did not block any of those sites in error, in case we ever decided to do the same experiment for their product. However, it turned out that none of the blocking companies did this except for N2H2 (see below), so we used the first 1,000 .com domains in our experiments with all the other programs.
In the case of N2H2 (makers of Bess), we ran a test in September using over 1,000 domains from the .com namespace. When we re-tested those sites in October, all of the errors were fixed up to site number 965, which had been blocked incorrectly but was now fixed. All of the errors from site number 1196 onward (which was the next wrongly blocked site) had not been fixed. So, we concluded that N2H2 had simply looked at the first 1,000 .com domains and made sure that their list contained no errors in that set. Since the results for the first 1,000 .com domains were no longer valid for Bess, we tested Bess using the second 1,000 .com domains.
Our SurfWatch report was published in August at:
http://www.peacefire.org/censorware/SurfWatch/first-1000-com-domains.html
All other reports were published in conjunction with this report.
Cyber Patrol:
http://www.peacefire.org/censorware/Cyber_Patrol/first-1000-com-domains.html
Bess:
http://www.peacefire.org/censorware/BESS/second-1000-com-domains.html
AOL Parental Controls:
http://www.peacefire.org/censorware/AOL/first-1000-com-domains.html
SafeServer:
http://www.peacefire.org/censorware/FoolProof/first-1000-com-domains.html
In the case of AOL Parental Controls, the numbers produced by the experiment were so small (5 blocked sites and 1 site blocked in error) that the "20%" figure cannot be taken as accurate without using a larger sample. The results for AOL Parental Controls were included for completeness, and to show the trade-off between accuracy and effectiveness (AOL had the lowest error rate, but also blocked fewer pornographic sites than any other program).
After this report is published, many of the sites listed here as "errors" will probably be unblocked by the company whose product is currently blocking them. This only means that the specific errors highlighted in this report would be fixed; the overall error rate for the program would not change. However, if you test the first 1,000 .com domains using these products at any point after this report is published, you may obtain different results.
This survey of selected filtering programs found that none of them had an error rate lower than 20%, and two of them had an error rate of about 80%. By contrast, a list of blocked sites that was composed by human reviewers would have a much lower error rate -- in that situation, less than 1% of the blocked sites should be mistakes (due to, for example, a clerical error). We conclude that any one of the given products blocks large amounts of innocuous material -- and that most of the sites blocked by these products have not been reviewed by staff to ensure that the sites meet the company's criteria.