[E-voting] (USA) Information Quality Professionals Issue Urgent Alert

[Catherine Ansbro]

[The quality assurance context is a new slant and very relevant.]
http://www.bbvforums.org/cgi-bin/forums/show.cgi?tpc=36349&post=28713#POST28713

MEDIA RELEASE

Date: October 12, 2006

Subject: *Information Quality Professionals Issue Urgent Alert Regarding 
United States Elections*

Contact:
Seth Johnson
(212) 543-4266

Attachment: Letter to Congress and Election Officials from Information 
Quality Professionals (text below)

In a recent letter addressed to Congress and chief election officials in 
every state, a group of information quality professionals has voiced 
concerns as fundamental changes are being introduced in United States 
election processes under the provisions of the 2002 Help America Vote Act.

The signers of the letter:

- see United States election processes at grave risk

- express concern about the lack of means to observe the present impact 
of changes being introduced in elections

- urge election officials to assess the accuracy of election outcomes 
through a count of a random sample of ballots

- call for the use of dependable quality control measures and the 
prevention of errors by incorporating reliable quality processes

Commenting on a July 19, 2006 Congressional hearing on voting technology 
standards [1], the letter observes that factual assessments of the 
accuracy of election processes have not been
offered, and that voting technology is being introduced in elections 
throughout the country without adequate means to observe the impact of 
the change.

The letter stresses that efforts to improve elections must be focused on 
the election process as a whole and the quality of the information it 
produces, and that a focus on voting technology certification is not 
sufficient to assure election integrity.

The signers of the letter point out that assessing the quality of voting 
technology is not adequate to understand the impact of
technology on the election process, and that automation in itself does 
not assure accuracy and reliability of an information production 
process. They warn that efforts to improve usability of technology in 
the vote capture step that have arisen in response to issues illustrated 
in Florida during the 2000 general election, should not lead to an 
assumption that simply adding automation to the process is sufficient to 
assure accuracy.

*Among the letter's key points:*

- A systems approach to assuring the reliability of elections should 
address the process as a whole, not the technology being introduced in 
itself.

- The election process produces vote counts as an information product.

- Assessing the quality of vote counts is a means of managing the 
complexity of the election process.

- Quality measures of information products reveal the effects of defects 
in a process, even if the causes are not yet known. Specific root 
causes, even if they relate to intricate details of technology or code, 
can be identified and corrected subsequently.

- A measure of election accuracy also helps assure security against fraud.

The letter urges its recipients to act quickly to address these concerns.

[1] Hearing testimony is available at: 
http://www.house.gov/science/hearings/full06/July%2019/index.htm

Contact:
Seth Johnson
(212) 543-4266
New York, NY

Formatted copy of the letter:
application/mswordInformation quality professionals election alert
*http://www.bbvforums.org/forums/messages/36349/IQ_on_VVSG_10-05-06-44355.doc*(26.8 
k)



Text of letter:

------------------------------------------------------------------------
*quote:*October 5, 2006

The Honorable Vernon Ehlers, Chairman
The Honorable Juanita Millender-McDonald, Ranking Member
Committee on House Administration
1309 Longworth Building
Washington, DC 20515

The Honorable Sherwood L. Boehlert, Chair
The Honorable Bart Gordon, Ranking Member
House Committee On Science
2320 Rayburn House Office Building
Washington, DC 20515

(See list below letter for additional recipients, including hearing 
witnesses.)

Dear Members of the House Administration and Science Committees, State 
Election Officials, and Hearing Witnesses:

We are writing as practitioners of quality management for information 
production processes, to offer some urgent comments and recommendations 
following the July 19, 2006 Joint Hearing of the House Science and 
Administration Committees on "Voting Machines: Will New Standards and 
Guidelines Prevent Future Problems?" (1)

This hearing effectively served as a status update on efforts to improve 
elections under the provisions of the 2002 Help America Vote Act. It 
sought to address the questions of how and whether voting equipment 
standards and testing can help improve accuracy and security and prevent 
errors and fraud. Spokespersons for the Election Assistance Commission 
(EAC), the National Institute for Standards and Technology (NIST), State 
election offices for Minnesota and Maryland, voting technology vendors 
and a computer scientist speaking in an individual capacity commented on 
and described plans to improve the Voluntary Voting System Guidelines 
(VVSG)(2), to establish processes to certify testing labs and voting 
systems, to encourage the adoption of standards by state election 
offices and voting technology vendors, and to assist election officials 
in improving voting processes.

These are all highly important tasks in improving elections. Our 
concerns relate to the need to manage and understand the present impact 
of the changes that are being introduced into election processes.

We have two main recommendations:

• The VVSG includes many requirements and tests for devices and 
software, but no mention of measures of the voting process as a whole. A 
profound change in election processes such as the introduction of voting 
technology demands that the entire voting process be placed under 
dependable, sound quality control with safeguards to prevent voter error 
and vote tampering.

• We strongly recommend election officials prepare to assess the 
accuracy of election outcomes in a transparent and publicly observable 
process, by comparing election results against the results of a manual 
count of a statistical random sample of ballots selected from the 
complete pool of cast ballots. Plans should address courses of action if 
discrepancies are found. Jurisdictions that do not produce a manually 
readable paper ballot that is verified by the voter before leaving the 
voting booth or that have regulations restricting their use 
unfortunately will not be able to perform this measurement, but should 
prepare to address questions about the basis for confidence in the 
reliability of the election process.

Detailed Commentary:

Lack of Recommendation for Quality Control
The question of the accuracy of voting processes was repeatedly raised 
during this hearing on technology standards. Yet responses to this 
question addressed it as an issue subject to ongoing research, and no 
factual assessments of how election process changes have affected the 
accuracy of election results were offered.

Witnesses described ongoing research regarding test procedures for 
testing labs to measure errors and performance of voting systems, 
guidelines for design and usability and election management, independent 
verification technologies, and the incorporation of a "paper trail." But 
nowhere in the testimony or other materials available from the EAC or 
the NIST Technical Guidelines Development Committee (TGDC) do we find a 
recommendation to establish quality control measures of the election 
outputs to observe the impact of the changes being introduced by new 
voting technologies.

Quality measures of process outcomes are critical to enable detection of 
negative side-effects that may be introduced by changes in a process, 
for addressing issues relating to questions of accountability that often 
arise between subprocesses, and to provide a picture of how reliable a 
process is as a whole.

Guidelines by the EAC and NIST that seek to foster improvements in the 
quality of elections must call for implementing such quality measures as 
a means of assuring the process is in control. Guidelines must also call 
for designing quality into the voting process to error-proof it against 
root causes of failure.

The processes presently established under HAVA are bringing about the 
replacement of known-defective technology with new voting technology in 
elections throughout the country without providing means to observe and 
control the potential negative impact that these changes may cause. This 
approach (of introducing new technology without assessing, controlling 
or improving the voting process) puts election processes further out of 
control in many ways.

The Voluntary Voting System Guidelines Address Voting Technology-Not the 
Process of Producing Election Information Accurately

While they speak of systems approaches, performance-based measures, and 
user-centered design and human factors in usability research, the 2005 
Voluntary Voting System Guidelines are nevertheless focused on assessing 
the voting technology, and do not address the fact that elections are 
complex information production processes, comprised of many other 
factors that work together to produce vote counts (including such 
functions as determining eligibility to vote and capturing votes). 
Assessing the quality of election results is essential to understanding 
the quality of the election process, not just addressing elements of the 
process in isolation.

In taking up this focus on the technology as such, NIST and the TGDC are 
addressing their specific charter under HAVA. However, as stewards of 
the Baldrige Award, NIST possesses a strong capability in process 
excellence methodology that would equip it to analyze and recommend how 
to apply appropriate forms of quality planning, control and improvement 
to processes that produce information in the critical and complex 
election process. In addition, there is a broad community of experts in 
the field who apply quality principles to information processes as a 
matter of their everyday practice.

Managing Election Information Quality

Many businesses have found that finding and correcting errors is 
unworkable as a way to achieve reliable improvements in information 
quality, because information is collected too rapidly and correcting 
errors after the fact is costly and difficult. This is particularly true 
for elections, which operate under time constraints and for which the 
cost of correcting errors includes recounts or the conduct of new 
elections. The worst-case scenario is that election process failure can 
cause the wrong person to be elected. Instead, the appropriate approach 
is to focus on preventing information errors (in voter registration, 
recorded votes and vote totals) revealed by factual assessment and by 
targeting their root causes.

The quality of complex information production processes is assured by 
designing quality in to them this way. This approach is essential to 
error-proof and control election processes and assure their integrity 
while fundamental changes are being introduced.

Automation and Accuracy

Simply adding automation to a process will not assure its reliability 
and accuracy. While automated devices generally execute programmed 
functions very consistently, the reliability of an information 
production process depends on many more factors than the automation that 
may be added to it. Election administration processes are complex 
information production processes, with a technical side responsible for 
systems and applications, and a “business” side that performs the rest 
of the process, including the operation of automated systems. Both sides 
must work together to produce accurate voting information, and both 
sides must be held accountable to the requirements for the product of 
the process - the vote count results. Even assuming voting devices 
function properly, the ways in which the introduction of automation may 
affect the election process go well beyond the precision of the devices' 
functioning. Many factors can introduce errors, including mismatching 
the designed ballot screens with the vote recording data store, to the 
usability design of the ballot screens, to poorly written voter 
instructions, to the vote counting procedures, just to name a few.

Usability Research and Accuracy: Applying principles of usability to 
voting devices in the vote capture step clearly promises reductions in 
the incidence of unclear or ambiguous records of voter intent and 
thereby may bring about improvements over the phenomena of hanging 
chads, unreadable marks on optical ballots, unclear ballot designs and 
similar problems that were on display in the 2000 general election in 
Florida. However, the promise of introducing voting technology at any 
step does not guarantee error-free election processes, and should not 
suggest that the reliability of elections is sufficiently assured by 
introducing automation alone. The 2004 elections revealed many different 
types of failure in automated technologies. The 2006 elections saw 
electronic vote counts that were changed several times. The truth is 
that the introduction of automation increases the need to manage the 
reliability and accuracy of the election process as a whole.

Defining Accuracy: Accuracy is defined as "the degree to which data 
correctly reflects the real world object or event being described" (3).

Accuracy of the voting process means that the captured and counted vote 
agrees with the intended vote of the voter. Accuracy is a measure of the 
information produced by the process, not of the technology employed by 
the process. It is not a measure that is performed electronically. 
Unlike characteristics that may be measured electronically, such as 
completeness of values, acceptable values, non-duplication, timeliness, 
or validity according to business rules (4), a measure of accuracy 
entails comparing of the electronic representation against the real 
world entity (or event) being represented.

Measuring Election Accuracy: For elections, which encompass a 
requirement of anonymous voting, direct manual comparisons of individual 
electronic records against cast ballots or voter intent are not 
appropriate. However, accuracy may be assessed by comparing vote total 
percentage results of the live process against the percentage results of 
a manual count of a representative random sample of human-readable 
ballots that have been verified by the voter before leaving the voting 
booth. This is a measure of the accuracy of the election process if both 
human-readable ballots and electronically recorded votes and vote totals 
are maintained with control of the chain of custody, and a statistically 
representative sample of ballots is selected randomly across all 
districts relevant to a contest.

This measure of accuracy will reveal the effects of defects in the 
process after the point of casting the ballot, whether inadvertent or 
willful, if the electronic vote total percents vary from the results of 
the accuracy measure by more than its statistical margin of error. Such 
a discrepancy might trigger a manual recount. This accuracy assessment 
serves as a reliable check of the integrity of election outcomes with 
respect to ballots cast. Identifying and addressing root causes for 
discovered defects should be performed ultimately for any voting anomaly 
causes found.

Security and Election Accuracy

In response to a question about expanding the model presented by the 
California Secretary of State's Voting Systems Technology Assessment 
Advisory Board, Dr. David Wagner commented that while he believes that 
testing for reliability is at a level of readiness such that it may be 
applied to the certification of voting systems, the status of security 
testing is still not adequate. The hearing's charter acknowledges the 
deficiencies of the 2005 Voluntary Voting System Guidelines with respect 
to security testing, noting in particular special difficulties 
associated with testing software due to its great degree of customization.

However, a properly conducted global quality control measure assessing 
the accuracy of the results produced by the process as a whole would 
serve very well as a means for assuring security, since it would detect 
the effects of successful attempts, at any step after the capture of the 
ballot, to fraudulently affect the election outcome, if the effects are 
larger than the margin of error of the accuracy measure relative to the 
election margin difference.

Observations Related to Hearing Participants' Comments

Managing Complexity: Indeed, such global measures of the quality of a 
product are generally useful in the management of complex processes. 
House Science Committee Chairman Sherwood Boehlert noted that as 
election processes are computerized, the things that can go wrong become 
harder to recognize, fix and prevent, including security issues.

Establishing countermeasures for all potential types of defects that can 
arise with computerized voting devices is intrinsically complex. But the 
key concern in managing the impact of changes in the election process is 
the reliability of the process as a whole, in which the devices are 
used. It is not enough to have "confidence" in computerized voting 
devices in and of themselves.

Quality principles provide the framework within which performance 
information may be used to improve the process, as suggested in the 
questions posed by Representative Darlene Hooley, though it is important 
to recognize that voting technology is only one factor in the overall 
process of producing vote counts, and that the proper focus must be on 
the process and the quality of its product. Testing of voting technology 
before, during and after elections, according to the recommendation of 
Britt Williams and others, and as mentioned by Representative Juanita 
Millender-McDonald, does not place the process under control and cannot 
address the full range of potential kinds of defects that may arise 
following the introduction of voting technology into elections.

Quality methods enable the protection of the integrity of the entire 
process, as Commissioner Donetta Davidson indicated was EAC's mandate 
under HAVA - and in particular given the way they address the issue of 
security, these methods should be incorporated in election management 
guidelines. The systems approach as described in the NIST's Human 
Factors Report (5) and the comments of the ACCURATE group on the 2005 
VVSG (6), is geared toward modeling protocols to test the devices, and 
is not designed to assess how well all factors of the process work 
together to produce quality election results. It is therefore not 
adequate in and of itself for the purpose of understanding and managing 
the impact of the changes being introduced in the process.

Likewise, the use of performance-based standards and measures is 
appropriate only if they measure the full election process, not just the 
election technology. The quality of a process as a whole such as 
elections is assessed through performance measures that address the 
requirements that its product must meet.

Accuracy of the election result is the most important characteristic the 
election process must meet.

Both Minnesota Secretary of State Mary Kiffmeyer and Maryland 
Administrator of Elections Linda Lamone expressed the need to 
incorporate recognition of the role of all factors in the election 
process. Established methods for quality management address such aspects 
identified by Ms. Lamone as the processes surrounding the technology and 
the role of people taking part in administration (as well as systems 
development), while it also provides a basis for assuring that elections 
are being performed well. It also provides safeguards against the 
increased risks that Ms. Kiffmeyer noted are brought by the introduction 
of technology in the process. It provides a framework for "wrapping the 
whole system" in the manner she described, enabling the observation of 
the effect of defects in complex aspects of the process such as source 
code and technology.

Representative Zoe Lofgren echoed Dr. Wagner's concerns about the lack 
of public reporting and transparency on the part of the technology 
testing labs, and Representative Robert Ney suggested that the 
relationship of the Election Assistance Commission with the labs might 
help facilitate the use of independent assessments.

In much the same manner that rigorously performed quality measures can 
help manage complexity and address issues of security (particularly as 
provided by a measure of accuracy), they can also serve to help obviate 
the problems of bias and undue influence of vendors in technology 
testing and standards development.

Bearing in mind our comments above generally, Dr. Wagner's 
recommendations, and his citing of the recent Brennan Center 
recommendations (7) and those of the ACCURATE group of which he is a 
member, are properly cognizant of the critical issues brought about by 
the introduction of technology into election processes and are 
consistent with quality principles - including feedback loops for 
continual improvement, broadening the focus of testing beyond functional 
criteria to incorporate evaluations of security, reliability and 
usability, strengthening usability and accessibility tests through 
professional methods, grounding standards in the best scientific and 
engineering understanding, employing manually readable votes to assess 
the process, and independently reviewing source code. We would add that 
feedback and "field data" should address the process and the quality of 
the output that it produces, and not solely address features of the 
process with respect to how they relate to the quality of the technology 
in itself.

Voting Technology Certification Is Not Sufficient to Assure Election 
Integrity
Federal certification of voting technology under the 2005 Voluntary 
Voting System Guidelines should not be regarded as establishing the 
appropriate controls for assuring the integrity of elections as voting 
technology is added to the process. Elections produce information.

Voting technology vendors produce voting technology. Among the criteria 
to which vendors should be held accountable is the role of the 
technology in the quality of information that results from the election 
as a whole, since they are participants in that process. However, in 
schematic terms their role relates to systems development, though in 
many cases they may also be engaged for a number of election 
administration services as well. The factors that contribute to the 
quality of election processes differ on the "business" side from those 
on the development side.

The bottom line is that the election processes must be managed and 
controlled from the beginning of the value chain-voter registration-to 
the end result-clear confidence that the election outcomes represent the 
intent of the voters.

We hope that you will address these concerns expeditiously, as currently 
we see the election process in the United States at grave risk.

Sincerely,
(The following list their names in support of the above statement. 
Affiliations are listed for identification only.)

Larry English, Election Assessment Advisor; Author, "Information Quality 
Mandate for Election Reform"
Seth Johnson, Information Quality Improvement Consultant

Christy Bryant, Six Sigma Black Belt
Robert Fragola, VP Sales and Marketing, ChoiceMaker Technologies, Inc.
Matthias Groh, Six Sigma Master Black Belt
Raymond C. Hager, Certified Data Management Professional
David C. Hay, President, Essential Strategies, Inc.
Bruce McTavish, Data Architect; Past VP, Seattle Chapter of the Data 
Management Association
David Rafner, VP of Industry Relations, DAMA International
Dawn M. Wolthuis, President, Tincat Group, Inc.

Hearing Witnesses:
Donetta Davidson, Commissioner, Election Assistance Commission
John S. Groh, Chairman, Election Technology Council, Information 
Technology Association of America
Dr. William Jeffrey, Director, National Institute of Standards and 
Technology
Mary Kiffmeyer, Secretary of State, Minnesota
Linda Lamone, Administrator of Elections, State Board of Elections, 
Maryland
Dr. David Wagner, Professor of Computer Science, University of 
California, Berkeley

cc: The Honorable Rush Holt, United States Representative for the State 
of New Jersey
State Election Officials, United States of America
National Voluntary Laboratory Accreditation Program
Malcolm Baldrige National Quality Award
Brennan Center for Justice at NYU Law School
A Center for Correct, Usable, Reliable, Auditable, and Transparent 
Elections
Council of State Governments
National Governors Association
National Conference of State Legislatures
National Association of State Election Directors
National Association of Secretaries of State
National Lieutenant Governors Association
National Association of Attorneys General
National Association of Counties
National Association of County Recorders, Election Officials and Clerks
International Association of Clerks, Recorders, Elections Officials and 
Treasurers
United States Conference of Mayors
United States Commission on Civil Rights
Federal Voting Assistance Program
Voting Section, Civil Rights Division, United States Department of Justice
Office of Public Integrity, Criminal Division, United States Department 
of Justice


Please Contact:
Seth Johnson
275 Fort Washington Avenue, Suite 3C
New York, NY 10032
(212) 543-4266

Notes:
(1) Hearing testimony may be found at: 
http://www.house.gov/science/hearings/full06/July%2019/index.htm 
http://www.house.gov/science/hearings/full06/July 19/index.htm
(2) The Voluntary Voting System Guidelines may be found at: 
http://guidelines.kennesaw.edu/vvsg/intro.asp
(3) See Larry English, "Defining and Measuring Accuracy", 
http://www.infoimpact.com/articles/DMR_7.03DefiningandMeasuringAccuracy.Pdf
(4) See Larry English, "Improving Data Warehouse and Business 
Information Quality", Wiley & Sons, 1999, pp. 178-179
(5) See National Institute of Standards and Technology Special 
Publication 500-256, "Improving the Usability and Accessibility of 
Voting Systems and Products", 
http://vote.nist.gov/Final%20Human%20Factors%20Report%20%205-04.pdf 
http://vote.nist.gov/Final Human Factors Report 5-04.pdf
(6) See A Center for Correct, Usable, Reliable, Auditable, and 
Transparent Elections, "Public Comment on the 2005 Voluntary Voting 
System Guidelines", 
http://accurate-voting.org/accurate/docs/2005_vvsg_comment.pdf
(7) See Brennan Center Task Force on Voting System Security, "The 
Machinery of Democracy: Protecting Elections in an Electronic World", 
http://www.brennancenter.org/programs/downloads/Full%20Report.pdf 
http://www.brennancenter.org/programs/downloads/Full Report.pdf