Requirements for mass digitization projects

["Joseph J. Esposito" <espositoj@gmail.com>]

Ongoing discussions about various mass digitization projects, 
driven primarily by the Google Libraries program but including 
the respective activities of Microsoft, the Open Content 
Alliance, and others, prompts these comments about what should be 
taken into account as these programs proceed.  My concern is a 
practical one:  Some projects are incomplete in their design, 
which will likely result in their having to be redone in the near 
future, an expense that the world of scholarly communications can 
ill afford.  There are at least four essential characteristics of 
any such project, and there may very well be more.

As many have noted, the first requirement of such a project is 
that it adopt an archival approach.  Some scanning is now being 
done with little regard for preserving the entire informational 
context of the original.  Scanning first editions of Dickens 
gives us nothing if the scans do not precisely copy first 
editions of Dickens; the corollary to this is that clearly 
articulated policies about archiving must be part of any mass 
digitization project.  Some commercial projects have little 
regard for this, as archival quality simply is not part of the 
business plan; only members of the library community are in a 
position to assert the importance of this.  An archival 
certification board is evolving as a scholarly desideratum.

Archives of digital facsimiles are important, but we also need 
readers' editions, the second requirement of mass digitization 
projects.  This goes beyond scanning and involves the editorial 
process that is usually associated with the publishing industry. 
The point is not simply to preserve the cultural legacy but to 
make it more available to scholars, students, and interested 
laypeople.  The high school student who first encounters 
Dickens's "Great Expectations" should not also be asked to fight 
with Victorian typography, not to mention orthography.  In the 
absence of readers' editions, broad public support for mass 
digitization projects will be difficult to come by.

As devotees of "Web 2.0" insist with increasing frequency, all 
documents are in some sense community documents.  Thus scanned 
and edited material must be placed into a technical environment 
that enables ongoing annotation and commentary.  The supplemental 
commentary may in time be of greater importance than the initial 
or "founding" document itself, and some comments may themselves 
become seminal.  I become uneasy, however, when the third 
requirement of community engagement is not paired with the first 
of archival fidelity.  What do we gain when "The Declaration of 
Independence" is mounted on a Web site as a wiki?  Sitting 
beneath the fascinating activities of an intellectually engaged 
community must be the curated archival foundation.

The fourth requirement is that mass digitization projects should 
yield file structures and tools that allow for machine process to 
work with the content.  Whether this is called "pattern 
recognition" or "data mining" or something else is not important. 
What is important is to recognize that the world of research 
increasingly will be populated by robots, a term that no longer 
can or should carry a negative connotation.  Some people call 
this "Web 3.0", but I prefer to think of it as "the post-human 
Internet," which may not even be a World Wide Web application.

To my knowledge, none of the current mass digitization projects 
fully incorporate all four of these requirements.

Note that I am not including any mention of copyright here, which 
is the topic that gets the most attention when mass digitization 
is contemplated. All four of these requirements hold for public 
domain documents.  Copyright is a red herring.

Joe Esposito