Re: Impact Factor, Open Access & Other Statistics-Based Quality

[Stevan Harnad <harnad@ecs.soton.ac.uk>]

Citation counts do not measure quality directly, but they are correlated
with it. So are download counts, and no doubt other digitometric measures
that are under development, and that will be derived from a growing OA
corpus. See http://citebase.eprints.org/ and
http://citebase.eprints.org/analysis/correlation.php

Some studies on the correlation:

    Lee KP, Schotland M, Bacchetti P, Bero LA (2002) Association of
    journal quality indicators with methodological quality of clinical
    research articles. AMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION
    287 (21): 2805-2808

        "High citation rates... and low manuscript acceptance rates...
        appear to be predictive of higher methodological quality scores
        for journal articles"

    Ray J, Berkwits M, Davidoff F (2000) The fate of manuscripts rejected
    by a general medical journal. AMERICAN JOURNAL OF MEDICINE 109
    (2): 131-135.

        "The majority of the manuscripts that were rejected... were
        eventually published... in specialty journals with lower impact
        factor..."

    Donohue JM, Fox JB (2000) A multi-method evaluation of journals in the
    decision and management sciences by US academics.  OMEGA-INTERNATIONAL
    JOURNAL OF MANAGEMENT SCIENCE 28 (1): 17-36

        "perceived quality ratings of the journals are positively
        correlated with citation impact factors... and negatively
        correlated with acceptance rate."

    Yamazaki S (1995) Refereeeng System of 29 Life-Science Journals
    Preferred by JapanesE Scientists SCIENTOMETRICS 33 (1): 123-129

        "There was a high correlation between the rejection rate and
        the impact factor"

Source: http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/1111.html

Stevan Harnad
 On Sat, 5 Jun 2004, Jan Velterop wrote:

> There is of course a distortion. If one is looking to measure quality, an
> impact factor is unlikely to be the right tool. Two equivalent papers, one
> OA and the other in a subscription journal, should have the same or a very
> similar IF. If not, they're not equivalent (or, more to the point in the
> current situation, their impact isn't measured properly, e.g. by arbitrary
> exclusion from the count by the 'impact factory').
> 
> But impact factors do not measure quality; they measure impact. Not nearly
> the same thing. The OA paper of two equivalent ones is likely to have the
> best impact (when measured, of course).
> 
> Everybody is playing the impact factor game. Authors and publishers
> (including BioMed Central with some pretty nice impact factors) do,
> because most funders and tenure committees do (though often deny it), so
> careers and business prospects depend on it. But it shouldn't be confused
> with quality.
> 
> On quality flaws in high impact journals, this may be illustrative 
> reading, too:  
> http://www.biomedcentral.com/content/pdf/1471-2288-4-13.pdf
> 
> Jan Velterop