Roger Clarke's Web-Site

 

© Xamax Consultancy Pty Ltd,  1995-2017


Roger Clarke's 'Citation Analysis and Australian IS'

A Citation Analysis of Australian Information Systems Researchers
Towards a New ERA?

Roger Clarke **

Final Version of 16 April 2008

Preprint of a paper published in the Australasian Journal of Information Systems 15, 2 (June 2008), at http://dl.acs.org.au/index.php/ajis/article/view/465/436

© Xamax Consultancy Pty Ltd, 2007-08

Available under an AEShareNet Free
for Education licence or a Creative Commons 'Some
Rights Reserved' licence.

This document is at http://www.rogerclarke.com/SOS/Cit-AJIS.html


Abstract

Citation analysis is a potentially valuable means of assessing the contributions of researchers, in Information Systems (IS) as in other disciplines. In particular, a combination of raw counts and deeper analysis of citation data can deliver insights into the impact of a researcher's publications on other researchers. Despite this potential, the limited literature in the IS discipline has paid very little attention to the use of citation analysis for this purpose.

Meanwhile, the federal department responsible for education funding has convinced successive federal governments to develop research quality measures that can be used as a basis for differential funding. The Howard Government's proposed Research Quality Framework (RQF) has been abandoned, but a number of aspects of it survive within the Rudd Government's Excellence in Research for Australia (ERA) initiative. The ERA also appears likely to involve a highly formalised process whereby 'research groupings' within individual universities will be evaluated, with (as yet unclear) impacts on the distribution of research funding.

Funding agencies have an interest in score-keeping, whether or not their enthusiasm is shared by Australian researchers. It is therefore highly advisable that Australian disciplines, and especially less well-established and powerful disciplines like Information Systems, achieve a clear understanding of their performance as indicated by the available measurement techniques applied to the available data.

This paper reports on citation analysis using data from both the longstanding Thomson/ISI collection and the more recently developed Google Scholar service. Few Australian IS researchers have achieved scores of any great significance in the Thomson/ISI collection, whereas the greater depth available in Google Scholar provides a more realistic picture.

Quality assessment of the Thomson/ISI collection shows it to be seriously inappropriate for relatively new disciplines generally, and for IS in particular. Yet the Howard Government had selected the Thomson/ISI collection as the data-source for citation analysis for the now-abandoned RQF, and the Rudd Government appears likely to re-select it for the ERA.

It is critical to the ongoing acessibility of research funding by Australian IS researchers that steps be taken to ensure that decision-makers are made very well aware of the inadequacies of the ISI data collection, and that alternative, more appropriate processes for impact assessment be developed.


Contents


1. Introduction

The last two decades have seen dramatic changes in tertiary education. Successive federal governments, through the department responsible for education (until the end of 2007, the Department of Education, Science and Training - DEST, now the Department of Education, Employment and Workplace Relations - DEEWR), have re-badged institutes of technology and colleges of advanced education as universities, strongly encouraged a rapid change from collegial processes to managerialism, reduced funding per student, imposed a range of wasteful administrative processes, forced most institutions into heavy financial dependence on full-fee-paying international students, and driven them into commercially risky overseas ventures.

During 2005-07, a further major intervention was in development. The basis for distribution of research funding was to be changed to a vastly bureaucratic process. The term 'Research Quality Framework' (RQF) referred to a scheme whereby DEST would determine the quality and impact of research undertaken by groups of academics in institutions. The scores arising from that process would in turn be a major factor in the allocation of research funding. The RQF was notionally abandoned by the Rudd Government which took office at the end of 2007. The alternative scheme currently in development appears very likely, however, to inherit some, and perhaps many, features of the predecessor RQF proposal.

Assessments of research quality and impact can be based on a wide range of measures. This paper considers one key element - the count of citations of a researcher's publications. It commences with a review of the now-dead RQF and the general frame of its replacement, and some of the challenges inherent in such schemes. An overview is then provided of the study that was undertaken, followed by an analysis of the results arising from the two selected citation databases, and of the quality of the collections. The implications for the Information Systems (IS) discipline in Australia are argued to be potentially negative, and very seriously so. Conclusions are drawn regarding actions that need to be taken.


2. Measures of the Quality of Research

In recent years, several countries in Australia's reference group have implemented mechanisms for evaluating the impact of individual researchers and research groups. These have generally been an element within a broader activity, particularly relating to the award of block grants to research centres. The key examples have been the U.K. Research Assessment Exercise (RAE 2001, 2005), which was undertaken in 1986, 1989, 1992, 1996 and 2001, and the New Zealand Performance-Based Research Fund (PBRF 2005). The U.K. RAE has involved lengthy and bureaucratic specifications of how research groupings and individuals are to fill in forms, including definitions of the publications that can be included, large evaluation panels comprising people from disparate disciplines, and lengthy, rule-bound assessment processes. It has been a very highly resource-intensive activity. It had been suggested that the RAE was to be abandoned, on the basis that it has achieved its aims (MacLeod 2006), but a 2008 process is under way, with only moderate changes from the previous iteration.

During 2005-07, the then DEST devised an Australian Research Quality Framework (RQF), along similar lines, and for similar purposes (DEST 2005, 2006a, 2006b, 2007a and 2007b). Critical consideration of the RQF's likely impact on IS is in Fisher et al. (2007).

Even as it moved towards implementation, many key aspects of the Australian RQF remained vague. It sought to measure:

The unit of study of the measurement process was to be 'research groupings' within institutions. These were to be decided by each institution, but were to be subject to considerable constraints in terms of disciplinary focus and minimum size. It was envisaged that each research grouping would be assessed against 5-point scales. The outcomes were to be rankings, which were to be used (in a manner that was never explained) in decision-making about the allocation of research-funding.

The iteration current at the time of its abandonment by the incoming government at the end of 2007 restricted each researcher to their "four best outputs" during "the six-year assessment period", although a case could be made for inclusion of impacts of "original research conducted during the preceding six years" (DEST 2006a, p. 15). This represented a very narrow window into each researcher's product, product quality, and product impact. Yet worse, this was coupled with a demand that each discipline nominate a fixed list of A+, A, B and C journals, with no more than 5% of them as A+. This left no scope at all for subtlety or diversity in relation to such matters as sub-disciplines, research domains, the emergence of new sub-disciplines and domains, and the progressive re-definition of discipline scope.

A further aspect of the scheme that warranted concern was the suggestion that a measure be used comprising "the proportion of [those] publications ... which have citations [i.e. citation-counts] that would put them in the top decile for the discipline" (DEST 2006a). Such an assessment would have required not only a clear definition of complex questions about the boundaries of disciplines, but also a comprehensive database capable of supporting analysis of all citations of all publications in all disciplines. The architects of the scheme appeared to be unaware of the realities of bibliometrics, or unprepared to permit practicalities to interfere with a political imperative.

There was considerable evidence of muddy thinking within the DEST documents. During the first two years, attempts were made by academics to achieve some clarity and logic in the scheme, but as the date of implementation was drawing near, their energy was diverted to desperate endeavours to defend the interests of individual universities and disciplines.

Of particular relevance were the following problems:

The abandoned RQF is to be replaced by an Excellence in Research for Australia (ERA) initiative. At the date of writing, only two authoritative documents were available, both Ministerial Media Releases (Carr 2008a, 2008b). ERA, "to be developed by the Australian Research Council (ARC) in conjunction with the Department of Innovation, Industry, Science and Research [DIISR], will assess research quality using a combination of metrics and expert review by committees comprising experienced, internationally-recognised experts". It is to be "a transparent, workable system to assess the quality of home-grown research". (All quotations in these paragraphs are from Carr 2008a).

The 'discipline cluster' notion has survived: "ERA will use leading researchers to evaluate research activity progressively in each of the ARC discipline clusters and several clusters covering health and medical research that will be informed by experts from the National Health and Medical Research Council (NHMRC)". Verbal presentations have indicated that there are to be precisely 6 discipline clusters within the ARC and 2 within NH&MRC. This inevitably lumps together virtually all physical sciences into one 'cluster', and similarly all social sciences, all humanities and all creative arts. Attitudes to the appropriateness and worth of different categories of publishing venue are contentious even within individual disciplines, and vary enormously between disciplines. Unless the ERA somehow enables differentiation within each 'discipline cluster', there will inevitably be vicious battles over evaluation methods, and inevitably the large, well-established and well-connected disciplines will win out.

The idea that the quality measures will influence the allocation of research resources has also survived: "Each cluster report will detail by institution and by discipline those areas that are internationally competitive, together with emerging areas where there are opportunities for development and further investment", and "It will also assist with the Government's plans to introduce funding compacts for universities".

Departmental officers also succeeded in retaining at least some aspects of the prior deliberations about metrics, despite their confused and unsatisfactory nature. This is apparent because "The ERA will build on work done to date", and "consultation [would continue] with [some?] disciplines about metrics appropriate to their disciplines, noting that some measures will be appropriate to all disciplines and that for all disciplines expert review of the metrics is essential". It remains unclear as to whether the word 'disciplines' is to be taken at face value, or really means 'discipline clusters'.

In the meantime, the ERA was to "start with those disciplines where the metrics were most widely accepted, for example, in the physical and biological sciences". This appears to have the advantage for IS of leaving more time for study - unless some parts of it are drawn into the 'physical sciences', which is possible because the codes for IS are split between 'Information and Computing Sciences' Division and the 'Business and Management' Group (ARC 2006, ANZSRC 2008). The deferral has the corresponding disadvantage that the evaluation framework might be set in stone without IS and disciplines with similar characteristics even being involved in the negotiations.

Schemes such as the RAE, PBRF, RQF and the emergent ERA are political in nature, designed to provide a justification for funds-allocation decisions. Moreover, any evaluative regime of this nature is inherently backward-looking. In rewarding prior performance, it overlooks those individuals and teams that are poised to undertake breakthrough research if only they are provided with sufficient resources.

The research reported on in this paper, while acknowledging the political dimension, seeks to adopt a rational approach to the question of how to appropriately measure one particular, key aspect of research quality.

The 'merit' component of 'research quality' might be measured by a count of publications, and/or a count weighted by the venue in which each publication appeared. This is not a primary focus of the research reported on in this paper, but it was reflected to some extent in the research design.

The concept 'impact on researchers' is multi-faceted, and a variety of indicators could be used. For example, re-publication of papers in collections and anthologies, and translations into foreign languages, both indicate that a researcher's work has been both noticed and considered to be worthy. Reputation or esteem is indicated by international appointments, prizes and awards, memberships of academies and editorial boards, keynote-speaker invitations, and collaborations with other highly-reputed researchers.

A count of citations within journals is an attractive metric for several reasons. It produces quantitative data and hence statistical techniques can be applied. Journal publication is mainstream across a great many disciplines, and hence cross-disciplinary comparison may be feasible.

Many issues arise, however. There is likely to be a preference for citation-counts to be weighted in some manner, depending on the publication-venue of the cited paper and/or of the citing paper. Conference proceedings may be ignored, or weighted very low, irrespective of the norms in the discipline or research domain. In addition, the publications impact of some researchers may be tightly focussed, even on a single work, whereas the impact of other researchers may may involve the summation of many works.

Citation-count was to be the primary focus of the RQF, and it appears likely to figure prominently in the ERA as well. The research reported on in this paper focusses on citation-counts, but places them in context, and considers various measures associated with them, rather than counts alone.


3. Overview of the Study

This study was undertaken within two important contexts. One was the increasing bureaucratic interest in measuring the quality of research, as discussed in the previous section. The other context was a study of the history of the Information Systems (IS) discipline in Australia. 'Retrospectives' on the discipline were published in Clarke (2006 and 2008), the first within a Special Issue of AJIS (2006) and the second, at greater length, in Gable & Smyth (2008).

A full description of the citation analysis method adopted is reported in Clarke (2008), which also reports the outcomes of analyses of the citation records of leading and 'middle-ground' academics internationally.

Briefly, the method involved the following steps:

The remainder of this section discusses two aspects that are specific to this part of the study and hence were not addressed in Clarke (2008).

The first topic is the manner in which the list of Australian IS researchers was developed. There are currently about 700 members of the IS discipline in Australia. In order to compile a list of names, the author drew on his experience in the field since about 1970. This includes the establishment of the Australian IS Academics Directory (Clarke 1988), and involvement in the subsequent incarnations as the Australasian IS Academics Directory (Clarke 1991), the Asia Pacific Directory of IS Researchers (Gable & Clarke 1994 and 1996), and the (A)ISWorld Faculty Directory (1997-). In addition, the broader project of which this was a part included the development of a directory of full IS Professors in Australia (Clarke 2007). Somewhat arbitrary decisions were taken as to who was an expatriate Australian, and how long immigrants needed to be active in Australia to be treated as being Australian for the purposes of this analysis. Data was sought in relation to about 100 leading Australian IS researchers, plus 4 well-known and successful expatriates.

The second aspect of the method relates to the set of refereed venues that was selected in early 2006 as a basis for evaluating the scope of citation databases. As explained in Clarke (2008), this reflected the considerable body of publications on publication venues, including the primary international register of IS journals (which was established, and is maintained and published, in Australia - Lamp 2005).

By the time this paper was completed, it was known that four other relevant projects had been undertaken in parallel with this research, and published after it was completed.

Firstly, the international body of IS researchers, the Association for Information Systems (AIS) has addressed concerns within the USA about inappropriate valuations of IS publications by Deans of Graduate Schools of Business. A 'senior scholars forum' proposed that the AIS publish "a 'basket' of journals that the AIS deems as 'excellent'", and nominated "MISQ, ISR, JAIS, JMIS, EJIS, and ISJ [plus] JSIS and JIT" (depending on whether six or eight journals was decided on being appropriate) (SSF 2007).

Secondly, two papers were published in a special Featured Theme in AJIS 14, 1 (June 2007) focussing on academic journals. Fisher et al. (2007) derived a ranking list for IS journals relevant to IS in Australia. This identified 10 A and 19 B journals. Another paper (Sellito 2007) used data submitted to DEST by universities to identify the journals in which Australian IS researchers published.

Finally, in late 2007, the Australian Council of Professors and Heads of Information Systems (ACPHIS) released a list of 182 IS journals, allocating 9 A+ (limited by RQF Rules to 5% of the total), 29 A, 32 B and 112 C-grade journals.

Comparisons of these later lists with that used in this study demonstrates several differences of detail, which reflect the specific purposes of five quite distinct projects. One aspect of significance is the distinction drawn in the present study among general, specialist and regional journals. There is, however, a sufficiently broad similarity that it appears reasonable to infer that the results of the study would not have been greatly different had any of the other lists of journals developed in those studies been available and been used instead.


4. Results of the Thomson/ISI Study

Thomson Scientific now owns a service previously known as the Institute for Scientific Information (ISI). ISI was established in 1960, and has been widely available since 1988, and on the Web since 1997. What is referred to here as Thomson/ISI comprises three sub-collections marketed under the brand-name 'Web of Science'. The sub-collections are the Science Citation Index (SCI), the Social Science Citation Index (SSCI) and the Arts & Humanities Citation Index (A&HCI). The collection is well-known and much-used for bibliometric analyses generally, and citation analyses in particular.

Sometime in late 2007, DEST announced that Thomson Scientific was to be the data supplier for the RQF. (The only authority that could be located was on a web-page entitled 'RQF 2008 Documents and Forms' - DEST 2007b, although a Thomson Media Release of 2 November 2007 also mentioned it). On the basis of comments made in presentations by ARC and DIISR executives in early 2008, it appears likely that the Rudd Government will select a single supplier of a citation database to support its ERA initiative, and that database will be Thomson/ISI. For these reasons, it was essential that it be utilised as part of this study.

4.1 Method

Data was extracted from the collection over several days in late January 2006. Access was gained through the ANU Library Reverse Proxy, by means of Thomson's 'Web of Science' offering. Both sets of searches were restricted to 1978-2006, across all three Citation Indices (SCI, SSCI and A&HCI). Multiple name-spellings and initials were checked, and where doubt arose were also cross-checked with the AIS eLibrary and the (A)ISWorld Faculty Directory.

Following the Google study (reported on in the following section), supplementary research was then undertaken within the Thomson/ISI database in April 2006. During the 3 months between the two rounds of analysis using ISI, the database and hence the citation-counts of course continued to accumulate. Re-sampling was undertaken in June 2007, in order to provide information about the stability of the data collections, and the rate of change of citation-counts. Further experiments were performed, in order to enhance understanding of the quality of the counts.

4.2 Citation-Counts for Australian IS Researchers

For a small number of Australian IS researchers moderate counts were found, but the majority of Australian IS academics barely register on the ISI scales. Reasons for this are discussed below.

An arbitrary cut-off of 100 total citations was applied. (As it happened, none fell between 86 and 166). This resulted in the inclusion in the table of the 4 expatriates and 7 local researchers. Column 1 shows the total citation-count, and column 2 the number of papers found. Column 3 shows the citation-count for the person's most-cited paper, primarily to provide an indication of the upper bounds on citation-count-per-article.

Exhibit 1: ISI Data for Leading Australian IS Researchers, January 2006

CAVEAT: For reasons that are discussed progressively in this paper and in greater depth in Clarke (2008), there are strong arguments for not utilising the data in this table, and for not utilising the ISI 'General Search', as a basis for assessing the impact of individual researchers or individual articles

 
Citation Count
Number of Articles

Largest Per-Article Count

Expatriates   
Iris Vessey (as I)
601
35
111
Rick Watson (as RT), since 1989
485
28
78
Ted Stohr (as EA)
217
12
108
Peter Weill (as P) , since 2000
178
13
47
Locals
Marcus O'Connnor (as M)
354
31
66
Ron Weber (as R)
328
22

38

Philip Yetton (as P and PW), since 1975
270
26
57
Michael Lawrence (as M)
208
27
66
Michael Vitale (as M and MR), since 1995
179
14
107
Ross Jeffery (as DR, and as R)
172
28
38
Marianne Broadbent (as M)
166
24
36

The Thomson/ISI service supports several alternative measures. As described in Clarke (2008), the 'Cited Ref Search' generates higher scores, because citations within papers in the ISI collection are counted even if the cited paper itself is not in the ISI collection. Although the analysis was problematical, the results do lead to insights of some value and are provided in Exhibit 2. The people selected include one of those in Exhibit 1, together with several others from the 'middle-ground' behind the citation-count leaders.

The first group of columns provides the same data as Exhibit 1, derived using ISI's 'General Search' function. The second group of columns shows the additional citation-counts disclosed by ISI's 'Cited Ref Search'. The last two columns show the sum of the two citation-counts and the expansion factor, i.e. the ratio of the summed citation-counts to the General Search citation-count.

Exhibit 2: ISI General Search cf. Cited Ref Search, April 2006

---- In ISI

Database ----

-- Not in

ISI Database -

Researcher
Citation-Count
Article-Count
Highest Cite-Count
Citation-Count
Article-Count
Highest Cite-Count
Total Citations
Expansion Factor
Iris Vessey
622
32
114
186
52
76
808
1.3
---
---
---
---
---
---
---
---
Philip Yetton
278
20
59
65
51
6
343
1.2
Peter Seddon (P, PB)
81
4
69
67
30
22
148
1.8
Graeme Shanks (G)
66
10
15
51
32
7
117
1.8
Paula Swatman (PMC)
43
4
31
102
39
23
145
3.4
Roger Clarke (R, RA)
41
11
17
176
131
8
217
5.3
Guy Gable (GG)
35
4
29
73
24
39
108
3.1

The data in Exhibit 2 enables the following inferences to be drawn:

4.3 Quality Assessment

A quality assessment of the ISI data-holdings was undertaken, reported on at length in Clarke (2008), and summarised in Appendix 8 of that paper. Key conclusions were that:

Further discussion of the implications arising from this part of the study is provided in section 5.3 below.


5. Results of the Google Study

Google Scholar is an experimental value-adding service provided by the well-known search-engine company. It applies the same search-engine, but to a specialised sub-set of its vast index. Although it was introduced in 2004, it was still labelled a 'beta' service in early 2008.

From a bibliometric perspective, Google Scholar is crude, because it is based on brute-force free-text analysis, without recourse to metadata, and without any systematic approach to testing venues for quality before including them. On the other hand, it has the advantages of substantial reach, ready accessibility, and popularity. It is inevitable that it will be used as a basis for citation analysis, and therefore important that it be compared against the more formal ISI database.

It was selected because it is very different from Thomson/ISI, it is in a growth-phase, it appears to be the most extensive collection available, it is readily available, and it has attracted widespread attention.

5.1 Method

Google Scholar was searched for the 30 Australian researchers who were most highly cited in the Thomson/ISI collection. These elements of the empirical work were performed during April 2006. The approach adopted was necessarily different from that used for Thomson/ISI data. The method is discussed in some detail in Clarke (2008).

Re-sampling was undertaken in June 2007, in order to provide information about the stability of the data collections, and the rate of change of citation-counts. It was apparent from instability in the results that changes had occurred during the intervening period in both the Google collection and the Google service. Further experiments were performed, in order to enhance understanding of the quality of the counts. Some further re-sampling was conducted in April 2008.

5.2 Citation-Counts for Australian IS Researchers

The results for the people who qualified for Exhibit 1 are shown in Exhibit 3.

Column 1 shows the 'h-index'. This is a measure of recent origin, proposed by Hirsch (2005) and enthusiastically adopted in the bibliometrics literature. The h-index is computed by sorting a person's publications in descending order of citations, then counting down them until the publication-count matches or exceeds the citation-count. Hence a person with an h-index of 15 has 15 papers with at least 15 citations each. Further discussion of the h-index method is provided in Clarke (2008).

Column 2 shows the 'h-count'. This is the total citation-count for the papers that are within each author's 'h-index'. Column 3 shows the largest citation-count for a single paper.

Exhibit 3: Google Data for Leading Australian IS Researchers, June 2007

CAVEATS:

1. This data is based on an experimental service, and a collection of undeclared extent

2. The scores for researchers marked with asterisks may be seriously under-stated due to the non-inclusion of the proceedings of conferences at the technical end of the IS discipline

h-Index
h-Count
Largest Per-Item Count
Expatriates   
Iris Vessey
26
1,481
196
Rick Watson, since 1989
26
1,755
236
Ted/Ed Stohr
15
1,077
689
Peter Weill, since 2000
21
1,673
268
Locals
** Marcus O'Connnor
11
357
84
Ron Weber
20
1,469

172

Philip Yetton, since 1975
16
1,000
488
** Michael Lawrence
12
252
42
Michael Vitale, since 1995
14
1,152
296
** Ross Jeffery
18
763
76
Marianne Broadbent
13
1,002
230

The Google Scholar data is deeper and more finely-grained than that in ISI. A test was therefore undertaken to determine whether meaningful impact measures could be generated from Google citation-counts for the next level of researchers. This was done using a purposive sub-sample of 8 Australians, all of whom are chaired or chairworthy, but whose ISI citation-counts fell between 34 and 70 - well below the threshhold of 100 used in Exhibit 1. In Exhibit 4 below, the comparative data for the 8 researchers is shown, sorted in descending order of ISI citation-count.

Exhibit 4: ISI/Google Comparison for 'the Middle Ground'

 
ISI-Derived Data

April 2006

 
Google-Derived Data

June 2007

Citation Count
Number of Articles

Largest

Per-Article Count

 
h-Index
h-Count

Largest

Per-Item Count

Peter Seddon (P, PB)
70
6
60
 
13
674
220
Graeme Shanks (G)
61
13
14
 
13
469
114
Paula Swatman (PMC)
53
9
29
 
15
597
165
Roger Clarke (R, RA)
44
22
16
 
18
684
113
Michael Rosemann (M), since 1999
41
14
18
 
17
618
97
Chris Sauer (C), until 2000
40
18
13
 
11
493
180
Simpson Poon (S), until 2003
37
3
29
 
12
496
165
Guy Gable (GG)
34
9
23
 
12
516
141

The ISI data in the left-hand section can be seen to be too thin to support any meaningful analysis. The Google data in the right-hand section, on the other hand, is much deeper. There are several instances where researchers' h-indices exceed the entire count of articles found on ISI. Although the data extraction exercises were conducted 18 months apart, ISI's collection is stable. For example, in one re-test, the ISI citation-count increased by 30% during the period. In contrast, the gap between the two count columns averages around a factor of 20. The Google Scholar data therefore appears to be capable of more reliable interpretation.

The Google data gives rise to substantial h-indices which overlap with the measures for the Australian leaders in Exhibit 3. On the other hand, the Google h-counts of between 469 and 684, and the largest per-item counts of between 97 and 220, are both noticeably lower. Although ranking within this sample of 'Middle Ground' researchers would be contentious, the data provides a basis for comparison both with the impact leaders in the main samples above, and with other researchers internationally.

Re-tests were performed in November 2007 and April 2008. In the Google collection, over the period from June 2007 to April 2008, as shown in Exhibits 5A and 5B, the h-Counts for a micro-sample of two researchers increased at annualised rates of 70% and 50% respectively. Among the leaders, each of whom was by definition building on a larger base, growth rates ranged downwards - Philip Yetton about 50%, Iris Vessey and Rick Watson about 35%, Michael Vitale about 30%, Peter Weill 20% and Ed Stohr 10%. In the ISI collection, on the other hand, over the period from April 2006 to April 2008, Swatman's and Clarke's total citation-counts grew at annualised rates of 28% (53 to 82) and 20% (44 to 62), reflecting the smaller number of items in the ISI collection and the narrower catchment of citing articles.

Exhibit 5A: Growth in Google Scores - Paula Swatman

 
June 2007
Nov 2007
Apr 2008
h-Index
15
17
17
h-Count
597

758

866

Largest

165
195
231

Exhibit 5B: Growth in Google Scores - Roger Clarke

 
June 2007
Nov 2007
Apr 2008
h-Index
18
19
21
h-Count
684
807
890

Largest

113
133
145

5.3 Quality Assessment

Further experiments were conducted, in order to throw light on the quality of the Google Scholar collection, but also on that of the Thomson/ISI data service.

The results of the tests on Google suggested that it is feasible to specify a procedure to generate meaningful measures from the collection, but that the undertaking was dependent upon a great deal of knowledge about the author's works, and on fuzzy matching.

Experiments were conducted to test the quality of the ISI collection, using Google as a benchmark. A sub-set of 7 of the Australian researchers was selected, including 3 of the 7 leaders and 4 of the 'middle ground' sample. Their top-10 Google citation-counts were extracted, and comparison made with the ISI results. In each case, careful comparison was necessary, to ensure accurate matching of the articles uncovered by Google against those disclosed by ISI. The data is shown in Exhibits A1 to A7, in Appendix A.

Google finds many more items than ISI, and finds many more citations of those items than ISI does. In the sample, the ISI count includes only 39/70 items, and even for those 39 the total ISI citation-count is only 45% of the total Google citation-count.

To some extent, this is a natural result of the very different approaches the two services adopt: indiscriminate inclusiveness on the one hand, and narrow exclusivity on the other. However, a number of aspects throw serious doubt on the adequacy of ISI as a basis on which to assess IS academics' research impact:

In the study of international IS researchers, further, serious and potentially very harmful anomalies in the ISI data collection were unearthed in respect of both an article (Delone & McLean's seminal 'Information Systems Success: The Quest for the Dependent Variable') and two individuals (Clarke 2008).

In short, this analysis confirmed the provisional conclusions reached earlier in the paper: Google Scholar suffers a number of inadequacies as a basis for score-keeping for Australian IS academics, whereas the Thomson/ISI service is so grossly inadequate that use of it would be to the serious detriment of the discipline.


6. Implications

A wide range of inferences from the overall study are identified in Clarke (2008). This section focusses on those with direct relevance to the Australian context.

Any form of citation analysis will have defects and undesirable consequences. On the other hand, it is feasible to describe a collection and analysis tool and supporting database that would be of sufficient quality to have value in supporting assessments of individuals' impacts on the research community.

For well-established disciplines, particularly in some branches of the sciences and social sciences, the quality of the Thomson/ISI service may be adequate to support assessments. It is likely to be generally seriously inadequate for relatively new disciplines, for such reasons as the long delay in accepting journals into its collection, its failure to add Volumes prior to its adoption date, and its exclusion of refereed conference proceedings. The analysis reported on in this paper and in Clarke (2008) shows it to be an extremely poor-quality means of measuring the impact of IS researchers generally, and particularly IS researchers outside the USA.

If Thomson/ISI General Search were imposed as a means of measuring Australian IS researchers' impact on other researchers, it would disadvantage every Australian IS researcher, and essentially disqualify from consideration for research funding all but about 7 people, of whom 2 have retired, 1 is employed as a consultant, 1 is a Dean, and 1 regards himself as adjacent to IS rather than firmly within it.

If instead the broader Thomson/ISI Cited Ref Search were imposed, the field might expand, but by an uncertain amount. The proposal in the now-dead RQF that each researcher would be constrained to their "four best outputs" would create a further serious hurdle. For example, if a threshhold of 100 total citations across each person's "four best outputs" were to be imposed, only a small middle-ground of Australian IS researchers would qualify; and if a threshhold of 100 citations for a single paper were imposed, virtually every IS academic in Australia would be excluded from consideration.

In comparison with Thomson/ISI, Google Scholar has much greater scope and depth; but to date it lacks in transparency, and in stability. However its advantages already outweigh its disadvantages, at least when compared with the complete inadequacy of Thomson/ISI, especially for Australian IS researchers.

A rational evaluation of a researcher's 'merit' and 'impact on researchers' would take into account multiple measures, such as the following:

On the basis of the research reported on in this paper, a process to generate measures of those kinds from Google Scholar data appears capable of being implemented. Because of the nature and quality of the data, it appears very unlikely that the process could be fully automated. On the other hand, automated tools appear feasible that would significantly reduce the resource requirements to perform, and re-perform the analysis.

If reduction to one or two measures is demanded, then the least-worst candidate would appear to be the Google Scholar h-index and h-count. The rationale underlying this proposition is that the two measures reflect both breadth and depth, rather than over-weighting the researcher's few most meritorious and/or most-heavily-cited papers. An Australian management academic has developed a tool to support such analysis, called Publish or Perish (Harzing 2007).

If a mechanism of this kind were adopted, the threshholds at the end of 2007 might be perceived to be, for an outstanding Australian IS researcher, an h-index of 25 and/or an h-count of 750, and for successful Australian IS researchers, an h-index of 12 or 15 combined with an h-count of 500. Because citations are cumulative, and the collections are not entirely stable, any such heuristic requires re-consideration at least annually.


7. Conclusions

The research reported in this paper was undertaken during the formative phases of the RQF initiative, and the paper was completed after the RQF's demise but before any clarity had emerged about the details of its replacement, the ERA. Some aspects of the conclusions must therefore be expressed in a qualified and contingent manner.

The U.K. RAE, the N.Z. PBRF, and the Australian RQF and ERA are political mechanisms aimed at focussing research funding on a small proportion of research centres within a small proportion of institutions. They are mass-production exercises, and are subject to heavily bureaucratic processes and definitions. Citation analysis used in all such processes will be inevitably largely mechanical, with simple rules applied to all disciplines, irrespective of their appropriateness.

The IS discipline lacks a power-base, and hence it appears unlikely that special treatment can be achieved on such grounds as the significance of refereed conference proceedings, the diversity and rapidity of change in the research domain and hence the substantial proportion of its publications that appear in specialist journals that attract low weightings, and the applied and instrumentalist nature of the discipline and hence the substantial proportion of its publications that appear in unrefereed venues.

The analysis undertaken in this paper shows that the use of Thomson/ISI would be a seriously inappropriate decision in respect of the IS discipline, and would result in very few IS researchers being recognised as having had a significant impact. This would be very likely to have a negative impact on access by Australian IS researchers to research funding from DEEWR and ARC sources.

Several measures were suggested in Clarke (2008) that could be taken by the IS discipline as a whole, presumably through the Association for Information Systems (AIS). Exhibit 6 identifies a number of measures that can be taken by Australian institutions, in particular the Australasian Chapter of AIS (AAIS) and the Australian Council of Professors and Heads of Information Systems (ACPHIS).

Exhibit 6: Actions Needed by AAIS and/or ACPHIS


Appendix A: ISI cf. Google Comparisons for Selected Researchers

This Appendix provides detailed comparisons of results extracted from both ISI and Google Scholar. Seven Australian academics were selected, from among both expatriates and local researchers. All but one were selected because of their relatively uncommon names, in order to ease the difficulties of undertaking the searches and thereby achieve reasonable quality in the data. The other (the author of this paper) was selected because, for researchers with common names, full knowledge of the author's publications list makes it much easier to confidently achieve a reasonable degree of accuracy. Hence no significance should be inferred from inclusion in or exclusion from this short list.

In each of the following tables:

Exhibit A1: ISI cf. Google - Iris Vessey, January/April 2006

Google Count
Thomson Count
Venue
145
111
Journal
92
Unindexed (!!)
Journal (ISR)
88
83
Journal
86
Unindexed (!!)
Journal (CACM)
56
26
Journal
52
Unindexed
Conference (ICIS)
52
Unindexed
Journal (IJMMS)
48
Unindexed (!!)
Journal (CACM)
41
Unindexed (!!)
Journal (IEEE Software)
31
9
Journal
691 or 320
229 (of 601)
Totals

Exhibit A2: ISI cf. Google - Ron Weber, January/April 2006

Google Count
Thomson Count
Venue
125
38
Journal
102
Unindexed
Journal (JIS)
106
30
Journal
87
36
Journal
72
26
Journal (Commentary)
65
20
Journal (Commentary)
45
Unindexed
Book
34
Unindexed
Journal (JIS)
34
22
Journal
31
24
Journal
701 or 520
196 (of 328)
Totals

Exhibit A3: ISI cf. Google - Philip Yetton, January/April 2006

Google Count
Thomson Count
Venue
302
Unindexed
Book
55
11
Journal
42
12
Journal
32
12
Journal
31
34
Journal (1988)
27
Unindexed
Book
26
57
Journal (1982)
20
23
Book
18
6
Journal (1985)
18
Unindexed
Government Report
571 or 224
155 (of 270)
Totals

Exhibit A4: ISI cf. Google - Peter Seddon, January/April 2006

Google Count
Thomson Count
Venue
133
60
Journal
47
Unindexed
Journal (CAIS)
43
Unindexed
Conference (ICIS)
33
Unindexed
Conference
22
Unindexed (!)
Journal (DB, 2002)
24
2
Journal (I&M, 1991)
18
2
Journal
18
Unindexed
Journal (JIS)
13
Unindexed
Conference (ECIS)
9
0
Journal (JIT, Editorial)
360 or 184
64 (of 70)
Totals

Exhibit A5: ISI cf. Google - Paula Swatman, January/April 2006

Google Count
Thomson Count
Venue
117
29
Journal
73
Unindexed
Journal (Int'l Mkting Rev)
61
Unindexed
Journal (TIS)
43
Unindexed
Journal (JSIS)
29
Unindexed (!)
Journal (IJEC)
26
12
Journal
26
Unindexed
Journal (JIS)
24
6
Journal
22
Unindexed
Conference
20
Unindexed
Journal (EM)
441 or 167
47 (of 53)
Totals

Exhibit A6: ISI cf. Google - Roger Clarke, January/April 2006

Position
Google Count
Thomson Count
Venue
57
81
14
Journal
59
85
16
Journal
102
60
Unindexed
Journal (IT&P)
148
47
Unindexed
Journal (TIS)
253
33
Unindexed
Conference
325
28
Unindexed
Conference
373
25
3
Journal
407
23
Unindexed
Journal (JSIS)
539
18
Unindexed
Journal
558
17
Unindexed
Conference
417 or 191
33 (of 44)
Totals

Exhibit A7: ISI cf. Google - Guy Gable, January/April 2006

Google Count
Thomson Count
Venue
102
Unindexed (!)
Journal (EJIS, 1994)
56
Unindexed (!)
Journal (ISF, 2000)
40
Unindexed
Journal (JGIM)
27
6
Journal (MS)
27
Unindexed
Conference
24
23
Journal (I&M, 1991)
23
Unindexed
Conference
14
Unindexed
Conference
13
Unindexed
Conference
10
Unindexed
Conference
336 or 51
29 (of 34)
Totals


References

Except where otherwise noted, all URLs were most recently access on 16 April 2008.

AJIS (2006) `AJIS Featured Theme: The Information Systems Discipline in Australian Universities' Australasian Journal of Information Systems 14, 1 (November 2006) 123-140, at http://dl.acs.org.au/index.php/ajis/issue/view/1

ANZSRC (2008) '1297.0 - Australian and New Zealand Standard Research Classification' Australian Bureau of Statistics, March 2008, at http://www.abs.gov.au/AUSSTATS/abs@.nsf/Latestproducts/1297.0Contents12008?opendocument&tabna

ARC (2006) 'Research Fields, Courses and Disciplines Classification (RFCD)' Australian Research Council, undated, apparently of 21 February 2006, withdrawn at the end of 2007 from http://www.arc.gov.au/apply_grants/rfcd_seo_codes.htm, and superseded by ANZSRC (2008)

CAIS (2007) 21 (2007) Special Volume on the IS Academic Discipline in Pacific Asia in 2006, Communications of the AIS, 2007

Carr K. (2008a) 'New ERA for research quality: Announcement of Excellence in Research for Australia initiative' Minister for Innovation, Industry, Science and Research, 26 February 2008, at http://www.arc.gov.au/media/releases/media_26feb08.htm

Carr K. (2008b) 'New classification to assist ERA development' Minister for Innovation, Industry, Science and Research, 31 March 2008, at http://www.arc.gov.au/media/releases/media_31march08.htm

Clarke R. (Ed.) (1988) 'Australian Information Systems Academics: 1988/89 Directory' Australian National University, November 1988

Clarke R. (Ed.) (1991) 'Australasian Information Systems Academics: 1991 Directory' Australian National University, April 1991

Clarke R. (2006) 'Key Aspects of the History of the Information Systems Discipline in Australia' Australasian Journal of Information Systems 14, 1 (November 2006) 123-140, at http://dl.acs.org.au/index.php/ajis/article/view/12/11

Clarke R. (2007) 'A Retrospective on the Information Systems Discipline in Australia: Appendix 4: Professors' Xamax Consultancy Pty Ltd, March 2007, at http://www.rogerclarke.com/SOS/AISHistApp4.html

Clarke R. (2008) 'An Exploratory Study of Information Systems Researcher Impact' Commun. AIS 22, 1 (January 2008), at http://cais.aisnet.org/articles/default.asp?vol=22&art=1, and at http://www.rogerclarke.com/SOS/Cit-CAIS.html

DEST (2005) 'Research Quality Framework: Assessing the quality and impact of research in Australia: Final Advice on the Preferred RQF Model' Department of Education, Science & Training, December 2005, withdrawn at the end of 2007 from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issue/research_quality_framework/final_advice_on_preferred_rqf_model.htm

DEST (2006a) 'Research Quality' Department of Education, Science and Training, undated but apparently of mid-2006, withdrawn at the end of 2007 from http://www.dest.gov.au/NR/rdonlyres/26AFCA18-A4CA-481E-B54F-2A83C29528B0/13523/RsearchQuality.pdf

DEST (2006b) 'Research Quality Framework: Assessing the Quality and Impact of Research in Australia: The Recommended RQF' Department of Education, Science and Training, October 2006, withdrawn at the end of 2007 from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issue/research_quality_framework/documents/the_recommended_RQF_pdf.htm

DEST (2007a) 'Research Quality Framework Fact Sheet' undated, but apparently of May 2007, withdrawn at the end of 2007 from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issue/research_quality_framework/documents/RQF_factsheet_2006/RQF_pdf.htm

DEST (2007b) 'RQF 2008 Documents and Forms', Department of Education, Science and Training, September 2007, , withdrawn at the end of 2007 from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issue/research_quality_framework/documents_and_forms.htm

Fisher J., Shanks G. & Lamp J. (2007) 'A Ranking List for Information Systems Journals' Australasian J. of Infor. Syst. 14, 2 (June 2007), at http://dl.acs.org.au/index.php/ajis/article/view/469/417

Gable G. & Clarke R. (Eds.) (1994) 'Asia Pacific Directory of Information Systems Researchers: 1994' National University of Singapore, 1994

Gable G. & Clarke R. (Eds.) (1996) 'Asia Pacific Directory of Information Systems Researchers: 1996' National University of Singapore, 1996

Gable G. & Smyth R. (Eds.) (2008) 'The Information Systems Discipline in Australian Universities' Forthcoming, ANU ePress, 2008

Harzing A.-W. (2007) 'Google Scholar - a new data source for citation analysis', University of Melbourne, 2007, at http://www.harzing.com/pop_gs.htm

Hirsch J.E. (2005) 'An index to quantify an individual's scientific research output' arXiv:physics/0508025v5, 29 September 2005, at http://arxiv.org/PS_cache/physics/pdf/0508/0508025v5.pdf, accessed July 2007

Lamp J. (2005) 'The Index of Information Systems Journals', Deakin University, version of 16 August 2005, at http://lamp.infosys.deakin.edu.au/journals/index.php

MacLeod D. (2006) 'Research exercise to be scrapped' The Guardian, 22 March 2006, at http://education.guardian.co.uk/RAE/story/0,,1737082,00.html

PBRF (2005) 'Performance-Based Research Fund', N.Z. Tertiary Education Commission, July 2005, at http://www.tec.govt.nz/downloads/a2z_publications/pbrf2006-guidelines.pdf

RAE (2001) 'A guide to the 2001 Research Assessment Exercise', U.K. Department for Employment and Learning, apparently undated, at http://www.hero.ac.uk/rae/Pubs/other/raeguide.pdf

RAE (2005) 'Guidance on submissions' Department for Employment and Learning, RAE 03/2005, June 2005, at http://www.rae.ac.uk/pubs/2005/03/rae0305.pdf

Sellito C. (2007) 'A Study of Journal Publication Diversity Within the Australian Information Systems Sphere' Australasian J. of Infor. Syst. 14, 1 (June 2007), at http://dl.acs.org.au/index.php/ajis/article/view/26/409

SSF (2007) 'Letter to the AIS President' AIS Senior Scholars Forum Subcommittee on Journals, 23 April 2007, at http://home.aisnet.org/associations/7499/files/Senior%20Scholars%20Letter.pdf


Acknowledgements

The work reported on in this paper was conducted within the context of a major collaborative project on the IS discipline, led by Guy Gable and Bob Smyth at QUT. A report on Australia was published in a Special Issue of the Australasian Journal of Information Systems 14, 1 (AJIS 2006), and a further report on Pacific Asia appeared as a Special Issue in CAIS (2007).

The author acknowledges valuable input from Peter Seddon of the University of Melbourne and Colin Steele of the ANU, and the feedback from the Associate Editor and reviewers. Responsibility for all aspects of the work rests, of course, entirely with the author.


Author Affiliations

Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., a Visiting Professor in the E-Commerce Programme at the University of Hong Kong, and a Visiting Professor in the Department of Computer Science at the Australian National University.



xamaxsmall.gif missing
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.

From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 50 million in early 2015.

Sponsored by Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916

Created: 15 November 2007 - Last Amended: 16 April 2008 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/SOS/Cit-AJIS.html
Mail to Webmaster   -    © Xamax Consultancy Pty Ltd, 1995-2017   -    Privacy Policy