Roger Clarke's Web-Site
© Xamax Consultancy Pty Ltd, 1995-2024
Infrastructure
& Privacy
Matilda
Roger Clarke's Web-Site© Xamax Consultancy Pty Ltd, 1995-2024 |
|
|||||
| HOME | eBusiness |
Information Infrastructure |
Dataveillance & Privacy |
Identity Matters | Other Topics | |
| What's New |
Waltzing Matilda | Advanced Site-Search | ||||
Final Version of 16 April 2008
Preprint of a paper published in the Australasian Journal of Information Systems 15, 2 (June 2008), at http://dl.acs.org.au/index.php/ajis/article/view/465/436
© Xamax Consultancy Pty Ltd, 2007-08
Available under an AEShareNet 
licence or a Creative
Commons 
licence.
This document is at http://www.rogerclarke.com/SOS/Cit-AJIS.html
Citation analysis is a potentially valuable means of assessing the contributions of researchers, in Information Systems (IS) as in other disciplines. In particular, a combination of raw counts and deeper analysis of citation data can deliver insights into the impact of a researcher's publications on other researchers. Despite this potential, the limited literature in the IS discipline has paid very little attention to the use of citation analysis for this purpose.
Meanwhile, the federal department responsible for education funding has convinced successive federal governments to develop research quality measures that can be used as a basis for differential funding. The Howard Government's proposed Research Quality Framework (RQF) has been abandoned, but a number of aspects of it survive within the Rudd Government's Excellence in Research for Australia (ERA) initiative. The ERA also appears likely to involve a highly formalised process whereby 'research groupings' within individual universities will be evaluated, with (as yet unclear) impacts on the distribution of research funding.
Funding agencies have an interest in score-keeping, whether or not their enthusiasm is shared by Australian researchers. It is therefore highly advisable that Australian disciplines, and especially less well-established and powerful disciplines like Information Systems, achieve a clear understanding of their performance as indicated by the available measurement techniques applied to the available data.
This paper reports on citation analysis using data from both the longstanding Thomson/ISI collection and the more recently developed Google Scholar service. Few Australian IS researchers have achieved scores of any great significance in the Thomson/ISI collection, whereas the greater depth available in Google Scholar provides a more realistic picture.
Quality assessment of the Thomson/ISI collection shows it to be seriously inappropriate for relatively new disciplines generally, and for IS in particular. Yet the Howard Government had selected the Thomson/ISI collection as the data-source for citation analysis for the now-abandoned RQF, and the Rudd Government appears likely to re-select it for the ERA.
It is critical to the ongoing acessibility of research funding by Australian IS researchers that steps be taken to ensure that decision-makers are made very well aware of the inadequacies of the ISI data collection, and that alternative, more appropriate processes for impact assessment be developed.
The last two decades have seen dramatic changes in tertiary education. Successive federal governments, through the department responsible for education (until the end of 2007, the Department of Education, Science and Training - DEST, now the Department of Education, Employment and Workplace Relations - DEEWR), have re-badged institutes of technology and colleges of advanced education as universities, strongly encouraged a rapid change from collegial processes to managerialism, reduced funding per student, imposed a range of wasteful administrative processes, forced most institutions into heavy financial dependence on full-fee-paying international students, and driven them into commercially risky overseas ventures.
During 2005-07, a further major intervention was in development. The basis for distribution of research funding was to be changed to a vastly bureaucratic process. The term 'Research Quality Framework' (RQF) referred to a scheme whereby DEST would determine the quality and impact of research undertaken by groups of academics in institutions. The scores arising from that process would in turn be a major factor in the allocation of research funding. The RQF was notionally abandoned by the Rudd Government which took office at the end of 2007. The alternative scheme currently in development appears very likely, however, to inherit some, and perhaps many, features of the predecessor RQF proposal.
Assessments of research quality and impact can be based on a wide range of measures. This paper considers one key element - the count of citations of a researcher's publications. It commences with a review of the now-dead RQF and the general frame of its replacement, and some of the challenges inherent in such schemes. An overview is then provided of the study that was undertaken, followed by an analysis of the results arising from the two selected citation databases, and of the quality of the collections. The implications for the Information Systems (IS) discipline in Australia are argued to be potentially negative, and very seriously so. Conclusions are drawn regarding actions that need to be taken.
In recent years, several countries in Australia's reference group have implemented mechanisms for evaluating the impact of individual researchers and research groups. These have generally been an element within a broader activity, particularly relating to the award of block grants to research centres. The key examples have been the U.K. Research Assessment Exercise (RAE 2001, 2005), which was undertaken in 1986, 1989, 1992, 1996 and 2001, and the New Zealand Performance-Based Research Fund (PBRF 2005). The U.K. RAE has involved lengthy and bureaucratic specifications of how research groupings and individuals are to fill in forms, including definitions of the publications that can be included, large evaluation panels comprising people from disparate disciplines, and lengthy, rule-bound assessment processes. It has been a very highly resource-intensive activity. It had been suggested that the RAE was to be abandoned, on the basis that it has achieved its aims (MacLeod 2006), but a 2008 process is under way, with only moderate changes from the previous iteration.
During 2005-07, the then DEST devised an Australian Research Quality Framework (RQF), along similar lines, and for similar purposes (DEST 2005, 2006a, 2006b, 2007a and 2007b). Critical consideration of the RQF's likely impact on IS is in Fisher et al. (2007).
Even as it moved towards implementation, many key aspects of the Australian RQF remained vague. It sought to measure:
The unit of study of the measurement process was to be 'research groupings' within institutions. These were to be decided by each institution, but were to be subject to considerable constraints in terms of disciplinary focus and minimum size. It was envisaged that each research grouping would be assessed against 5-point scales. The outcomes were to be rankings, which were to be used (in a manner that was never explained) in decision-making about the allocation of research-funding.
The iteration current at the time of its abandonment by the incoming government at the end of 2007 restricted each researcher to their "four best outputs" during "the six-year assessment period", although a case could be made for inclusion of impacts of "original research conducted during the preceding six years" (DEST 2006a, p. 15). This represented a very narrow window into each researcher's product, product quality, and product impact. Yet worse, this was coupled with a demand that each discipline nominate a fixed list of A+, A, B and C journals, with no more than 5% of them as A+. This left no scope at all for subtlety or diversity in relation to such matters as sub-disciplines, research domains, the emergence of new sub-disciplines and domains, and the progressive re-definition of discipline scope.
A further aspect of the scheme that warranted concern was the suggestion that a measure be used comprising "the proportion of [those] publications ... which have citations [i.e. citation-counts] that would put them in the top decile for the discipline" (DEST 2006a). Such an assessment would have required not only a clear definition of complex questions about the boundaries of disciplines, but also a comprehensive database capable of supporting analysis of all citations of all publications in all disciplines. The architects of the scheme appeared to be unaware of the realities of bibliometrics, or unprepared to permit practicalities to interfere with a political imperative.
There was considerable evidence of muddy thinking within the DEST documents. During the first two years, attempts were made by academics to achieve some clarity and logic in the scheme, but as the date of implementation was drawing near, their energy was diverted to desperate endeavours to defend the interests of individual universities and disciplines.
Of particular relevance were the following problems:
The abandoned RQF is to be replaced by an Excellence in Research for Australia (ERA) initiative. At the date of writing, only two authoritative documents were available, both Ministerial Media Releases (Carr 2008a, 2008b). ERA, "to be developed by the Australian Research Council (ARC) in conjunction with the Department of Innovation, Industry, Science and Research [DIISR], will assess research quality using a combination of metrics and expert review by committees comprising experienced, internationally-recognised experts". It is to be "a transparent, workable system to assess the quality of home-grown research". (All quotations in these paragraphs are from Carr 2008a).
The 'discipline cluster' notion has survived: "ERA will use leading researchers to evaluate research activity progressively in each of the ARC discipline clusters and several clusters covering health and medical research that will be informed by experts from the National Health and Medical Research Council (NHMRC)". Verbal presentations have indicated that there are to be precisely 6 discipline clusters within the ARC and 2 within NH&MRC. This inevitably lumps together virtually all physical sciences into one 'cluster', and similarly all social sciences, all humanities and all creative arts. Attitudes to the appropriateness and worth of different categories of publishing venue are contentious even within individual disciplines, and vary enormously between disciplines. Unless the ERA somehow enables differentiation within each 'discipline cluster', there will inevitably be vicious battles over evaluation methods, and inevitably the large, well-established and well-connected disciplines will win out.
The idea that the quality measures will influence the allocation of research resources has also survived: "Each cluster report will detail by institution and by discipline those areas that are internationally competitive, together with emerging areas where there are opportunities for development and further investment", and "It will also assist with the Government's plans to introduce funding compacts for universities".
Departmental officers also succeeded in retaining at least some aspects of the prior deliberations about metrics, despite their confused and unsatisfactory nature. This is apparent because "The ERA will build on work done to date", and "consultation [would continue] with [some?] disciplines about metrics appropriate to their disciplines, noting that some measures will be appropriate to all disciplines and that for all disciplines expert review of the metrics is essential". It remains unclear as to whether the word 'disciplines' is to be taken at face value, or really means 'discipline clusters'.
In the meantime, the ERA was to "start with those disciplines where the metrics were most widely accepted, for example, in the physical and biological sciences". This appears to have the advantage for IS of leaving more time for study - unless some parts of it are drawn into the 'physical sciences', which is possible because the codes for IS are split between 'Information and Computing Sciences' Division and the 'Business and Management' Group (ARC 2006, ANZSRC 2008). The deferral has the corresponding disadvantage that the evaluation framework might be set in stone without IS and disciplines with similar characteristics even being involved in the negotiations.
Schemes such as the RAE, PBRF, RQF and the emergent ERA are political in nature, designed to provide a justification for funds-allocation decisions. Moreover, any evaluative regime of this nature is inherently backward-looking. In rewarding prior performance, it overlooks those individuals and teams that are poised to undertake breakthrough research if only they are provided with sufficient resources.
The research reported on in this paper, while acknowledging the political dimension, seeks to adopt a rational approach to the question of how to appropriately measure one particular, key aspect of research quality.
The 'merit' component of 'research quality' might be measured by a count of publications, and/or a count weighted by the venue in which each publication appeared. This is not a primary focus of the research reported on in this paper, but it was reflected to some extent in the research design.
The concept 'impact on researchers' is multi-faceted, and a variety of indicators could be used. For example, re-publication of papers in collections and anthologies, and translations into foreign languages, both indicate that a researcher's work has been both noticed and considered to be worthy. Reputation or esteem is indicated by international appointments, prizes and awards, memberships of academies and editorial boards, keynote-speaker invitations, and collaborations with other highly-reputed researchers.
A count of citations within journals is an attractive metric for several reasons. It produces quantitative data and hence statistical techniques can be applied. Journal publication is mainstream across a great many disciplines, and hence cross-disciplinary comparison may be feasible.
Many issues arise, however. There is likely to be a preference for citation-counts to be weighted in some manner, depending on the publication-venue of the cited paper and/or of the citing paper. Conference proceedings may be ignored, or weighted very low, irrespective of the norms in the discipline or research domain. In addition, the publications impact of some researchers may be tightly focussed, even on a single work, whereas the impact of other researchers may may involve the summation of many works.
Citation-count was to be the primary focus of the RQF, and it appears likely to figure prominently in the ERA as well. The research reported on in this paper focusses on citation-counts, but places them in context, and considers various measures associated with them, rather than counts alone.
This study was undertaken within two important contexts. One was the increasing bureaucratic interest in measuring the quality of research, as discussed in the previous section. The other context was a study of the history of the Information Systems (IS) discipline in Australia. 'Retrospectives' on the discipline were published in Clarke (2006 and 2008), the first within a Special Issue of AJIS (2006) and the second, at greater length, in Gable & Smyth (2008).
A full description of the citation analysis method adopted is reported in Clarke (2008), which also reports the outcomes of analyses of the citation records of leading and 'middle-ground' academics internationally.
Briefly, the method involved the following steps:
The remainder of this section discusses two aspects that are specific to this part of the study and hence were not addressed in Clarke (2008).
The first topic is the manner in which the list of Australian IS researchers was developed. There are currently about 700 members of the IS discipline in Australia. In order to compile a list of names, the author drew on his experience in the field since about 1970. This includes the establishment of the Australian IS Academics Directory (Clarke 1988), and involvement in the subsequent incarnations as the Australasian IS Academics Directory (Clarke 1991), the Asia Pacific Directory of IS Researchers (Gable & Clarke 1994 and 1996), and the (A)ISWorld Faculty Directory (1997-). In addition, the broader project of which this was a part included the development of a directory of full IS Professors in Australia (Clarke 2007). Somewhat arbitrary decisions were taken as to who was an expatriate Australian, and how long immigrants needed to be active in Australia to be treated as being Australian for the purposes of this analysis. Data was sought in relation to about 100 leading Australian IS researchers, plus 4 well-known and successful expatriates.
The second aspect of the method relates to the set of refereed venues that was selected in early 2006 as a basis for evaluating the scope of citation databases. As explained in Clarke (2008), this reflected the considerable body of publications on publication venues, including the primary international register of IS journals (which was established, and is maintained and published, in Australia - Lamp 2005).
By the time this paper was completed, it was known that four other relevant projects had been undertaken in parallel with this research, and published after it was completed.
Firstly, the international body of IS researchers, the Association for Information Systems (AIS) has addressed concerns within the USA about inappropriate valuations of IS publications by Deans of Graduate Schools of Business. A 'senior scholars forum' proposed that the AIS publish "a 'basket' of journals that the AIS deems as 'excellent'", and nominated "MISQ, ISR, JAIS, JMIS, EJIS, and ISJ [plus] JSIS and JIT" (depending on whether six or eight journals was decided on being appropriate) (SSF 2007).
Secondly, two papers were published in a special Featured Theme in AJIS 14, 1 (June 2007) focussing on academic journals. Fisher et al. (2007) derived a ranking list for IS journals relevant to IS in Australia. This identified 10 A and 19 B journals. Another paper (Sellito 2007) used data submitted to DEST by universities to identify the journals in which Australian IS researchers published.
Finally, in late 2007, the Australian Council of Professors and Heads of Information Systems (ACPHIS) released a list of 182 IS journals, allocating 9 A+ (limited by RQF Rules to 5% of the total), 29 A, 32 B and 112 C-grade journals.
Comparisons of these later lists with that used in this study demonstrates several differences of detail, which reflect the specific purposes of five quite distinct projects. One aspect of significance is the distinction drawn in the present study among general, specialist and regional journals. There is, however, a sufficiently broad similarity that it appears reasonable to infer that the results of the study would not have been greatly different had any of the other lists of journals developed in those studies been available and been used instead.
Thomson Scientific now owns a service previously known as the Institute for Scientific Information (ISI). ISI was established in 1960, and has been widely available since 1988, and on the Web since 1997. What is referred to here as Thomson/ISI comprises three sub-collections marketed under the brand-name 'Web of Science'. The sub-collections are the Science Citation Index (SCI), the Social Science Citation Index (SSCI) and the Arts & Humanities Citation Index (A&HCI). The collection is well-known and much-used for bibliometric analyses generally, and citation analyses in particular.
Sometime in late 2007, DEST announced that Thomson Scientific was to be the data supplier for the RQF. (The only authority that could be located was on a web-page entitled 'RQF 2008 Documents and Forms' - DEST 2007b, although a Thomson Media Release of 2 November 2007 also mentioned it). On the basis of comments made in presentations by ARC and DIISR executives in early 2008, it appears likely that the Rudd Government will select a single supplier of a citation database to support its ERA initiative, and that database will be Thomson/ISI. For these reasons, it was essential that it be utilised as part of this study.
Data was extracted from the collection over several days in late January 2006. Access was gained through the ANU Library Reverse Proxy, by means of Thomson's 'Web of Science' offering. Both sets of searches were restricted to 1978-2006, across all three Citation Indices (SCI, SSCI and A&HCI). Multiple name-spellings and initials were checked, and where doubt arose were also cross-checked with the AIS eLibrary and the (A)ISWorld Faculty Directory.
Following the Google study (reported on in the following section), supplementary research was then undertaken within the Thomson/ISI database in April 2006. During the 3 months between the two rounds of analysis using ISI, the database and hence the citation-counts of course continued to accumulate. Re-sampling was undertaken in June 2007, in order to provide information about the stability of the data collections, and the rate of change of citation-counts. Further experiments were performed, in order to enhance understanding of the quality of the counts.
For a small number of Australian IS researchers moderate counts were found, but the majority of Australian IS academics barely register on the ISI scales. Reasons for this are discussed below.
An arbitrary cut-off of 100 total citations was applied. (As it happened, none fell between 86 and 166). This resulted in the inclusion in the table of the 4 expatriates and 7 local researchers. Column 1 shows the total citation-count, and column 2 the number of papers found. Column 3 shows the citation-count for the person's most-cited paper, primarily to provide an indication of the upper bounds on citation-count-per-article.
CAVEAT: For reasons that are discussed progressively in this paper and in greater depth in Clarke (2008), there are strong arguments for not utilising the data in this table, and for not utilising the ISI 'General Search', as a basis for assessing the impact of individual researchers or individual articles
Citation Count | Number of Articles | Largest Per-Article Count | |
| Expatriates | |||
| Iris Vessey (as I) | 601 | 35 | 111 |
| Rick Watson (as RT), since 1989 | 485 | 28 | 78 |
| Ted Stohr (as EA) | 217 | 12 | 108 |
| Peter Weill (as P) , since 2000 | 178 | 13 | 47 |
| Locals | |||
| Marcus O'Connnor (as M) | 354 | 31 | 66 |
| Ron Weber (as R) | 328 | 22 | 38 |
| Philip Yetton (as P and PW), since 1975 | 270 | 26 | 57 |
| Michael Lawrence (as M) | 208 | 27 | 66 |
| Michael Vitale (as M and MR), since 1995 | 179 | 14 | 107 |
| Ross Jeffery (as DR, and as R) | 172 | 28 | 38 |
| Marianne Broadbent (as M) | 166 | 24 | 36 |
The Thomson/ISI service supports several alternative measures. As described in Clarke (2008), the 'Cited Ref Search' generates higher scores, because citations within papers in the ISI collection are counted even if the cited paper itself is not in the ISI collection. Although the analysis was problematical, the results do lead to insights of some value and are provided in Exhibit 2. The people selected include one of those in Exhibit 1, together with several others from the 'middle-ground' behind the citation-count leaders.
The first group of columns provides the same data as Exhibit 1, derived using ISI's 'General Search' function. The second group of columns shows the additional citation-counts disclosed by ISI's 'Cited Ref Search'. The last two columns show the sum of the two citation-counts and the expansion factor, i.e. the ratio of the summed citation-counts to the General Search citation-count.
---- In ISI
Database ---- | -- Not in
ISI Database - | |||||||
Researcher | Citation-Count | Article-Count | Highest
Cite-Count | Citation-Count | Article-Count | Highest
Cite-Count | Total
Citations | Expansion
Factor |
| Iris Vessey | 622 | 32 | 114 | 186 | 52 | 76 | 808 | 1.3 |
--- | --- | --- | --- | --- | --- | --- | --- | |
| Philip Yetton | 278 | 20 | 59 | 65 | 51 | 6 | 343 | 1.2 |
| Peter Seddon (P, PB) | 81 | 4 | 69 | 67 | 30 | 22 | 148 | 1.8 |
| Graeme Shanks (G) | 66 | 10 | 15 | 51 | 32 | 7 | 117 | 1.8 |
| Paula Swatman (PMC) | 43 | 4 | 31 | 102 | 39 | 23 | 145 | 3.4 |
| Roger Clarke (R, RA) | 41 | 11 | 17 | 176 | 131 | 8 | 217 | 5.3 |
| Guy Gable (GG) | 35 | 4 | 29 | 73 | 24 | 39 | 108 | 3.1 |
The data in Exhibit 2 enables the following inferences to be drawn:
A quality assessment of the ISI data-holdings was undertaken, reported on at length in Clarke (2008), and summarised in Appendix 8 of that paper. Key conclusions were that:
Further discussion of the implications arising from this part of the study is provided in section 5.3 below.
Google Scholar is an experimental value-adding service provided by the well-known search-engine company. It applies the same search-engine, but to a specialised sub-set of its vast index. Although it was introduced in 2004, it was still labelled a 'beta' service in early 2008.
From a bibliometric perspective, Google Scholar is crude, because it is based on brute-force free-text analysis, without recourse to metadata, and without any systematic approach to testing venues for quality before including them. On the other hand, it has the advantages of substantial reach, ready accessibility, and popularity. It is inevitable that it will be used as a basis for citation analysis, and therefore important that it be compared against the more formal ISI database.
It was selected because it is very different from Thomson/ISI, it is in a growth-phase, it appears to be the most extensive collection available, it is readily available, and it has attracted widespread attention.
Google Scholar was searched for the 30 Australian researchers who were most highly cited in the Thomson/ISI collection. These elements of the empirical work were performed during April 2006. The approach adopted was necessarily different from that used for Thomson/ISI data. The method is discussed in some detail in Clarke (2008).
Re-sampling was undertaken in June 2007, in order to provide information about the stability of the data collections, and the rate of change of citation-counts. It was apparent from instability in the results that changes had occurred during the intervening period in both the Google collection and the Google service. Further experiments were performed, in order to enhance understanding of the quality of the counts. Some further re-sampling was conducted in April 2008.
The results for the people who qualified for Exhibit 1 are shown in Exhibit 3.
Column 1 shows the 'h-index'. This is a measure of recent origin, proposed by Hirsch (2005) and enthusiastically adopted in the bibliometrics literature. The h-index is computed by sorting a person's publications in descending order of citations, then counting down them until the publication-count matches or exceeds the citation-count. Hence a person with an h-index of 15 has 15 papers with at least 15 citations each. Further discussion of the h-index method is provided in Clarke (2008).
Column 2 shows the 'h-count'. This is the total citation-count for the papers that are within each author's 'h-index'. Column 3 shows the largest citation-count for a single paper.
CAVEATS:
1. This data is based on an experimental service, and a collection of undeclared extent
2. The scores for researchers marked with asterisks may be seriously under-stated due to the non-inclusion of the proceedings of conferences at the technical end of the IS discipline
h-Index | h-Count | Largest Per-Item
Count | |
| Expatriates | |||
| Iris Vessey | 26 | 1,481 | 196 |
| Rick Watson, since 1989 | 26 | 1,755 | 236 |
| Ted/Ed Stohr | 15 | 1,077 | 689 |
| Peter Weill, since 2000 | 21 | 1,673 | 268 |
| Locals | |||
| ** Marcus O'Connnor | 11 | 357 | 84 |
| Ron Weber | 20 | 1,469 | 172 |
| Philip Yetton, since 1975 | 16 | 1,000 | 488 |
| ** Michael Lawrence | 12 | 252 | 42 |
| Michael Vitale, since 1995 | 14 | 1,152 | 296 |
| ** Ross Jeffery | 18 | 763 | 76 |
| Marianne Broadbent | 13 | 1,002 | 230 |
The Google Scholar data is deeper and more finely-grained than that in ISI. A test was therefore undertaken to determine whether meaningful impact measures could be generated from Google citation-counts for the next level of researchers. This was done using a purposive sub-sample of 8 Australians, all of whom are chaired or chairworthy, but whose ISI citation-counts fell between 34 and 70 - well below the threshhold of 100 used in Exhibit 1. In Exhibit 4 below, the comparative data for the 8 researchers is shown, sorted in descending order of ISI citation-count.
ISI-Derived Data
April 2006
| Google-Derived Data June 2007 | ||||||
Citation Count | Number of Articles | Largest Per-Article Count | h-Index | h-Count | Largest Per-Item Count | ||
| Peter Seddon (P, PB) | 70 | 6 | 60 | 13 | 674 | 220 | |
| Graeme Shanks (G) | 61 | 13 | 14 | 13 | 469 | 114 | |
| Paula Swatman (PMC) | 53 | 9 | 29 | 15 | 597 | 165 | |
| Roger Clarke (R, RA) | 44 | 22 | 16 | 18 | 684 | 113 | |
| Michael Rosemann (M), since 1999 | 41 | 14 | 18 | 17 | 618 | 97 | |
| Chris Sauer (C), until 2000 | 40 | 18 | 13 | 11 | 493 | 180 | |
| Simpson Poon (S), until 2003 | 37 | 3 | 29 | 12 | 496 | 165 | |
| Guy Gable (GG) | 34 | 9 | 23 | 12 | 516 | 141 | |
The ISI data in the left-hand section can be seen to be too thin to support any meaningful analysis. The Google data in the right-hand section, on the other hand, is much deeper. There are several instances where researchers' h-indices exceed the entire count of articles found on ISI. Although the data extraction exercises were conducted 18 months apart, ISI's collection is stable. For example, in one re-test, the ISI citation-count increased by 30% during the period. In contrast, the gap between the two count columns averages around a factor of 20. The Google Scholar data therefore appears to be capable of more reliable interpretation.
The Google data gives rise to substantial h-indices which overlap with the measures for the Australian leaders in Exhibit 3. On the other hand, the Google h-counts of between 469 and 684, and the largest per-item counts of between 97 and 220, are both noticeably lower. Although ranking within this sample of 'Middle Ground' researchers would be contentious, the data provides a basis for comparison both with the impact leaders in the main samples above, and with other researchers internationally.
Re-tests were performed in November 2007 and April 2008. In the Google collection, over the period from June 2007 to April 2008, as shown in Exhibits 5A and 5B, the h-Counts for a micro-sample of two researchers increased at annualised rates of 70% and 50% respectively. Among the leaders, each of whom was by definition building on a larger base, growth rates ranged downwards - Philip Yetton about 50%, Iris Vessey and Rick Watson about 35%, Michael Vitale about 30%, Peter Weill 20% and Ed Stohr 10%. In the ISI collection, on the other hand, over the period from April 2006 to April 2008, Swatman's and Clarke's total citation-counts grew at annualised rates of 28% (53 to 82) and 20% (44 to 62), reflecting the smaller number of items in the ISI collection and the narrower catchment of citing articles.
June 2007 | Nov 2007 | Apr 2008 | |
| h-Index | 15 | 17 | 17 |
| h-Count | 597 | 758 | 866 |
Largest | 165 | 195 | 231 |
June 2007 | Nov 2007 | Apr 2008 | |
| h-Index | 18 | 19 | 21 |
| h-Count | 684 | 807 | 890 |
Largest | 113 | 133 | 145 |
Further experiments were conducted, in order to throw light on the quality of the Google Scholar collection, but also on that of the Thomson/ISI data service.
The results of the tests on Google suggested that it is feasible to specify a procedure to generate meaningful measures from the collection, but that the undertaking was dependent upon a great deal of knowledge about the author's works, and on fuzzy matching.
Experiments were conducted to test the quality of the ISI collection, using Google as a benchmark. A sub-set of 7 of the Australian researchers was selected, including 3 of the 7 leaders and 4 of the 'middle ground' sample. Their top-10 Google citation-counts were extracted, and comparison made with the ISI results. In each case, careful comparison was necessary, to ensure accurate matching of the articles uncovered by Google against those disclosed by ISI. The data is shown in Exhibits A1 to A7, in Appendix A.
Google finds many more items than ISI, and finds many more citations of those items than ISI does. In the sample, the ISI count includes only 39/70 items, and even for those 39 the total ISI citation-count is only 45% of the total Google citation-count.
To some extent, this is a natural result of the very different approaches the two services adopt: indiscriminate inclusiveness on the one hand, and narrow exclusivity on the other. However, a number of aspects throw serious doubt on the adequacy of ISI as a basis on which to assess IS academics' research impact:
In the study of international IS researchers, further, serious and potentially very harmful anomalies in the ISI data collection were unearthed in respect of both an article (Delone & McLean's seminal 'Information Systems Success: The Quest for the Dependent Variable') and two individuals (Clarke 2008).
In short, this analysis confirmed the provisional conclusions reached earlier in the paper: Google Scholar suffers a number of inadequacies as a basis for score-keeping for Australian IS academics, whereas the Thomson/ISI service is so grossly inadequate that use of it would be to the serious detriment of the discipline.
A wide range of inferences from the overall study are identified in Clarke (2008). This section focusses on those with direct relevance to the Australian context.
Any form of citation analysis will have defects and undesirable consequences. On the other hand, it is feasible to describe a collection and analysis tool and supporting database that would be of sufficient quality to have value in supporting assessments of individuals' impacts on the research community.
For well-established disciplines, particularly in some branches of the sciences and social sciences, the quality of the Thomson/ISI service may be adequate to support assessments. It is likely to be generally seriously inadequate for relatively new disciplines, for such reasons as the long delay in accepting journals into its collection, its failure to add Volumes prior to its adoption date, and its exclusion of refereed conference proceedings. The analysis reported on in this paper and in Clarke (2008) shows it to be an extremely poor-quality means of measuring the impact of IS researchers generally, and particularly IS researchers outside the USA.
If Thomson/ISI General Search were imposed as a means of measuring Australian IS researchers' impact on other researchers, it would disadvantage every Australian IS researcher, and essentially disqualify from consideration for research funding all but about 7 people, of whom 2 have retired, 1 is employed as a consultant, 1 is a Dean, and 1 regards himself as adjacent to IS rather than firmly within it.
If instead the broader Thomson/ISI Cited Ref Search were imposed, the field might expand, but by an uncertain amount. The proposal in the now-dead RQF that each researcher would be constrained to their "four best outputs" would create a further serious hurdle. For example, if a threshhold of 100 total citations across each person's "four best outputs" were to be imposed, only a small middle-ground of Australian IS researchers would qualify; and if a threshhold of 100 citations for a single paper were imposed, virtually every IS academic in Australia would be excluded from consideration.
In comparison with Thomson/ISI, Google Scholar has much greater scope and depth; but to date it lacks in transparency, and in stability. However its advantages already outweigh its disadvantages, at least when compared with the complete inadequacy of Thomson/ISI, especially for Australian IS researchers.
A rational evaluation of a researcher's 'merit' and 'impact on researchers' would take into account multiple measures, such as the following:
On the basis of the research reported on in this paper, a process to generate measures of those kinds from Google Scholar data appears capable of being implemented. Because of the nature and quality of the data, it appears very unlikely that the process could be fully automated. On the other hand, automated tools appear feasible that would significantly reduce the resource requirements to perform, and re-perform the analysis.
If reduction to one or two measures is demanded, then the least-worst candidate would appear to be the Google Scholar h-index and h-count. The rationale underlying this proposition is that the two measures reflect both breadth and depth, rather than over-weighting the researcher's few most meritorious and/or most-heavily-cited papers. An Australian management academic has developed a tool to support such analysis, called Publish or Perish (Harzing 2007).
If a mechanism of this kind were adopted, the threshholds at the end of 2007 might be perceived to be, for an outstanding Australian IS researcher, an h-index of 25 and/or an h-count of 750, and for successful Australian IS researchers, an h-index of 12 or 15 combined with an h-count of 500. Because citations are cumulative, and the collections are not entirely stable, any such heuristic requires re-consideration at least annually.
The research reported in this paper was undertaken during the formative phases of the RQF initiative, and the paper was completed after the RQF's demise but before any clarity had emerged about the details of its replacement, the ERA. Some aspects of the conclusions must therefore be expressed in a qualified and contingent manner.
The U.K. RAE, the N.Z. PBRF, and the Australian RQF and ERA are political mechanisms aimed at focussing research funding on a small proportion of research centres within a small proportion of institutions. They are mass-production exercises, and are subject to heavily bureaucratic processes and definitions. Citation analysis used in all such processes will be inevitably largely mechanical, with simple rules applied to all disciplines, irrespective of their appropriateness.
The IS discipline lacks a power-base, and hence it appears unlikely that special treatment can be achieved on such grounds as the significance of refereed conference proceedings, the diversity and rapidity of change in the research domain and hence the substantial proportion of its publications that appear in specialist journals that attract low weightings, and the applied and instrumentalist nature of the discipline and hence the substantial proportion of its publications that appear in unrefereed venues.
The analysis undertaken in this paper shows that the use of Thomson/ISI would be a seriously inappropriate decision in respect of the IS discipline, and would result in very few IS researchers being recognised as having had a significant impact. This would be very likely to have a negative impact on access by Australian IS researchers to research funding from DEEWR and ARC sources.
Several measures were suggested in Clarke (2008) that could be taken by the IS discipline as a whole, presumably through the Association for Information Systems (AIS). Exhibit 6 identifies a number of measures that can be taken by Australian institutions, in particular the Australasian Chapter of AIS (AAIS) and the Australian Council of Professors and Heads of Information Systems (ACPHIS).
This Appendix provides detailed comparisons of results extracted from both ISI and Google Scholar. Seven Australian academics were selected, from among both expatriates and local researchers. All but one were selected because of their relatively uncommon names, in order to ease the difficulties of undertaking the searches and thereby achieve reasonable quality in the data. The other (the author of this paper) was selected because, for researchers with common names, full knowledge of the author's publications list makes it much easier to confidently achieve a reasonable degree of accuracy. Hence no significance should be inferred from inclusion in or exclusion from this short list.
In each of the following tables:
Google Count | Thomson Count | Venue |
145 | 111 | Journal |
92 | Unindexed (!!) | Journal (ISR) |
88 | 83 | Journal |
86 | Unindexed (!!) | Journal (CACM) |
56 | 26 | Journal |
52 | Unindexed | Conference (ICIS) |
52 | Unindexed | Journal (IJMMS) |
48 | Unindexed (!!) | Journal (CACM) |
41 | Unindexed (!!) | Journal (IEEE
Software) |
31 | 9 | Journal |
691 or 320 | 229 (of 601) | Totals |
Google Count | Thomson Count | Venue |
125 | 38 | Journal |
102 | Unindexed | Journal (JIS) |
106 | 30 | Journal |
87 | 36 | Journal |
72 | 26 | Journal (Commentary) |
65 | 20 | Journal (Commentary) |
45 | Unindexed | Book |
34 | Unindexed | Journal (JIS) |
34 | 22 | Journal |
31 | 24 | Journal |
701 or 520 | 196 (of 328) | Totals |
Google Count | Thomson Count | Venue |
302 | Unindexed | Book |
55 | 11 | Journal |
42 | 12 | Journal |
32 | 12 | Journal |
31 | 34 | Journal (1988) |
27 | Unindexed | Book |
26 | 57 | Journal (1982) |
20 | 23 | Book |
18 | 6 | Journal (1985) |
18 | Unindexed | Government Report |
571 or 224 | 155 (of 270) | Totals |
Google Count | Thomson Count | Venue |
133 | 60 | Journal |
47 | Unindexed | Journal (CAIS) |
43 | Unindexed | Conference (ICIS) |
33 | Unindexed | Conference |
22 | Unindexed (!) | Journal (DB, 2002) |
24 | 2 | Journal (I&M, 1991) |
18 | 2 | Journal |
18 | Unindexed | Journal (JIS) |
13 | Unindexed | Conference (ECIS) |
9 | 0 | Journal (JIT, Editorial) |
360 or 184 | 64 (of 70) | Totals |
Google Count | Thomson Count | Venue |
117 | 29 | Journal |
73 | Unindexed | Journal (Int'l Mkting Rev) |
61 | Unindexed | Journal (TIS) |
43 | Unindexed | Journal (JSIS) |
29 | Unindexed (!) | Journal (IJEC) |
26 | 12 | Journal |
26 | Unindexed | Journal (JIS) |
24 | 6 | Journal |
22 | Unindexed | Conference |
20 | Unindexed | Journal (EM) |
441 or 167 | 47 (of 53) | Totals |
Position | Google Count | Thomson Count | Venue |
57 | 81 | 14 | Journal |
59 | 85 | 16 | Journal |
102 | 60 | Unindexed | Journal (IT&P) |
148 | 47 | Unindexed | Journal (TIS) |
253 | 33 | Unindexed | Conference |
325 | 28 | Unindexed | Conference |
373 | 25 | 3 | Journal |
407 | 23 | Unindexed | Journal (JSIS) |
539 | 18 | Unindexed | Journal |
558 | 17 | Unindexed | Conference |
417 or 191 | 33 (of 44) | Totals |
Google Count | Thomson Count | Venue |
102 | Unindexed (!) | Journal (EJIS, 1994) |
56 | Unindexed (!) | Journal (ISF, 2000) |
40 | Unindexed | Journal (JGIM) |
27 | 6 | Journal (MS) |
27 | Unindexed | Conference |
24 | 23 | Journal (I&M, 1991) |
23 | Unindexed | Conference |
14 | Unindexed | Conference |
13 | Unindexed | Conference |
10 | Unindexed | Conference |
336 or 51 | 29 (of 34) | Totals |
Except where otherwise noted, all URLs were most recently access on 16 April 2008.
AJIS (2006) `AJIS Featured Theme: The Information Systems Discipline in Australian Universities' Australasian Journal of Information Systems 14, 1 (November 2006) 123-140, at http://dl.acs.org.au/index.php/ajis/issue/view/1
ANZSRC (2008) '1297.0 - Australian and New Zealand Standard Research Classification' Australian Bureau of Statistics, March 2008, at http://www.abs.gov.au/AUSSTATS/abs@.nsf/Latestproducts/1297.0Contents12008?opendocument&tabna
ARC (2006) 'Research Fields, Courses and Disciplines Classification (RFCD)' Australian Research Council, undated, apparently of 21 February 2006, withdrawn at the end of 2007 from http://www.arc.gov.au/apply_grants/rfcd_seo_codes.htm, and superseded by ANZSRC (2008)
CAIS (2007) 21 (2007) Special Volume on the IS Academic Discipline in Pacific Asia in 2006, Communications of the AIS, 2007
Carr K. (2008a) 'New ERA for research quality: Announcement of Excellence in Research for Australia initiative' Minister for Innovation, Industry, Science and Research, 26 February 2008, at http://www.arc.gov.au/media/releases/media_26feb08.htm
Carr K. (2008b) 'New classification to assist ERA development' Minister for Innovation, Industry, Science and Research, 31 March 2008, at http://www.arc.gov.au/media/releases/media_31march08.htm
Clarke R. (Ed.) (1988) 'Australian Information Systems Academics: 1988/89 Directory' Australian National University, November 1988
Clarke R. (Ed.) (1991) 'Australasian Information Systems Academics: 1991 Directory' Australian National University, April 1991
Clarke R. (2006) 'Key Aspects of the History of the Information Systems Discipline in Australia' Australasian Journal of Information Systems 14, 1 (November 2006) 123-140, at http://dl.acs.org.au/index.php/ajis/article/view/12/11
Clarke R. (2007) 'A Retrospective on the Information Systems Discipline in Australia: Appendix 4: Professors' Xamax Consultancy Pty Ltd, March 2007, at http://www.rogerclarke.com/SOS/AISHistApp4.html
Clarke R. (2008) 'An Exploratory Study of Information Systems Researcher Impact' Commun. AIS 22, 1 (January 2008), at http://cais.aisnet.org/articles/default.asp?vol=22&art=1, and at http://www.rogerclarke.com/SOS/Cit-CAIS.html
DEST (2005) 'Research Quality Framework: Assessing the quality and impact of research in Australia: Final Advice on the Preferred RQF Model' Department of Education, Science & Training, December 2005, withdrawn at the end of 2007 from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issue/research_quality_framework/final_advice_on_preferred_rqf_model.htm
DEST (2006a) 'Research Quality' Department of Education, Science and Training, undated but apparently of mid-2006, withdrawn at the end of 2007 from http://www.dest.gov.au/NR/rdonlyres/26AFCA18-A4CA-481E-B54F-2A83C29528B0/13523/RsearchQuality.pdf
DEST (2006b) 'Research Quality Framework: Assessing the Quality and Impact of Research in Australia: The Recommended RQF' Department of Education, Science and Training, October 2006, withdrawn at the end of 2007 from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issue/research_quality_framework/documents/the_recommended_RQF_pdf.htm
DEST (2007a) 'Research Quality Framework Fact Sheet' undated, but apparently of May 2007, withdrawn at the end of 2007 from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issue/research_quality_framework/documents/RQF_factsheet_2006/RQF_pdf.htm
DEST (2007b) 'RQF 2008 Documents and Forms', Department of Education, Science and Training, September 2007, , withdrawn at the end of 2007 from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issue/research_quality_framework/documents_and_forms.htm
Fisher J., Shanks G. & Lamp J. (2007) 'A Ranking List for Information Systems Journals' Australasian J. of Infor. Syst. 14, 2 (June 2007), at http://dl.acs.org.au/index.php/ajis/article/view/469/417
Gable G. & Clarke R. (Eds.) (1994) 'Asia Pacific Directory of Information Systems Researchers: 1994' National University of Singapore, 1994
Gable G. & Clarke R. (Eds.) (1996) 'Asia Pacific Directory of Information Systems Researchers: 1996' National University of Singapore, 1996
Gable G. & Smyth R. (Eds.) (2008) 'The Information Systems Discipline in Australian Universities' Forthcoming, ANU ePress, 2008
Harzing A.-W. (2007) 'Google Scholar - a new data source for citation analysis', University of Melbourne, 2007, at http://www.harzing.com/pop_gs.htm
Hirsch J.E. (2005) 'An index to quantify an individual's scientific research output' arXiv:physics/0508025v5, 29 September 2005, at http://arxiv.org/PS_cache/physics/pdf/0508/0508025v5.pdf, accessed July 2007
Lamp J. (2005) 'The Index of Information Systems Journals', Deakin University, version of 16 August 2005, at http://lamp.infosys.deakin.edu.au/journals/index.php
MacLeod D. (2006) 'Research exercise to be scrapped' The Guardian, 22 March 2006, at http://education.guardian.co.uk/RAE/story/0,,1737082,00.html
PBRF (2005) 'Performance-Based Research Fund', N.Z. Tertiary Education Commission, July 2005, at http://www.tec.govt.nz/downloads/a2z_publications/pbrf2006-guidelines.pdf
RAE (2001) 'A guide to the 2001 Research Assessment Exercise', U.K. Department for Employment and Learning, apparently undated, at http://www.hero.ac.uk/rae/Pubs/other/raeguide.pdf
RAE (2005) 'Guidance on submissions' Department for Employment and Learning, RAE 03/2005, June 2005, at http://www.rae.ac.uk/pubs/2005/03/rae0305.pdf
Sellito C. (2007) 'A Study of Journal Publication Diversity Within the Australian Information Systems Sphere' Australasian J. of Infor. Syst. 14, 1 (June 2007), at http://dl.acs.org.au/index.php/ajis/article/view/26/409
SSF (2007) 'Letter to the AIS President' AIS Senior Scholars Forum Subcommittee on Journals, 23 April 2007, at http://home.aisnet.org/associations/7499/files/Senior%20Scholars%20Letter.pdf
The work reported on in this paper was conducted within the context of a major collaborative project on the IS discipline, led by Guy Gable and Bob Smyth at QUT. A report on Australia was published in a Special Issue of the Australasian Journal of Information Systems 14, 1 (AJIS 2006), and a further report on Pacific Asia appeared as a Special Issue in CAIS (2007).
The author acknowledges valuable input from Peter Seddon of the University of Melbourne and Colin Steele of the ANU, and the feedback from the Associate Editor and reviewers. Responsibility for all aspects of the work rests, of course, entirely with the author.
Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., a Visiting Professor in the E-Commerce Programme at the University of Hong Kong, and a Visiting Professor in the Department of Computer Science at the Australian National University.
| Personalia |
Photographs Presentations Videos |
Access Statistics |
 |
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax. From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 75 million in late 2024. Sponsored by the Gallery, Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer |
Xamax Consultancy Pty Ltd ACN: 002 360 456 78 Sidaway St, Chapman ACT 2611 AUSTRALIA Tel: +61 2 6288 6916 |
Created: 15 November 2007 - Last Amended: 16 April 2008 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/SOS/Cit-AJIS.html
Mail to Webmaster - © Xamax Consultancy Pty Ltd, 1995-2024 - Privacy Policy