Roger Clarke's Web-Site

 

© Xamax Consultancy Pty Ltd,  1995-2017


Roger Clarke's 'IS Citation Analysis'

An Exploratory Study of Information Systems Researcher Impact

Roger Clarke **

Review Version of 29 April 2006

© Xamax Consultancy Pty Ltd, 2006

Available under an AEShareNet Free
for Education licence or a Creative Commons 'Some
Rights Reserved' licence.

This document is at http://www.anu.edu.au/people/Roger.Clarke/SOS/CitAnal0605.html


Abstract

Citation-counts of refereed articles are a potentially valuable measure of the impact of a researcher's work, in the information systems discipline as in many others. This paper reports on a pilot study of the apparent impact of IS researchers, as disclosed by citation-counts of their works. Citation analysis using currently available indexes is found to be fraught with many problems, and must be handled with great care.


Contents


1. Introduction

Information systems (IS) is a maturing discipline, with a considerable specialist literature, and relationships with reference disciplines that are now fairly stable and well-understood. In a mature discipline, various forms of 'score-keeping' are undertaken. One reason for this is as a means to distinguish among applicants for promotion, and contenders for senior appointments. An increasingly significant application of score-keeping, however, is as a factor in the allocation of resources to support research.

Two intuitively obvious approaches to score-keeping are to count the number of works that a researcher publishes, and moderate it by the time-span over which they were published, the categories of publication (such as books, conference papers and journal articles), and the quality of the venues; and to count the number of citations of a researcher's publications. The weighted count of publications represents a measure of research quality. The citation-count represents a measure of the researcher's impact.

This paper performs an analysis of citations of IS researchers, in order to examine the extent to which currently available data provides satisfactory measures of researcher impact. It is motivated by the concern that, whether or not such analyses are performed by members of the IS discipline, it appears increasingly likely that others will do it for us.

The paper commences by briefly reviewing formal schemes for appraising researcher quality and impact that have been implemented or proposed in several countries in recent years. It then discusses citation analysis, and its hazards. The research objectives and research method are described. The raw scores are presented and issues arising from the analysis are identified and examined.


2. Schemes to Assess Researcher Impact

In a number of countries in recent years, there have been endeavours to implement mechanisms for evaluating the impact of individual researchers and research groups. These have generally been an element within a broader activity, particularly relating to the award of block grants to research centres. Examples include the U.K. Research Assessment Exercise (RAE 2001, 2005), which has been operational since 1986, the New Zealand Performance-Based Research Fund (PBRF 2005), and the emergent Australian Research Quality Framework (RQF - DEST 2005), currently in train.

The procedure in the most recent RAE in 2001 was described as follows: "Every higher education institution in the UK may make [submissions] ... Such submissions consist of information about the academic unit being assessed, with details of up to four publications and other research outputs for each member of research-active staff. The assessment panels award a rating on a scale of 1 to 5, according to how much of the work is judged to reach national or international levels of excellence" (RAE 2001, p. 3). The forthcoming 2008 Exercise also involves evaluation of "up to four items ... of research output produced during the publication period (1 January 2001 to 31 December 2007) by each individual named as research active and in post on the census date (31 October 2007)" (RAE 2005, p. 13). Similarly, in the New Zealand scheme, a major part of the 'quality evaluation' depends on assessment of the 'evidence portfolio' prepared by or for each eligible staff-member, which contains up to four 'nominated research outputs' (PBRF 2005, p. 41).

The Australian RQF seeks to measure firstly "the quality of research", which includes "its intrinsic merit and academic impact", and secondly the "broader impact or use of the research", which refers to "the extent to which it is successfully applied in the broader community". The outcomes of the measurements would be rankings, which would be used in decision-making about the allocation of research-funding. The unit of study is 'research groupings', which are to be decided by each institution, but will be subject to considerable constraints in terms of disciplinary focus and minimum size (DEST 2005, 2006).

The design of all three schemes involves lengthy and bureaucratic specifications of how research groupings and individuals are to fill in forms, including definitions of the publications that can be included, large evaluation panels comprising people from disparate disciplines, and lengthy and bureaucratic assessment processes. The RAE in particular has been a very highly resource-intensive activity. A recent report suggests that itthe RAE is shortly to be abandoned (MacLeod 2006), on the basis that it has achieved its aims.

A conventional indicator of research quality is publications in refereed venues, primarily in 'quality, refereed journals', and perhaps secondarily in the stronger refereed conferences. This may be supplemented by the rankings of those journals and conference proceedings. The RQF, for example, appears to have adopted a modified form of the RAE approach, with each research grouping to be allocated a rating, on a scale of 1 (low) to 5 (high), based on the relevant panel's assessment of the "contribution" of the research grouping's "research work", and the "significance" of the area in which the work is undertaken.

Research impact is distinguished from research quality. A potential indicator of research impact is evidence of the use of refereed publications, in particular citations of them. Citations within journals and refereed conferences would indicate the researcher's impact on other researchers; whereas citations within less formal literatures such as professional magazines and newsletters, government reports, and the trade press, might be used as indicators of impact in the broader community. Other indicators of impact include re-publications, translations and inclusion in collections and anthologies. Researcher impact is, however, largely a cumulative matter, rather than being limited to a single publication. The focus might therefore be more broadly on reputation or 'esteem', in which case further indicators could include international appointments, prizes and awards, membership of academies and editorial boards, keynote-speaker invitations, and collaborations with other highly-reputed researchers. The RQF envisages the impact of each research grouping as being assessed against a 3-point scale of High, Moderate and Limited.

Schemes such as the RAE, PBRF and RQF are political in nature, designed to provide a justification for funds-allocation decisions. There are many aspects of such evaluation processes that could influence the accessibility of research support by IS academics. This paper is concerned with whether the use of citation analysis would provide a reasonable basis for evaluating the impact of IS researchers.


3. Citation Analysis

"Citations are references to another textual element [relevant to] the citing article. ... In citation analysis, citations are counted from the citing texts. The unit of analysis for citation analysis is the scientific paper" (Leydesdorff 1998). Leydesdorff and others apply 'citation analysis' to the study of cross-references within a literature, in order to document the intellectual structure of a discipline. This paper is concerned with its use for the somewhat different purpose of evaluating the quality and/or impact of works and their authors by means of the references made to them in refereed journal articles.

Authors have cited prior works for centuries. Gradually, the extent to which a work was cited in subsequent literature emerged as an indicator of the work's influence, which in turn implied significance of the author. Whether the influence of work or author was of the nature of notability or notoriety was, and remains, generally ignored by citation analysis. Every citation counts equally, always provided that it is in a work recognised by whoever is doing the counting.

Attempts to formally measure the quality and/or impact of works, and of their authors, on the basis of the number of citations that they gather, is a fairly recent phenomenon. Indeed, the maintenance of citation indices appears to date only to about 1960, with the establishment of the Science Citation Index (SCI), associated with Garfield (1964). Considerable progress in the field of 'bibliometrics' has ensued.

The advent of the open, public Internet, particularly since the Web exploded in 1993, has stimulated many developments. Individual journal publishing companies such as Elsevier, Blackwell, Kluwer, Springer and Taylor & Francis have developed automated cross-linkage services, at least within their own journal-sets. Meanwhile, the open access movement is endeavouring to produce not only an open eLibrary of refereed works, but also full and transparent cross-referencing within the literature. A leading project in the area was the Open Citation (OpCit) project in 1999-2002. An outgrowth from the OpCit project, the Citebase prototype, was referred to as 'Google for the refereed literature'. It took little time for Google Inc. to discover the possibility of a lucrative new channel. It launched Google Scholar in late 2004.

These need to be regarded as initial forays. All still fall far short of the vision of the electronic library, which was conceived by Vannevar Bush (1945), and articulated by Ted Nelson in the 1960s as 'hypertext'. As outlined in Nelson's never-completed Project Xanadu, the electronic library would include the feature of 'transclusion', that is to say that quotations are included by precise citing of the source, rather than by replicating some part of the content of the source.

In the complex, inter-locking and ambiguous field of learned publishing, it is to be expected that citation analysis will gave rise to a degree of contention. Dissatisfaction with it as a means of evaluating the quality and impact of works and of researchers has a long history. See, for example, Hauffe (1994), MacRoberts & MacRoberts (1997) and Adam (2002).

Within the IS discipline, there is a long history of attention being paid to citations. The primary references appear to be Culnan (1978, 1986), Culnan & Swanson (1986), Culnan (1987), Cheon et al. (1992), Cooper et al. (1993), Eom et al. (1993), Holsapple et al. (1993), Eom (1996), Walstrom & Leonard (2000), Vessey et al. (2002), Schlogl (2003) and Katerattanakul & Han (2003). That is a fairly short list of articles. Moreover, a General Search on the ISI database shows that the most cited among them (Holsapple et al.) has only accumulated a count of 24 (or, if a deep and well-informed analysis is conducted, using the ISI Cited Ref facility, then the most cited appears to be achieved by combining several counts for a total of 61 for Culnan 1987), and on Google Scholar the largest citation-count appears as 63, for each of Culnan (1986) and Culnan (1987). As will be shown, these are not insignificant counts, but they are not large ones either; and the difficulties and ambiguities involved in generating them represent a mini-case study in the application of citation analysis.

The primary purposes of the research reported in the papers listed above have been to develop an understanding of the intellectual structure of the IS discipline, of patterns of development within the discipline, and of the dependence of IS on reference disciplines. In some cases, the impact of particular journals has been in focus (in particular Cooper et al. 1993, Holsapple et al. 1993 and Katerattanakul & Han 2003). In one instance (Walstrom & Leonard 2000), highly-cited articles were the primary concern. The literature search conducted as part of this project did not identify any articles in which the primary focus of the citation analysis was on individual researchers or research groupings.

A number of deficiencies in the use of citation analysis for this purpose are apparent from the outset. In the course of presenting the analysis, more will emerge, and a consolidated list is provided at the end of the paper. Despite these deficiencies, 'score-keeping' is increasingly being applied to the allocation of research resources. The work reported on here is accordingly justified at least as much by political pragmatics as for intellectual reasons.


4. The Research Purposes and Method

Because little prior research has been conducted in this specific area, the essential purpose was to provide insights into the effectiveness of citation analysis applied to individual IS researchers.

Because of the vagaries of databases that are organised primarily on names, considerable depth of knowledge of individuals active in IS research is needed in order to achieve a reasonable degree of accuracy in the analysis. The analysis accordingly focusses primarily on researchers active in the author's country of long-term residence, Australia. This was appropriate not only as a means of achieving reasonable data quality, but also because the scale was manageable (with c. 700 members of the IS discipline, but no more than 150 with well-established publishing records over the last decade), and the issue of researcher impact assessment is current. Balance was sought by complementing the Australian analysis by also assessing the impact of some leading researchers in North America and Europe.

One important insight that was sought was the extent to which the publishing venues that are generally regarded by IS researchers as carrying quality refereed articles are reflected in the databases on which citation analysis can be performed. Rather than simply tabulating citation-counts, the research accordingly commenced by establishing a list of relevant journals and conference proceedings.

Consideration was given to generating lists of highly-cited articles, and working down the list, accumulating total counts for each individual. Preliminary experimentation showed, however, that this was impractical. It was concluded that a 'census' approach was to be preferred.

The following method was adopted:

Further detail on each of these steps is now provided.

The set of venues was developed by reference to the now well-established literature on IS journals and their rankings, for which a bibliography is provided at Saunders (2005). Consideration was given to the lists and rankings there, including the specific rankings used by several universities, and available on that site. Details of individual journals were checked in the most comprehensive of the several collections, which is maintained at Deakin University (Lamp 2005).

The author believes that the set selected, listed in Exhibit 2, represents a fairly conventional view of the key refereed journals on the management side of the IS discipline. It significantly under-represents those journals that are in the reference discpline of computer science, and at the intersection between IS and computer science. The reason is that these fields are themselves highly diverse, and a very large number of venues would need to be considered, and many included, in which the heavy majority of IS researchers neither read nor publish. Less conventionally, the list separates a few 'AA-rated' journals, and divides the remainder into general, specialist and regional journals. There is, needless to say, ample scope for debate on all of these matters.

Consideration could be given to supplementing the journals with the major refereed conferences. In this author's experience, ICIS is widely regarded as approaching AA status, ECIS as a generic A, and AMCIS, PACIS and ACIS may be considered by some as being generic A as well. These are accessible and indexed in the Association for Information Systems' AIS eLibrary, in the case of ICIS since it commenced in 1980, ECIS since 2000, and ACIS since 2002. However, no attempt was made to extend the analysis undertaken in this work to even the strongest of the refereed conferences.

A survey was conducted of available citation indices. It was clear that Thomson / ISI needed to be included, because it is well-known and would be very likely to be used by evaluators. Others considered included:

Thomson Scientific, previously known as the Institute for Scientific Information (ISI), publishes the Science Citation Index (SCI) and the Social Science Citation Index (SSCI). In January 2006, its site stated that SCI indexes 6,496 'journals' (although some are proceedings), and that SSCI indexes 1,857 'journals'. The company's policies in relation to inclusion (and hence exclusion) of venues is explained at http://scientific.thomson.com/mjl/selection/. An essay on the topic is at Thomson (2005), but access is unreliable.

Elsevier's Scopus has only been operational since late 2004. The next three are computer science indexes adjacent to IS, and at the time the research was conducted the last on the list was still experimental. The decision was taken to utilise the Thomson/ISI SCI and SSCI Citation Indices, and to extract comparable data from Google Scholar. A more comprehensive project would be likely to add Scopus into the mix.

Exhibit 2 shows the set of journals that were selected, together with indications as to whether the journal is included in the Thomson/ISI SCI or SSCI Citation Indices. The final column shows the inferences drawn by the author regarding the extent of the Thomson/ISI coverage of the journal.

Exhibit 2: Refereed Venues Selected

Journal NameJournal Abbrev.SSCISCIIssues Included
  
AA Journals (3)    
Information Systems ResearchISRY Only from 1994, Vol. 4 ?
Journal of Management Information SystemsJMISY Only from 1999, Vol. 16!
Management Information Systems QuarterlyMISQY Only from 1984, Vol. 8!
  
AA Journals in the Major Reference Disciplines (4)    
Communications of the ACM (Research Articles only)CACM YFrom 1958, Vol. 1
Management ScienceMSY From 1955, Vol. 1
Academy of Management JournalAoMJY From 1958, Vol. 1
Organization ScienceOSY From 1990, Vol. 1?
  
A Journals - General (9)    
Communications of the AIS (Peer Reviewed Articles only)CAIS  None!
DatabaseData Base YOnly from 1982 Vol. 14 ?
Information Systems FrontiersISF YOnly from 2001, Vol. 3
Information Systems JournalISJY Only from 1995, Vol. 5
Information & ManagementI&MY Only from 1983, Vol. 6
Journal of the AISJAIS  None!
Journal of Information SystemsJIS  None!
Journal of Information TechnologyJITY Only 18 articles
WirtschaftsinformatikWI YOnly from 1990, Vol. 32
  
A Journals - Specialist (15)    
Decision Support SystemsDSS YOnly from 1985, Vol. 1
Electronic MarketsEM  None
International Journal of Electronic CommerceIJECY From 1996, Vol. 1
Information & OrganizationI&O  None
Information Systems ManagementISMY Only from 1994, Vol. 11
Information Technology & PeopleIT&P  None!
Journal of End User ComputingJEUC  None
Journal of Global Information ManagementJGIM  None
Journal of Information Systems EducationJISE  None
Journal of Information Systems ManagementJISM  None
Journal of Management SystemsJMS  None
Journal of Organizational and End User ComputingJOEUC  None
Journal of Organizational Computing and Electronic CommerceJOCEC  None
Journal of Strategic Information SystemsJSIS YFrom 1992, Vol. 1 ?
The Information SocietyTISY Only from 1997, Vol. 13! - ?
  
A Journals - Regional (3)    
Australian Journal of Information SystemsAJIS  None
European Journal of Information SystemsEJIS YOnly from 1995, Vol. 4
Scandinavian Journal of Information SystemsSJIS  None

When assembling a list of individuals active in the field, it is challenging to be sure of being comprehensive. People enter and depart from the discipline. There are overlaps with the Computer Science discipline, with various management disciplines, and with the IS profession. When determining the set of IS academics in a particular country, immigration, emigration and expatriates create definitional challenges.

The set of names of Australian academics was developed based on the author's experience in the field since about 1970, but in particular by reference to the Australian IS Academics Directory (Clarke 1988), the Australasian IS Academics Directory (Clarke 1991), the Asia Pacific Directory of IS Researchers (Gable & Clarke 1994 and 1996), and the ISWorld Directory (1997-). Arbitrary decisions were taken as to who was an expatriate Australian, and how long immigrants needed to be active in Australia to be treated for the purposes of this analysis as being Australian. Data was sought in relation to about 100 leading Australian IS researchers, plus 4 well-known and successful expatriates.

Data was extracted from the SCI and SSCI citation indices over several days in late January 2006. Access was gained through the ANU Library Reverse Proxy, by means of the company's 'Web of Science' offering. Both sets of searches were restricted to 1978-2006, across all Citation Indices (Science Citation Index - SCI-Expanded, Social Sciences Citation Index - SSCI and Arts & Humanities Citation Index - A&HCI). Multiple name-spellings and initials were checked, and where doubt arose were also cross-checked with the AIS eLibrary.

In order to provide comparison with leaders in the IS discipline outside Australia, the same process was applied to a small sample of leading overseas researchers. The selection was not intended to be random. It relied on this author's longstanding involvement in the field, and his knowledge of the literature and the individuals concerned. Relatively uncommon surnames were used, in order to reduce the likelihood of pollution through the conflation of articles by multiple academics.

Google Scholar was then searched for a sub-set of the 30 apparently most-highly-cited Australian researchers. Supplementary research was then undertaken within the Thomson/ISI database. These elements were performed in respectively early and late April 2006. During the 3 months between the two rounds of analysis using ISI, the database and hence the citation-counts of course continued to accumulate.


5. Thomson/ISI

This section presents the results of the pilot citation analyses of the Thomson/ISI collections.

5.1 Method

The data collected for each of the authors was the apparent count of articles, and the apparent total count of citations. The Thomson/ISI site provides several search-techniques. The search-technique used was the 'General Search'. In each case, the search-terms used were <Author includes author-surname> combined with <Author includes author-initial or -initials>, with the date-range restricted to 1978 onwards. In the case of Australian researchers with common names, this was qualified with <Address includes 'Australia'>. Each list generated by a search was inspected, in order to remove articles that appeared to be by people other than the person being targetted.

The ISI General Search provides a list of articles by all authors sharing the name in question, provided that they were published in venues that are in the ISI database. For each such article, a citation-count is provided, which is defined as the number of other articles in the ISI database that cite it. (It should be noted that although the term 'citation' is consistently used by all concerned, the analysis appears to actually utilise the entries in the reference list provided with the article, rather than the citations that appear within the text of the article).

This measure was selected primarily because it is the most obvious, and hence the most likely to be used by someone evaluating the apparent contribution of a particular academic or group of academics. It is also the most constrictive definition available, and hence could be argued to be the most appropriate to use when evaluating applicants for the most senior appointments, and when evaluating the reputations of the most prominent groups of researchers. Other possible approaches are discussed at the end of this section.

5.2 Citation-Counts for Australian IS Researchers

Exhibit 3 provides data for the highest-scoring Australian IS academics, using an arbitrary cut-off of 40 total citations. This resulted in the inclusion in the table of the 4 expatriates and 19 local researchers. For each person, the data shown is the total citation-count, the number of papers in the index, and the citation-count for that person's most-cited paper.

CAVEAT: For reasons that are discussed progressively through the remainder of this paper, there are strong arguments for not utilising the data in this table, and for not utilising the ISI 'General Search', as a basis for assessing the impact of individual articles or individual researchers.

Exhibit 3: Citation-Counts for Australian IS Researchers

Citation Count
Number of Articles

Largest Per-Article Count

Expatriates   
Iris Vessey (as I.)
601
35
111
Rick Watson (as R.T.)
485
28
78
Ted Stohr (as E.A.)
217
12
108
Peter Weill (as P.)
178
13
47
Locals
Marcus O'Connnor (as M.)
354
31
66
Ron Weber (as R.)
328
22

38

Philip Yetton (as P. and P.W.)
270
26
57
Michael Lawrence (as M.)
208
27
66
Michael Vitale (as M. and M.R.)
179
14
107
Ross Jeffery (as D.R., and as R.)
172
28
38
Marianne Broadbent (as M.)
166
24
36
Bob Edmundson (as R.H.)
86
4
66
Graham Low (as G. and G)
83
19
38
Peter Seddon (as P. and P.B.)
70
6
60
June Verner (as J. and J.M.)
65
18
22
Graeme Shanks (as G.)
61
13
14
Paula Swatman (P.M)
53
9
29
Kit Dampney (as C and C.N.G.)
47
16
25
John D'Ambra (as J.)
47
7
16
Roger Clarke (as R. and R.A.)
44
22
16
Michael Rosemann (as M.)
41
14
18
Graham Winley (as G.)
40
13
13
Chris Sauer (as C.)
40
18
13

5.3 Citation-Counts for Some Leading International IS Academics

Some form of benchmark is required, against which this measure of each local researcher's impact can be compared. The appropriate comparison is not with people in other disciplines, because their data reflects different numbers of academics, and different numbers of journals, which are differentially indexed. Exhibit 4 shows the same data as for Australian IS academics, but for some well-known leaders in the discipline in North America and Europe. A small number of individuals were selected whose surnames were unusual, in order to reduce the risks of conflating their publications with those of other people, and of overlooking relevant publications.

CAVEAT: For reasons that are discussed progressively through the remainder of this paper, there are strong arguments for not utilising the data in this table, and for not utilising the ISI 'General Search', as a basis for assessing the impact of individual articles or individual researchers.

Exhibit 4: Citation-Counts for Some Leading International IS Academics

Citation Count
Number of Articles

Largest Per-Article Count

North American   
Lynne Markus (as M.L.)
1,335
39
296
Izak Benbasat (as I.)
1,281
71
155
Dan Robey (as D.)
1,247
45
202
Sirkka Jarvenpaa (as S.L.)
960
40
107
Detmar Straub (as D. and D.W.)
873
49
160
Rudy Hirschheim (as R.)
600
44
107
Gordon Davis (as G.B.)
428
48
125
Peter Keen (as P.G.W.)
427
21
188
Eph(raim) McLean (as E. and E.R.)
119
30
31
Europeans
Kalle Lyytinen (as K.)
458
55
107
Leslie Willcocks (as L.)
231
42
28
Bob Galliers (as R.D. and R.)
173
34

25

Guy Fitzgerald (as G.)
121
50
38
Enid Mumford (as E.)
103
21
42
Claudio Ciborra (as C. and C.U.)
60
13
26
Frank Land (as F.)
59
18
19
David Avison (as D.)
51
37
44
Ron Stamper (as R. and R.K.)
32
13
16
Niels Bjorn-Andersen (as N.)
3
3
2

5.4 Quality Assessment

An important aspect of this exploratory study was the question as to how effectively the available databases reflected the extent to which the individuals concerned were actually cited. Several tests were applied. The first was the ISI database's coverage of the selected publishing venues. Nothing was found on the Thomson/ISI site that declared which Issues of journals were in the database, and it was necessary to conduct experiments in order to infer the extent of the coverage. The examination disclosed that:

These are not the only reasons why citation analysis based on the Thomson/ISI collection leaves a great deal to be desired. The following problems were detected:

Two further quality tests were undertaken. In the European list, I was surprised by the low counts for Ron Stamper and Niels Bjorn-Andersen. I report later on the results of an investigation of their citations as indicated by Google Scholar. In the North American list, I was surprised by the low results for Eph McLean, and consequently I examined his list more closely. An article that, in my view at that time, could have been expected to be among the most highly-cited in the discipline (Delone & McLean's 'Information Systems Success: The Quest for the Dependent Variable') did not appear in Eph McLean's list. It was published in ISR, which would have been AA-rated by most in the discipline from its inception. But ISR is indexed only from 5, 1 (March 1994), whereas the Delone & McLean paper appeared in 3, 1 (March 1992). Using the 'Cited Ref Search' provide by ISI also fails to detect it under McLean E.R., but does detect a single citation when the search is performed on 'INFORM SYST RES' and '1992'. ISI finds it with <Author = Delone W.H.>, with 7 variants of the citation, all of which are misleadingly foreshortened to 'INFORMATION SYSTEMS'. These disclose the very substantial sum of 448 hits. A later section reports on the citations of this paper as indicated by Google Scholar.

The final quality test applied was a comprehensive assessment of the inclusion and exclusion of the refereed works of a single author. A highly convenient sample of 1 was selected: the author of this paper. In the author's view, this is legitimate, for a number of reasons. This was exploratory research, confronted by many challenges, not least the problems of false inclusions and exclusions, especially in the case of researchers with common names. I have a very common surname, I have a substantial publications record, those publications are scattered across a wide range of topics and venues, I have had some impact, and I am well-positioned to ensure accuracy in this particular analysis because I know all of my own works. The outcome of the analysis was as follows:

The many deficiencies in the Thomson/ISI database identified from these tests result in differential bias against researchers. Some of the deficiencies would appear to affect all disciplines (e.g. the apparent incompleteness of journals that are claimed to be indexed, and the failure to differentiate refereed from unrefereed content in at least some journals). Many others would appear to mainly affect relatively new disciplines, such as the long delay before journals are included, and refusals to include some journals even when representations are made. When applied to IS researchers, the extent of the under-reporting is not easily predictable, although it does appear to depend heavily on the individual's areas of interest and preferred venues:

These problems suggest that comparisons within a discipline are problematic, and that comparisons between disciplines would be extraordinarily difficult. The following appear to be important factors, with observations about how IS compares with disciplines generally:

5.5 Alternative Methods

This sub-section considers alternative ways in which the Thomson/ISI database could be applied to the purpose. Two other categories of search are available. One is 'Advanced Search', which provides the ability to apply Boolean operators on combinations of fields. This would be valuable when conducting tightly focussed searches. If a common evaluation method were able to be defined, it may be feasible to use 'Advanced Search' to apply it. It was not considered relevant to the pilot study conducted here.

The other category of search is called 'Cited Ref Search'. This is looser than the 'General Search' used in this pilot study. It lists all articles by all authors sharing the name in question, that have been cited by any article in any publishing venue in the database. It therefore includes articles that were published in venues that are not in the ISI database. It is arguable that this scope of citation-count represents a more appropriate measure of the reputation, worth or impact of an individual researcher or group of researchers. The measure counts citations in articles in the ISI database, rather than citations in articles in the ISI database of articles that are also in the ISI database.

In order to test the likely impact of applying this alternative approach to IS researchers, a small number of researchers in each category were selected, and the study repeated. The design of the search-facility creates challenges because it makes only a small number of parameters available. For example, it does not enable restriction to <Address includes 'Australia'>. In addition, very little information is provided for each hit, the sequence provided is alphabetical by short journal title, and common names generate in excess of 1,000 hits. A further problem is that there is enormous variation in citation styles, and in the accuracy of the data in reference lists. This results in the appearance of there being many more articles than there actually are - many articles were found to have 2 or 3 entries, and the most variants detected during the analysis were 5 and 7.

Appendix 1 provides the results of this part of the study. All of the researchers for whom the analysis was undertaken had substantial numbers of articles that were cited in the ISI database, but that were not themselves in the ISI database. In some cases, relaxing the criterion resulted in an increase in the citation-count by 20-30%, but in others in an increase by factors of as much as 5. The data in Exhibit 6 enables the following inferences to be drawn:

Dependence on the General Search alone provides only a restricted measure of the reputation, worth or impact of an academic. Moreover, it may give a seriously misleading impression of the impact of researchers who publish in non-ISI venues such as books, and journals targetted at the IS profession and management. To the extent that citation analysis of Thomson/ISI data is used for evaluation purposes, a method needs to be carefully designed that reflects the objectives of the analysis. In addition, it would be essential for an opportunity to be provided for the individuals concerned to consider the results and submit supplementary information.


6. Google Scholar

Google Scholar is still an experimental service. From a bibliometric perspective, it is crude, because it is based on brute-force free-text analysis, without recourse to metadata, and without any systematic approach to testing venues for quality before including them. On the other hand, it has the advantages of substantial reach, ready accessibility, and popularity. It is inevitable that it will be used as a basis for citation analysis, and therefore important that it be compared against the more formal Thomson/ISI database.

The analysis presents considerable challenges. The method adopted was to conduct searches using the names of a sample of the researchers whose data from Thomson/ISI appears in Exhibits 2 and 3. Searches generate long lists of hits, each of which is either an article indexed by Google, or is inferred from a citation in an article indexed by Google. (As is the case with ISI, it appears that the 'citations' counted are actually the entries in the reference-list to each article, rather than the citations within the article's text). Each article has a citation-count shown, inferred from the index; and the hits appear to be sorted in approximate sequence of apparent citation-count, most first. (Only limited documentation appears to be available; and the service, although producing interesting and even valuable results, appears to be anything but stable).

Various approaches had to be experimented with, in order to generate useful data. (From a researcher's perspective, Google's search facilities are among the weakest offered by search-engines). For example, 'A' and 'I' are stop-words in the indexing logic, and hence searches for names including those initials required careful construction. The common words 'is' and 'it' are also stopwords, and hence it is difficult to use the relevant expressions 'IS' and 'IT' to restrict the hits to something more manageable. Search-terms of the form <"I Vessey" OR "Vessey I"> appeared to generate the most useful results, and this format was generally applied. Experiments with searching based on article-titles gave rise to other challenges, in particular the need to develop an even richer starting-point for the analysis: a comprehensive list of article-titles for each researcher.

A very small sample was used, because of the resource-intensity involved, and the experimental nature of the procedure. An intentionally biassed sample was selected, firstly because the intention was to test the comparability of the two sets of data, both in terms of coverage and citation-counts, and secondly in order to avoid conflation among multiple authors and the omission of entries. To simplify matters, only the first 10 articles for each author were gathered (roughly, but not necessarily reliably, those with the highest-citation-count).

The intensity of the 'multiple authors with the same name' problem is highly varied. For many of the researchers for whom data is presented below, there was no evident conflation with others, i.e. their Top-10 appeared on the first page of 10 entries displayed by Google Scholar. For a few, it was necessary to skip some papers, and move to the second or even third page. To reach Eph McLean's 10th-ranked paper, it was necessary to check 60 titles, and to reach Ron Weber's 10th, 190 titles were inspected. A test on my own, rather common name on the above basis resulted in 12,700 hits. To reach the 10th-most-cited, it was necessary to inspect 558 entries. The challenges involved in this kind of analysis are underlined by the fact that those first 558 entries included a moderate number of papers by other R. Clarkes on topics and in literatures that are at least adjacent to topics I have published on and venues I have published in. These could have easily been mistakenly assigned to me by a researcher who lacked a detailed knowledge of my publications list. Similarly, false-negatives would have easily arisen. There are many researchers with common names, and hence accurate citation analysis based on name alone is difficult to achieve.

Appendix 2 contains tables (numbered 6A through 6G) which show comparisons between the raw Google citation-count and the Thomson/ISI citation-count. Results are shown for seven researchers. In each case, careful comparison was necessary, to ensure accurate matching of the articles uncovered by Google against those disclosed by Thomson/ISI.

A number of conclusions can be drawn, some of which are intuitive, some less so; and some of which are disturbing. In particular:

An important further test was the measure generated for Delone & McLean's 'Information Systems Success: The Quest for the Dependent Variable'. The test was undertaken by keying the search-term <E McLean W DeLone> into Google Scholar, and critically considering the results. The test was performed twice, in early April 2006 and late April 2006. The results differed in ways that suggested that, during this period, Google was actively working on the manner in which its software counts citations and presents hits. The later, apparently better organised counts are used here. The analysis was complicated by the following:

The raw results comprised 824 citations for the main entry (and a total of 832 hits). Based on a limited pseudo-random sample from the first 40, many appeared to be indeed attributable to the paper. This is a citation-count of a very high order. An indication of this is that the largest Thomson/ISI citation-count for an IS paper that was located during this research was 296, for a paper in CACM by Lynne Markus. In Google, that paper scored 472. The Delone & McLean scored 75% more Google-citations than the Google-citation score of the highest-ranked IS paper so far located in the Thomson/ISI database. In short, it appears that, as a result of what is most likely data capture error, Thomson/ISI denies the authors the benefit of being seen to have co-authored what may be the single most-referenced paper in the entire discipline.

A further test was undertaken to compare the citation-counts generated by ISI and Google Scholar for two European researchers whose ISI counts were lower than had been anticipated:

Some perspective on what 100 citations means in the IS discipline can be gained from an assessment of the total citation-count of the top 10 items that are discovered in response to some terms of considerable popularity in recent years. The terms were not selected in any systematic manner. The count of the apparently 10th-most-cited article containing the expression is also highlighted, because it provides an indication of the depth of the heavily-cited literature using that term:

Another aspect of interest is the delay-factor before citations begin to accumulate. Some insight was gained from an informal sampling of recent MISQ articles, supplemented by searches for the last few years' titles of this author's own refereed works. A rule of thumb appears to be that there is a delay of 6 months before any citations are detected by Google, and of 18 months before any significant citation-count is apparent. The delay is rather longer on Thomson/ISI. This is to be expected, because of the inclusion of edited and lightly-refereed venues in Google, which have a shorter review-and-publication cycle than Thomson/ISI's journals, most of which are heavily refereed.

In summary, citation-counts above about 75-100 on Google Scholar appear to indicate a high-impact article, and above 40-50 a significant-impact article. Appropriate threshholds on Thomson/ISI would appear to be somewhat less than half of those on Google. These threshholds are of course specific to IS, and other levels would be likely to be appropriate in other disciplines, and in other countries.


7. Implications

Reputation is a highly multi-dimensional construct, and reduction to a score is morally dubious, intellectually unsatisfying, and economically and practically counter-productive. On the other hand, the impact of a researcher's publications, as measured by the frequency with which they are cited by other authors, is a factor that an assessment of reputation would ignore at its peril.

The research presented in this paper has demonstrated that there are enormous problems to be confronted in applying currently available databases to the purpose. Exhibit 5 summarises them.

Exhibit 5: Deficiencies in Citation Analysis as a Means of Assessing Researcher Impact

Citation Databases, particularly Thomson/ISI

The Articles

Contextual Factors

Despite these problems, the scope exists for citation analysis to be used constructively, as an aid in evaluation, with care taken to avoid or overcome the problems. For example, members of search and selection committees responsible for the appointment of key staff-members apply the technique, but incorporate subtlety, sophistication and insight into their work. This could be summed up with the epithet: 'if you can't be good, be careful'. One approach would be to apply both ISI and Google Scholar; and another would be for citation analysis to be undertaken openly, with the opportunity for researchers to submit additional information needed to fill out the picture.

Where impact assessment is institutionalised, however, as in such schemes as the U.K. RAE, the N.Z. PBRF and the Australian RQF, the likelihood of subtlety, sophistication and insight being applied is remote. These are political mechanisms aimed at focussing research funding on a small proportion of research centres within a small proportion of institutions. They are mass-production exercises, and are subject to heavily bureaucratic processes and definitions. Citation analysis used in such processes is inevitably largely mechanical.

Simplistic application of raw citation-counts to evaluate the performance of individual researchers and of research groupings would disadvantage some disciplines, many research groupings, and many individual researchers. The IS discipline is highly exposed to the risk of simplistic application of citation analysis. For the many reasons identified above, citation-counts will suggest that most IS researchers fall short of the criteria demanded for the higher rankings.

The IS discipline in at least some countries is confronted by the spectre of reduced access to research funding, as a result of unreasonable application of citation analysis. Options available include representations to Thomson in order to achieve back-loading into ISI of large volumes of missing publications, and the development of the discipline's own comprehensive database of publication titles and reference lists (possibly building on the AIS eLibrary). At the very least, it appears to be necessary for a defensive measure to be adopted, in the form of a set of guidelines explaining how to utilise citation analysis to evaluate the impacts of articles, individuals and research groupings, including an exhortation that results never be applied without the affected individuals having the opportunity to review the data and submit supplementary information.


8. Conclusions

There may be a world in which the electronic library envisioned by Bush and Nelson has come into existence, in which all citations are counted, and in which venues are subject to a weighting scheme that reflects differences among disciplines and research-domains, and that is subject to progressive adaptation.

Back in the real world, however, the electronic library is deficient in a great many ways. It is fragmented and very poorly cross-linked. And the interests of copyright-owners (including discipline associations but particularly the for-profit corporations that publish and exercise control over the majority of journals) are currently in building more and more substantial barriers rather than working towards integration. It remains to be seen whether that will be broken down by the communitarian open access movement, or by the new generation of corporations spear-headed by Google.

In the current and near-future contexts, citation analysis is a very blunt weapon, which should be applied only with great care, but which appears very likely to harm the interests of the less politically powerful disciplines such as IS.


References

In all cases in which URLs are shown, the web-page was last accessed in early April 2006.

Adam D. (2002) 'Citation analysis: The counting house' Nature 415 (2002) 726-729

ARC (2006) 'Research Fields, Courses and Disciplines Classification (RFCD)' Australian Research Council, undated, apparently of 21 February 2006, at http://www.arc.gov.au/apply_grants/rfcd_seo_codes.htm

Bush V. (1945) 'As We May Think' The Atlantic Monthly. July 1945, at http://www.theatlantic.com/doc/194507/bush

Cheon M.J., Lee C.C. & Grover V. (1992) 'Research in MIS - Points Of Work and Reference - A Replication and Extension of the Culnan and Swanson Study' Data Base 23, 2 (September 1992) 21-29

Clarke R. (Ed.) (1988) 'Australian Information Systems Academics: 1988/89 Directory' Australian National University, November 1988

Clarke R. (Ed.) (1991) 'Australasian Information Systems Academics: 1991 Directory' Australian National University, April 1991

Clarke R. (2006) 'Plagiarism by Academics: More Complex Than It Seems' J. Assoc. Infor. Syst. 7, 2 (February 2006)Cooper R.B., Blair D. & Pao M. (1993) 'Communicating MIS research: a citation study of journal influence' Infor. Processing & Mngt 29, 1 (Jan.-Feb. 1993) 113 - 127  Culnan M.J. (1978) 'Analysis of Information Usage Patterns of Academics and Practitioners in Computer Field - Citation Analysis of a National Conference Proceedings' Infor. Processing & Mngt 14, 6 (1978) 395-404

Culnan M.J. (1986) 'The Intellectual-Development of Management-Information-Systems, 1972-1982 - A Cocitation Analysis' Mngt Sci. 32, 2 (February 1986) 156-172

Culnan M.J. (1987) 'Mapping the Intellectual Structure of MIS, 1980-1985: A Co-Citation Analysis' MIS Qtly 11, 3 (September 1987) 341-353

Culnan M.J. & Swanson E.B. (1986) 'Research In Management-Information-Systems, 1980-1984 - Points Of Work And Reference' MIS Qtly 10, 3 (September 1986) 289-302

DEST (2005) 'Research Quality Framework: Assessing the quality and impact of research in Australia: Final Advice on the Preferred RQF Model' Department of Education, Science & Training, December 2005 , at http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issues/research_quality_framework/final_advice_on_preferred_rqf_model.htm

DEST (2006) 'Research Quality Framework Guidelines Scoping Workshop : Workshop Summary' Department of Education, Science & Training, 9 February 2006 , at http://www.dest.gov.au/NR/rdonlyres/1D5A7163-A754-48B7-88B7-5530F486EDD4/9772/RQFGuidelinesScopingWorkshopOutcomesFINAL17March06.pdf

Eom S.B. (1996) 'Mapping The Intellectual Structure Of Research In Decision Support Systems Through Author Cocitation Analysis (1971-1993)' Decision Support Systems 16, 4 (April 1996) 315-338

Eom S.B., Lee S.M. & Kim J.K. (1993) 'The Intellectual Structure Of Decision-Support Systems (1971-1989)' Decision Support Systems 10, 1 (July 1993) 19-35

Gable G. & Clarke R. (Eds.) (1994) 'Asia Pacific Directory of Information Systems Researchers: 1994' National University of Singapore, 1994

Gable G. & Clarke R. (Eds.) (1996) 'Asia Pacific Directory of Information Systems Researchers: 1996' National University of Singapore, 1996

Garfield E. (1964) ''Science Citation Index - A New Dimension in Indexing' Science 144, 3619 (8 May 1964) 649-654 , at http://www.garfield.library.upenn.edu/essays/v7p525y1984.pdf

Hauffe H. (1994) 'Is Citation Analysis a Tool for Evaluation of Scientific Contributions?' Proc. 13th Winterworkshop on Biochemical and Clinical Aspects of Pteridines, St.Christoph/Arlberg, 25 February 1994, at http://www.uibk.ac.at/ub/mitarbeiter_innen/publikationen/hauffe_is_citation_analysis_a_tool.html

Holsapple C.W., Johnson L.E., Manakyan H. & Tanner J. (1993) 'A Citation Analysis Of Business Computing Research Journals' Information & Management 25, 5 (November 1993) 231-244

Katerattanakul P. & Han B. (2003) 'Are European IS journals under-rated? an answer based on citation analysis' Euro. J. Infor. Syst. 12, 1 (March 2003) 60-71

Lamp J. (2005) 'The Index of Information Systems Journals', Deakin University, version of 16 August 2005, at http://lamp.infosys.deakin.edu.au/journals/index.php

Leydesdorff L. (1998) 'Theories of Citation?' Scientometrics 43, 1 (1998) 5-25, at http://users.fmg.uva.nl/lleydesdorff/citation/index.htm

MacLeod D. (2006) 'Research exercise to be scrapped' The Guardian, 22 March 2006, at http://education.guardian.co.uk/RAE/story/0,,1737082,00.html

MacRoberts M.H. & MacRoberts B.R. (1997) 'Citation content analysis of a botany journal' J. Amer. Soc. for Infor. Sci. 48 (1997) 274-275

PBRF (2005) 'Performance-Based Research Fund', N.Z. Tertiary Education Commission, July 2005, at http://www.tec.govt.nz/downloads/a2z_publications/pbrf2006-guidelines.pdf

Perkel J.M. (2005) 'The Future of Citation Analysis' The Scientist 19, 20 (2005) 24

RAE (2001) 'A guide to the 2001 Research Assessment Exercise', U.K. Department for Employment and Learning, apparently undated, at http://www.hero.ac.uk/rae/Pubs/other/raeguide.pdf

RAE (2005) 'Guidance on submissions' Department for Employment and Learning, RAE 03/2005, June 2005, at http://www.rae.ac.uk/pubs/2005/03/rae0305.pdf

Saunders C. (2005) 'Bibliography of MIS Journals Citations', Association for Information Systems, undated but apparently of 2005, at http://www.isworld.org/csaunders/rankings.htm

Schlogl C. (2003) 'Mapping the intellectual structure of information management' Wirtschaftsinformatik 45, 1 (February 2003) 7-16

Vessey I., Ramesh V. & Glass R.L. (2002) 'Research in information systems: An empirical study of diversity in the discipline and its journals' J. Mngt Infor. Syst. 19, 2 (Fall 2002) 129-174

Walstrom K.A. & Leonard L.N.K. (2000) 'Citation classics from the information systems literature' Infor. & Mngt 38, 2 (December 2000) 59-72


Appendix 1: Thomson/ISI Cited Ref Search

This Appendix provides comparisons of results obtained using the ISI General Search and ISI Cited Ref Search. A small number of academics were selected, from among expatriates, local researchers, North Americans and Europeans. All but one were selected because of their relatively uncommon names, in order to ease the difficulties of undertaking the searches and thereby achieve reasonable quality in the data, and no significance should be inferred from inclusion in or exclusion from this short list. One, this author, was selected because, for researchers with common names, full knowledge of the publications list makes it much easier to ensure accuracy.

The first three columns of the table show the number of citations for each author of articles that are in the ISI database, together with the count of those articles, and the largest citation-count found. (This data should correspond with that for the same researcher in Exhibits 3 and 4, but in practice there are many small variations, some caused by the 3-month gap between the studies that gave rise to those two tables and the study that gave rise to Exhibit 6). The next three columns show the same data for articles by the author that are not in the ISI database. The final two columns show the sum of the two Citation-Count columns, and the apparent Expansion Factor, computed by dividing the Total Citations by the Citation-Count for articles in the ISI database.

Exhibit 6: Thomson/ISI Cited Ref Search

---- In ISI Database ----
-- Not in ISI Database -
Researcher
Citation-Count
Article-Count
Highest Cite-Count
Citation-Count
Article-Count
Highest Cite-Count
Total Citations
Expansion Factor
Sirkka Jarvenpaa (as S.L.)
973
34
110
575
118
88
1548

1.6

Peter Keen (as P.G.W.)
425
14
190
1625
325
463
2050

4.8

Eph McLean (as E. and E.R.)
132
14
41
84
29
42
216
1.6
(Eph Mclean, corrected)
132
14
41
532
20
448
664
5.0
---
---
---
---
---
---
---
---
David Avison (as D.)
66
9
46
99
37
16
165
2.5
Ron Stamper (as R. and R.K.)
59
13
16
255
149
19
314
5.3
Frank Land (as F.)
74
16
19
161
88
25
235
3.2
---
---
---
---
---
---
---
---
Iris Vessey (as I.)
622
32
114
186
52
76
808
1.3
---
---
---
---
---
---
---
---
Philip Yetton (as P. and P.W.)
278
20
59
65
51
6
343
1.2
Peter Seddon (as P. and P.B.)
81
4
69
67
30
22
148
1.8
Graeme Shanks (as G.)
66
10
15
51
32
7
117
1.8
Paula Swatman (as P.M)
43
4
31
102
39
23
145
3.4
Roger Clarke (as R. and R.A.)
41
11
17
176
131
8
217
5.3
Guy Gable (as G.G.)
35
4
29
73
24
39
108
3.1


Appendix 2: Thomson/ISI cf. Google Comparisons for Selected Researchers

This Appendix provides detailed comparisons of results for both ISI and Google Scholar. Seven Australian academics were selected, from among both expatriates and local researchers. All but one were selected because of their relatively uncommon names, in order to ease the difficulties of undertaking the searches and thereby achieve reasonable quality in the data, and no significance should be inferred from inclusion in or exclusion from this short list. One, this author, was selected because, for researchers with common names, full knowledge of the publications list makes it much easier to ensure accuracy.

In each of the following tables:

Exhibit 7A: Thomson/ISI cf. Google - Iris Vessey

Google Count
Thomson Count
Venue
145
111
Journal
92
Unindexed (!!)
Journal (ISR)
88
83
Journal
86
Unindexed (!!)
Journal (CACM)
56
26
Journal
52
Unindexed
Conference (ICIS)
52
Unindexed
Journal (IJMMS)
48
Unindexed (!!)
Journal (CACM)
41
Unindexed (!!)
Journal (IEEE Software)
31
9
Journal
691 or 320
229 (of 601)
Totals

Exhibit 7B: Thomson/ISI cf. Google - Ron Weber

Google Count
Thomson Count
Venue
125
38
Journal
102
Unindexed
Journal (JIS)
106
30
Journal
87
36
Journal
72
26
Journal (Commentary)
65
20
Journal (Commentary)
45
Unindexed
Book
34
Unindexed
Journal (JIS)
34
22
Journal
31
24
Journal
701 or 520
196 (of 328)
Totals

Exhibit 7C: Thomson/ISI cf. Google - Philip Yetton

Google Count
Thomson Count
Venue
302
Unindexed
Book
55
11
Journal
42
12
Journal
32
12
Journal
31
34
Journal (1988)
27
Unindexed
Book
26
57
Journal (1982)
20
23
Book
18
6
Journal (1985)
18
Unindexed
Government Report
571 or 224
155 (of 270)
Totals

Exhibit 7D: Thomson/ISI cf. Google - Peter Seddon

Google Count
Thomson Count
Venue
133
60
Journal
47
Unindexed
Journal (CAIS)
43
Unindexed
Conference (ICIS)
33
Unindexed
Conference
22
Unindexed (!)
Journal (DB, 2002)
24
2
Journal (I&M, 1991)
18
2
Journal
18
Unindexed
Journal (JIS)
13
Unindexed
Conference (ECIS)
9
0
Journal (JIT, Editorial)
360 or 184
64 (of 70)
Totals

Exhibit 7E: Thomson/ISI cf. Google - Paula Swatman

Google Count
Thomson Count
Venue
117
29
Journal
73
Unindexed
Journal (Int'l Mkting Rev)
61
Unindexed
Journal (TIS)
43
Unindexed
Journal (JSIS)
29
Unindexed (!)
Journal (IJEC)
26
12
Journal
26
Unindexed
Journal (JIS)
24
6
Journal
22
Unindexed
Conference
20
Unindexed
Journal (EM)
441 or 167
47 (of 53)
Totals

Exhibit 7F: Thomson/ISI cf. Google - Roger Clarke

Position
Google Count
Thomson Count
Venue
57
81
14
Journal
59
85
16
Journal
102
60
Unindexed
Journal (IT&P)
148
47
Unindexed
Journal (TIS)
253
33
Unindexed
Conference
325
28
Unindexed
Conference
373
25
3
Journal
407
23
Unindexed
Journal (JSIS)
539
18
Unindexed
Journal
558
17
Unindexed
Conference
417 or 191
33 (of 44)
Totals

Exhibit 7G: Thomson/ISI cf. Google - Guy Gable

Google Count
Thomson Count
Venue
102
Unindexed (!)
Journal (EJIS, 1994)
56
Unindexed (!)
Journal (ISF, 2000)
40
Unindexed
Journal (JGIM)
27
6
Journal (MS)
27
Unindexed
Conference
24
23
Journal (I&M, 1991)
23
Unindexed
Conference
14
Unindexed
Conference
13
Unindexed
Conference
10
Unindexed
Conference
336 or 51
29 (of 34)
Totals

Acknowledgements

This paper has benefitted from comments from several colleagues on an earlier draft, in particular from Peter Seddon of the University of Melbourne, who identified a methodological flaw that needed to be addressed. Responsibility for the work rests, however, entirely with the author.


Author Affiliations

Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., a Visiting Professor in the E-Commerce Programme at the University of Hong Kong, and a Visiting Professor in the Department of Computer Science at the Australian National University.



xamaxsmall.gif missing
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.

From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 50 million in early 2015.

Sponsored by Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916

Created: 1 April 2006 - Last Amended: 29 April 2006 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/SOS/CitAnal0605.html
Mail to Webmaster   -    © Xamax Consultancy Pty Ltd, 1995-2017   -    Privacy Policy