Roger Clarke's Web-Site
© Xamax Consultancy Pty Ltd, 1995-2018
|Identity Matters||Other Topics||Waltzing Matilda||What's New|
Roger Clarke **
Notes of 20 June 2006, minor revs. 21 June 2006, 31 July 2007
See also Draft Guidelines for the Bled Conference
Prepared for the 19th Bled eCommerce Conference 5-7 June 2006, when presenting the Outstanding Paper Award
© Xamax Consultancy Pty Ltd, 2006
Available under an AEShareNet licence or a Creative Commons licence.
This document is at http://www.rogerclarke.com/SOS/PaperRev.html
2 slides to support a brief presentation on the topic are at http://www.rogerclarke.com/SOS/PaperRev.ppt
Academic communities are heavily dependent upon the efforts of their members in relation to the editing of journals, the management of conference programs, the refereeing of journal articles and the review of conference papers. Despite the importance of these roles, few journals or conferences provide guidance about what is expected of a referee or reviewer. These Notes are intended to contribute to the filling of that gap. I'd appreciate your thoughts about how to improve this document.
Although there are differences in the level of expectation for a refereed journal article in comparison with the reviewing of a conference paper, the two activities are sufficiently similar that these Notes treat them as though they were the same activity. The term 'review' is used throughout to encompass 'refereeing' as well.
The reviewing of papers is an act of quality assurance. The dimensions of quality that a journal or conference is concerned about vary, particularly in the weighting given to them. The quality factors that are considered by the Bled Outstanding Paper Award Committee are as follows:
Regrettably, some journals have sacrificed Relevance in recent years and prioritised Rigour for its own sake. Some journals, and most conferences, adopt a more balanced approach. The reviewer needs to take into account the the particular venue's attitude towards these quality factors.
There are several customers or stakeholders whose interests need to be taken into account.
Clearly, the immediate purpose of a review is to provide the Editor or Program Chair with information about the paper's acceptability or otherwise. But, important as those people are, they operate as a proxy for the journal's readers or the conference delegates.
The next purpose of a review is to inform the Author(s) about what needs to be done to improve the paper, possibly to the level needed to get across the threshhold and achieve acceptance, possibly not.
But it's advantageous to think about the Community with whom the reviewer is engaging. This may be formed around the nucleus of a discipline, a research domain, a geographical region, or intersections among two or more of them. All members of that community stand to gain from a professional approach to reviewing.
A treatise could be written on what constitutes a good review; but there are a couple of key characteristics that enable a review to serve the needs of the editor, the author and the community alike:
There are also some traps that it's important for reviewers to avoid. Three are ethical:
There are also a couple of corollaries of the four quality indicators discussed above:
Directly relevant articles and notes from within the IS discipline are:
Bieber M. (1997?) 'How to Review' at http://www-ec.njit.edu/~bieber/review.html
Davison R.M., de Vreede G-J. & Briggs R.O. (2006) 'On Peer Review Standards For The Information Systems Literature' Commun. AIS 16, 49 (2005) 967-980, at http://cais.isworld.org/articles/default.asp?vol=16&art=49
Koh C. (2003) 'IS journal review process: a survey on IS research practices and journal review issues' Infor. & Mngt 40 (2003) 743-756
Lee A.S. (1995) 'Reviewing a Manuscript for Publication' Invited Note in J. Ops Mngt 13, 1 (July 1995) 87-92, at http://www.people.vcu.edu/~aslee/referee.htm
Zmud R. (1998) 'A Personal Perspective on the State of Journal Refereeing' MIS Qtly 22, 2 (September 1998), at http://www.misq.org/archivist/vol/no22/issue3/edstat.html
A couple of relevant items from other disciplines are:
Black N., van Rooyen S., Godlee F., Smith R. & Evans S. (1998) 'What Makes a Good Reviewer and a Good Review for a General Medical Journal' J. Am. Med. Assoc. 280, 3 (1998) 231-233
Finney D.J. (1997) 'The Responsible Referee' Biometrics 53, 2 (June 1997) 715-719
Hames I. (2007) 'Peer Review and Manuscript Management in Scientific Journals' Blackwell/ALPSP, 2007, from http://www.blackwellpublishing.com/book.asp?ref=9781405131599
Smith, A.J. (1990) 'The task of the referee' IEEE Computer 23, 4 (April 1990) 65-71
Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the E-Commerce Programme at the University of Hong Kong, a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., and a Visiting Professor in the Department of Computer Science at the Australian National University.
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.
From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 50 million in early 2015.
Sponsored by Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916
Created: 20 June 2006 - Last Amended: 31 July 2007 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/SOS/PaperRev.html