Roger Clarke's Web-Site

 

© Xamax Consultancy Pty Ltd,  1995-2018


Roger Clarke's 'Critical Analysis of Works'

The Critical Analysis of Published Works:
How to Establish a New Research Technique
in the IS Discipline?

Review Version of 28 June 2018

Roger Clarke **

© Xamax Consultancy Pty Ltd, 2018

Available under an AEShareNet Free
for Education licence or a Creative Commons 'Some
Rights Reserved' licence.

This document is at http://www.rogerclarke.com/SOS/CAPW.html


Abstract

The Information Systems (IS) discipline has accumulated a large corpus of published works. Constructive criticism, both of individual works and of collections of works, is an essential element of a living discipline. On the other hand, there is resistance among IS editors and reviewers to articles whose primary contribution is a critique of prior publications. A project has been undertaken that is intended to provide conceptual foundations and methodological guidance to enable papers of that nature to achieve publication in quality IS venues. This paper provides an overview of the challenges involved in establishing a sufficient basis for a new research technique, and the approach being adopted to address them.


Contents


1. Introduction

Since the Information Systems (IS) discipline emerged c. 1965, a substantial body of publications has accumulated. For example, in mid-2018, the AIS eLibrary contains over 15,000 refereed works, John Lamp's directory of IS journals identifies almost 700 active venues that publish IS works, of the order of 7,000 articles have been published in the `Basket of 8' journals alone, and the major five IS conferences alone publish an additional 1500 refereed papers per annum.

The accumulated IS literature evidences two strong desires. A great deal of emphasis is placed on empirical research involving the observation and measurement of some aspect of IS practice. In addition, many articles are published that seek to establish or extend theories about IS practices, which can then provide a basis for further empirical research.

Important though these approaches are, progress in the discipline depends on some further elements as well. Meta-discussions are needed, in order to clarify such aspects as the discipline's scope, the meanings of key terms, and the suitability of particular methods for particular purposes. This paper is a contribution to one such meta-discussion.

The starting-point for the project reported on here is a conviction that critical thought about prior IS literature has a vital role to play. Criticism is, after all, central to the notion of science, in that all propositions must be in principle refutable, all propositions must be regarded as provisional, and all propositions must be subjected to testing (Popper 1963). For this reason, criticism is inherent in the review process whereby works are evaluated as a condition of publication in refereed venues. However, the IS discipline has to date demonstrated considerable nervousness about works that contain criticism of prior works, and especially about works whose specific purpose is to criticise prior works.

The author contends that the conduct and publication of critiques of prior IS works is vital to progress in the discipline. The research methodology literature does not currently provide a sufficient basis for work of this nature. A project has accordingly been undertaken to identify the conceptual foundations for a research technique for the critical analysis of published works, and to formulate and test guidance for the conduct of such research.

The purpose of the present paper is to provide an overview of the project as a whole. It commences by discussing methodological considerations in the establishment of a research technique. The nature of prior publications is examined, and examples are identified of applications which the technique should support. A survey is undertaken of the various forms of content analysis used in research. This is combined with a brief discussion of the nature and role of criticism, in order to enable conceptualisation of the critical analysis of published works. The second part of the paper then utilises existing research techniques in order to develop specific guidance for the conduct of the technique, and provides some exemplars of its application.

The number of elements involved is considerable. In order to produce an article of conventional length, each section necessarily provides an overview only. Further details on most aspects are available in a small number of papers that have achieved publication to date, together with the author's Working Papers.


2. Methodological Considerations

This paper uses the terms `research technique' to refer to a specific process (such as structured interview, case study or action research). Each project uses a `research method', comprising one or more research techniques. The term `research methodology' is used in its appropriate sense of the disciplined study of research methods, rather than as a synonym for either or both of `research technique' or `research method'. In these terms, the purpose of the present project is the establishment of a new research technique.

The objective of the project is the development of a research technique for the critical analysis of published works relevant to the IS discipline. This is a socio-technical artefact as that term is used in design science (Niedermann & March 2012, Gregor & Hevner 2013, p.337), and hence the design science approach is broadly relevant to the project.

In principle, the project needs to be founded on generic guidance in relation to the establishment of a new research technique. Sources of that nature are proving elusive, however, and the search continues. What is clear is that a body of knowledge about a research technique comprises contextual information, conceptual foundations, guidance in relation to the process to be used, and exemplars of processes that have been articulated in order to address particular kinds of research questions. The body of this paper addresses each of these elements in turn.

The term `published works' is intended to encompass refereed articles in journals, refereed papers in conference proceedings, refereed chapters in academic books, and refereed academic books. It might reasonably be extended to papers in workshops and symposia, and for some purposes may include PrePrints or Working Papers published by institutions with appropriate standing, and possibly also research reports commissioned by government agencies, foundations, industry associations and perhaps individual corporations. Some other forms of publication may be relevant, depending on the specific research purpose. For example, technical media or corporate white papers might be included, if the research purpose were to, for example, assess the impact of academic work on thinking among consultancies or within industry sectors.

The venues that are `relevant to the IS discipline' depend on the context of the specific project. Beyond mainstream IS journals and conferences, venues used by particular reference and cognate disciplines may also be relevant, e.g. in management, social sciences, or computer sciences.

It might be feasible to conduct studies of the complete corpus of `published works relevant to the IS discipline'. It is far more likely, however, for each project's focus to be on a particular population segment, with a sampling frame and sample selected from within that segment. A segment might be defined to be a particular venue (e.g. the complete set of ICIS Proceedings), or a time-span within one or more venues (e.g. the last 10 years of the `Basket of 8' IS journals), or a focussed collection (e.g. one or more journal special issues, narrowly-specialised conferences, or academic books). Another approach would be to define the population segment on the basis of the research domain that the papers address, or the research technique that they apply. The smallest unit of study would be a single work, although this would only be likely to be relevant in the case of papers that are judged to be particularly significant, on the basis of citation-count or some other measure of influence.


3. Content Analysis

A significant proportion of research involves the appraisal of content previously uttered by other people. This section briefly reviews categories of research technique whose focus is adjacent to the topic addressed in this paper.

Qualitative research techniques such as ethnography, grounded theory and phenomenology involve the disciplined examination of content. However, the content is of a kind materially different to refereed papers. The text may be generated in natural settings (field research), in contrived settings (laboratory experiments), or in a mix of the two settings (e.g. interviews conducted in the subject's workplace). The materials may originate as text, or as communications behaviour in verbal form (speech in interviews that is transcribed into text), as natural non-verbal behaviour ('body-signals'), or as non-verbal, non-textual communications behaviour (such as answering structured questionnaires). In other cases, text that arises in some naturalistic setting is exploited by the researcher (e.g. voice transcripts, emails, social media postings). The issues arising with analysis of these kinds of content are very different from those associated with the analysis of carefully-considered, formalised content in refereed articles.

A context that is more closely related to the present purpose is literature reviews that examine substantial bodies of prior research. Traditional `narrative' reviews have been criticised as being insufficiently rigorous (Oakley 2003, p.23). There are now expectations of structure, transparency and replicability, and the term `systematic' review is commonly applied (Webster & Watson 2002, Okoli & Schabram 2010). Examples of relevance to this project include Galliers & Whitley (2002, 2007), which analysed themes in the ECIS conference series, and Clarke (2012) and Clarke & Pucihar (2015), which reviewed the corpus of Bled Conference papers. Grover & Lyytinen (2015) and Tarafdar & Davison (2018) reported on meta-analyses of papers in the `Basket of 8' IS journals. Such approaches have relevance to the present project, but their focus is on description and interpretation, and the research techniques used were not devised in order to support critical analysis.


4. Criticism

The focus of this project is the critical analysis of publications. Two aspects of the notion of criticism are particularly relevant. The first is the role that criticism plays in scientific research generally and in IS research in particular. The second is the critical theory research genre.

4.1 The Role of Criticism in Research

The purpose of undertaking content analysis may be simply exposition, that is to say the identification, extraction and summarisation of content, without any significant degree of evaluation. There are benefits in undertaking content analysis in a positive frame of mind, and in assuming that all that has to be done is to present existing information in brief and readily-accessible form (as, for example, much of the present paper does).

Alternatively, the researcher may bring a questioning and even sceptical attitude to the work. A common purpose of literature reviews is to depict the current state of theory in an area. The purpose may be to infer one or more implications for the particular context relevant to the project. Alternatively, it may be to identify gaps in the present body of theory. Gap-identification is a gentle form of criticism in the sense in which the term is used in this paper. The notion of criticism does, however, go rather further than merely identifying previously unresearched corners of theory. A critic asks hard questions that necessarily cut to the core of academic work. Is it reasonable to assume that all relevant published literature is of high quality? that the measurement instruments and research techniques have always been good, well-understood by researchers, and appropriately applied? that there have been no material changes in the relevant phenomena? and that there have been no material changes in the intellectual contexts within which research is undertaken?

The term 'criticism' is often used in a pejorative sense, implying that the critic is merely finding fault, is being destructive rather than constructive, and is failing to propose improvements to sustain the merits and overcome the faults. The sense in which the term is used here, however, is related to `literary criticism', and is less focussed on negative sentiments and more on positive ones. Criticism presents an analysis of both the merits and faults of a body of work. The focus may be on the process (the sequence of actions undertaken by the original authors) or the product (the expression of the analysis and the conclusions that they reached). Such terms as `constructive criticism' and 'critique' may be preferred, in an endeavour to avoid the negative impressions, to indicate that the process is systematic, and to bring focus to bear on the contribution being made by both the critical comments and the work that is being subjected to analysis.

The justification for applying a sceptical eye to a body of work is that criticism plays a vital role in the scientific process. The conventional Popperian position is that the criterion for recognising a scientific theory is that it deals in statements that are empirically falsifiable, and that progress depends on scrutiny of theories and attempts to demonstrate falsity of theoretical statements: "The scientific tradition ... passes on a critical attitude towards [its theories]. The theories are passed on, not as dogmas, but rather with the challenge to discuss them and improve upon them" (Popper 1963, p.50).

However, senior members of a discipline commonly behave in ways that are not consistent with the Popperian position. This might be explained by the postulates of 'normal science', which view the vast majority of research work as being conducted within a 'paradigm' and subject to its conventions (Kuhn 1962). In more practical terms, the problem may arise because senior members of any discipline have strong psychic investment in the status quo, and - nomatter how cogent and important the argument - react negatively against revolutionary propositions. Firmly-worded criticism is normal in reviewers' reports on as-yet unpublished submissions, and may be acceptable if uttered by a senior about a contrarian idea, whereas it commonly attracts opprobrium if made by an outsider or relative newcomer about the contemporary wisdom.

In an influential commentary, Webster & Watson (2002) recommended that "In contrast to specific and critical reviews of individual papers, tell the reader what patterns you are seeing in the literature ... Do not fall into the trap of being overly critical" (p.xviii). Although, on a literal reading, merely warns against unduly strong or negative expression, it is capable of being interpreted as valuing politeness among researchers more highly than scientific insight and progress. Similarly, Straub (2009, p.viii) advised authors that "papers should be in apposition [the positioning of things side by side or close together] rather than in opposition". Unfortunately, this too is all-too-easily interpreted as expressing a moral judgement that 'criticism is a bad thing'. On the contrary, it is an obligation of researchers to 'think critically' and to 'apply their critical faculties.

4.2 Critical Theory Research

Positivism and interpretivism are well-established schools of research in IS. They have been joined by design science. And they have an odd bedfellow, in the form of what is variously termed 'critical research' and 'critical theory research'. The term 'critical' in this context is a little different from, but related to, the sense of 'analysis of the merits and faults of a work' discussed in the previous section.

Both positivism and interpretivism are concerned with description and understanding of phenomena and behaviours. Sometimes the focus is on natural phenomena, but frequently the interest is in natural phenomena that have been subjected to an intervention. Importantly for the present project, however, both positivism and interpretivism involve strenuous avoidance of moral judgements and of 'having an agenda'. Design research, in contrast, is expressly purposeful and value-laden, in that the features designed into the artefact embody judgements about what is good in the particular context, and whose perspective the goodness is to serve.

Critical theory research is concerned with description and understanding of phenomena and behaviours, but, like design science, it `has an agenda'. It recognises the effects of power and the tendency of some stakeholders' interests to dominate those of other stakeholders. It brings to light "the restrictive and alienating conditions of the status quo" and expressly sets out to "eliminate the causes of alienation and domination" (Myers 1997). "Critical research generally aims to disrupt ongoing social reality for the sake of providing impulses to the liberation from or resistance to what dominates and leads to constraints in human decision-making. Typically critical studies put a particular object of study in a wider cultural, economic and political context, relating a focused phenomenon to sources of broader asymmetrical relations in society ... (Alvesson & Deetz 2000, p.1). "Critical IS research specifically opposes technological determinism and instrumental rationality underlying IS development and seeks emancipation from unrecognised forms of domination and control enabled or supported by information systems" (Cecez-Kezmanovic 2005, p.19).

In Myers & Klein (2011), three elements of critical research are identified:


5. Conceptualisation of the Critical Analysis of Published Works

The preceding discussion provides a basis for defining the desired technique at a conceptual level.

Critical analysis of published works (CAPW) begins with a research question that is not about the real world of IS practice, but is about some segment of the corpus of published works relevant to the IS discipline.

CAPW is concerned with more than explication of the content of the selected works, and more than its interpretation. It presents a critique, by which is meant analysis of the merits and faults of the selected works, and perhaps by implication of the segment and even of the corpus.

CAPW conducts critical analysis of works, not critical analysis of authors, and expresses criticism in a measured manner, avoiding unduly strong or negative expression and overly colourful speech.

CAPW seeks such structuredness and auditability as is feasible in the particular context, and expresses criticism with a degree of confidence that is commensurate with the nature of the evidence.

On the basis of these conceptual foundations, the following sections consider existing techniques, and propose an expression of process guidance in relation to the critical analysis of published works.


6. Relevant Existing Techniques

This section outlines two categories of existing research technique that are directly relevant to the present purpose. The first is a form of literature review compatible with critical theory research. The second is a range of techiques referred to generically as `content analysis'.

6.1 A Hermeneutic Approach to Literature Review

The emphasis on 'systematic' literature reviews noted earlier has itself been subjected to criticism, in that it "suppresses aspects of quality in research and scholarship that are at least as important as clarity, countability and accountability - such as intertextual connectivity, critique, interest, expertise, independence, tacit knowledge, chance encounters with new ideas, and dialogic interactions between researcher, 'literature' and 'data'" (MacLure 2005, p.394).

In Boell & Cecez-Kecmanovic (2014) it is argued that a constructively loose and iterative process is needed, to avoid undue constraints and unlock insight and creativity: "Highly structured approaches downplay the importance of reading and dialogical interaction between the literature and the researcher; continuing interpretation and questioning; critical assessment and imagination; argument development and writing - all highly intellectual and creative activities, seeking originality rather than replicability [MacLure, 2005, Hart, 1998]" (p.258).

To address these issues, Boell & Cecez-Kecmanovic (2014) "propose hermeneutic philosophy as a theoretical foundation and a methodological approach for studying literature reviews as inherently interpretive processes in which a reader engages in ever exp[a]nding and deepening understanding of a relevant body of literature. Hermeneutics does not assume that correct or ultimate understanding can be achieved, but instead is interested in the process of developing understanding" (p.259). The framework, reproduced in Figure 1, comprises two intertwined cycles: a search and acquisition circle, and a wider analysis and interpretation circle (p.263).

Rather than a carefully-planned and closed-ended process, this approach embodies "questioning and critical assessment ... of previous research" (p.258). "Critical assessment ... not only reveals but also ... challenges the horizon of possible meanings and understanding of the problem and the established body of knowledge" (p.267).

6.2 Content Analysis

The term `content analysis' refers to a cluster of techniques that seek to classify text, or specific aspects of text, into a manageable number of categories. Two definitions are:

Content Analysis is the semantic analysis of a body of text, to uncover the presence of strong concepts (Indulska et al. 2012, p.4, citing Weber 1990) Content Analysis is the interpretation of the content of text data through the systematic classification process of coding and identifying themes or patterns (Hsieh & Shannon 2005, p.1278)

The Hsieh & Shannon (2005) paper indicates a 7-step process which the authors attribute to Kaid (1989). See also vom Brocke & Simons (2008):

  1. formulation of the research questions
  2. sample selection
  3. definition of the categories to be applied
  4. specification of the coding process
  5. implementation of the coding process
  6. quality control
  7. analysis

As with any research technique, all aspects need to be subject to quality controls. Krippendorff (1980), Weber (1990) and Stemler (2001) emphasise steps 3-5 in relation to the coding scheme and its application, highlighting the importance of achieving reliability. Approaches include coding by individuals with strong experience in both the review of articles and the subject-matter, parallel coding by multiple individuals, review of individuals' coding by other parties, and publication of both the source materials and the detailed coding sheets, in order to enable audit by other parties.

Content analysis techniques exhibit varying degrees of structure and rigour, from impressionistic to systematic, and they may involve qualitative and/or quantitative assessment elements. Quantitative data may be on nominal, ordinal, cardinal or ratio scales. Data collected on higher-level scales, especially on a ratio scale, is able to be subjected to more powerful inferencing techniques. Qualitative data, on the other hand, may be gathered on a nominal scale (whereby differences are distinguished, but no ordering is implied) or on an ordinal scale (such as 'unimportant', 'important', 'very important').

Quantification generally involves measurement, most fundamentally by counting. This raises questions about the arbitrariness of boundaries, and about configuration and calibration of the measuring instrument(s). Some quantification techniques involve sleight of hand. A significant example is the frequently-encountered but unjustified assumption that 'Likert-scale' data is not merely ordinal, but is cardinal (i.e. the spaces between the successive terms are identical), and even ratio (i.e. the scale also features a natural zero), in order to justify the application of powerful statistical techniques to the data.

Many authors implicitly equate quantification with rigour, and qualitative data with subjectivity. They accordingly deprecate qualitative analysis, or at least relegate it to pre-theoretical research, which by implication should be less common than research driven by strong theories. However, the majority of authors spend only limited time considering the extent to which the assumptions and the processes underlying the act of quantification may be arbitrary or themselves 'subjective'. Positivism embodies an implicit assumption that computational analysis necessarily leads to deep truth. The assumption needs to be tested in each particular circumstance, yet such testing is seldom evident.

A positivist approach to categorising content analysis "along a continuum of quantification" distinguishes "narrative reviews, descriptive reviews, vote counting, and meta-analysis" (King & He 2005, p.666):

As regards the approach taken to establishing a coding scheme for content, a four-way classification scheme is relevant to the present purpose. The first three forms are usefully discussed in Hsieh & Shannon (2005).

Firstly, the categories may be defined a priori, in particular arising from existing theory. To the extent that the declared or inferred content of the text does not fit well to the predefined categories, there may be a need to consider possible revisions of the coding scheme, or even of the theory on which the research design was based. It may be feasible to draw inferences based on counts of the occurrences of categories and/or on the intensity of the statements in the text, such as the confidence inherent in the author's choice of language (e.g. "this shows that" cf. "a possible explanation is that"). However, as with any theory-driven research, the evidence extracted from the text may have a self-fulfilling-prophecy quality about it, i.e. there is an inevitable tendency to find more evidence in support of a theory than in conflict with it, and contextual factors may be overlooked. In order to enable auditability, it is important that not only the analysis be published, but also the raw material and the coding scheme.

Alternatively, the categories may emerge from the data. This is of value, for example, when there is an absence of suitable theories to guide the establishment of a priori categories. The process of necessity involves assumptions, and hence external validity of conclusions arising from this approach is likely to be limited. Depending on the degree of generality of the conclusions claimed by the author, full disclosure of the text selection, coding and inferencing procedures may be merely desirable or vital.

A third approach has been dubbed summative context analysis. It involves "counting and comparisons, usually of keywords or content, followed by the interpretation of the underlying context" (Hsieh & Shannon 2005, p.1277). The first step is to explore usage, by "identifying and quantifying certain words or content in text with the purpose of understanding the contextual use of the words or content" (p.1283).

Because of the complexity and variability of language use, and the ambiguity of a large proportion of words and phrases, a naive approach to counting words is problematic. At the very least, a starting-set of terms needs to be established and justified. A thesaurus of synonyms and perhaps antonyms and qualifiers is needed. Allowance must be made for both manifest or literal meanings, on the one hand, and latent, implied or interpreted meanings (in semiotic terms, `pragmatics'), on the other. Counts may be made not only of the occurrences of terms, but also of the mode of usage (e.g. active versus passive voice, dis/approval indicators, associations made).

The degree of analytical rigour that quantification can actually deliver depends a great deal on a number of factors. Critical among them are:

Publication of details of text selection and the analytical process is in all cases important, and essential where a degree of rigour and external validity is claimed.

A fourth approach is usefully described as programmatic content analysis. This enables much larger volumes of text to be analysed. The coding scheme may be defined manually, cf. directed content analysis / a priori coding. However, some techniques involve purely computational approaches to establishing the categories, cf. 'machine-intelligent' (rather than human-intelligent) emergent coding. The processing depends, however, on prior data selection, data scrubbing and data-formatting. In addition, interpretation of the results involves at least some degree of human activity.

Debortoli et al. (2016), distinguishes three alternative approaches to programmatic coding:

Given that the 'big data analytics' movement is highly fashionable, vast volumes of data are available, and there is a comfort factor involved in office-based work much of which is automated, it would appear reasonable to anticipate that programmatic analysis techniques may be a growth-area in the coming few years - at least until their limitations are better appreciated (Clarke 2016a, 2016c).

The four approaches to content analysis outlined above appear to be accepted within the IS discipline, but their use has been somewhat limited. For example, in a survey of the papers published in six leading IS journals during the 1990s, Mingers (2003) found that the use of content analysis as a research technique was evident in only four of the journals, and even in those four in only 1-3% of all papers published during that time. In February 2017, of the nearly 15,000 refereed papers indexed in the AIS electronic library, 13 had the term 'content analysis' in the title, and 69 in the Abstract. In recently-published papers, the most common forms of text that have been subjected to content analysis appear to be social media and other message content, with other categories including newspaper articles and corporations' 'letters to shareholders'. There appears to have been very limited application of these techniques to published works.

The four forms of content analysis outlined above are primarily descriptive, and at best interpretive. Their focus is on concepts, themes, patters and relationships. Moreover, the extent to which the semantics of the source-materials are reflected is quite low. A technique for `critical analysis' must delve much more deeply not only into the tenable meanings of the content, but also into the context and the assumptions inherent in the work. It may also be necessary to follow trails such as cited works and terms used, in order to place the work within a genre, or within a `school of thought'.

A recent proposal endeavours to address these weaknesses. Wall et al. (2015) describe an approach that they refer to as critical discourse analysis, which they say they have based on Habermasian strains of critical discourse analysis (CDA). Their starting-point is that "the information systems (IS) discipline is subject to ideological hegemony" (p.258). By `ideological hegemony' the authors mean "the conscious or unconscious domination of the thought patterns and worldviews of a discipline or subdiscipline that become ingrained in the epistemological beliefs and theoretical assumptions embedded in scientific discourse (Fleck, 1979; Foucault, 1970; Kuhn, 2012). They argue that "ideologies can be harmful to individuals who are disadvantaged or marginalized by them, and they can be problematic to scientific research because they represent blind spots" (p.258), and hence that "review papers [should] ... challenge ideological assumptions by critically assessing taken-for-granted assumptions" (p.257).

The authors propose a seven-step process (pp. 265-9):

  1. Identifying the Problem
  2. Specifying the Literature
  3. Developing Codes for Validity Claims
  4. Analyzing Content and Coding
  5. Reading and Interpreting
  6. Explaining the Findings
  7. Engaging in Critical Reflexivity

The hermeneutic approach to literature review of Boell & Cecez-Kezmanovic and the critical discourse analysis approach to content, overlaid on the prior literature, assist in the design of a content analysis research method with the desired attributes.


7. A Technique for the Critical Analysis of Published Works

The methodological discussion at the beginning of this paper identified that the proposed technique has considerable scope, with many different aspects of published works subject to various kinds of criticism. Some of these are criticisms within the terms of the work itself, such as evaluation of the quality of the application of the research technique, whereas others are critical in the sense used by critical theory research, in that they reflect power relationships among stakeholders. The description of a research technique must therefore be sufficiently abstract to encompass all variants, but also guide the researcher in how to customise a technique appropriate to the particular need.

Several strongly-desirable characteristics can be derived from the most relevant of the research techniques outlined above. These include:

Appendix 1 provides the current version of the proposed guidance.


8. Applications

As indicated in earlier sections, there are multiple contexts of use for the guidance in Appendix 1. In each case, the guidance needs to be customised for the nature of the research question and particularised to the approach adopted to categorisation.

The following is suggested as a representative set of use cases:

Successive iterations of the process have been trialled in a particular context. The author has been developing a line of research into the question of `researcher perspective'. By this is meant the viewpoint from which the phenomena are observed. Several collections of works have been examined, including a cross-sectional sample of the author's national IS conference (ACIS) and journal (AJIS) (Clarke 2015), a longitudinal sample of papers from an annual conference (Clarke 2016b), a journal Special Issue on personal data markets (Clarke 2017), and (as-yet unpublished) a longitudinal sample of papers from the `Basket of 8' IS journals (Clarke et al. 2018). The analyses have shown that it is very common for IS research to adopt a single stakeholder's perspective as an objective, relegating the interests of other stakeholders to the role of constraints. Further, these studies have shown that this primary perspective is, in about 90% of cases, that of the system's sponsor.

The projects completed to date attest to the practicality of the technique when articulated for, and applied in, a specific critical theory research context. They do not represent a sufficiently rich set to illustrate or demonstrate the usefulness of the guidance across all of the various project-types. Far less do the projects conducted to date represent the evaluation step called for in design science, which involves observation and measurement of the new artefact's effectiveness in addressing stated objectives.


9. Conclusions, Limitations and Implications

This paper has outlined the elements of a project to establish adequate conceptual foundations and guidance in relation to the implementation of a technique for critical analysis of published works relevant to the IS discipline. It is contended that research of this nature is important to progress within the discipline, because constructive criticism lies at the very heart of scientific method. Criticism is, however, challenging and unpopular, particularly to the gate-keepers of a discipline. Both the conceptual foundations and the guidance in relation to the technique must accordingly be sufficiently robust to withstand the negative reactions that the proposition engenders, and to support the performance, reporting and publication of critiques that will assist in improvements to the quality of IS research, and the advancement of the discipline's standing.

All of the conceptual aspects outlined in this paper require further connections with the existing literature, deeper consideration, and application to the purpose. The guidance relating to the technique needs to reflect the ongoing engagement with prior literature. Multiple instantiations of the technique need to be articulated in respect of exemplar research projects. As indicated by the set suggested in the previous section, these projects need to encompass diverse senses of the term `criticism', some performed within the terms of the works that are being subjected to examination, and some adopting a broader view of the research segment, in line with the norms of critical theory research.


Appendix: A Technique for the Critical Analysis of Published Works

1. Specify the Objective

Determine and document the purpose of the research project.

2. Specify the Segment, Sampling Frame and Sample

Determine and document the part of the corpus of works relevant to IS that are to be subjected to examination.

3. Specify the Sources of Categorisations

Determine and document the means whereby code-sets are to be defined, and the protocol whereby codes are to be assigned and checked (e.g. through directed content analysis using a priori coding, emergent coding, or some other approach).

4. Review each Work in its own Terms

Perform 'orientational reading' of each paper, in order to "gain an overall impression" (Boell & Cecez- Kecmanovic, 2014, p.265).

5. Identify Key Passages

Perform deep 'analytical reading' (Boell & Cecez-Kecmanovic 2014, p.265).

6. Perform the Coding

Apply the coding protocol, including appropriate quality assurance measures.

7. Record Key Passages and Coding Rationale

Record sufficient data about, and quotations from, each paper to enable review and moderation of the coding, to provide contextual information and thereby support analysis of the results, and enable the research to be subjected to audit.

8. Review all Aspects of the Analysis

Review the approach adopted to quality assurance, and to the reliability of selection, coding, interpretation and presentation. Reflect on whether any aspect of the work is unreasonable or biassed: "By intentionally expressing, questioning, and reflecting upon their subjective experiences, beliefs, and values, critical researchers expose their [own] ideological and political agendas" (Cecez-Kezmanovich 2001, p.147).

9. Re-Work the Analysis as Necessary

Revise any selections, coding, interpretations or presentation about which doubt exists, carry the effects through the entire analytical process.

10. Consider Ways to Convey the Nature of the Content and the Results

Devise graphical forms to convey the nature of the paper (Boell & Cecez-Kecmanovic 2014, p.266), and the results of the analysis and coding, e.g. counts of relevant passages and indicators of their intensity, such as qualifications to statements. In the absence of reliable evidence, avoid attributing intent on the part of authors, and instead assume that the behaviour is "unconscious hegemonic participation" (Wall et al. 2015, p.261).


References

Alvesson M. & Deetz S. (2000) 'Doing Critical Management Research' Sage, 2000

Boell S.K. & Cecez-Kecmanovic D. (2014) 'A Hermeneutic Approach for Conducting Literature Reviews and Literature Searches' Communications of the Association for Information Systems 34, 12, at http://tutor.nmmu.ac.za/mgerber/Documents/ResMeth_Boell_2014_Literature%20Reviews.pdf

vom Brocke J. & Simons A. (2008) 'Towards a Process Model for Digital Content Analysis - The Case of Hilti' Proc. Bled Conference, June 2008, at http://aisel.aisnet.org/bled2008/2

Cecez-Kecmanovic D. (2001) 'Doing Critical IS Research: The Question of Methodology' Ch.VI in 'Qualitative Research in IS: Issues and Trends: Issues and Trends' (ed. Trauth E.M.), pp. 141-163, Idea Group Publishing, 2001, at https://pdfs.semanticscholar.org/37b1/e4c060b93fcfa04d81f03b750e746ba42f2d.pdf

Cecez-Kecmanovic D. (2005) 'Basic assumptions of the critical research perspectives in information systems' Ch. 2 in Howcroft D. & Trauth E.M. (eds) (2005) 'Handbook of Critical Information Systems Research: Theory and Application', pp.19-27, Edward Elgar, 2005

Clarke R. (2012) 'The First 25 Years of the Bled eConference: Themes and Impacts' Proc. 25th Bled eConference Special Issue, PrePrint at .. http://www.rogerclarke.com/EC/Bled25TA.html

Clarke R. (2015) 'Not Only Horses Wear Blinkers: The Missing Perspectives in IS Research' Keynote Presentation, Proc. Austral. Conf. in Infor. Syst. (ACIS 2015), at https://arxiv.org/pdf/1611.04059, Adelaide, December 2015, PrePrint at http://www.rogerclarke.com/SOS/ACIS15.html

Clarke R. (2016a) 'Big Data, Big Risks' Information Systems Journal 26, 1 (January 2016) 77-90, PrePrint at http://www.rogerclarke.com/EC/BDBR.html

Clarke R. (2016b) 'An Empirical Assessment of Researcher Perspectives' Proc. Bled eConf., Slovenia, June 2016, at http://www.rogerclarke.com/SOS/BledP.html

Clarke R. (2016c) 'Quality Assurance for Security Applications of Big Data' Proc. EISIC'16, Uppsala, 17-19 August 2016, PrePrint at http://www.rogerclarke.com/EC/BDQAS.html

Clarke R. (2017) 'Content Analysis in Support of Critical Theory Research: How to Deliver an Unwelcome Message Without Being Shot' Proc. 30th Bled eConference, June 2017, PrePrint at http://www.rogerclarke.com/SOS/CACT.html

Clarke R., Davison R. & Jia W. (2018) 'Researcher Perspective in the 'Basket of 8' IS Journals: A Weakness in Disciplinary Scope?' Working Paper, Xamax Consultancy Pty Ltd, at http://www.rogerclarke.com/SOS/RP8.html

Clarke R. & Pucihar A. (2013) 'Electronic Interaction Research 1988-2012 through the Lens of the Bled eConference' Electronic Markets 23, 4 (December 2013) 271-283, PrePrint at http://www.rogerclarke.com/EC/EIRes-Bled25.html

Debortoli S., Müller O., Junglas I. & vom Brocke (2016) 'Text Mining For Information Systems Researchers: An Annotated Topic Modeling Tutorial' Communications of the Association for Information Systems 39, 7, 2016

Galliers R.D. & Whitley E.A. (2002) 'An Anatomy of European Information Systems Research - ECIS 1993 - ECIS 2002: Some Initial Findings' Proc. ECIS 2002, June 2002, Gdask, Poland, PrePrint at http://personal.lse.ac.uk/whitley/allpubs/ECIS2002.pdf

Galliers R.D. & Whitley E.A. (2007) 'Vive les differences? Developing a profile of European information systems research as a basis for international comparisons' Euro. J. of Infor. Syst. 16, 1 (2007) 20-35

Gregor S. & Hevner A. (2013) 'Positioning Design Science Research for Maximum Impact' MIS Quarterly 37, 2 (June 2013 ) 337-355, at https://ai.arizona.edu/sites/ai/files/MIS611D/gregor-2013-positioning-presenting-design-science-research.pdf

Grover V. & Lyytinen K. (2015) `New State of Play in Information Systems Research: The Push to the Edges' MIS Quarterly 39:2 (2015) 271-296

Hsieh H.-S. & Shannon S.E. (2005) 'Three Approaches to Qualitative Content Analysis' Qualitative Health Research 15, 9 (November 2005) 1277-1288, at http://www33.homepage.villanova.edu/edward.fierros/pdf/Hsieh%20Shannon.pdf

Indulska M., Hovorka D.S. & Recker J.C. (2012) 'Quantitative approaches to content analysis: Identifying conceptual drift across publication outlets' European Journal of Information Systems 21, 1, 49-69, at http://eprints.qut.edu.au/47974/

Kaid L.L. (1989) 'Content analysis' In Emmert P. & Barker L.L. (Eds.) 'Measurement of communication behavior', pp. 197-217, Longman, 1989

King W.R. & He J. (2005) 'Understanding the Role and Methods of Meta-Analysis in IS Research' Communications of the Association for Information Systems 16, 32

Krippendorff K. (1980) 'Content Analysis: An Introduction to Its Methodology' Sage, 1980

Kuhn T.S. (1962) 'The Structure of Scientific Revolutions' University of Chicago Press, 1962

MacLure M. (2005) '`Clarity bordering on stupidity': where's the quality in systematic review?' Journal of Education Policy 20, 4 (2005) 393-416, at http://www.esri.mmu.ac.uk/respapers/papers-pdf/Paper-Clarity%20bordering%20on%20stupidity.pdf

Mingers J. (2003) 'The paucity of multimethod research: a review of the information systems literature' Information Systems Journal 13, 3 (2003) 233-249

Myers M.D. (1997) 'Qualitative research in information systems' MISQ Discovery, June 1997, at http://www.academia.edu/download/11137785/qualitative%20research%20in%20information%20systems.pdf

Myers M.D. & Klein H.K. (2011) 'A Set of Principles for Conducting Critical Research in Information Systems' MIS Quarterly 35, 1 (March 2011) 17-36, at https://pdfs.semanticscholar.org/2ecd/cb21ad740753576215ec393e499b1af12b25.pdf

Niederman F. & March S. (2012) 'Design Science and the Accumulation of Knowledge in the Information Systems Discipline' ACM Transactions on MIS 3, 1 (2012), 1

Oakley, A. (2003) Research evidence, knowledge management and educational practice: early lessons from a systematic approach, London Review of Education, 1, 1: 21-33, at http://www.ingentaconnect.com/contentone/ioep/clre/2003/00000001/00000001/art00004?crawler=true&mimetype=application/pdf

Okoli C. & Schabram K. (2010) 'A Guide to Conducting a Systematic Literature Review of Information Systems Research' Sprouts: Working Papers on Information Systems, 10, 26 (2010), at http://sprouts.aisnet.org/10-26

Popper K. (1963) 'Conjectures and Refutations: The Growth of Scientific Knowledge' Harper & Row, 1963

Stemler S. (2001) 'An overview of content analysis' Practical Assessment, Research & Evaluation 7, 17 (2001), at http://PAREonline.net/getvn.asp?v=7&n=17

Straub D. W. (2009) 'Editor's comments: Why top journals accept your paper' MIS Quarterly, 33, 3 (September 2009) iii-x, at http://misq.org/misq/downloads/download/editorial/3/

Tarafdar M. & Davison R.M. (2018) `Research in Information Systems: Intra-disciplinary and Inter-disciplinary Approaches' Forthcoming, Journal of the Association for Information Systems

Wall J.D., Stahl B.C. & Salam A.F. (2015) 'Critical Discourse Analysis as a Review Methodology: An Empirical Example' Communications of the Association for Information Systems 37, 11 (2015)

Webster J. & Watson R.T. (2002) 'Analyzing The Past To Prepare For The Future: Writing A Literature Review' MIS Quarterly 26, 2 (June 2002) xiii-xxiii, at http://intranet.business-science-institute.com/pluginfile.php/247/course/summary/Webster%20%20Watson.pdf

Weber R.P. (1990) 'Basic Content Analysis' Sage, 1990


Acknowledgements

This work has benefitted from feedback by colleagues, including during presentations of the emergent argument and analysis in a Keynote at the Australasian Conference in Information Systems (ACIS'15) in Adelaide in 2015, and at the Bled Conference in 2016 and 2017.


Author Affiliations

Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in Cyberspace Law & Policy at the University of N.S.W., and a Visiting Professor in the Research School of Computer Science at the Australian National University.



xamaxsmall.gif missing
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.

From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 50 million in early 2015.

Sponsored by Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916

Created: 23 June 2018 - Last Amended: 28 June 2018 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/SOS/CAPW.html
Mail to Webmaster   -    © Xamax Consultancy Pty Ltd, 1995-2017   -    Privacy Policy