A Literature Review of Enterprise Architecture Success Measures, since 2015
This post is a copy of my recent assignment in Uni to do a literature review for Enterprise Architecture.
Abstract
Enterprise Architecture is widely adopted across many, particular large, organisations. This paper is a literature review of academic studies since 2015 to understand “the evidence that good Enterprise Architecture practice is strongly linked to positive business outcomes”. The question is asked in the context of today’s businesses being under continual pressure to drive cost efficiencies, improve innovation, reduce cycle times, increase shareholder value or meet or enhance Corporate Social Responsibility.
While there are many frameworks, models and constructs, empirical proof of Enterprise Architecture success from tangible measurements is not well described in the literature (noting that each paper selected here includes a review of literature over the previous 2 decades). Enterprise Architecture, as a discipline and investment area for organisations, is weakened as a result. This review highlights the need for more research in empirical evidence gathering for Enterprise Architecture Success using the scientific method, including experiments and control groups.
Keywords Enterprise Architecture, Measurement, Success, Metrics, Positive Business Outcomes, Framework, Balanced Scorecard, OKRs
Introduction
Langhorst (2017) Describes Enterprise Architecture (EA) as ‘a coherent whole of principles, methods, and models that are used in the design and realisation of an enterprise’s organisational structure, business processes, information systems, and infrastructure.’ It follows that, in order to understand the value that Enterprise Architecture can bring to an organisation, the organisation needs to measure its effectiveness. Without compelling evidence through metrics, the benefits that EA brings will be attributed to other reasons with stronger evidence, leaving EA vulnerable to funding cuts in the organisation.
Corporate and public funding is under constant pressure with competing investment required for:
- Cost efficiencies,
- Innovation
- Reducing cycle times,
- Increasing Shareholder Value or
- Corporate Social Responsibility
This paper asks the question:
> “What is the evidence that good Enterprise Architecture (EA) practice is strongly linked to positive business outcomes (such as those in the above bulleted list)?\” This literature review is organised with a description of the methodology, followed by the analysis of the reviewed papers, and finally a conclusion section. The analysis section includes descriptions of concepts and concept categories identified in the papers, before detailing the analysis of the concepts in the papers. The conclusion offers recommendations for further research.
Methodology
The term ‘\”enterprise architecture success\” measures’ was searched for using Google Scholar with a date filter for results since 2015. This returned 13 results. Seven of those results did not have Enterprise Architecture measures as the focus of the paper and were discarded. The date filter used is based on the assumption that work since this time will include a literature review up until that point.
Table 1 Google Scholar Search Terms
The remaining six results each contain a literature review including material from the previous 25 years. A seventh paper was added manually as it was referenced multiple times in the other 6 papers — even though it was written well before the start of the date filter used. Table 2 below shows the papers included in this review, along with an ‘Index’ used to refer to each paper in the remainder of this document, plus the number of citations for each paper (a proxy indication of their quality).
Table 2 Summary of Selected and Reviewed Research Articles
Concepts and concept categories were identified for each reviewed paper. Each concept is grouped into categories of “Analysis Domain”, “Approach” (to Enterprise Architecture) and “Research Method”. Concepts within each concept category are respectively described in Tables 3, 4 and 5 below.
Concept Category — Analysis Domain
The Analysis Domain category contains the high-level concepts the papers reviewed used in their analysis. For this review, this category contains the following concepts in Table 3 below:
Table 3 Analysis Domain Concepts
Concept Category — Approach
The Approach category contains concepts that describe how the papers approached the creation of their measurement model or framework for Enterprise Architecture. For this review, this category contains the following concepts in Table 4 below:
Table 4 Approach Concepts
Concept Category — Research Method
The Research Method category contains concepts describing the research methodology the papers used in their work. For this review, this category contains the following concepts in Table 5 below:
Table 5 Research Method Concepts
Table 6, below, shows the mapping matrix that was created using the ‘index’ from Table 2 and the concepts described in Tables 3, 4 and 5 above. Table 6 therefore highlights the core concepts and research approaches discussed in each reviewed paper. This matrix is used as the focal point for discussion and analysis in this review.
Analysis
All selected papers (n=7) discuss or propose some kind of measurement model for Enterprise Architecture. A key theme from each is that alignment between business and IT is either a benefit (n=6) or a requirement for success (n=3). Each paper proposes a model or framework (n=7) to either help assure success (n=4), describe benefits (n=6) or both (n=3). Where the model discusses success factors (n=4) the papers describe pragmatic measures to implement to track business and EA success.
Table 6 Concept to Research Paper Mapping Matrix
The Balanced Scorecard (Kaplan & Norton 1992) is a well-known method for aligning corporate goals and metrics across lines of business to help assure collaborative working. Over half of the papers selected (n=4) either referenced or leveraged Balanced Scorecard (BSC) as part of its measurement model as BSC supports success factors identified in the papers such as executive support, governance, communication and business/IT alignment. BSC appears to be a promising candidate for measurement models given the level of existing research and ability to align business and IT.
It is noteworthy that more recently popular business/IT alignment frameworks like OKRs (Objectives and Key Results) popularised at organisations such as Google and Facebook (Schrage 2018) are not referenced in any of the reviewed papers.
Three papers used Balanced Scorecard as the basis of a reference model, while the remainder (n=4) developed customised frameworks or models. The proposed measurement model from each paper included a range of concepts in their analysis domain (see Table 2 above).
Three named research methods were identified, Design Science Research (DSR) (n=3), Quantitative Research (n=1) and Systematic Mapping Study (n=1). The two remaining papers were not allocated a specific research method. Most papers (n=6) showed strong links to industry and practice in their methods and analysis.
However, none of the papers were able to define measures that directly link Enterprise Architecture practice (or the absence of EA practice) to specific business outcomes. That is, of all the benefits attributed to EA, those benefits could only be weekly connected to the specific implementation of EA practice, for example through expert opinion. There were no control groups defined in any of the studies to compare the presence of good EA practice with the absence of EA practice.
One paper (P3) offered a damning assessment of the Enterprise Architecture Measurement field in its abstract:
“In addition, the EA measurement literature presents indications that its theoretical grounding is lacking with regards to the meaning of EA measurement, the evaluation of EA, the assessment of EA, and the analysis of EA.” (Abdallah, Lapalme & Abran 2016)
Even so, most of the papers (n=6) have practical application in offering measurement models that can be used to compare the periods before and after Enterprise Architecture practice has been adopted, and therefore have a mechanism to prove the effectiveness of EA practice localised to that organisation. The exception to this is P6, which only had weak endorsement from experts.
Conclusion
The literature contains clear evidence that academia and industry have strong links which has been used to create frameworks and models that can be applied in industry today. However, despite these links, the literature is still lacking in scientific rigour around empirical evidence to support investment in Enterprise Architecture. Industry practice clearly favours Enterprise Architecture, but without this empirical evidence, supported by the control group experimentation, EA remains vulnerable to a ‘by the numbers’ cost cutting exercise.
Balanced Scorecard is a popular method for formulating Enterprise Architecture measures, models or frameworks and is well suited to the task given its focus on cross organisational alignment. Other models for aligning the business across the organisation to key goals or objectives, such as OKRs, were conspicuous by their absence. There is room for the development of new models to be investigated in the research. Such new models that complement emerging business management models will offer more choice to Enterprise Architects and CIOs when developing strategy for EA and business alignment. The benefit of an increased range of models will be to allow business leaders to align the Enterprise Architecture measurement model more closely to the business operating model, increasing synergy between business and IT.
Recommendations for Further Research This review highlights the need for more research in empirical evidence gathering for Enterprise Architecture Success using the scientific method, including experiments and control groups. There will also be some benefit in developing competing models that complement emerging leading business management models, in order to provide a range of options to executives and practitioners when aligning Enterprise Architecture to business strategy.
References Abdallah, A, Lapalme, J & Abran, A 2016, Enterprise Architecture Measurement: A Systematic Mapping Study — IEEE Conference Publication, Unimelb.edu.au, viewed 7 June 2019,
https://ieeexplore-ieee-org.ezp.lib.unimelb.edu.au/abstract/document/7880427.
Baddeley, M 2017, ‘Behavioural economics\u202f: a very short introduction’, Oxford University Press, Oxford, viewed 19 May 2019,
Bradley, RV, Pratt, RME, Byrd, TA, Outlay, CN & Wynn Jr., DE 2011, ‘Enterprise architecture, IT effectiveness and the mediating role of IT alignment in US hospitals’, Information Systems Journal, vol. 22, no. 2, pp. 97–127, viewed 5 June 2019,
https://onlinelibrary-wiley-com.ezp.lib.unimelb.edu.au/doi/full/10.1111/j.1365-2575.2011.00379.x .
Cane, S 2007, ‘MEASURING THE IMPACT OF ENTERPRISE ARCHITECTURE’, Issues in Information Systems, viewed 3 June 2019,
https://pdfs.semanticscholar.org/17d6/57ac95dd5ac916ac9d8b280c6ae795566471.pdf.
Hjort-Madsen, K 2006, ‘Enterprise Architecture Implementation and Management: A Case Study on Interoperability — IEEE Conference Publication’, Unimelb.edu.au, viewed 3 June 2019,
https://ieeexplore-ieee-org.ezp.lib.unimelb.edu.au/abstract/document/1579432.
Ilin, I, Levina, A, Abran, A & Iliashenko, O 2017, ‘Measurement of enterprise architecture (EA) from an IT perspective’, Proceedings of the 27th International Workshop on Software Measurement and 12th International Conference on Software Process and Product Measurement on
- IWSM Mensura ’17, viewed 7 June 2019,
https://dl-acm-org.ezp.lib.unimelb.edu.au/citation.cfm?id=3143457.
Jusuf, Muhammad Baharudin & Kurnia, S 2017, ‘Understanding the Benefits and Success Factors of Enterprise Architecture’, Hawaii.edu, viewed 7 June 2019,
https://scholarspace.manoa.hawaii.edu/handle/10125/41755.
Kaplan, RS & Norton, DP 1992, The Balanced Scorecard: Measures That Drive Performance, Harvard Business Review, viewed 9 June 2019,
https://hbr.org/2005/07/the-balanced-scorecard-measures-that-drive-performance.
Kluge, C, Ch, A & Rosemann, M 2006, Association for Information Systems AIS Electronic Library (AISeL) How to realise corporate value from enterprise architecture A. Dietzsch, viewed 6 May 2019,
https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1077&context=ecis2006 [https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1077&context=ecis2006].
Labusch, N & Aier, S 2015, Information Provision as a Success Factor in the Architectural Support of Enterprise Transformations — IEEE Conference Publication, Unimelb.edu.au, viewed 3 June 2019,
https://ieeexplore-ieee-org.ezp.lib.unimelb.edu.au/abstract/document/6904315.
Lange, M, Mendling, J & Recker, J 2012, REALIZING BENEFITS FROM ENTERPRISE ARCHITECTURE: A MEASUREMENT MODEL, AIS Electronic Library (AISeL), viewed 6 May 2019, https://aisel.aisnet.org/ecis2012/10/.
Lankhorst, M 2009, Enterprise Architecture at Work — 1 Introduction to Enterprise Architecture, Unimelb.edu.au, viewed 27 April 2019,
https://link-springer-com.ezp.lib.unimelb.edu.au/content/pdf/10.1007%2F978-3-642-01310-2_1.pdf .
Lankhorst, M 2009, ‘Enterprise Architecture at Work: Modelling, Communication and Analysis’.
Mocker, M, Ross, JW & Hopkins, C 2015, How USAA Architected its Business for Life Event Integration: DISCOVERY, Unimelb.edu.au, viewed 3 May 2019,
https://eds-a-ebscohost-com.ezp.lib.unimelb.edu.au/eds/detail/detail?vid=0&sid=924389a9-e1b4-4970-bc4b-caa49bf54cf1%40sdc-v-sessmgr04&bdata=JnNpdGU9ZWRzLWxpdmUmc2NvcGU9c2l0ZQ%3d%3d#AN=000368249800001&db=edswss [https://eds-a-ebscohost-com.ezp.lib.unimelb.edu.au/eds/detail/detail?vid=0&sid=924389a9-e1b4-4970-bc4b-caa49bf54cf1%40sdc-v-sessmgr04&bdata=JnNpdGU9ZWRzLWxpdmUmc2NvcGU9c2l0ZQ%3d%3d#AN=000368249800001&db=edswss] .
Murer, S, Bonati, B & Furrer, FJ 2011, Managed Evolution | SpringerLink, Unimelb.edu.au, viewed 27 April 2019,
https://link-springer-com.ezp.lib.unimelb.edu.au/book/10.1007%2F978-3-642-01633-2 .
Nkundla-Mgudlwa, S & C. Mentz, J 2017, ‘A Synthesis of Enterprise Architecture Effectiveness Constructs’, Proceedings of the 19th International Conference on Enterprise Information Systems, viewed 7 June 2019,
https://pdfs.semanticscholar.org/f8c1/2fc6bee05e532932b7654bd2cbb1ab186e48.pdf.
Rakgoale, MA & Mentz, JC 2015, ‘Proposing a Measurement Model to Determine Enterprise Architecture Success as a Feasible Mechanism to Align Business and IT — IEEE Conference Publication,’ Unimelb.edu.au, viewed 7 June 2019,
https://ieeexplore-ieee-org.ezp.lib.unimelb.edu.au/abstract/document/7406866.
Schelp, J & Stutz, M 2007, ‘A Balanced Scorecard Approach to Measure the Value of Enterprise Architecture
- Alexandria’, Unisg.ch, viewed 6 May 2019,
https://www.alexandria.unisg.ch/213190/.
Schmidt, C & Buxmann, P 2008, ‘Outcomes and success factors of enterprise IT architecture management: empirical insight from the international financial services industry’, European Journal of Information Systems, viewed 5 June 2019,
https://orsociety-tandfonline-com.ezp.lib.unimelb.edu.au/doi/full/10.1057/ejis.2010.68#.XPUCzdMzZ0s .
Schmidt, R, Möhring, M, Härting, R-C, Reichstein, C, Zimmermann, A & Luceri, S 2015, ‘Benefits of Enterprise Architecture Management — Insights from European Experts’, Lecture Notes in Business Information Processing, pp. 223–236, viewed 7 June 2019,
https://link-springer-com.ezp.lib.unimelb.edu.au/chapter/10.1007/978-3-319-25897-3_15 .
Schrage, M 2018, ‘Leading With Next-Generation Key Performance Indicators’, MIT Sloan Management Review, viewed 9 June 2019,
Sessions, R 2007, ‘A Comparison of the Top Four Enterprise-Architecture Methodologies’, Gsu.edu, viewed 25 May 2019,
Winter, R & Fischer, R 2006, ‘Essential Layers, Artifacts, and Dependencies of Enterprise Architecture — IEEE Conference Publication’, Unimelb.edu.au, viewed 20 May 2019,
https://ieeexplore-ieee-org.ezp.lib.unimelb.edu.au/abstract/document/4031290.",
Originally published at https://datajps.com on June 9, 2019.
