The Future of Software Development Methods: January 2018
The Future of Software Development Methods: January 2018
The Future of Software Development Methods: January 2018
net/publication/319543740
CITATION READS
1 1,792
2 authors:
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Klaas-Jan Stol on 10 September 2018.
9
THE FUTURE OF SOFTWARE
DEVELOPMENT METHODS
Introduction
There is a distinct half-life obsolescence with respect to software development methods, in that the
principles and tenets that become enshrined in methods can become obsolete due to the emergence of
new challenges and issues as the technological environment evolves. This has been labeled the ‘problem
of tenses’ (cf. Fitzgerald 2000; Friedman 1989). The software development methods and processes that
are popular today have been derived from principles that were first identified many decades ago – the
systems development life cycle, object orientation, agile and lean methods, open source software,
software product lines, and software patterns, for example. However, there are a number of
fundamentally disruptive technological trends that suggest that we need an ‘update of tenses’ in relation
to the software development methods that can cater to this brave new world of software development.
Fitzgerald (2012) has coined the term Software Crisis 2.0 to refer to this new disruptive age. The original
software crisis (Software Crisis 1.0, as we might refer to it), identified in the 1960s, referred to the fact
that that software took longer to develop and cost more than estimated, and did not work very well
when eventually delivered. The diligent efforts of many researchers and practitioners has had a positive
outcome in that this initial software crisis has been resolved to the extent that software is one of the
success stories of modern life.
and wearable computing. Also, the increasing role of software is evident in the concept of software
defined * (where * can refer to networking, infrastructure, data center, or enterprise). The Software
Crisis 2.0 bottleneck arises from the inability to produce the volume of software necessary to leverage
the absolutely staggering increase in the volume of data being generated in turn allied to the enormous
amount of computational power offered by the many hardware devices also available, and both
complemented by the demands of the newly emerged digital native consumer in a world where
increasingly software is the key enabler. Figure 9.1 summarizes these ‘push’ and ‘pull’ factors.
Our organization has become a software company. The problem is that our engineers haven’t
realized that yet!
This is how the vice president for research of a major semiconductor manufacturing company,
traditionally seen as the classic hardware company, characterized the context in which software solutions
were replacing hardware in delivering his company’s products. This organization knew precisely the
threshold of reuse level for its hardware components before designing for reuse became cost-effective.
However, this level of sophistication was not yet present in its software development processes. There
was a tendency to approach each software project in a once-off fashion, and a repeatable and formalized
software development process had not been fully enacted.
Hardware Advances
• Moore’s Law, Butter’s
Law, Kryder’s Law
• Parallel computing
• IBM Watson platform
• Quantum computer/
Memcomputer
Digital Natives
• Crowdsourcing/
Big Data
crowd-science
•
•
IoT
Ubiquitous devices
Software • Quantified self,
• Systems of Systems Crisis 2.0 Life-logging,
Wearable computing
• Columnar databases
• Mass customization
• Continuous Deployment/
Innovation
Software-Defined *
• SD Networking
• SD Infrastructure
• SD Data Center
• SD Enterprise
We have seen this situation replicated across several business domains as the transformation to software
has been taking place for quite some time. The telecommunications industry began the move to
softwareization in the 1970s with the introduction of computerized switches, and currently, the mobile
telephony market is heavily software focused. The automotive industry has very noticeably been moving
toward softwareization since the 1960s – today, 80%–90% of innovations in the automotive industry are
enabled by software (Mossinger 2010; Swedsoft 2010). This is evidenced in the dramatic increase in the
numbers of software engineers being employed in proportion to the numbers employed in traditional
engineering roles. A striking example of the growing importance of software in the automotive industry
is conveyed in the following: In 1978, a printout of the lines of code would have made a stack about
twelve centimeters high; by 1995, this was already a stack three meters high; and by 2015, it is a staggering
830 meters, higher than the Burj Khalifa – the tallest man-made structure in the world (see Figure 9.2).
Another example of a domain in which software has increased dramatically in importance is the
medical device sector. Traditionally, medical devices were primarily hardware with perhaps some
embedded software. However, a 2010 EU Medical Device Directive classifies stand-alone software
applications as active medical devices (McHugh et al. 2011). This has major implications for the software
development process in many organizations, as they now find themselves subject to regulatory bodies
such as the US Food and Drug Administration (FDA).
Thus, we have a context where software, traditionally seen as secondary and a means to an end in
many sectors, moves center stage. The implications of this global shift are frequently underestimated. It
requires the software development function to transform itself in order to provide the necessary
foundation to fulfill this central role, which in turn requires an expansion of the set of concerns that need
to be integrated into any software solution. The increasing importance of software is not only a matter
of ‘scale’ in the traditional sense, measured in lines of code, transactions per second, or capacity, but the
scaling of software causes the need for changes in other dimensions, too. This has implications for the
methods that we use to guide software development practice.
The Software
Enterprise Exemplar trends:
DevOps / Continuous Delivery
Enterprise Agile
Processes and Methods Lean software development
Continuous Software Engineering
Agile in regulated domains
It is estimated that 35 billion devices are currently connected in the IoT scenario, a figure that is
expected to rise to 100 billion by 2020 (Feki et al. 2013). The exponential increase in the volume of
available data – Big Data – allied to the ubiquitous devices that are available to process the data also leads
to a demand for additional software systems to process into market relevant knowledge. So, not only do
software systems become bigger, the number of systems (however small these may be) is growing
exponentially.
Another trend is that of servitization. Rather than selling products to customers, organizations sell
services that could not exist without software. Airbnb is the world’s largest supplier of accommodation,
but the company owns no real estate. Likewise, Uber is the world’s largest taxi company, but owns no
taxis of its own. This shift from products to services is continuing and has significant implications for
how companies conduct business.
organizations are faced with mission-critical legacy software systems that are written in older technologies
such as COBOL or even assembly language. This represents major risks for many organizations that wish
to modernize their systems. Again, there is a tension in relation to the use of agile methods for such
modernization projects. Certainly, the use of agile methods in large development projects remains a
significant challenge (Booch 2015), with little empirical evidence of the successful deployment of agile
in large projects (e.g., Cao et al. 2004; Kähkönen 2004). The extant literature tends to assume that scaling
agile methods can be done in a linear fashion; Rolland et al. (2016) have argued that this approach has
limitations and that such assumptions must be re-evaluated.
Industry vignettes
We believe the three dimensions in Figure 9.3 are inter-related and suggest that to scale successfully in
any one dimension requires that related challenges in the other dimensions be resolved simultaneously.
This can be difficult as the software development function in many organizations today is not in a position
to mandate organizational change in other organizational functions, yet the latter is critical for success.
A case in point is the current focus on DevOps and continuous deployment, which requires buy-in from
an operations team to be able to deliver new versions at a fast pace (Stol and Fitzgerald 2014).
We illustrate these challenges with a number of industry vignettes describing real-world scenarios
where organizations are faced with these challenges. These vignettes represent real organizational
B Fitzgerald and K Stol (2018) The Future of Software Development Methods. RD Galliers and MK
Stein (Eds.) The Routledge Companion to Management Information Systems. Routledge. Pp. 125-137
contexts in which we have been involved through funded research projects (Fitzgerald et al. 2013). Each
vignette is primarily driven from one dimension but challenges in terms of the other dimensions are
clearly evident. The first vignette presents the case of QUMAS and relates to the Scaling Processes and
Methods dimension.
QUMAS is an example of a company developing software for a regulated market. The software
development process in QUMAS must adhere to a number of standards. QUMAS previously
employed a waterfall-based approach. However, this approach resulted in a long time-to-market
and a large release overhead, which were seen as drawbacks in the quickly changing market that
QUMAS is operating in. As a consequence, they have adopted and augmented the Scrum
methodology.
Agile methods were initially assumed to be limited to small development projects with co-
located teams working on non-safety critical development projects. In order to introduce a new
agile process, QUMAS have to interact with external organizations in the regulatory bodies that
monitor compliance with the relevant standards. Also, the agile process that QUMAS have adopted
requires the Quality Assurance (QA) function to assess compliance of the software produced at the
end of each three-week sprint. This was a fundamental change in work practice for QA as they
currently assess regulatory compliance more or less annually to coincide with new releases of the
product. Because QA are required to be independent of the software development function in a
regulated environment, senior management need to support such a large organizational change
initiative.
QUMAS see this approach as also altering the product and services they offer. New
functionality in interim software products at the end of sprints can be demonstrated to customers
and their feedback sought more frequently than in the typically annual big-bang release of a new
product as per the traditional waterfall-based process.
QUMAS’s transformation started on the processes and methods dimension, but resulted in significant
changes in the product and organizational dimensions. The need to involve other parts of the
organization to make a process transformation is clearly relevant to the organizational dimension. In
regulated domains, the QA function is required to be an independent department from the software
development function, and thus the agile transformation also required other parts of the organization to
transform. The change to an incremental process facilitated the concept of continuous compliance.
Whereas the waterfall approach validated the product under development only at the very end, in the
new process the compliance auditing took place incrementally as well. The incremental development
and frequent delivery also facilitated presales of the product before the software was finished – something
unheard of in a waterfall approach. Finally, given the new incremental development approach also
affected the implementation of the product, as its design now had to facilitate incremental addition of
features.
The second vignette presents the case of Husqvarna, a leading manufacturer of gardening machinery,
and represents a company transforming primarily in relation to the Products, Systems and Services
dimension, but with major implications for the Organization and Business Domains and the Processes
and Methods dimensions.
B Fitzgerald and K Stol (2018) The Future of Software Development Methods. RD Galliers and MK
Stein (Eds.) The Routledge Companion to Management Information Systems. Routledge. Pp. 125-137
There are several key lessons in the vignette. First, as Husqvarna was moving from gas-driven to
electrically powered machinery, they found themselves in a new business domain that was already
inhabited by other companies producing similar devices, and hence were faced with new competitors.
In order to introduce new products and services, new approaches such as software product lines and a
formalized development process were necessary. Furthermore, the need to deliver more software
required an organizational expansion as a result of the hiring of trained software developers.
The final vignette relates to the Scaling Organizations dimension, and we draw on the experiences of
Sony Mobile.
developers across the organization to collaborate on common projects without formal team
membership. Engagement with OSS projects and adopting Inner Source has implications for the
product and process dimensions as well. Teams interacting with communities of ‘unknown’
developers must comply with community norms and common practices so as to ensure that their
contributions can be integrated.
Similar to the companies in the other vignettes, Sony Mobile has also been facing the challenge to deliver
more software and more quickly. In their domain, they chose to adopt the Android platform for their
mobile phones, and consequently were facing the need to collaborate with other stakeholders outside
their organization – in this case, open source communities. At the same time, the company is also
adopting inner source: managing projects that involve such external and internal communities requires
considering new and unknown forces (Höst et al. 2014). Many other large organizations are actively
involved in open source communities, including Hewlett-Packard and Wipro. These organizations must
learn how to interact with these external communities of developers that may have a different agenda
and motivations. To that end, many of these companies are now employing dedicated Open Source
Community Experts.
In each of the three vignettes described earlier, the primary focus was on scaling in one dimension,
but the companies involved ultimately had to deal with changes in all three dimensions. For example,
QUMAS started their scaling transformation in the process dimension, but consequentially had to make
changes to their organization and product dimensions. Likewise, Husqvarna is scaling primarily in the
product dimension, leading to subsequent changes to their processes and organization. Sony Mobile
focused primarily on expanding their organization by leveraging open source and inner source
communities, which in turn led to changes in the product architecture (e.g., dependency on the open
source Android platform) and process changes.
Scaling Dimension Exemplar Scaling Scenario Implications for Other Dimensions Implications for Practice, Research and
Education
Products, Systems Organizations are changing their Companies need to scale on organizational aspects, e.g., hire trained software The range of software development
and Services product offerings and moving to engineers instead of relying on hardware engineers, and adopt an appropriate contexts will become much more
‘softwareization’ and software development process that suits the product development context. varied, which increases the importance
‘servitization.’ of context in research and education of
software engineers.
Organizations and Organizations are becoming Trends such as open sourcing, outsourcing and crowdsourcing require Organizations must carefully consider
Business Domains increasingly dependent on changes to an organization’s internal development processes. Outsourcing the implications of their sourcing and
external suppliers and workforces, organizations should also consider the product architecture in coordinating partnership strategies. Software may
and offer participation to third- external workforces and integrating externally developed software. Keystone take an organization to new business
party suppliers in ecosystems. players in ecosystems offering platforms to third parties to develop plug-ins domains that are already inhabited by
or apps must consider constraints of the product architecture. others, thus facing new competition.
Processes and Organizations are adopting Organizations need to consider other stakeholders in the organization Software transcends the software
Methods contemporary and emerging affected by a changing process; e.g., to adopt DevOps and continuous development function, which must be
software development practices delivery, teams responsible for operations must be involved. Top-level linked to the whole organization – the
and techniques, including agile management support is required to get different departments involved. If software enterprise. Software
methods, DevOps, continuous applicable, constraints due to regulatory compliance must be considered. engineering education must address
deployment, and continuous Product architectures must be amenable to a continuous delivery approach. the interactions with other functions,
software engineering. including marketing and sales.
B Fitzgerald and K Stol (2018) The Future of Software Development Methods. RD Galliers and MK
Stein (Eds.) The Routledge Companion to Management Information Systems. Routledge. Pp. 125-137
Conclusion
The domain of software development is expanding and reaching far beyond mere technical solutions that
involve algorithms, software architectures, and ultra-large systems. The ability to deliver this increasing
amount of software is an immediate and pressing concern for many organizations – this has been termed
Software Crisis 2.0 (Fitzgerald 2012). Software development must take a holistic view and consider the
various interdependencies between trends such as softwareization and servitization, the organizational
aspects that are affected by an increasing ‘pull’ for software-based solutions, and the processes and methods
that are used to deliver this software.
These trends will have far-reaching consequences. First, the tension that arises due to the push and
pull factors discussed earlier may increase the pressure to deliver software more quickly, which inevitably
will lead to compromises in the quality of the software produced. Software that is constructed quickly,
without proper design, review, and quality assurance is far more likely to exhibit defects down the road,
thus increasing the cost of maintenance, which in turn exacerbates Software Crisis 2.0.
Second, the pull factors also imply an increasing need for talented software developers. As software
systems are becoming increasingly complex, the reliance on self-educated and software hobbyists is no
longer sufficient; instead, companies need well-trained staff. The lack of well-trained staff is an oft-cited
reason for companies to outsource their software development.
Another challenge that is perhaps not receiving enough attention is our society’s ever-increasing
reliance on software systems. Software systems are truly ubiquitous, but many of these systems are very
dated legacy systems. Many critical systems (e.g., operating at banks, insurance companies, and
government institutions) were developed in the sixties and seventies on technology platforms that are
now considered dated, such as COBOL. While developers who are able to maintain these systems are
currently still available, this may no longer be the case 20 or 30 years from now. An incredible amount
of software is being written today on a variety of technologies that rapidly become obsolete and will
inevitable have to be replaced at some point. As systems are becoming ever more intertwined and
interdependent, this will become increasingly problematic.
With the realization that software development transcends algorithms and techniques, we believe
research should include considerations across all three key dimensions: products, services, and systems;
processes and methods; and organizations and business domains. An increased understanding of the
different types of scaling challenges that organizations face will help the software industry to overcome
them, and will help in deriving new software development methods better suited to the needs of the
prevailing software development environment.
Acknowledgements
This work was supported, in part, by Enterprise Ireland, grant IR/2013/0021 to ITEA-SCALARE,
Science Foundation Ireland grant 15/SIRG/3293 and 13/RC/2094 and co-funded under the European
Regional Development Fund through the Southern & Eastern Regional Operational Programme to Lero
– the Irish Software Research Centre (www.lero.ie).
References
Ågerfalk, P. J., B. Fitzgerald and K. Stol (2015a) Software Sourcing in the Age of Open: Leveraging the Unknown
Workforce, Springer.
Ågerfalk, P. J., B. Fitzgerald and K. Stol (2015b) Not so shore anymore: The new imperatives when sourcing in the
age of open, Proceedings of the European Conference on Information Systems (ECIS) Münster, Germany.
Boehm, B. (2002) “Get ready for agile methods, with care,” IEEE Computer, vol. 35, pp. 64–69;
doi:10.1109/2.976920.
B Fitzgerald and K Stol (2018) The Future of Software Development Methods. RD Galliers and MK
Stein (Eds.) The Routledge Companion to Management Information Systems. Routledge. Pp. 125-137
Booch, G. (2015) Keynote at the International Conference on Software Engineering, SEIP Track, Florence, Italy.
Bosch, J. and P. Bosch-Sijtsema, “From integration to composition: On the impact of software product lines, global
development and ecosystems,” Journal of Systems and Software, vol. 83, no. 1, 2010, pp. 67–76;
doi:10.1016/j.jss.2009.06.051.
Cao, K., P. Mohan, P. Xu and B. Ramesh, (2004) “How extreme does extreme programming have to be? Adapting
XP practices to large-scale projects,” in 37th Hawaii International Conference on System Science.
Conboy, K., S. Coyle, X. Wang and M. Pikkarainen, “People over process: Key challenges in agile development,”
IEEE Software, vol. 28, no. 4, 2011, pp. 48–57.
Dybå, T., D.I.K. Sjøberg and D. S. Cruzes, “What works for whom, where, when, and why? On the role of Context
in empirical software engineering,” Proceedings of Empirical Software Engineering and Measurement , 2012,
pp. 19–28.
Feki, M. A., F. Kawsar, M. Boussard and L. Trappeniers, “The internet of things: The next technological revolution,”
IEEE Computer, vol. 46, no. 2, 2013, pp. 24–25.
Feller, J., B. Fitzgerald, S. Hissam, and K. Lakhani. (2005) Perspectives on Free and Open Source Software. MIT
Press. Cambridge, MA.
Fitzgerald, B. “Systems development methodologies: the problem of tenses,” Information Technology & People, vol.
13, no. 3, 2000, pp. 174–185.
Fitzgerald, B., “Software Crisis 2.0,” IEEE Computer, vol. 45, no. 4, 2012, pp. 89–91.
Fitzgerald, B., M. Musial and K. Stol, “Evidence-based decision making in lean software project management,” in
Proceedings of International Conference on Software Engineering , vol. 2, Hyderabad, India, 2014.
Fitzgerald, B. and K. Stol, “Continuous software engineering and beyond: A roadmap and agenda,” Journal of Systems
and Software, vol. 123, 2017, pp. 176-189.
Fitzgerald, B., K. Stol, R. O’Sullivan and D. O’Brien, “Scaling agile methods to regulated environments: An industry
case study,” Proceedings of International Conference on Software Engineering , vol. 2, 2013, pp. 863–872.
Friedman, A. (1989) Computer Systems Development: History, Organisation and Implementation , Wiley & Sons,
Chichester.
Höst, M., K. Stol and A. Orucevic-Alagic (2014) “Inner source project management,” in: G. Ruhe and C. Wohlin
(Eds.) Software Project Management in a Changing World , Springer, pp. 343-367.
Kahkonen, T. (2004) “Agile methods for large organisations – building communities of practice,” in Agile
Development Conference.
McHugh, M., F. McCaffery and V. Casey, “Standalone software as an active medical device,” Proceedings of 11th
International SPICE Conference, 2011, pp. 97–107.
Mössinger, J., “Software in automotive systems,” IEEE Software, vol. 27, no. 2, 2010, pp. 92–94.
Müller, H., K. Wong, and S. Tilley, “Understanding software systems using reverse engineering technology,”
Proceedings of the 62nd Congress of L’Association Canadienne Française pour l’Avancement des Sciences
(ACFAS), vol. 26, no. 4, 1994, pp. 41–48.
Poppendieck, M. and M. Cusumano, “Lean software development: A tutorial,” IEEE Software, vol. 29, no. 5, 2012,
pp. 26–32.
Rolland, K., B. Fitzgerald, T. Dingsøyr and K. Stol, “Problematizing agile in the large: Alternative assumptions for
large-scale agile development,” Proceedings 37th International Conference on Information Systems , Dublin,
Ireland, 2016.
Schneider, J. “Software-innovations as key driver for a green, connected and autonomous mobility,” ARTEMIS-
IA/ITEA-Co-Summit, 2015.Stol, K. and B. Fitzgerald, “Inner source – adopting open source development
practices in organizations: A tutorial,” IEEE Software, vol. 32, no. 4, 2015.
Stol, K. and B. Fitzgerald (2014) “Two’s company, three’s a crowd: A case study of crowdsourcing software
development,” Proceedings of the 36th International Conference on Software Engineering (Technical Track),
Hyderabad, India
Swedsoft, A Strategic Research Agenda for the Swedish Software Intensive Industry , 2010.
Tamburri, D., P. Lago and H. van Vliet, “Uncovering latent social communities in software development,” IEEE
Software, vol. 30, no. 1, 2013, pp. 29–36.
B Fitzgerald and K Stol (2018) The Future of Software Development Methods. RD Galliers and MK
Stein (Eds.) The Routledge Companion to Management Information Systems. Routledge. Pp. 125-137
Turk, D., R. France and B. Rumpe, “Assumptions underlying agile software-development processes,” Journal of
Database Management, vol. 16, no. 4, 2005, pp. 62–87.
van der Linden, F., K. Schmid and E. Rommes (2007) Software Product Lines in Action, Springer, New York, Inc.
Secaucus, NJ, USA.