ACM Sigsoft Software Engineering Notes, Mar 28, 2018
Software product and service companies need capabilities to evaluate their development decisions ... more Software product and service companies need capabilities to evaluate their development decisions and customer and user value. Continuous experimentation, as an experiment-driven development approach, may reduce such development risks by iteratively testing product and service assumptions that are critical to the success of the software. Experimentdriven development is gaining increasing attention by the practitioners, yet it is a novel research area to be investigated. Involving users and customers in the software development process and understanding their needs and behaviors are essential aspects when building successful software products and services. Continuous experimentation approach in software development comprises user involvement by its nature, however, due to novelty of the concept, there is no clear understanding of the user involvement practices. My doctoral project aims to investigate how software development organizations can involve the users in continuous experimentation. The contributions of the research are to a) find out how involve users in software companies that begin to adopt continuous experimentation approach, b) identify the barriers hindering involving the users in the approach, and c) investigate what kinds of strategies have been used for experimentation with different software product and services.
Software product and service companies need capabilities to evaluate their development decisions ... more Software product and service companies need capabilities to evaluate their development decisions and customer and user value. Continuous experimentation, as an experiment-driven development approach, may reduce such development risks by iteratively testing product and service assumptions that are critical to the success of the software. Experimentdriven development is gaining increasing attention by the practitioners, yet it is a novel research area to be investigated. Involving users and customers in the software development process and understanding their needs and behaviors are essential aspects when building successful software products and services. Continuous experimentation approach in software development comprises user involvement by its nature, however, due to novelty of the concept, there is no clear understanding of the user involvement practices. My doctoral project aims to investigate how software development organizations can involve the users in continuous experiment...
Product-Focused Software Process Improvement, 2020
Processes and practices used in data science projects have been reshaping especially over the las... more Processes and practices used in data science projects have been reshaping especially over the last decade. These are different from their software engineering counterparts. However, to a large extent, data science relies on software, and, once taken to use, the results of a data science project are often embedded in software context. Hence, seeking synergy between software engineering and data science might open promising avenues. However, while there are various studies on data science workflows and data science project teams, there have been no attempts to combine these two very interlinked aspects. Furthermore, existing studies usually focus on practices within one company. Our study will fill these gaps with a multi-company case study, concentrating both on the roles found in data science project teams as well as the process. In this paper, we have studied a number of practicing data scientists to understand a typical process flow for a data science project. In addition, we studied the involved roles and the teamwork that would take place in the data context. Our analysis revealed three main elements of data science projects: Experimentation, Development Approach, and Multidisciplinary team(work). These key concepts are further broken down to 13 different sub-themes in total. The found themes pinpoint critical elements and challenges found in data science projects, which are still often done in an ad-hoc fashion. Finally, we compare the results with modern software development to analyse how good a match there is.
Software development in highly dynamic environments imposes high risks to development organizatio... more Software development in highly dynamic environments imposes high risks to development organizations. One such risk is that the developed software may be of only little or no value to customers, wasting the invested development efforts. Continuous experimentation, as an experiment-driven development approach, may reduce such development risks by iteratively testing product and service assumptions that are critical to the success of the software. Although several experiment-driven development approaches are available, there is little guidance available on how to introduce continuous experimentation into an organization. This article presents a multiple-case study that aims at better understanding the process of introducing continuous experimentation into an organization with an already established development process. The results from the study show that companies are open to adopting such an approach and learning throughout the introduction process. Several benefits were obtained, such as reduced development efforts, deeper customer insights, and better support for development decisions. Challenges included complex stakeholder structures, difficulties in 2 defining success criteria, and building experimentation skills. Our findings indicate that organizational factors may limit the benefits of experimentation. Moreover, introducing continuous experimentation requires fundamental changes in how companies operate, and a systematic introduction process can increase the chances of a successful start.
Highlights 1 • 16 patterns and 16 anti-patterns for experiential project based start-up 2 educati... more Highlights 1 • 16 patterns and 16 anti-patterns for experiential project based start-up 2 education. 3 • Physical and virtual environments as well as pedagogics are covered. 4 • Student, teacher, and customer perspectives are considered.
2021 IEEE International Conference on Big Data (Big Data), 2021
Data science projects have become commonplace over the last decade. During this time, the practic... more Data science projects have become commonplace over the last decade. During this time, the practices of running such projects, together with the tools used to run them, have evolved considerably. Furthermore, there are various studies on data science workflows and data science project teams. However, studies looking into both workflows and teams are still scarce and comprehensive works to build a holistic view do not exist. This study bases on a prior case study on roles and processes in data science. The goal here is to create a deeper understanding of data science projects and development processes. We conducted a survey targeted at experts working in the field of data science (n=50) to understand data science projects’ team structure, roles in the teams, utilized project management practices and the challenges in data science work. Results show little difference between big data projects and other data science. The found differences, however, give pointers for future research on how agile data science projects are, and how important is the role of supporting project management personnel. The current study is work in progress and attempts to spark discussion and new research directions.
2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2017
Today's students are prospective entrepreneurs and employees in modern, start-up like environ... more Today's students are prospective entrepreneurs and employees in modern, start-up like environments within established companies. In these settings, software development projects face extreme requirements in terms of innovation and attractiveness of the end-product. They also suffer severe consequences of failure such as termination of the development effort and bankruptcy. As the abilities needed in start-ups are not among those traditionally taught in universities, new knowledge and skills are required to prepare students for the volatile environment that new market entrants face. This paper reports experiences gained during seven years of teaching start-up knowledge and skills in a higher-education institution. We offer a collection of patterns that help educational institutions to design, implement and operate physical environments, curricula and teaching materials, and to plan interventions that may be required for project-based start-up education.
Software experiments are presently often used by big technology pioneers, such as Microsoft, Face... more Software experiments are presently often used by big technology pioneers, such as Microsoft, Facebook and Google, in order to learn about their users and to guide their research and development activities. Continuous experimentation (CE) is reported to be an integral part of software development in these organisations, however, how they transitioned to the approach is not publicly shared. Therefore, there is a lack of guidance for other organisations that are willing to adopt CE. In the current competitive markets, investing time and money in a new approach might be risky for these organisations, especially if they do not know how to initiate this transition process. This dissertation focuses on how organisations can initiate the transition towards CE, i.e., an approach to enhance development decisions by running experiments in an iterative and sustainable fashion. The dissertation was designed to acquire descriptive and observational knowledge through empirical studies and was cond...
2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2018
Software companies need capabilities to evaluate the user value and the success of their products... more Software companies need capabilities to evaluate the user value and the success of their products. This is especially crucial for highly competitive markets, such as the mobile game industry, where thousands of new games are introduced every month. Game companies often run continuous experiments as an integrated part of the overall development process. This paper presents a game company's journey on experimentation, and describes how the experiments are used at different stages of the development cycle to produce reliable, meaningful data for developers as well as how to balance between different data collection methods. Our study indicates that experiments are important in all stages of the development in different forms. Early stages in the development experiments can be run with proxy users due to lack of real users, whereas later in the development Key Performance Indicator (KPI) metrics play the most important role in experiments. Establishing concrete goals for the experim...
2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Nov 1, 2017
Background: Experiment-driven development with the help of real usage data helps to build softwar... more Background: Experiment-driven development with the help of real usage data helps to build software products and services that are of high value to their users. As more software companies use experimentation in their development practises, ethical concerns are increasingly important. Objective: There is a need for understanding the ethical issues companies must take into account when practising experimentation as a development strategy. This paper examines how software development practitioners experience the need for notifying users when involving them in experimentation. Method: We conducted a survey within four software companies, inviting employees in different functional roles to indicate their attitudes and perceptions through a number of statements. Results: Employees working in different roles have different viewpoints on ethical issues. While managers are more conscious about companycustomer relationships, UX designers appear more familiar with involving users. Developers think that details of experiments can be withheld from users if the results depend on it. Conclusion: Barriers to successfully conducting experiment-driven development are different for different roles. Clear and specific guidelines are needed for ethical aspects of experimentation. Index Terms-User involvement, customer involvement, continuous experimentation, experiment-driven development, ethics, human factors, software development
Processes and practices used in data science projects have been reshaping especially over the las... more Processes and practices used in data science projects have been reshaping especially over the last decade. These are different from their software engineering counterparts. However, to a large extent, data science relies on software, and, once taken to use, the results of a data science project are often embedded in software context. Hence, seeking synergy between software engineering and data science might open promising avenues. However, while there are various studies on data science workflows and data science project teams, there have been no attempts to combine these two very interlinked aspects. Furthermore, existing studies usually focus on practices within one company. Our study will fill these gaps with a multi-company case study, concentrating both on the roles found in data science project teams as well as the process. In this paper, we have studied a number of practicing data scientists to understand a typical process flow for a data science project. In addition, we studied the involved roles and the teamwork that would take place in the data context. Our analysis revealed three main elements of data science projects: Experimentation, Development Approach, and Multidisciplinary team(work). These key concepts are further broken down to 13 different sub-themes in total. The found themes pinpoint critical elements and challenges found in data science projects, which are still often done in an ad-hoc fashion. Finally, we compare the results with modern software development to analyse how good a match there is.
Context: Companies need capabilities to evaluate the customer value of software-intensive product... more Context: Companies need capabilities to evaluate the customer value of software-intensive products and services. One way of systematically acquiring data on customer value is running continuous experiments as part of the overall development process. Objective: This paper investigates the first steps of transitioning towards continuous experimentation in a large company, including the challenges faced. Method: We conduct a single-case study using participant observation, interviews, and qualitative analysis of the collected data. Results: Results show that continuous experimentation was well received by the practitioners and practising experimentation helped them to enhance understanding of their product value and user needs. Although the complexities of a large multi-stakeholder business-to-business (B2B) environment presented several challenges such as inaccessible users, it was possible to address impediments and integrate an experiment in an ongoing development project. Conclusion: Developing the capability for continuous experimentation in large organisations is a learning process which can be supported by a systematic introduction approach with the guidance of experts. We gained experience by introducing the approach on a small scale in a large organisation, and one of the major steps for future work is to understand how this can be scaled up to the whole development organisation.
[Context and motivation] In order to build successful software products and services, customer in... more [Context and motivation] In order to build successful software products and services, customer involvement and an understanding of customers' requirements and behaviours during the development process are essential. [Question/Problem] Although continuous deployment is gaining attention in the software industry as an approach for continuously learning from customers, there is no common overview of the topic yet. [Principal ideas/results] To provide a common overview, we conduct a secondary study that explores the state of reported evidence on customer input during continuous deployment in software engineering, including the potential benefits, challenges, methods and tools of the field. [Contribution] We report on a systematic literature review covering 25 primary studies. Our analysis of these studies reveals that although customer involvement in continuous deployment is highly relevant in the software industry today, it has been relatively unexplored in academic research. The field is seen as beneficial, but there are a number of challenges related to it, such as misper-ceptions among customers. In addition to providing a comprehensive overview of the research field, we clarify the gaps in knowledge that need to be studied further.
ACM Sigsoft Software Engineering Notes, Mar 28, 2018
Software product and service companies need capabilities to evaluate their development decisions ... more Software product and service companies need capabilities to evaluate their development decisions and customer and user value. Continuous experimentation, as an experiment-driven development approach, may reduce such development risks by iteratively testing product and service assumptions that are critical to the success of the software. Experimentdriven development is gaining increasing attention by the practitioners, yet it is a novel research area to be investigated. Involving users and customers in the software development process and understanding their needs and behaviors are essential aspects when building successful software products and services. Continuous experimentation approach in software development comprises user involvement by its nature, however, due to novelty of the concept, there is no clear understanding of the user involvement practices. My doctoral project aims to investigate how software development organizations can involve the users in continuous experimentation. The contributions of the research are to a) find out how involve users in software companies that begin to adopt continuous experimentation approach, b) identify the barriers hindering involving the users in the approach, and c) investigate what kinds of strategies have been used for experimentation with different software product and services.
Software product and service companies need capabilities to evaluate their development decisions ... more Software product and service companies need capabilities to evaluate their development decisions and customer and user value. Continuous experimentation, as an experiment-driven development approach, may reduce such development risks by iteratively testing product and service assumptions that are critical to the success of the software. Experimentdriven development is gaining increasing attention by the practitioners, yet it is a novel research area to be investigated. Involving users and customers in the software development process and understanding their needs and behaviors are essential aspects when building successful software products and services. Continuous experimentation approach in software development comprises user involvement by its nature, however, due to novelty of the concept, there is no clear understanding of the user involvement practices. My doctoral project aims to investigate how software development organizations can involve the users in continuous experiment...
Product-Focused Software Process Improvement, 2020
Processes and practices used in data science projects have been reshaping especially over the las... more Processes and practices used in data science projects have been reshaping especially over the last decade. These are different from their software engineering counterparts. However, to a large extent, data science relies on software, and, once taken to use, the results of a data science project are often embedded in software context. Hence, seeking synergy between software engineering and data science might open promising avenues. However, while there are various studies on data science workflows and data science project teams, there have been no attempts to combine these two very interlinked aspects. Furthermore, existing studies usually focus on practices within one company. Our study will fill these gaps with a multi-company case study, concentrating both on the roles found in data science project teams as well as the process. In this paper, we have studied a number of practicing data scientists to understand a typical process flow for a data science project. In addition, we studied the involved roles and the teamwork that would take place in the data context. Our analysis revealed three main elements of data science projects: Experimentation, Development Approach, and Multidisciplinary team(work). These key concepts are further broken down to 13 different sub-themes in total. The found themes pinpoint critical elements and challenges found in data science projects, which are still often done in an ad-hoc fashion. Finally, we compare the results with modern software development to analyse how good a match there is.
Software development in highly dynamic environments imposes high risks to development organizatio... more Software development in highly dynamic environments imposes high risks to development organizations. One such risk is that the developed software may be of only little or no value to customers, wasting the invested development efforts. Continuous experimentation, as an experiment-driven development approach, may reduce such development risks by iteratively testing product and service assumptions that are critical to the success of the software. Although several experiment-driven development approaches are available, there is little guidance available on how to introduce continuous experimentation into an organization. This article presents a multiple-case study that aims at better understanding the process of introducing continuous experimentation into an organization with an already established development process. The results from the study show that companies are open to adopting such an approach and learning throughout the introduction process. Several benefits were obtained, such as reduced development efforts, deeper customer insights, and better support for development decisions. Challenges included complex stakeholder structures, difficulties in 2 defining success criteria, and building experimentation skills. Our findings indicate that organizational factors may limit the benefits of experimentation. Moreover, introducing continuous experimentation requires fundamental changes in how companies operate, and a systematic introduction process can increase the chances of a successful start.
Highlights 1 • 16 patterns and 16 anti-patterns for experiential project based start-up 2 educati... more Highlights 1 • 16 patterns and 16 anti-patterns for experiential project based start-up 2 education. 3 • Physical and virtual environments as well as pedagogics are covered. 4 • Student, teacher, and customer perspectives are considered.
2021 IEEE International Conference on Big Data (Big Data), 2021
Data science projects have become commonplace over the last decade. During this time, the practic... more Data science projects have become commonplace over the last decade. During this time, the practices of running such projects, together with the tools used to run them, have evolved considerably. Furthermore, there are various studies on data science workflows and data science project teams. However, studies looking into both workflows and teams are still scarce and comprehensive works to build a holistic view do not exist. This study bases on a prior case study on roles and processes in data science. The goal here is to create a deeper understanding of data science projects and development processes. We conducted a survey targeted at experts working in the field of data science (n=50) to understand data science projects’ team structure, roles in the teams, utilized project management practices and the challenges in data science work. Results show little difference between big data projects and other data science. The found differences, however, give pointers for future research on how agile data science projects are, and how important is the role of supporting project management personnel. The current study is work in progress and attempts to spark discussion and new research directions.
2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2017
Today's students are prospective entrepreneurs and employees in modern, start-up like environ... more Today's students are prospective entrepreneurs and employees in modern, start-up like environments within established companies. In these settings, software development projects face extreme requirements in terms of innovation and attractiveness of the end-product. They also suffer severe consequences of failure such as termination of the development effort and bankruptcy. As the abilities needed in start-ups are not among those traditionally taught in universities, new knowledge and skills are required to prepare students for the volatile environment that new market entrants face. This paper reports experiences gained during seven years of teaching start-up knowledge and skills in a higher-education institution. We offer a collection of patterns that help educational institutions to design, implement and operate physical environments, curricula and teaching materials, and to plan interventions that may be required for project-based start-up education.
Software experiments are presently often used by big technology pioneers, such as Microsoft, Face... more Software experiments are presently often used by big technology pioneers, such as Microsoft, Facebook and Google, in order to learn about their users and to guide their research and development activities. Continuous experimentation (CE) is reported to be an integral part of software development in these organisations, however, how they transitioned to the approach is not publicly shared. Therefore, there is a lack of guidance for other organisations that are willing to adopt CE. In the current competitive markets, investing time and money in a new approach might be risky for these organisations, especially if they do not know how to initiate this transition process. This dissertation focuses on how organisations can initiate the transition towards CE, i.e., an approach to enhance development decisions by running experiments in an iterative and sustainable fashion. The dissertation was designed to acquire descriptive and observational knowledge through empirical studies and was cond...
2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2018
Software companies need capabilities to evaluate the user value and the success of their products... more Software companies need capabilities to evaluate the user value and the success of their products. This is especially crucial for highly competitive markets, such as the mobile game industry, where thousands of new games are introduced every month. Game companies often run continuous experiments as an integrated part of the overall development process. This paper presents a game company's journey on experimentation, and describes how the experiments are used at different stages of the development cycle to produce reliable, meaningful data for developers as well as how to balance between different data collection methods. Our study indicates that experiments are important in all stages of the development in different forms. Early stages in the development experiments can be run with proxy users due to lack of real users, whereas later in the development Key Performance Indicator (KPI) metrics play the most important role in experiments. Establishing concrete goals for the experim...
2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Nov 1, 2017
Background: Experiment-driven development with the help of real usage data helps to build softwar... more Background: Experiment-driven development with the help of real usage data helps to build software products and services that are of high value to their users. As more software companies use experimentation in their development practises, ethical concerns are increasingly important. Objective: There is a need for understanding the ethical issues companies must take into account when practising experimentation as a development strategy. This paper examines how software development practitioners experience the need for notifying users when involving them in experimentation. Method: We conducted a survey within four software companies, inviting employees in different functional roles to indicate their attitudes and perceptions through a number of statements. Results: Employees working in different roles have different viewpoints on ethical issues. While managers are more conscious about companycustomer relationships, UX designers appear more familiar with involving users. Developers think that details of experiments can be withheld from users if the results depend on it. Conclusion: Barriers to successfully conducting experiment-driven development are different for different roles. Clear and specific guidelines are needed for ethical aspects of experimentation. Index Terms-User involvement, customer involvement, continuous experimentation, experiment-driven development, ethics, human factors, software development
Processes and practices used in data science projects have been reshaping especially over the las... more Processes and practices used in data science projects have been reshaping especially over the last decade. These are different from their software engineering counterparts. However, to a large extent, data science relies on software, and, once taken to use, the results of a data science project are often embedded in software context. Hence, seeking synergy between software engineering and data science might open promising avenues. However, while there are various studies on data science workflows and data science project teams, there have been no attempts to combine these two very interlinked aspects. Furthermore, existing studies usually focus on practices within one company. Our study will fill these gaps with a multi-company case study, concentrating both on the roles found in data science project teams as well as the process. In this paper, we have studied a number of practicing data scientists to understand a typical process flow for a data science project. In addition, we studied the involved roles and the teamwork that would take place in the data context. Our analysis revealed three main elements of data science projects: Experimentation, Development Approach, and Multidisciplinary team(work). These key concepts are further broken down to 13 different sub-themes in total. The found themes pinpoint critical elements and challenges found in data science projects, which are still often done in an ad-hoc fashion. Finally, we compare the results with modern software development to analyse how good a match there is.
Context: Companies need capabilities to evaluate the customer value of software-intensive product... more Context: Companies need capabilities to evaluate the customer value of software-intensive products and services. One way of systematically acquiring data on customer value is running continuous experiments as part of the overall development process. Objective: This paper investigates the first steps of transitioning towards continuous experimentation in a large company, including the challenges faced. Method: We conduct a single-case study using participant observation, interviews, and qualitative analysis of the collected data. Results: Results show that continuous experimentation was well received by the practitioners and practising experimentation helped them to enhance understanding of their product value and user needs. Although the complexities of a large multi-stakeholder business-to-business (B2B) environment presented several challenges such as inaccessible users, it was possible to address impediments and integrate an experiment in an ongoing development project. Conclusion: Developing the capability for continuous experimentation in large organisations is a learning process which can be supported by a systematic introduction approach with the guidance of experts. We gained experience by introducing the approach on a small scale in a large organisation, and one of the major steps for future work is to understand how this can be scaled up to the whole development organisation.
[Context and motivation] In order to build successful software products and services, customer in... more [Context and motivation] In order to build successful software products and services, customer involvement and an understanding of customers' requirements and behaviours during the development process are essential. [Question/Problem] Although continuous deployment is gaining attention in the software industry as an approach for continuously learning from customers, there is no common overview of the topic yet. [Principal ideas/results] To provide a common overview, we conduct a secondary study that explores the state of reported evidence on customer input during continuous deployment in software engineering, including the potential benefits, challenges, methods and tools of the field. [Contribution] We report on a systematic literature review covering 25 primary studies. Our analysis of these studies reveals that although customer involvement in continuous deployment is highly relevant in the software industry today, it has been relatively unexplored in academic research. The field is seen as beneficial, but there are a number of challenges related to it, such as misper-ceptions among customers. In addition to providing a comprehensive overview of the research field, we clarify the gaps in knowledge that need to be studied further.
Uploads
Papers by Sezin Yaman