Search

Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusion: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve. © 2021 The Authors, open access

Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusion: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve. © 2021 The Authors, open access

Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusion: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve. © 2021 The Authors, open access

Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusion: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve. © 2021 The Authors, open access

Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusion: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve. © 2021 The Authors, open access

Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusion: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve. © 2021 The Authors, open access

Context: The increased use of Open Source Software (OSS) affects how software-intensive product development organizations (SIPDO) innovate and compete, moving them towards Open Innovation (OI). Specifically, software engineering tools have the potential for OI, but require better understanding regarding what to develop internally and what to acquire from outside the organization, and how to cooperate with potential competitors. Aim: This paper aims at synthesizing a theory of openness for software engineering tools in SIPDOs, that can be utilized by managers in defining more efficient strategies towards OSS communities. Method: We synthesize empirical evidence from a systematic mapping study, a case study, and a survey, using a narrative method. The synthesis method entails four steps: (1) Developing a preliminary synthesis, (2) Exploring the relationship between studies, (3) Assessing the validity of the synthesis, and (4) Theory formation. Result: We present a theory of openness for OSS tools in software engineering in relation to four constructs: (1) Strategy, (2) Triggers, (3) Outcomes, and (4) Level of openness. Conclusion: The theory reasons that openness provides opportunities to reduce the development cost and development time. Furthermore, OI positively impacts on the process and product innovation, but it requires investment by organizations in OSS communities. By betting on openness, organizations may be able to significantly increase their competitiveness. © 2017 Elsevier B.V.

Context: The increased use of Open Source Software (OSS) affects how software-intensive product development organizations (SIPDO) innovate and compete, moving them towards Open Innovation (OI). Specifically, software engineering tools have the potential for OI, but require better understanding regarding what to develop internally and what to acquire from outside the organization, and how to cooperate with potential competitors. Aim: This paper aims at synthesizing a theory of openness for software engineering tools in SIPDOs, that can be utilized by managers in defining more efficient strategies towards OSS communities. Method: We synthesize empirical evidence from a systematic mapping study, a case study, and a survey, using a narrative method. The synthesis method entails four steps: (1) Developing a preliminary synthesis, (2) Exploring the relationship between studies, (3) Assessing the validity of the synthesis, and (4) Theory formation. Result: We present a theory of openness for OSS tools in software engineering in relation to four constructs: (1) Strategy, (2) Triggers, (3) Outcomes, and (4) Level of openness. Conclusion: The theory reasons that openness provides opportunities to reduce the development cost and development time. Furthermore, OI positively impacts on the process and product innovation, but it requires investment by organizations in OSS communities. By betting on openness, organizations may be able to significantly increase their competitiveness. © 2017 Elsevier B.V.

Context: The increased use of Open Source Software (OSS) affects how software-intensive product development organizations (SIPDO) innovate and compete, moving them towards Open Innovation (OI). Specifically, software engineering tools have the potential for OI, but require better understanding regarding what to develop internally and what to acquire from outside the organization, and how to cooperate with potential competitors. Aim: This paper aims at synthesizing a theory of openness for software engineering tools in SIPDOs, that can be utilized by managers in defining more efficient strategies towards OSS communities. Method: We synthesize empirical evidence from a systematic mapping study, a case study, and a survey, using a narrative method. The synthesis method entails four steps: (1) Developing a preliminary synthesis, (2) Exploring the relationship between studies, (3) Assessing the validity of the synthesis, and (4) Theory formation. Result: We present a theory of openness for OSS tools in software engineering in relation to four constructs: (1) Strategy, (2) Triggers, (3) Outcomes, and (4) Level of openness. Conclusion: The theory reasons that openness provides opportunities to reduce the development cost and development time. Furthermore, OI positively impacts on the process and product innovation, but it requires investment by organizations in OSS communities. By betting on openness, organizations may be able to significantly increase their competitiveness. © 2017 Elsevier B.V.

Context: The increased use of Open Source Software (OSS) affects how software-intensive product development organizations (SIPDO) innovate and compete, moving them towards Open Innovation (OI). Specifically, software engineering tools have the potential for OI, but require better understanding regarding what to develop internally and what to acquire from outside the organization, and how to cooperate with potential competitors. Aim: This paper aims at synthesizing a theory of openness for software engineering tools in SIPDOs, that can be utilized by managers in defining more efficient strategies towards OSS communities. Method: We synthesize empirical evidence from a systematic mapping study, a case study, and a survey, using a narrative method. The synthesis method entails four steps: (1) Developing a preliminary synthesis, (2) Exploring the relationship between studies, (3) Assessing the validity of the synthesis, and (4) Theory formation. Result: We present a theory of openness for OSS tools in software engineering in relation to four constructs: (1) Strategy, (2) Triggers, (3) Outcomes, and (4) Level of openness. Conclusion: The theory reasons that openness provides opportunities to reduce the development cost and development time. Furthermore, OI positively impacts on the process and product innovation, but it requires investment by organizations in OSS communities. By betting on openness, organizations may be able to significantly increase their competitiveness. © 2017 Elsevier B.V.

Context: The increased use of Open Source Software (OSS) affects how software-intensive product development organizations (SIPDO) innovate and compete, moving them towards Open Innovation (OI). Specifically, software engineering tools have the potential for OI, but require better understanding regarding what to develop internally and what to acquire from outside the organization, and how to cooperate with potential competitors. Aim: This paper aims at synthesizing a theory of openness for software engineering tools in SIPDOs, that can be utilized by managers in defining more efficient strategies towards OSS communities. Method: We synthesize empirical evidence from a systematic mapping study, a case study, and a survey, using a narrative method. The synthesis method entails four steps: (1) Developing a preliminary synthesis, (2) Exploring the relationship between studies, (3) Assessing the validity of the synthesis, and (4) Theory formation. Result: We present a theory of openness for OSS tools in software engineering in relation to four constructs: (1) Strategy, (2) Triggers, (3) Outcomes, and (4) Level of openness. Conclusion: The theory reasons that openness provides opportunities to reduce the development cost and development time. Furthermore, OI positively impacts on the process and product innovation, but it requires investment by organizations in OSS communities. By betting on openness, organizations may be able to significantly increase their competitiveness. © 2017 Elsevier B.V.

Context: The increased use of Open Source Software (OSS) affects how software-intensive product development organizations (SIPDO) innovate and compete, moving them towards Open Innovation (OI). Specifically, software engineering tools have the potential for OI, but require better understanding regarding what to develop internally and what to acquire from outside the organization, and how to cooperate with potential competitors. Aim: This paper aims at synthesizing a theory of openness for software engineering tools in SIPDOs, that can be utilized by managers in defining more efficient strategies towards OSS communities. Method: We synthesize empirical evidence from a systematic mapping study, a case study, and a survey, using a narrative method. The synthesis method entails four steps: (1) Developing a preliminary synthesis, (2) Exploring the relationship between studies, (3) Assessing the validity of the synthesis, and (4) Theory formation. Result: We present a theory of openness for OSS tools in software engineering in relation to four constructs: (1) Strategy, (2) Triggers, (3) Outcomes, and (4) Level of openness. Conclusion: The theory reasons that openness provides opportunities to reduce the development cost and development time. Furthermore, OI positively impacts on the process and product innovation, but it requires investment by organizations in OSS communities. By betting on openness, organizations may be able to significantly increase their competitiveness. © 2017 Elsevier B.V.