DevOps Challenges and Risk Mitigation Strategies by DevOps Professionals Teams

Professionals describe DevOps as a software engineering culture, work practice or even a philosophy. If we observe the scientific community, different views, perceptions and stances have been developed and suggested regarding DevOps. DevOps describes how cross-functional teams work together to build, test and release faster software more reliably [18]. Automation plays a vital role in DevOps operations as its goal is to improve collaboration between two teams in terms of software development.

Organizations are increasingly adopting DevOps practices to enhance their software delivery process [23]. By effectively implementing and adopting DevOps principles, the gap between development and operations teams can be minimized. The development process triggers software deployment, which is crucial for software organizations to move software into production [8]. The key aspect of DevOps in an organization is to ensure continuous delivery and deployment, resulting in faster software delivery cycles [10]. As a result, DevOps has become an essential part of modern software development, providing organizations with a competitive edge and enabling them to stay ahead in the market.

Krey et al. [15] have identified six major challenges faced by small and medium-sized enterprises in DevOps implementation: costs, risks, scope, quality, business value, and time. However, a lack of communication among teams can be a major contributor to unsuccessful DevOps adoption. Operations teams have specific responsibilities, and they often don’t pass or monitor different performance metrics that could help developers execute tasks [21].

Companies are increasingly adopting DevOps practices in response to customer and user expectations for software applications that meet their needs [13]. To meet this demand, organizations are striving to release frequently and deploy faster, but this requires an efficient process environment and proper utilization of resources. DevOps helps address miscommunications and gaps in the process with four guiding principles: automation, culture, collaboration, and measurement [13]. Gupta et al. [13] also identified four variables that impact the implementation process: source control, automation, cohesive teams, and continuous delivery. By addressing these factors, organizations can successfully implement and adopt DevOps practices.

Effective collaboration between development teams and operations teams is crucial for successful software development and deployment. To facilitate this, it is important to have a common set of tools used by both teams, as using different toolsets can create problems and inefficiencies in the collaboration process [6]. Communication between the Dev and Ops teams is also of utmost importance, as lack of communication can lead to delays in the operating process of both teams [21]. DevOps leverages a variety of tools to streamline the software development process. However, the COVID-19 pandemic has forced most of the work to go remote, which has had a significant impact on the working process [20]. It is important to note that electronic tools alone cannot solve all problems and some issues are best addressed in person. Furthermore, integrating different tools can be challenging and require additional maintenance and execution efforts [5].

Companies can employ various strategies to effectively address risks and challenges. One such strategy is to move away from the traditional Dev and Ops mindset and embrace continuous delivery practices. Adopting microservices-based infrastructure and architecture, implementing test automation techniques, prioritizing tools, delegating release ownership to teams, and fostering a culture of continuous learning are also effective strategies. Jones et al. [14] recommend the introduction of job crafting as a means to help DevOps professionals achieve their personal goals. Job crafting is an individualized design process that allows employees to proactively modify job characteristics to align personal growth with work objectives. Through job crafting, employees gain greater control over their tasks, determine how their work is perceived, and shape the social context and relationships within the workplace [4]. According to Jones et al. [14], task, relational, and cognitive job crafting can significantly enhance work performance while adopting DevOps in companies. Liete et al. [16] suggest three approaches for implementing DevOps adoption in companies: department collaboration, DevOps teams, and cross-functional teams.

The aim of this paper is to identify the challenges and risks that IT companies face when adopting DevOps, and how they mitigate these risks by implementing various strategies. We have conducted in-depth interviews (N=26) with DevOps professionals from different companies around the world to investigate these issues. As a result, we will try to answer the following research questions in this paper:

Throughout our research, we had the privilege of interviewing multiple accomplished DevOps professionals in order to gather valuable data. Our research methodology involved conducting thorough interviews to pinpoint prevalent obstacles and potential hazards that professionals face, examine professional practices, address security concerns, and deeply explore the ethical considerations within DevOps teams. To ensure our interviews were comprehensive, we created a set of 18 questions organized into three themes: challenges and risks overview with mitigation, security risk and mitigation, and team ethics and mitigation strategies from technical, social, and cultural viewpoints.

During the course of the study, respondents represented companies ranging from 80 to 15,000 employees. The respondents held various positions within their respective organizations, including Head of Technology, Tech Lead, Scrum Master, Site Reliability Engineer, DevOps Engineer, Software Specialist, Business Analyst, Cloud Engineer, Technical Project Manager, and Software Engineer. With working experience in the software development industry ranging from one to twenty years, respondents were contacted via email for participation in the interview. The interviews were scheduled for a duration of thirty minutes, during which in-depth questions were asked, focusing on specific areas of DevOps practices. The researcher worked diligently to ensure that the data collected was accurate and relevant to the study’s objectives.

During the interview process, we ensured that each interviewee provided their consent to being recorded. For those who declined to be recorded, we respectfully opted to take notes instead. In total, we conducted 26 interviews with distinguished DevOps professionals occupying diverse roles across numerous companies. These interviews were conducted during the first quarter of 2023, specifically from March to April. Subsequently, we performed a comprehensive analysis of the findings based on the interviews. The 26 interviewed individuals represented 26 distinct companies, which we labeled with different alphabets in the presentation of our results.

For analyzing the data, we have used the Gioia method presented by Gioia et al. [12]. An iterative process has been followed which ensures the repetition of steps for the data analysis. We have followed open coding for extracting data from the interviews. As a guideline, we have followed Strauss and Cobin [22]for assigning codes for the analysis. We started the coding process with the interview transcripts, then we marked specific areas and assigned the codes suggested by [19]. For the first-round coding, we used the research questions as guidelines. From the empirical data, we checked what are the similar codes in the various segments of the data. Then we checked the dissimilarities present in the codes and identified those codes.

In our research, We utilized constant comparison and followed the grounded theory approach [22]. We have prepared a table that showcases the coding activities created from the interview data, providing an explicit understanding of the coding process. The table includes a detailed list of codes, their corresponding descriptions, and quotes from professionals. An exemplary table called Table 1 illustrates the coding activities. This table can be used as a reference to gain insights about the coding methodology.

After the first coding ended, we moved to the second phase of coding. In the second phase, we have started categorizing the first phase codes. According to Charmaz, to create second-order codes for concepts it is necessary to categorize the first-phase codes [7]. Then we merged the first-phase codes with second-phase codes [11]. To make the data analysis process accurate we have also used memoing techniques. Memoing helped us to understand more insights and perspectives of professionals’ views regarding critical success factors and organizational practice. A total of 910 pages were generated from the interview data transcription. According to our understanding, we have used an iterative process for data analysis [22].

In the third phase of the data analysis, we have aggregated the themes into four main aggregate categories including Organizational risk, Cultural and Social risk, and Technical risk and Ethics and security breach risk. In Fig. 1, we have shown the data analysis process with themes.

A professional quoted:

“In our teams, there are sometimes miscommunications, and due to that DevOps practices get hampered (Development and Operations) and lack of collaboration between clients and developer teams make the process risky, improper communication creates difficulties for better outcomes”.

Strategic Planning. Based on the extensive research by Azad and Hyrynsalmi Azad and Hyrynsalmi [2], the product management team is responsible for maintaining the business requirements, while the technology team handles the technical requirements, emphasizing the need for meticulous planning related to resources, initiatives, and budget for the overall software process. It is crucial for the IT and business plan to share similar goals and objectives. Adopting continuous development and continuous delivery would ensure top-notch quality of the product. Therefore, strategic planning should prioritize company pressure, change management, meeting deadlines, and reducing the time to market Azad and Hyrynsalmi [2].

Our findings suggest that improper allocation of budget for the toolset is a risk for DevOps practices. The budget allocation for toolsets is important because wrong choices create risks for the project. According to professionals risky change and development are challenging for the teams. People in the team are reluctant to new changes as those are uncomfortable and people fear changes.

A professional quoted that:

“Risk mitigation through automated testing and quality assurance is essential for the development process. If automated tests are in place, a developer can immediately get feedback about their newly written programs/features. Then the process becomes less risky”.

Quality assurance acts as a bridge between development and operations teams and supports developers by testing new iterations in real-time with continuous quality checks to keep the testing cycle running smoothly.

Another professional quoted that:

“Balancing security and risk management for the DevOps process is crucial. For good balancing the team needs to make sure that they do not release anything if not properly tested”.

Social and cultural risk factors are one of the leading factors for DevOps risks in the organization [2]. Below we discuss the team dynamics and social and cultural shifts risks.

A professional quoted that:

“When a team has skilled and knowledgable people with a diversified culture that helps the team to progress better. A sudden change like someone leaving the team might slow down the process as DevOps teams are connected with each other and that’s the way the team progress”.

A professional quoted that:

“Lack of collaboration and organizational culture does not help for better building products for clients. The company culture should be collaborative, flexible, and supportive. To make it secured from the beginning DevSecOps should be a part of the process”.

There are several technical risks associated with DevOps practices. Some of those include improper code review by team members, security in a DevOps environment, and human error as a DevOps risk.

A professional quoted that:

“Uh, of the project experiences within the company they at first understand the requirements and set up the tools which are actually secured. So the important thing is that selection of the tools that make a secured environment for the development process”.

According to our findings, the professionals stated that security vulnerabilities in DevOps pipelines are risky for the companies. Security vulnerabilities include missing data inscription, missing authentication for critical functions, and buffer overflows with insecure interactions between software companies. Whatever the developer has done and if the test is an improper code review, it is the number one risk for the process.

A professional quoted that:

“For maintaining security vulnerabilities, developers need to check if the web service is running and the Azure function can send requests and get the response back each hour. There should be access restrictions so only certain IPs are allowed if that is required”.

Human errors are one of the most unpredictable situations for any DevOps team which might create several risks for the DevOps environment. There are many steps as a part of DevOps work. People may forget to test certain codes or follow best practices. Maybe one port remains open by mistake, Data Storage is open to public access, Databases does not have IP restrictions, forgets to stop an expensive during holiday/weekend, no cost tracking of the cloud services. These errors could impact the development process hugely.

A professional quoted that:

“DevOps practices align with our organization’s values and ethical principles and require timely release features, Deployment frequencies, Time to recover in case of any issues, data protection, and scalability. ”.

A professional quoted that:

“For ethical considerations, a company should take into account where, and how to collect, store, and analyze data in our DevOps operations”.

A professional quoted that:

“Involve users and other stakeholders in ethical decision-making processes related to our DevOps operations is essential. A good communication can solve most of the issues. ”

To mitigate risks and improve performance, there are various approaches that professionals can adopt. Respondents have highlighted different strategies that can assist in managing organizational risks. According to the research findings some of the risk mitigation strategies could be continuous testing, using infrastructure as a code, security audit and monitoring, disaster recovery planning, cross-functional training, proper documentation, continuous learning, continuous improvement, making process visible to the team members, prioritize personnel so they feel valued, enforce security policy, introduce DecsecOps, involvements of experts from outside, and improved management strategies. In the example Table 2, we have given a short list of risks and risk mitigation strategies proposed by the professionals.

It is imperative to establish a comprehensive framework that can effectively address the issues of security and ethics. To achieve this, it is crucial to facilitate effective communication and establish a robust system of governance. An effective security system or a set of cybersecurity approaches should be implemented to ensure that the security processes are straightforward, transparent, and comprehensible. The security process should encompass a wide range of issues, including code review, access restrictions, and management configuration, among others.

In order to produce a well-secured application, it’s crucial for DevOps teams and security teams to work closely together. This collaboration helps to ensure that robust policies and effective tools are implemented to protect the application from potential security threats. By working together, DevOps and security teams can identify potential vulnerabilities in the application and take proactive steps to address them. Additionally, this collaboration can help to streamline the development process by incorporating security measures early on, reducing the likelihood of costly delays or security breaches down the line.

According to the respondents, the challenges for DevOps adoption is insufficient knowledge in industries and the engineers also have a knowledge gap. Though they might have some strong understanding or knowledge or background in some specific part of the software development but DevOps practices still needs to be understood by many of them. DevOps needs proper communication with software developer. The engineers witness that sometimes a developer only working on his coding but when deployment comes, he doesn’t have really much idea what’s happening in the back end or in the cloud system and also the automation is unclear to him.

This study addresses two aspects of DevOps. Firstly, the risk factors identified by industry professionals and, secondly, risk mitigation strategies for DevOps risks. Our findings also discuss security issues and organizational ethics along with risk factors in DevOps operations. From the interviews with professionals, we have learned several DevOps practical risks faced by organizations. However, these risk factors are not universal. These are the professional’s own views regarding the risks and the ways to mitigate them when necessary.

According to our findings, there are four major risk factors including organizational risk factors, social and cultural risk factors, technical risk factors, and ethics and security breach factors.

Misunderstanding between Development and Operations teams poses a significant obstacle to the success of the DevOps process. According to research, there is a lack of coordination among team members when working together [1, 2, 15, 21]. This lack of communication can hinder the adoption of DevOps, making the process unsuccessful [1, 15]. One of the most significant risks faced by DevOps teams is the need to balance performance and the speed of releases [3]. Professionals have reported that fast release cycles can enhance performance while reducing the time required for development [3].

Based on the feedback received from participants, it is clear that implementing DevOps in a company can be a difficult and risky task, which may result in an unsuccessful implementation. Employees often struggle to accept and adapt to changes, leading to confusion and delays. The process of change is perceived as complex and time-consuming, which adds to the challenge of implementing DevOps [2, 15].

Lack of focus or differences in development is another challenge for DevOps practices. Often devlopers faced that there is a lack of focus in the development process. They are not sure of what they are doing, there could be miscommunication with team members. There could be misconceptions between development teams and operations team members. Due to these reasons, differences occur in the development process [1, 3, 15, 21].

Creating proper test and production environments is a significant challenge. Both testing and production environments are crucial for the production process, and it is essential to have a well-designed testing process for the code. The production environment should support the testing process seamlessly. Poor integration can hamper the testing process, which is why it is essential to set up proper test setups to ensure that the rest of the process functions effectively [9].

Choosing the right tools for DevOps operations is another obstacle companies face. They select the tools based on their project needs and requirements. However, finding or selecting the appropriate tools is often difficult for companies.

We witnessed certain limitations in conducting the research. First of all, the research did not consider the psychological aspects of the DevOps working environment and could not cover the emotional aspects of employees working in teams.

Second, In the study, practices of IT organizations were observed but the focus was developed countries IT practices. Therefore, if we could consider developing countries’ DevOps operations then we could compare the scenario of developed and developing countries’ IT practices to understand better views on DevOps challenges and risk mitigation strategies.

Third, due to lack of time we could not conduct longitudinal studies. A prospective study would be a great way to focus on DevOps practices which might help the researchers to understand management practices with experts’ perspectives over time.

Fourth, our topic is narrowed to DevOps operations and organizational practices. Due to this reason, the domain became more specific. Identifying DevOps professionals for interviews was specifically a real challenge. We had to use various techniques to find professionals for interviews. Finding professionals was difficult and considered one of this research’s major limitation, as there is a possibility of response bias and selection bias.

Fifth, the respondents could not share some information that they consider confidential for their companies. Due to those issues, we could not ask them questions as planned.

We have identified several areas in the DevOps domain that require further study.