Evaluation and improvement using interviews
In this chapter the SMF is evaluated and improved using interviews. Therefore, the objective of the interviews, the chosen research method as well as the chosen sample and the interview approach are described. The initial SMF version developed in the previous chapter acts as input for the improvement and evaluation demonstrated in the course of this paragraph.
Objective
The objective of the third step in the development process of the SMF is to incorporate information about real-world SOA adoptions to ensure the applicability of the developed maturity model. The research process described in this chapter focuses on the derivation and incorporation of the relevant information. In addition to that the SMF developed so far should be peer-reviewed by professionals with relevant experience. Hence, the research objective is to disclose inconsistencies in the model and collect suggestions for improvements in order to improve the overall quality of the SMF.
Research method
Semi-structured qualitative interviews were considered as most appropriate research tool to gather the relevant information. In contrast to a structured interview the semi-structured interview allows bringing up new questions during the interview because it does not restrict the interviewer to a fixed set of questions. This additional flexibility is important to give the interviewed experts the freedom to state their experiences as uninfluenced as possible. During the interviews experts try to reconstruct experiences they made themselves and hence the results only reflect experiences made in individual cases [GLA04]. Therefore the representativeness of the individual results is of limited representativeness [FRO03]. However, for the purpose of improving the SMF, the chosen research method seems to be a promising one.
Interview approach
In order to be able to conduct expert interviews it is essential to be well prepared because it shows real interest in the experts’ area of expertise. In addition to that it is very difficult to conduct an interview which reveals the required information without a thorough understanding of the topic [GLA04]. However, during the development of the first SMF version and research in the area of SOA and the financial services industry, sufficient knowledge in this field was acquired.
An interview guide was prepared to be able to focus on the interviewee instead of thinking about new questions. In the interview guide relevant keywords were grouped to the five areas of the maturity model: Monetary, Architecture, Organization, Infrastructure and Governance & Security. These keywords proved to be very useful during the interviews because they ensured that all relevant aspects of an area were covered but still provided enough freedom to both the interviewer and the interviewee. The advantage of using only keywords instead of questions is that a too strict adherence to a set of predefined questions is not possible. It also proved to be helpful for the interviewees if questions were backed up with examples.
The duration of the interviews varied from 45 to 95 minutes. Five interviews could be conducted face-to-face and two had to be conducted via phone because the interview partners were in India and Germany at the time of the interview. The initial version of the SMF was published as a website in order to give these two interview partners the possibility to conveniently display the model on their computer during the evaluation phase of the interview.
At the beginning of each interview the interviewer briefly introduced the purpose of the interview, assured that all statements are used for the purpose of this thesis only, and all received information is treated anonymously, preventing conclusions to be drawn about the interview partners. After the interview partners had agreed to recording of the interview, they were asked to give their view on SOA and their role in SOA projects as an introduction. This enabled the interviewer to choose the adequate areas of questions for the following two main parts of the interviews.
The first main part of the interview covered questions about their individual experiences in the relevant areas. Thereby the interviewees were asked to provide examples based on their own experiences to avoid superficial repetition of opinions based on articles from popular science journals. The first part took about two thirds of the average interview time. In the second part the interviewees were asked to evaluate the first version of the SMF. They were urged to mention all inconsistencies and suggest improvements according to their opinion. In addition to that the technical SOA benefits were briefly evaluated and assigned a maturity level. Due to time constraints only six interview partners could complete the second part of the interview.
After the second part was finished, the interviewees were finally asked if anything important concerning a SOA adoption remained untouched during the interview in order to avoid missing essential points.
First interview part: Details of past experiences with SOA
In the first main part the interview partners were basically asked questions concerning their main area of expertise.
Second interview part: Evaluation of the SMF
Although many necessary adjustments to the model component could be derived out of the first part of the interviews, this second part was considered as important to verify the practical applicability of the model structure and the defined maturity levels.
Improved SMF version
Out of the interviews described in the previous paragraphs, essential information that can be used to improve the [[SOA Maturity Model for the Financial Services Industry|maturity model] could be retrieved. Therefore, the most important weaknesses in the first version of the SMF which became obvious during the interviews and the resulting improvements are summarized. Minor changes only affecting single model components are not listed here but can be seen comprehensively in the thesis.
- The gap between maturity level one and two has to be reduced. This step was obviously perceived as a step from trial projects to having all systems integrated into the SOA. Although this was not the intention during the development of the first version, it has to be changed in order to avoid further confusions. Therefore the concerning generic goals were worded differently and minor changes to other model components were done.
- Level five is perceived as a very ambitious aim which is probably difficult to achieve within the next few years even for very progressive players within the financial services industry. In addition to that the specific practices and some goals were perceived as buzzwords. Hence, two solutions to this issue were considered as possible: Either to remove maturity level five or to reduce the model components in level five to more abstract goals only. On the one hand, removing maturity level five would lead to a more tangible maturity model because the remaining four levels are more concrete. On the other hand, it would restrict the model’s extensibility because the model could not suggest any more improvements for a SOA at maturity level four. Hence, it was decided to keep maturity level five but define only goals at this level. This offers the benefit of having an ultimate aim which can be targeted and be continuously concretized by adding additional goals or practices whenever there is substantial progress in one of the areas. Defining specific goals without connected specific practices also complies with the SMF metamodel (see figure 28).
- In addition to these two major fundamental changes many minor improvements could be made based on the interview partner’s experience, especially at the maturity levels one to three. At the higher levels the interview partners had no practical experience, leading to a smaller amount of changes. Nevertheless, they critically evaluated these levels as well and gave advice based on their theoretical knowledge in the field of SOA.
[1] shows exemplarily the first level of the second version of the SMF. Model components which have not been changed since the first version are gray, improved model components are marked green, and newly added model components are highlighted yellow. In total 25 model components have been improved (3 generic goals, 4 specific goals, 18 specific practices) and 43 new model components have been added (2 generic goals, 17 specific goals, 24 specific practices) based on the interviews.
Letzte Änderung: 11.05.2009, 23:48 | 1360 Worte