Qualitative Research Methods for IB - Lecture notes - RUG

Lecture 1

The goal of qualitative research methods (QRM): to perform all stages of a qualitative research project, i.e. perform interviews in a systematic and methodologically responsible way.

Essentials of interviewing:

A good interview is a focus on the subject’s (Interviewee/IE) world

What to do?:

• Create a collaborative atmosphere The more comfortable interviewees feel, the more they are prepared to open up and talk, the better your data are likely to be

• Interviewer’s (IR) contribution: prepare both the task-oriented and the interaction-oriented role before the interview

• Develop a genuine interest in the interviewee during the interview, listen, prompt, encourage and direct.

Interviewers first attention generally goes to content (focus on data). What you SHOULD focus on: interviewee as a way to collect those data  IE mirrors the IR’s attitude: if lack of interest, so does IE.

Basic classification of interviews:

a. structured

b. unstructured

c. semi-structured

a.) Structured interview:

• participant responds to list of preconceived topics
• pre-formulated questions
• minimal role of interviewer

Advantage: consistency during multiple interviews

Disadvantage: no space for new insights on the basis of voluntary information from the interviewee

The most highly structured interview format is undoubtedly the survey questionnaire: an interview guide that pre-specifies both the content and the possible responses to each question. It involves the use of pre-formulated questions.

The whole idea of structured interviews is to ensure consistency across multiple interviews. They tend to be in market research, polling, telephone interviews and with intercept research such as in shopping centres.

b.) Unstructured interviews:

The opposite of structured interviews. There may not be a time limit. Questions are only used to jog someone’s memory.

Characteristics:

• very few pre-formulated questions.

• no attempt to maintain consistency across interviews

advantage: allows the interviewee to talk freely and tell you everything he or she considers important

disadvantage: if interviewee is not talkative, you may end up with too little; if the interviewee is very talkative, you may end up with too much. Also hard to compare answers by different Ies

c.) Semi-structured interview: in-depth interview (What we have to do in this course):

• some pre-formulated questions
• some control over the direction and content to be discussed,
• some consistency across interviews

Advantages: takes the best of structured/unstructured approaches while minimizing the risks; it is cost-efficient and relatively easy to conduct.

Limitations:

1. possibly limited ability of the participant to recall his experiences
2. possibly limited ability of the participant to articulate
3. possibly limited ability of the researchers to ask the “right” questions and to prompt a more detailed discussion

NB: for these reasons semi-structured interviews are often combined with other forms of data, such as observations, diaries, documents.

The composition of a focus group is of vital importance: participants need to feel comfortable talking with each other about the research topic. If you want a lively conversation, you need participants who are actively interested in talking to each other about the topic.

Point of attention: style of moderating by the interviewer.

Individual interviews: more useful when the goal is to obtain depth and detail about each participant.

Focus groups: more useful when the goal is to hear from a range of participants.

Combination: for example focus groups could provide an introduction to the views of a variety of participants. Next a number of participants could be selected for in-depth interviews.

Advantage: focus groups enable a researcher to elicit opinions, attitudes and beliefs held by members of a group

Disadvantage: often time-consuming and expensive

Point of attention: interviewer moderating style

Points of attention using any kind of interview:

• Lack of trust:

  interviewer is complete stranger, so interviewee may decide not to open up completely. As he may thus be withholding important information, the data gathering will be incomplete

• Lack of time:

  time pressure may either be the cause of omissions or answers that are not entirely true for lack of fine-tuning

• Level of entry / Elite bias:

  the level at which the researcher enters the organization is crucial: who is the interviewer identified with and how does that affect the relationship with interviewees.

Hawthorne effect:

the interviewer is not a neutral factor in the interview and may influence the interaction: IE knows that the answers are used for academic purposes so that may affect their response (is this what you want to hear?)

Ambiguity of language:

the meaning of the interviewer’s words can be ambiguous and it is not always certain that the interviewee fully understands the questions.

Interviews can go wrong:

it is possible that the interviewer unintentionally offends the interviewee in which case the interview might be abandoned altogether.

Conclusion:

When you are aware: opportunity for intervention

When you are prepared, you know:

• how to present yourself
• check your questions for unambiguity and evaluate response
• how to present the context

A good interview helps us to focus on the subject’s world. The idea is to use their language rather than impose our own.

• The role of the interviewer is to listen, prompt, encourage and direct. Overall, the more comfortable interviewees are, and the more they are prepared to open up and talk, the better the disclosure is likely to be.

If your interviewee does not open up during the conversation, then the primary data you collect may be of limited value. Both the quantity and the quality of your data will be affected negatively. As a general rule, the more proficient you become at interviewing, the better your qualitative data will be

Also: how much do you know for sure after an answer has been given?

We interpret what we see and hear all the time but we need to offer our interpretation to the IE for confirmation or correction!

In order to achieve this, the interviewer is focused on two activities at the same time: to listen, prompt, encourage and direct to collect valid and relevant data + to create an atmosphere where the interviewee is willing to provide you with data.

So the two roles of the interviewer are:

• Task-oriented = organize the interview in such a way that you reach your goal, i.e. elicit valid and reliable data for your research.

Criteria: interview structure, clear goal, active listening skills

• Relation-oriented = conduct the interview in such a way that the interviewee is willing to provide you with valid and relevant data. Focus is on interaction, contact and trust.

Criteria: create collaborative atmosphere, expectation management = paying genuine attention to the other person’s needs

Interview guide

Interview guides provide the researchers with a script or protocol for the interview

The topics of the interview guide are based on the research question and the tentative conceptual model that underlies the research.

Benefits of this approach:

• interviewee can come up with information or aspects of your topic that you had not considered before
• you collect data from individual people that can be compared because of the standard set-up
• for the validity and reliability, semi-standardization is required

As researcher you will work on grounded theory: that is, you start from scratch; you have concepts in your mind based on theory. You will build up theory from the data.

A question-based guide outlines the expected content of the interview in terms of a series of questions the interviewer intends to ask.

 A topic-based guide consists of a list of areas and issues the interviewer wants to hear about. Often they are already outlined in a format which serves as a checklist for the interviewer to make it easier to monitor which topics have already been covered.

Type of questions:

a. Open: what, where, how, who, why

Closed: only yes/no answer possible

Choice: answer is either .. or

b. Neutral: How would you characterize the atmosphere in your team?

Leading/Suggestive: Don’t you think the computer system in this organisation is out-dated?

Evaluating

By evaluating answers you checks their usability for your research focus:

1. Is the answer valid? Valid = what the interviewee thinks and says the same?

2. Is the answer complete?

Complete = checking if the answer is indeed all there is (e.g. I don’t know)

3. Is the answer relevant?

Relevant = a question might appear to be answered whereas in fact it is not

4. Is the answer clear?

Clear = are you sure you know what the IE means?

Whenever the interviewer (you) concludes that an answer does not meet any of the required criteria, or when you are in doubt, or when an interviewee has trouble finding an answer, then it is time to probe (ask further questions).

Probing does not add content to that introduced in the opening Q. It does not raise new issues.

Different types of probing:

1. Repeating or clarifying the opening question

2. Repeating or summarizing the answer

3. Focused / unfocussed probing

4. Silent probing and listening behaviour

Interviewing in a cross-cultural context requires awareness of:

• potential difference in language proficiency
• possible cultural differences
• potentially different belief systems
• different social conventions

What do you think + feel when you read this word?: Autumn

• Season of falling leaves in the Netherlands – “no such thing as autumn: Iceland – increasingly hot: South Africa

• Rain

• Gloomy season

• Moody, sad

versus

• Bright colours

• Long shadows

• Moving towards Xmas

• Bracing weather

• Lovely season

• Happy feeling

Lesson: we have a certain perspective on the world around us and we may not share IE’s experience and parameters. How to find out: probing!!

Interaction tips to consider (cross-cultural interview):

1. how do other people behave, how do they interact, their body language. As far as possible mirror and match their body language; e.g. leaning forward / back

2. Accept that social rules may be very different. Be adaptable to other people’s needs

3. Don’t assume because someone represents a particular country, that they behave accordingly

4. Keep going even when it (interview) takes longer than you had expected

Lecture 2

Before introducing the definition and specific methods of the qualitative approach, a general discussion about research methodology is required.

In any empirical research, you will always find a chapter about “methodology”.

Methodology refers to the choices made about cases to study, data gathering techniques, data analysis methods, etc.

In your own bachelor and master thesis, you will have to include this chapter too.

Methodology comprises:

• A theory of scientific knowledge, or set of assumptions (about the nature of reality, the role of researcher, etc.)
• A preference for certain methods (qualitative and / or quantitative)
• A systematic sequence of procedural steps to be followed once the method(s) has been selected

In this course, a number of qualitative methods are going to be discussed. You already know about some quantitative methods.

Essential in case of quantitative methods is that numbers is the basic material which is analysed, while in case of qualitative research, essential is text.

Recently, thanks to an increasing usage of social media, new online methods became relevant.

The traditional distinction between QUAL and QUANT is being broken down by these new opportunities, to find out what people are saying and feeling.

Quantitative: Survey, Laboratory experiment , Simulation , Mathematical modelling (based on numbers)

Using numbers, it can objectively describe, test, segment and predict (as long as the sampling is sound).

It tells you:

• main behaviours

• trends

• satisfaction levels, attitudes, awareness …

Qualitative: Case study, Interviewing, Focus Group, Grounded theory, Action research (based on text) Online: Listening, netnography, bulletin boards

Gives detail, understanding and emotional response: Emotion is always involved in decision making – even if people aren’t aware of it, or if they rationalise it

Definitions:

Quantitative research

“Measuring things that can be counted using predetermined categories that can be treated as interval or ordinal data and subjected to statistical analysis” (Patton, 1997)

Qualitative research

“Focuses on people’s experiences and the meanings they place on events, processes and structures of their normal social setting. Such research may involve prolonged or intense contact with people and groups in their everyday situations. This provides a holistic view, through the participants’ own words and perceptions, of how they understand, account for and act within these situations” (Miles and Huberman, 1994)

• understanding people and (their) context

The choice between using a qualitative or quantitative approach in research is not a trivial one. When you will be in the position to choose the most appropriate approach for your bachelor or master thesis, you may face this challenge.

Depending which approach is chosen for a particular research (qualitative or quantitative), a research method or a data collection method can be used differently.

For example, in a quantitative approach, observation can be used as a preliminary work to identify key informants within a large company. Alternatively, in a qualitative approach, observation can yield the data necessary to be analysed in order to identify the behaviour of different groups of people.

In a quantitative approach, text analysis can be used to count, for instance, the frequency of all terms related to “radical innovation”, “incremental innovation”, “sustainable innovation”.

In a qualitative approach, text analysis can be used to understand what “radical innovation”, “incremental innovation”, and “sustainable innovation” means for a group of experts in product development.

In a quantitative approach, interviews consisting on closed questions can be used in a survey research about “which social media platform do you use and how often”, involving 30 randomly selected employees. In a qualitative approach, open interviews can be performed with 10 people, about why and how do they use social media applications.

Research Design:

1. Research objectives: what is the research goal/objective?

2. Research setting

Comparative study: comparing joint-ventures from US and China with respect to leadership styles

Snapshot: high employee turnover in a specific company. It is a short study

Retrospective: the biography of Thomas Edison, the challenges of liberalisation of gas market in NL

Case study: Enron as an accountancy bad practice, also example of retrospective case

Longitudinal studies

• Repeated examination of a set of subjects over time

• The evolution of selected indicators is of concern

Panel longitudinal studies

• Explore micro-level change at individual level

• Determine the type of change – QUAL goal

• Measuring change is more a QUANT goal

• Ex: how merging affects job satisfaction, three waves of interviews every 6 months

3. Data collection

Cross-sectional (longitudinal) studies

• Explore macro-level change at context level

• changing societal influences on attitudes

• Ex: what shapes views in social-media usage?

Naturally occurring data

• Observation, conversation data

• Need: researcher’s interpretation

Generated data (primary data)

• In-depth (unstructured), semi-structured interviews, focus groups

• Participants give their own meanings and interpretations, interviewer can probe

Secondary data

• Existing materials, collected for different purpose

• Is it enough relevant and quality data for the new research?

4. sampling methods

Purposeful sampling:

• Homogeneous sampling- cases having same characteristics

• Heterogeneous sampling – cases which vary from each other

• Extreme case or deviant sampling – cases are unusual or special

• Typical case – cases characterized ‘normal’ or average to provide detailed profiling; it requires prior knowledge about overall patterns of response.

Convenience sampling and opportunistic sampling

• Convenience sampling: ease of access

• Opportunistic sampling: as events arise

Theoretical sampling

• Type of purposive sampling

• Cases are selected to base the development of new concepts

• In connection with grounded theory (in GT, theory emerges from qualitative data)

• After first data analysis step, additional data collection stages may occur: more interviewees, more groups

• Data analysis and data collection continues, until theoretical saturation is reached

Sample size:

Not too small, not too big

• Interviews: no more than 50 people

• Focus groups: max 90-100 people (12 to 14 groups)

Issues determining sample size:

• Heterogeneity of population

• Number of selection criteria

• Multiple samples within one study

• Type of data collection methods

• Budget and resources available

Small sample size if purposive or theoretical sampling has taken place!

Sampling frame:

Samples are generated from a population, using a sampling frame

Existing sources: ex. published lists

Generated sampling frames: ex. snowballing, flow population

5. Data analysis:

• Grounded theory

• Analytic induction

• Text (content) analysis

Explained in Lecture 3

Lecture 3

Difference qualitative and quantitative research:

• Quantitative: ‘amount’ of certain qualitites

• Qualitative: properties of objects, phenomena, people

Look back at the notes of the first lecture where the differences are shortly discussed.

In fact, there is quite an ideological discussion about these differences between different schools of thinking. You should be aware that, in essence, the difference should only be between good or bad research.

The research process of quantitative research is mainly a linear process (continuing the cycle is also possible).

In qualitative research, the research process has the tendency of being more circular. This happens because the considering of new information during the analysis process is possible.

Aimed more at theory building than theory testing

A circular model

Suppose that when we question about the (lack of) data sharing within company XYZ, we would consider in the beginning only 2 departments, then we would perform the analysis, and see that we do not have enough information.

Then we could continue to interview people from other departments as well.

Text analysis (content analysis)

In qualitative research, text is essential. There are 2 streams of thinking:

1. Content analysis involves quantification of qualitative data (e.g. words, gestures, art forms) for the purpose of statistical inferences)

2. Content analysis based on grounded-based theory (the constant comparative method – invented by Glaser and Strauss)

Hypothesis testing or Theory Development?

Quantification of qualitative data: Text analysis is used for counting entities such as words, phrases, codes within blocks of texts.

• Entity (item) counts are aggregated in categories or themes
• Categories/themes are used to make inferences about contexts (social, business)
• Text analysis is also used to depict relations between categories/themes
• Kwalitan software: e.g. matrices of codes by codes, words by words -- example: matrix words by words (using Kwalitan demo)

See slide 11 and 10: an illustration of words aggregated in themes. Words are counted.

Grounded theory: also used in qualitative research methods. The discovery of the Grounded Theory (1967) was done by Barony Glaser and Anselm Strauss. They wrote a book on methodology, after their success with ‘Awareness of Dying book’

It is also called ‘Constant comparative method’:

- inductive generation of hypotheses and theories from empirical data

- no preconceived theory prior to and during theory building

- aims at developing increasingly abstract and complex conceptual structure (s)

- emerge is essential

Other qualitative methods, such as case study, action research will be presented in Lecture 4

After first data analysis step, additional data collection stages may occur through additional interviewees, additional groups of people, additional decision concerning methods for data collection. See slide 14 for an example of a text analysis

Qualitative (text) analysis is based on assigning meanings to “things”!
Catching signals à combining/summarizing à pattern à possible connections à possible insights à new signals à etc.

There are two approaches of how to perform a text analysis:

1. Having at hand an existing list with predefined concepts: existence of preliminary theory: predefined themes (words, codes etc. are found to support these themes and eventually new concepts emerge).

Imagine that you know a-priori (from theory) that communication is an important factor responsible of data sharing in companies. You will read the text trying to identify all words/expressions/etc. which are related to the concept of communication.

2. Starting completely from scratch, there are no predefined concepts. There is no preliminary theory. Ground Theory: all themes/concepts emerge by coding and developing new themes (categories).

There are concepts relevant to your research objective that you simply do not have a clue about: you ‘dive’ into available text to identify new aspects.

Theoretical coding: the process of coding and categorizing qualitative data with the goal of developing a theory in an emergent manner.

Consists of 3 procedures (Strauss, 1984)

1. Open coding: coding and developing categories of related codes

2. Axial coding: relating codes (or categories) to each other, by combining inductive and deductive thinking

3. Selective coding: building a story line that connects the categories. It results a theory (framework, conceptual model, set of hypotheses)

Open coding:

Step 1: Establish segments (data fragmentation)

• Units of meaning: line-by-line, sentence-by sentence, paragraph-by-paragraph

Step 2: Assign codes within segments

• ‘Coding’ means assigning meaning to data, in terms of constant comparative method
• A code can refer to: one word, two (more) words, sentence, paragraph, etc.

Step 3: Group and order codes

• Related codes are grouped in categories
• Term(s) used by respondents – “in vivo” category
• Higher level categories and their relations form a theory

Axial coding:

• relating codes (or categories) by identifying relations

• a category can be included in a broader category

• a category can be collapsed or combined with other categories

• a category can be dropped

High level categories are candidates for high level concepts (or factors)

Selective coding:

• Takes place after specifying categories and their connections/relations
• Select the most important ‘core’ categories
• Eventually continue the research cycle around these concepts until saturation is reached
• Build a story line that connects the categories

Thematic coding: In different social worlds or groups different views can be found. It is used to compare group or cases with different perspectives

• multi-stage procedure
• from each group of interest, a case is selected

1. First, one case is analysed - open coding is used first and then selective coding is used to generate thematic domains and categories.

2. Next, developed categories are cross-checked with other cases; eventually new categories will emerge

3. The result is a thematic structure

Components of a conceptual model: It is a visual representation of a theory:

• concepts (with factors)
• relationships (what kind of relation)

Why and when is not represented, this has to be clarified in the text.

How to test a conceptual model: predictions (propositions / hypotheses)

Article discussion: A. Ordanini, L. Miceli, and M. Piazzetti, Crowd-funding: transforming customers into investors through innovative service platforms, Journal of Service Management, Vol. 22, No.4, 2011, pp. 443-470.

Phenomena not understood:

• Why consumers become investors?
• What functions service providers play?
• Which roles do consumers play?

Crowd-funding: “collective effort by people who network to pool their money together, via the Internet, in order to invest in and support efforts initiated by other people and organizations” (Ordanini, 2009)

Illustrative for…

• Process of qualitative research
• Research design (sample, data collection, data analysis methods
• Research dimensions
• Data collection and data analysis
• An example related to your assignment
• Regulative cycle (Lecture6)

Nestor -> Assignments -> crowdsourcing/crowd-funding materials

Lecture 4

What is case study research? (Yin 1981, 2009)

• Research technique aimed at examining a contemporary phenomenon in its natural setting

• Especially relevant when boundaries between phenomenon and context are not evident

Types of case study research:

• Descriptive Case Study: To describe the incidence or prevalence of a phenomenon of interest - “How have the attitudes of governmental institutions towards citizens changed over time?”

• Explanatory Case Study: To trace causal linkages among actions, decisions and events over time - “Why do organizations decline?”

• Exploratory Case Study: To develop pertinent hypothesis and propositions for further inquiry - “How can university education be rendered more effective?”

Five Elements of Case Study research:

1. Designing the case study

2. Preparing for data collection

3. Collecting data in the field

4. Evaluating and analyzing data

5. Sharing the findings / writing the report

1. Designing case studies:

1. What are the questions of the study?

• Case studies focus on “how” and “why” questions

• Look at previous literature to narrow down your questions

2. What are the propositions of the study (if any)?

• Clear statements you want to test

• Point at specific things that should be examined

• Not if exploratory study

   

3. What is the unit of analysis?

• What is the “case”?

• At what level do you need information to provide answers to your questions?

• Research questions should give you clear indication. If not: bad questions!

4. How can we link the data to the propositions?

• What type of analysis methods can we use to obtain insights?

5. What are the criteria to interpret the findings?

• No traditional statistical proof!

• Try to find rival explanations for your findings

Role of theory

• Case studies build on existing theory to come up with research questions

• Case studies test existing theory

• Case studies build new theory - Not by statistical proof - By analytical generalization

Selecting case study designs:

Single- or Multiple-Case designs?

• Single case: artifacts of specific situation?

• Better use multiple cases

Possibilities of direct replication

Possibilities of contrasting situations

Closed or Flexible designs?

• Data collection provides new info

• New info may require adjustment of setup

• But: be cautious and careful!

Mixed methods designs?

• Combining different methods in one single study

Quality of case study designs:

Four tests, and tactics to apply

• Construct validity:

1. Multiple sources of evidence
2. Chain of evidence
3. Key informants review the draft report

• Internal validity (only explanatory studies):

1. Pattern matching
2. Explanation building
3. Rival explanations
4. Logic models

• External validity:

1. Use theory in single-case studies
2. Use replications in multiple-case studies

• Reliability:

1. Case study protocol
2. Case study database

2. Preparing for data collection: What skills should a good investigator have?

• Ask good questions

• Be a good “listener”

• Be adaptive and flexible

• Have a good understanding of the issues that are being investigated

• Be unbiased by prior knowledge and ideas; sensitive and open to new information

What is a case study protocol?: Written document to be used as a guide during the case study  Keeps the researcher focused and forces anticipation of certain problems

What should be the elements of the protocol?

• Overview of the project - Background info, objectives, issues, relevant readings.

• Field procedures - Credentials, sites, language, source of data, procedural reminders

• Case study questions - What should the researcher focus on when collecting data, together with potential sources of information for each question.

• Guide for the final report

Selecting cases

Site Selection:

• Geographical location

• Business parameters

• Market segments

• Extreme ends of phenomenon of interest

Two-stage approach:

• Collection and analysis of quantitative data to get about 20-30 possible candidates

• Get more qualitative insights on these candidates, and then choose the case(s) for your study

• Do not create mini-cases for each!

Pilot case

Selection of the pilot case:

• very accessible people

• Convenient location

• Unusual amount of information

• The most complicated case of all - Any possible issue can be found there

Goals of the pilot case:

• Provides clarification of concepts

• Helps in refining the research plan and questions (>< pretest)

• Helps with narrowing down the research focus (hence: broader than final data collection)

• Helps with solving methodological issues

3. Collecting data

Six main sources of data

• Documents (see also next session)

• Archives (see also next session)

• Interviews

• Direct observation (see also next session)

• Participant-observation (see also next session)

• Physical artifacts

1. Technological instruments, tools, devices, art works, computer print-outs…
2. Perhaps somewhat less important?

Principles of data collection

1. Employ multiple sources of data

• Single source

Problems of accuracy and trustworthiness

• Triangulation

Rationale for utilizing multiple sources of empirical evidence

Advantage: Construct validity

Disadvantages: - Expensive - Time Consuming - Requirement for Diverse Skills

2. Create a case study database

• Separation between gathered evidence and final report

• Increases reliability

• Database Content

Documents, notes, quantitative data, narratives etc.

• Database should be accessible to external parties

3. Maintain a chain of evidence

• To enable an external observer to follow the derivation to any evidence

• Increases reliability

• Trace steps

From research questions to conclusions and vice versa

4. Evaluating and analyzing data: Before ‘really’ analyzing the data: stare and glare, play with them

• Put info in different arrays

• Make atrices of categories & place evidence in these categories

• Create displays, graphs

• Tabulate the frequency of events

• Provide descriptive statistics

• Put info in chronological order

Analysis strategies

Four main strategies to analyze the data:

1. Rely on theoretical propositions:

Bases for your research

Have shaped your data collection

2. Develop a case description:

Alternative when the first strategy does not work

e.g. lots of data without clear set of research questions or propositions

Interesting when original objective was descriptive

Can help to identify the appropriate causal link to be investigated

3. Combine qualitative and quantitative data:

Not either/or, but and/and

4. Examine rival explanations:

Can be combined with all three previous strategies - Rival hypotheses (1) - Rival descriptive frameworks (2) - Data from other groups (3)

Types of rivals

• Craft (no-real-life) rivals -Null hypothesis / Validity threats / Investigator bias

• Real-life rivals - - Direct rival / Commingled rival / Implementation rival / Rival theory / Super rival / Societal rival

Threats to Validity: Effects that take place during the study - Mrs. Smith

• Maturation: Physiological processes occurring within participants that can affect their behaviour

• Regression to the mean: Participants with extreme scores when tested first time, tend to have less extreme scores on subsequent retesting

• Selection of subjects: Bias in selecting and assigning participants to groups such that systematic differences between participants in each group exist

• Selection by Maturation interaction: Treatment and No-treatment groups, similar at start, have developed differently, even without any treatment

• Mortality: Subjects dropping out of the groups before the experiment is finished resulting in differences between groups unrelated to the treatment effects

• Instrumentation: Changes in measurement procedures may result in differences between groups that are confused with treatment effects

• Testing: After repeated testing of participants, changes in test scores may be the result of practice or knowledge of the procedure rather than treatment effects

• History: Extraneous events occurring during the course of the experiment that may affect the participants’ responses on the dependent measure

Analysis techniques

Five possible techniques to analyze the data

1. Pattern matching

Nonequivalent dependent variables

Rival explanations as patterns

Simpler patterns

2. Explanation building

Build an explanation about the case

“how” or “why” something has happened

Iterative in nature

Builds on well-developed analytic insight of the researcher

3. Time-series analysis

Simple or multiple

Trends, structural changes

Chronologies - Putting events in chronological order and covering many types of variables: can be richer and more insightful

4. Logic models

Deliberate stipulation of a complex chain of events over an extended period of time

Cause-effect-cause-effect patterns

Can be at individual / organizational / … level

5. Cross-case synthesis

When dealing with multiple-case setup

Cases part of the same or different studies

Can you find patterns across these different cases when similar conditions?

Can you find the opposite outcomes in cases with opposite conditions?

5. Preparing the report:

Goal of Case Reporting:

• Convey vicarious experience to readers about the phenomenon of interest

• Render solutions to problems publicly accessible

• Facilitate readers to apply the experience in real-life situations

Components of ‘Good’ Case Reporting:

• Displaying sufficient evidence for claims

• Communicating case boundaries

• Paying special attention to conflicting propositions

Techniques for Case Reporting

• Chronological Recounting

• Story Telling

• Reader-Centric

Characteristics of case study research: Most important characteristics

• Phenomenon of interest is examined in natural setting

• Data collected via multiple means

• One or few entities (e.g., person, group or organization) is examined

• Complexity of the unit is studied intensively

• Investigator(s) should have a receptive attitude towards exploration

• No experimental controls or manipulation are involved

• Investigator(s) may or may not specify the set of dependent and independent variables in advance

• Derived findings depend heavily on the integrative powers of investigator(s)

• Changes in site selection and data collection methods could take place as investigator(s) develops and/or updates hypotheses

• Useful in the investigation of “how” and “why” questions

• Focus on contemporary events

Advantages / Disadvantages of case study research depend on:

• Type of research question(s) that can be answered

• Control over actions, decisions and events

• Focus on contemporary phenomena as opposed to historical ones

Preferred research technique when:

• Answering “How” and “Why” questions

• Little control over actions, decisions and events

• Focus on contemporary phenomena within natural settings

Action research: Interactive inquiry process that balances collaborative problem solving with data-driven analysis to understand underlying causes and facilitate future predictions.

The researcher deliberately intervenes while at the same time studying the effect of that intervention.

Action research is a cyclical process with 5 phases:

• Diagnosing - Identification of problem(s) and issue(s) related to phenomenon of interest

• Action Planning - Development of an action plan to specify activities for resolving identified problem(s)

• Action Taking - Implementation of planned actions

• Evaluating - Assessment of actions taken and their impact on identified problem(s)

• Specifying Learning - Consolidation of knowledge from preceding phases to determine future course of action

Research is action research if the following elements are present (to some degree):

• Purpose and value choice - Scientific enquiry and practical problem solving

• Contextual focus - Research in a ‘real-world’ setting, focus on wider context

• Change-based data and sense-making - Data allow for tracking the consequences of changes, and for their interpretation

• Participation in the research process - Involving the people dealing with the ‘real-world’ problem at hand

• Knowledge diffusion - Translating findings into scientific literature, and linking to existing literature

Advantage of Action research:

• Helps to ensure practical relevance - Not too theoretical, but direct impact

Disadvantages of Action research:

• Difficult to combine action with research

• Tendency to overstate the importance of the intervention

• Tendency to overstate the contribution to academic research

Lecture 5

Observational research: Method of data collection by noting a phenomenon, often with instruments, in its natural setting, while recording it for scientific purposes.

Large set of different methods:

• Personal observation

• Mechanical observation (Scanner - Audimeter - (eye)camera)

• Audit

• Trace analysis

Classification of observation:

• Structured versus unstructured: How prespecified is the observation?

• Disguised versus undisguised: Bias?

• Natural versus contrived: Own kitchen or test kitchen? (Dis)Advantages of natural?

• Which are true for the Kitchen Stories example?

Roles of the observer:

Complete Observer:

• Scrutinizes the phenomenon of interest without direct physical intervention or interaction - Watching other people ‘from the outside’ - People are not aware of the fact that they are being observed

• Ideal objectivity: no possibility that results are affected by actions of observer

• Ethical issues

Observer-as-Participant

• People know that they are being observed

• Observer only has a role as researcher

• No direct action by researcher: only observation

Participant-as-Observer

• Takes part in the action (up to a certain level), creates connection with the people observed

• Actor and researcher

• People still aware of research focus

Complete Participant

• Observer is fully engaged and takes full part in the action

• People are not always aware of (hidden) research agenda

• Ethical issues

Levels of membership: Researcher may adopt different levels of engagement in community when doing participant observational research

• Peripheral membership:

1. Close interaction with observed people
2. Considered insiders
3. No participation in core activities

• Active membership:

1. Close interaction with observed people
2. Considered insiders
3. Active participation in core activities
4. No commitment to group’s values, goals, attitudes

• Complete membership:

1. Researcher becomes active and engaged member
2. Advocates positions of the group

Observational elements:

• Space – physical place or places

• Actor – people involved

• Activity – set of related acts that people do

• Object – physical things that are present

• Act – single actions that people do

• Event – a set of related activities that people carry out

• Time – sequencing that takes place over time

• Goal – things people are trying to accomplish

• Feeling – emotions felt and expressed

See slide 16 for an example

Trace analysis: People leave traces!

• E.g. tiles erosion: signs of popularity

• Radio dials at car service

• Shopping patterns in supermarkets

• Browsing behaviour on the Internet - Tracking cookies!

Quality of observational research: 5 main criteria:

• Objectivity - Do conclusions come from the data, or from biases induced by the researcher?

• Reliability - Has the research process been consistent and stable over time and across methods and researchers?

• Internal validity - Do the conclusions make sense and are they credible?

• External validity - Are the conclusions relevant beyond the study itself?

• Utilization - Can the findings be used for actions?

Advantages of Observational Research:

• Ability to observe phenomenon in natural setting

• Avoids possibility of self-censorship

Disadvantages of Observational Research:

• Access

• Labor intensive

• Possible bias induced by researcher

But: triangulation is possible!

Documents: Many different types: Written materials, Pictures, Diagrams, Photographs, Videos, Television programmes, Websites/electronic documents, Software etc.

Quality of documents: Four main criteria

• Authenticity - Is the object what it claims it is?

• Credibility - To what extent can the author be believed?

• Representativeness - To what extent is the document a good sample of a larger set of documents?

• Meaning - How should the document be interpreted and understood?

Advantages of documents:

• Relatively cheap

• Easy to access

• Make things visible and traceable

Disadvantages of documents:

• Some types of documents are hard to get access to

• Assessment of the authenticity, credibility, representativeness and meaning can be difficult - E.g. when no access to the original author

See slide 30/31 Huge amounts of money invested in advertising:

Good insights about returns to adspend, but what about content?

• Much anecdotal evidence

• Much experimental evidence on “soft” outcomes - Mainly on overlap - Some on variation

• No longitudinal evidence, no evidence on “hard” outcomes

Should brands try to be consistent in their message over time?

Should brands try to be different from competitors?

Findings of Linguistic Inquiry and Word Count (LIWC2007) (Pennebaker et al. 2007):

It pays off to clearly link the product to the category, and to resemble your competitors in that sense

• Not just once, but in a sustained way

When positioning the product, it is important to be clearly different than competitors

• What makes the product so unique? - Look into those product characteristics that do matter to customers, and stress unique features

When positioning the brand, it is important to be consistent over time

• What is the brand identity/image? - What are the personal concerns of customers the brand appeals to?

Internet and Social Media:

Enormous amounts of information are out there

• People leave traces - Browsing behaviour on the Internet

• People express opinions

1. Review websites
2. Consumer forums
3. Social media sites
4. Fan/hate sites

Be aware of possibly biased information!

New opportunities › Usage of Internet focus groups/panels

• Testing new product concepts

• Testing design of new websites

Same panel selection criteria as in ordinary focus groups/panels apply.

Lecture 6

Science = (from the Latin scientia, meaning "knowledge") builds and organizes knowledge in the form of testable explanations and predictions about the natural world.

Astronomy: is a natural science that is the study of celestial objects.

Astrology: is the belief system which claims that human affairs are correlated with the positions of celestial objects.

Going beyond the difference qualitative/quantitative, at a higher level, when choosing a research approach the choice is also about the nature of scientific research:

1. applied scientific research(design-focused)

• problem-solving questions

• specific for one situation

• interventions

• structure: regulative cycle(based on Van Strien, 1997)

Imagine the following business problem- management question: “Why customer data is not yet actively shared and used, despite the large amount of available data within company A?”

apply regulative cycle.

In the design phase we are finding solutions to solve the problem.

For instance, the solution chosen is to facilitate frequent meetings with project managers.

The management decided to organize meetings on weekly basis between all project managers of all 8 departments.

As implementation, the management offered the possibility of a (free) lunch during 45 minutes, where all incoming customer data are plenary discussed with project managers

After 6 months, this initiative is evaluated and checked whether indeed, customer data is appropriately shared and whether (new) knowledge emerged between departments.

2. theoretical scientific research (theory-based)

• general explanatory questions
• applicable to more than one situation
• goal: theories (explanation)
• structure: empirical cycle (De Groot, 1969)

Example: “What is the influence of national cultures in managing multinational companies” (Hofstede)

1. We observe that employees from, for instance, Asiatic countries behave differently than employees from Western countries. One of the dimensions considered by Hofsteade is “power distance”.
2. We may observe that the attitude versus authority is different from country to country. We may also observe that countries share the same geographical area (and/or history) have comparable attitude towards authority.
3. We can then induce that indeed, there can exist such a dimension related to the attitude towards authority, which may explain why managing companies in Asia is different than managing them in Europe. Going further, we identify more relevant dimensions. We say that we induced a theory.
4. Next we deduce empirical testable hypotheses
5. and then we test these hypotheses. T
6. Testing hypotheses can either lead to accepting or rejecting hypotheses (evaluation)  a new cycle may occur.

Quality criteria in research:

Validity: do I measure what I want to measure?

Reliability: repeated measurement: will I find the same values?

How to assess validity & reliability in qualitative research?

translates into: How to really convince that qualitative reasearch methods are not biased or innacurate or imprecise? (Miles and Huberman, 1994, pp. 277)

Assessing validity & reliability in qualitative research (Flick, 2009):

• applying traditional validity and reliability criteria, if possible

• reformulating traditional validity and reliability criteria

Reliability: the process of the study is consistent, reasonably stable over time, across researchers and methods, and replicable.

Criteria used in qualitative research:

• Procedural reliability (text analysis/ interviews)

• Inter-rater reliability

Procedural reliability (a classic approach)

• Use low-inference descriptors (Record/ document as concrete and detailed as possible, make the research process as transparent as possible (concerning methods, data and theory)

• Standardise procedures in measuring instruments (interviews/ field notes/ data analysis)

Procedural reliability – Text analysis:

• Reliability issues arise in coding/categorizing

• Codes/categories are developed in a standardized way (themes, sub-themes, topics etc.)

Procedural reliability – Interviews:

At respondent site: understanding questions in the same way

At interviewer site:

• pre-testing interviews

• training interviewer(s)

• Using (semi-)closed questions

• Using inter-rater reliability

• Using low-inference descriptors

• Recording all face-to-face interviews (accurate transcription)

• Presenting (long) extracts of data into the research report

   

Inter-rater reliability:

• Same data given to a number of analysts

• The analysts analyze the data according to an agreed set of codes/categories

• Their reports are examined and any differences are discussed and ironed out

Validity: What do you say that you have seen, is it actually happening in reality?

What is reality? (see textbook chapter 1) and how is reality approached in your findings?

3 errors:

• Type 1: To see relations/concepts when they do not exist

• Type 2: To neglect relations/concepts when they exist

• Type 3: To ask the wrong questions

Validity types

• Content validity: Do we measure what we want to measure?

• Internal validity: (credibility/authenticity): Do the finding of the study make sense in context of the study? Are they credible to the people we study and to the readers of the research?

• External validity: (transferability/generalizability): Are the finding transferable to other contexts? How far can they be generalized?

• Procedural validity: - In production of the data

In presentation and interpretation of the phenomenon

How do we ensure procedural validity?

In case of interviews (based on Wolcott, quoted in Flick 2009):

• Do not talk yourself too much but listen

• Take exact notes

• Write early, while facts are still fresh

• Show notes to the interviewee

• Provide a complete and detailed report

Other criteria for validity:

• trustworthiness/credibility - by triangulation

• transferability - by generalization of results

By reporting with sufficient amount of detail and providing arguments may increase both: validity and reliability.

Triangulation: combining several qualitative methods or several qualitative methods within or not within the same case.

Types (Flick, 2009):

• Data triangulation: different data sources are used

• Investigator triangulation: different investigators are used to analyze data

• Theory triangulation: different perspective/hypotheses are used

• Methodological triangulation: - Within method: e.g. use of different subscales for measurement of an item in a questionnaire

Between method: e.g. combination of a survey method with a semi-structured interview

Triangulation: validity assessment tool: if findings obtained by all these methods correspond and draw similar conclusions, the validity of those findings and conclusions has been established.

Problem: accounts are ‘situated’ in particular contexts: Therefore methods, drawn from different theories, cannot give an ‘objective truth’.

Triangulation is not a way of obtaining ‘true’ reading but rather a strategy that adds rigor, breath, complexity, richness and depth to any inquiry.

Generalization in qualitative research: developing a theory involves the generalization of concepts and relations.

Problem: findings are often bounded to a specific context

Choice of cases (sampling) is important:

Again: Constant comparative method (grounded theory)

Generalization in qualitative research is the gradual transfer of findings from one particular case study to more general relations.

However:

• Generalization is simplifying our understanding about a phenomenon

• Generalization is a push towards breath, and qualitative research is often more concerned with depth

• Generalization should be used when it seems to fit the research aims. Not a precondition in qualitative research.

See slide 32: Figure: Target population, operational population, sampling frame, sample

Sampling in qualitative research (See lecture 2: Research design)

Qualitative research: mostly non-probability sampling (units are not statistically representative)

• Most deliberate: purposeful sampling = criterion based

Homogeneous and heterogeneous sampling, typical case sampling, extreme case or deviant sampling

• Convenience sampling (ease of access) & Opportunistic sampling (as events arise)

• Theoretical sampling: type of purposive sampling. Cases are selected to the base of development of new concepts.

In connection with grounded theory (in GT, theory emerges from qualitative data).

after first data analysis step, additional data collection stages may occur: more interviewees, more groups.

Data analysis and data collection continues, until theoretical saturation is reached.

Combining qualitative and quantitative research:

Sequencing: different approaches in different phases of the research.

e.g. qualitative in Problem Analysis and Conceptual Analysis (formulating hypotheses) and quantitative in Empirical Analysis (testing hypotheses)

Triangulation: for validation purposes. Combining several qualitative methods or several qualitative and quantitative methods.

Within or not within the same case

Quantitative methods (data: numbers): survey, experiment, simulation, structured interview, statistics as secondary data

Qualitative methods (data: text): action research, case study, grounded theory, narratives.

Lecture 7

Lecture 7 was a wrap-up, no more extra information included then in the lectures before (just a summary). There was no guest lecture.

Difference between text book and lecture slides:

• Sometimes terminology is slightly different but same notions:

Book (p.28-33): functions of qualitative research: contextual, explanatory, evaluative, generative.

Lecture 2 – slide 25: research goals: description, exploration, explanation

• In textbook they distinguish between:

Essential knowledge: concepts, notions, theories, models, methods...

know this!

Support knowledge: explanation, examples, illustrations..

Be able to provide comparable explanations in your own words!