Https www youtube com watch v bgroitwkbhu

Evidence-based Management

MGMT 7250

TOPIC 3: Acquiring and appraising professional evidence

Plan for the day

• Managers and their professional expertise • Bias and noise in professional judgement • How to acquire and appraise professional evidence

Managers and their professional expertise & intuitive judgements

Professional expertise in management

• Cumulative specialised knowledge and skills used to solve professional problems and issues • Differs from personal intuition and judgement which is a more instinctive line of reasoning. • Expertise is best developed through: • Prolonged practice • Highly controlled conditions and environments • Regular direct and objective feedback

Activity

• Think of: a. A question/Questions you can ask of a professional expert b. What sort(s) of professional experts could you ask question(s)? c. How would you ask them the question(s)?

Types of questions you can ask Example Questions about non-effects Needs: What do people want or need? Attitude: What do people think or feel? Experience: What are peoples’ experiences? Prevalence: How many / often do people / organizations ...? Procedure: How can we implement ...? Process: How does it work? Economics: How much does it cost?

Critical thinking in the context of professional expertise

Descriptive/exploratory

What is the question Descriptive question (e.g. what is, how many, to what degree….)

What are the alternative answers/claims to answers?

What do professional experts see as alternative descriptions?

What support exists for each alternative claim?

What is professional opinion on which description is more likely to be true

How good is support for each claim?

Critical appraisal of this professional opinion

Professional expertise in management

• Expertise in management is hard to develop given the nature of management decision problems, issues and opportunities. • By extension is management expertise and knowledge illusory and mythic? • Meta-analytic evidence suggests that job experience does not correlate well with job success (Schmidt et al., 1988).

Judgement in management

• Managers have to make decisions all the time, some decisions are more complex than others. Managers often make decisions under uncertainty that require judgements about: ØPrediction and probability ØValue ØCause and effect relationships

• Intuition, professional expertise, attitudes, beliefs, and knowledge all form the basis of judgements (Tversky and Kahneman, 1973). • Making these decisions based on only a managers’ professional expertise and intuitive judgements alone may be commonplace, however this is highly problematic given the nature of management expertise and intuitive judgement.

The limits of intuitive judgement • Bounded rationality • Meta-cognitive bias where they overestimate their own rationality • Decision neglect: limited search and appraisal of information, incomplete survey of alternatives, selective attention and not drawing on the full range of resources and information available to make a decision

Fast, Intuitive, Impulsive, Heuristics, Emotional

Slow, Deliberate, Rational

System 1 (Dominant) System 2

System 1 & System 2 Thinking

Bias & noise in professional judgement

Bias vs Noise • System 1 thinking produces bias and noise in decision- making and reflects the fallacies in unaided human judgement and intuition. • Bias refers to systematic inaccuracy in decisions and can be divided into cognitive and social bias. • cognitive bias where inferences about people and events are drawn in an illogical fashion. • social bias which involves judgements against people and cases on the basis of categories that we assign them to. • Noise refers to unsystematic inaccuracy in decisions – random and due to chance variation and thus more difficult to mitigate.

• https://www.youtube.com/watch?v=8YyWbOAu6S8

• Identify the cognitive bias that this campaign aims to address or ameliorate, does it represent system 1 or system 2 thinking?

Activity on overconfidence

Purpose of this activity is to development a better understanding of the role of overconfidence in decision-making. Instructions: • Think about a time when you (someone else) had over-estimated your (their) abilities. • What was the situation? What happened? What information do you wish you had before you acted?

EBP capabilities: Meta-cognitive skills, communication skills

Meta-cognitive bias

Confirmation bias

Availability bias

Illusory of causation/Patternicity

Overconfidence

Outcome bias

Optimism bias

Overgeneralisation

Primacy and recency effects

Cognitive Biases

Gender stereotyping

Attribution bias or self-serving bias

False consensus effect

Similarity bias

Conformity and Groupthink https://www.youtube.com/watch?v=BgRoiTWkBHU

Halo effect

Social Biases

According to Lewandowsky et al., 2012:

ØAssumption often made that coherent information received is truthful and relevant – default of acceptance.

ØInformation is accepted if it is consistent with one’s beliefs and knowledge – cognitive consistency effect

ØInformation is accepted if it is perceived as being part of a coherent story – face validity or when it is from someone who is perceived as credible or an authority (authority bias)

ØInformation is accepted when it is perceived to be believed by a consensus or majority – a false consensus effect

ØWhat makes matters worse is that these processes also mean that misinformation can be quite resistant to correction and retraction referred to as the continued influence effect

Cognitive processes involved in the acceptance of misinformation

How to mitigate bias?

1. Taking an evidence-based approach to decision-making and practice helps neutralise bias as suggested by research on bias (Kahneman, 2011).

2. Bayesian thinking: ØBeliefs can sometimes be too dichotomous , ie you believe or you don’t believe.

ØBayes’ Rule tries to get around this fallacy by prescribing degrees of beliefs which precludes having absolute certainty in anything.

ØHaving absolute certainty about something means that beliefs cannot be revised in light of new information.

How to acquire & appraise professional evidence

Types of questions & professional evidence

• Different types of evidence generated through different types of research are methodologically appropriate to address different types of questions. • The questions that need to be asked to acquire evidence from professionals to identify problems and solutions tend to be (and need to be) more exploratory prompting the use of qualitative research methods

The image part with relationship ID rId4 was not found in the file.

Understanding reliability • Methods used to acquire and obtain information determine how valid and reliable it is. • Managers should assume that any information they gather, whether it be from professionals, information systems, scientific studies, and stakeholders for example contains a degree of measurement error or inaccuracy, some of this error can be quite systematic (bias) and others quite random (i.e noise). • In acquiring professional evidence you are measuring attitudes, attributions, beliefs, and ?? • Always systematic and unsystematic error in the measurement of these psychological constructs. Classical Test Theory:

Observed score = True score + error

Limitations (error) of professional evidence

Know the limitations of professional evidence (systematic error): •Cognitive and social biases •Sampling bias – limits of professional expertise ØSmall n ØIdiosyncratic rather than generalisable

Who do you ask?

• Knowing who to ask is important • Credibility and dependability

• The more people you ask the better

Methods to acquire professional evidence

Interviews: •Unstructured •Semi-structured *always think about Emic validity – ‘getting inside their heads’

Questionnaires: •Open-ended •Closed-ended – lower levels of systematic error if tested scales are used

Methods to acquire professional evidence

Group methods: •Delphi method •Focus groups