Consultancy in elicitation

My expertise

I have been a leading proponent in the Bayesian statistics community of the use of elicitation to formulate expert knowledge about uncertain quantities.

I led a major research project, funded by the National Health Service, into methods for elicitation. One of the outputs of that research was the most comprehensive survey of the field to date, published by Wiley in 2006.
I am co-developer of the SHELF package of elicitation tools, and have pioneered methods to quantify the imprecision in elicited distributions.
I was an expert member of a Working Group of the European Food Safety Authority which developed guidance on expert elicitation. The guidance forms a part of EFSA's standard procedures in risk assessment.

I can offer various options for training in elicitation. I have two established training courses in the theory and practice of elicitation, and can also arrange mock elicitation sessions to further develop facilitation skills. The next open course will be in January 2018.

My clients

I am currently working with GlaxoSmithKline to develop an online training course for experts who will take part in workshops to elicit their expert judgements.

I have conducted expert elicitations for HMIP (now part of the Health Protection Agency), Risk Management Solutions Ltd, AstraZeneca, Pfizer, the European Food Safety Authority, GlaxoSmithKline and the Centre for Workforce Intelligence. Methods that I developed have been coded into software which has been widely used in the water and rail industries, and I have advised a number of users of those methods.

My one-day general course on elicitation has been given to Evira (Finland), Pfizer (UK), Novartis (Switzerland and USA), Procter and Gamble (USA), the University of Texas at Austin (USA), the University of St Andrews (UK), the Netherlands Forensic Institute (Netherlands), Humboldt University (Germany), Chalmers University (Sweden) and also as an open-entry course at Warwick University.

My three-day intensive course has been given at GlaxoSmithKline (UK and USA), the University of St Andrews (UK) and Chalmers University (Sweden).

Typical challenges

Expert judgement is a crucial part of decision-making under uncertainty. Eliciting the knowledge of experts in the form of probability distributions for uncertain parameters of a decision model is a complex task. If done badly, the resulting probability distributions do not represent the expert's knowledge accurately, may not bear any resemblance to reality and can be dominated by arbitrary choices that the expert might make entirely differently if asked another way. Getting accurate, reproducible and defensible judgements requires careful management of the elicitation by a skilled facilitator. Skill and understanding of group psychology are particularly important when working with several experts (which is often needed to capture the full range of expert opinion). I can offer a comprehensive service as an experienced facilitator in elicitations across a range of areas including nuclear waste disposal, environmental hazards, project costing, manpower planning and pharmaceutical development.

Risk assessment and management is an important field that has special challenges. Experts are often called upon to assess risks smaller than can be informed by real data. Risk typically depends on the occurrence of extreme events, and particularly on the concurrence of several extreme values. From the perspective of elicitation, this poses particular difficulties in assessing the tails of probability distributions and correlations between variables. I can offer some novel methods to address these challenges.

I am happy to consult on these or other problems in elicitation.

Updated: 18 January 2017

Maintained by: Tony O'Hagan