Evaluation · 2-4 weeks

ROMPER (RAND/USC OPTIC Method for Policy Expert Ratings)

A validated, open-access approach specifically designed to elicit and summarize expert views on the effectiveness and implementability of population-level interventions. It provides a replicable framework for translating expert consensus into actionable policy data.

When to use ROMPER (RAND/USC OPTIC Method for Policy Expert Ratings)

When needing to prioritize multiple intervention options based on their likelihood of success and ease of implementation.

How ROMPER (RAND/USC OPTIC Method for Policy Expert Ratings) works

Instructional designers can use this method to evaluate the potential impact and feasibility of various training interventions by having subject matter experts rate proposed solutions before full-scale development.

Phases of ROMPER (RAND/USC OPTIC Method for Policy Expert Ratings)

  1. Identification of Policy/Intervention Options: The research team defines the set of population-level policies or interventions to be rated and recruits a balanced panel of experts. Each option is described clearly so experts rate the same thing.
  2. Online Expert Rating (Effectiveness and Implementability): In a three-round online modified Delphi, experts rate each policy on effectiveness and implementability using GRADE Evidence-to-Decision criteria and add comments. Round two is an anonymised discussion of results; round three is a re-rating.
  3. Consensus Summarization: Ratings are analysed with the RAND/UCLA IPRAS method to determine agreement and shown as forest plots, and comments are analysed thematically. The result shows which policies experts agree are, or are not, effective and implementable.
  4. Policy Recommendation Reporting: Each policy is summarised in a Policy Profile adapted from GRADE Evidence-to-Decision tables, and materials and data are shared openly. Decision makers use these profiles to discuss options.

Key principles

  • Validated rating scales
  • Dual-focus on effectiveness and implementability
  • Replicable and transparent methodology
  • Expert elicitation

Best for

  • Impact evaluation
  • Feasibility studies
  • Prioritizing organizational interventions

Considerations

  • Requires clearly defined intervention options to be rated
  • Success depends on the diversity and expertise of the rating panel

Attribution & sources

Developed by RAND and USC Schaeffer Center (OPTIC), 2024

Primary source

Design a session with ROMPER (RAND/USC OPTIC Method for Policy Expert Ratings)

METODIC builds a complete ROMPER (RAND/USC OPTIC Method for Policy Expert Ratings)-based session — methods, timing, worksheets, and facilitator guides — in minutes.

Try METODIC free