IntervalWAgg.RdCalculate one of several types of linear-weighted best estimates where the weights are dependent on the lower and upper bounds of three-point elicitation (interval widths).
IntervalWAgg(
expert_judgements,
type = "IntWAgg",
name = NULL,
placeholder = FALSE,
percent_toggle = FALSE
)A dataframe in the format of data_ratings.
One of "IntWAgg", "IndIntWAgg", "AsymWAgg", "IndIntAsymWAgg", "VarIndIntWAgg",
"KitchSinkWAgg".
Name for aggregation method. Defaults to type unless specified.
Toggle the output of the aggregation method to impute placeholder data.
Change the values to probabilities. Default is FALSE.
A tibble of confidence scores cs for each paper_id.
The width of the interval provided by individuals may be an indicator of certainty, and arguably of accuracy of the best estimate contained between the bounds of the interval.
type may be one of the following:
IntWAgg: Weighted according to the interval width across individuals for that claim, rewarding narrow interval widths. \[w\_Interval_{i,c}= \frac{1}{U_{i,c} - L_{i,c}}\] \[\hat{p}_c( IntWAgg) = \sum_{i=1}^N \tilde{w}\_Interval_{i,c}B_{i,c}\]
where \(U_{i,d} - L_{i,d}\) are individual \(i\)'s judgements for claim \(d\). Then
IndIntWAgg: Weighted by the rescaled interval width (interval width relative to largest interval width provided by that individual)
Because of the variability in interval widths between individuals across claims, it may be beneficial to account for this individual variability by rescaling interval widths across all claims per individual. This results in a re-scaled interval width weight, for individual \(i\) for claim \(c\), relative to the widest interval provided by that individual across all claims \(C\):
\[w\_nIndivInterval_{i,c}= \frac{1}{\frac{U_{i,c} - L_{i,c}}{max({ (U_{i,d} - L_{i,d}): d = 1,\dots, C})}}\]
\[\hat{p}_c\left( IndIntWAgg \right) = \sum_{i=1}^N \tilde{w}\_nIndivInterval_{i,c}B_{i,c}\]
AsymWAgg: Weighted by the asymmetry of individuals' intervals, rewarding increasing asymmetry.
We use the asymmetry of an interval relative to the corresponding best estimate to define the following weights:
\[w\_asym_{i,c}= \begin{cases} 1 - 2 \cdot \frac{U_{i,c}-B_{i,c}}{U_{i,c}-L_{i,c}}, \text{for}\ B_{i,c} \geq \frac{U_{i,c}-L_{i,c}}{2}+L_{i,c}\cr 1 - 2 \cdot \frac{B_{i,c}-L_{i,c}}{U_{i,c}-L_{i,c}}, \text{otherwise} \end{cases}\]
then,
\[\hat{p}_c(AsymWAgg) = \sum_{i=1}^N \tilde{w}\_asym_{i,c}B_{i,c}.\]
IndIntAsymWAgg: Weighted by individuals’ interval widths and asymmetry
This rewards both asymmetric and narrow intervals. We simply multiply the weights calculated
in the "AsymWAgg" and "IndIntWAgg" methods.
\[w\_nIndivInterval\_asym_{i,c} = \tilde{w}\_nIndivInterval_{i,c} \cdot \tilde{w}\_asym_{i,c}\]
\[\hat{p}_c( IndIntAsymWAgg) = \sum_{i=1}^N \tilde{w}\_nIndivInterval\_asym_{i,c}B_{i,c}\]
VarIndIntWAgg: Weighted by the variation in individuals’ interval widths
A higher variance in individuals' interval width across claims may indicate a higher responsiveness to the supporting evidence of different claims. Such responsiveness might be predictive of more accurate assessors. We define:
\[w\_varIndivInterval_{i}= var{(U_{i,c} - L_{i,c}): c = 1,\dots, C},\]
where the variance (\(var\)) is calculated across all claims for individual \(i\). Then,
\[\hat{p}_c(VarIndIntWAgg) = \sum_{i=1}^N \tilde{w}\_varIndivInterval_{i}B_{i,c}\]
KitchSinkWAgg: Weighted by everything but the kitchen sink
This method is informed by the intuition that we want to reward narrow and asymmetric intervals,
as well as the variability of individuals' interval widths (across their estimates). Again, we multiply
the weights calculated in the "AsymWAgg", "IndIntWAgg" and "VarIndIntWAgg" methods above.
\[w\_kitchSink_{i,c} = \tilde{w}\_nIndivInterval_{i,c} \cdot \tilde{w}\_asym_{i,c} \cdot \tilde{w}\_varIndivInterval_{i}\]
\[\hat{p}_c(KitchSinkWAgg) = \sum_{i=1}^N \tilde{w}\_kitchSink_{i,c}B_{i,c}\]
if (FALSE) IntervalWAgg(data_ratings)