ReasoningWAgg.RdCalculate one of several types of linear-weighted best estimates using supplementary participant reasoning data to create weights.
ReasoningWAgg(
expert_judgements,
reasons = NULL,
type = "ReasonWAgg",
name = NULL,
beta_transform = FALSE,
beta_param = c(6, 6),
placeholder = FALSE,
percent_toggle = FALSE,
flag_loarmean = FALSE
)A dataframe in the format of data_ratings.
A dataframe in the form of data_supp_reasons
One of "ReasonWAgg", "ReasonWAgg2".
Name for aggregation method. Defaults to type unless specified.
Toggle switch to extremise confidence scores with the beta distribution. Defaults to FALSE.
Length two vector of alpha and beta parameters of the beta distribution. Defaults to c(6,6).
Toggle the output of the aggregation method to impute placeholder data.
Change the values to probabilities. Default is FALSE.
A toggle to impute LOArMean instead of ArMean when no participants have a reasoning weight for a specific claim (defaults FALSE).
A tibble of confidence scores cs for each paper_id.
Weighted by the breadth of reasoning provided to support the individuals’ estimate.
type may be one of the following:
ReasonWAgg: Weighted by the number of supporting reasons
Giving greater weight to best estimates that are accompanied by a greater number of supporting reasons may be beneficial. We will consider \(w\_{reason}_{i,c}\) to be the number of unique reasons provided by that individual \(i\) in support of their estimate for claim \(c\).
\[\hat{p}_c(ReasonWAgg) = \sum_{i=1}^N \tilde{w}\_reason_{i,c}B_{i,c}\]
See Hanea et al. (2021) for an example of reason coding.
ReasonWAgg2: Incorporates both the number of reasons and their diversity across claims.
The claim diversity component of this score is calculated per individual from all claims they assessed. We assume each individual answers at least two claims. If an individual has assessed only one claim, there weighting for that claim is equivalent to "ReasonWAgg".
We will consider \(w\_{varReason}_{i,c}\) to be the weighted "number of unique reasons" provided by participant \(i\) in support of their estimate for claim \(c\). Assume there are \(R\) total unique reasons any participant can use to justify their numerical answers. Then, for each participant \(i\) we can construct a matrix \(\mathbf{CR_i}\) with \(R\) columns, each corresponding to a unique reason, \(r\), and \(C\) rows, where \(C\) is the number of claims assessed by that participant. Each element of this matrix \(\mathbf{CR_i}(r,c)\) can be either 1 or 0. \(\mathbf{CR_i}(r,c) = 1\) if reason \(R_r\) was used to justify the estimates assessed for \(c\), and \(\mathbf{CR_i}(r,c) = 0\) if reason \(R_r\) was not mentioned when assessing claim \(c\). The more frequently that a participant uses a given reason reduces the amount it contributes to the weight assigned to that participant.
\[w\_{varReason}_{i,c} =\sum_{r=1}^{R} \mathbf{CR_i}(c,r) \cdot (1 - \frac{\sum_{c=1}^C \mathbf{CR_i}(c,r)}{C})\]
\[\hat{p}_c(ReasonWAgg2) = \sum_{i=1}^N \tilde{w}\_varReason_{i,c}B_{i,c}\]
if (FALSE) ReasoningWAgg(data_ratings)