--- title: "How the armor score is calculated" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{How the armor score is calculated} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") ``` `get.optimal.armor.combos()` ranks armor combinations using a single number, `SCORE` (and its human-readable form, `SCORE_QUALITY`). This vignette explains where that number comes from. `get.armor.tradeoffs()` uses the same score: it's the stat maximized with `metric = "SCORE"`, and the tie-break when maximizing any other stat. ## The problem: metrics on different scales An armor combination is described by ten metrics: physical, strike, slash, and thrust defense; magic, fire, and lightning defense; and bleed, poison, and curse resistance. They vary between combinations by very different amounts: across every combination, slash defense has a standard deviation of about 33, and bleed resistance about 13. Added up raw with equal weights, a point of slash would count as much as a point of bleed, so slash's wider spread would decide far more of which combination comes out on top, not because the `weights` argument said so. (A stat's typical size doesn't matter on its own: adding the same amount to every combination's total changes no ranking. What matters is how far apart combinations are.) The fix is standardization: express each metric in units of "how unusual is this value", not its raw number. `get.optimal.armor.combos()` does this using the population mean and standard deviation of each metric, computed once across every possible armor combination (including every upgrade level) when the package's data was built - see `create_rda/create_rda.R`. These live in the package's internal data as `means`, `stddevs`, and `corrs` (the population correlation matrix across the ten metrics). Because the four slots are chosen independently, these follow exactly from each slot's own means and covariances (the combination's mean is the sum of the slots' means, and likewise its covariance matrix), so no combination ever needs to be enumerated. ## Building a score from weights Given a `weights` vector (one weight per metric, after the function normalizes it to sum to 1), the package computes a set of per-metric scalars: ```r score.scalars <- weights / (stddevs * sqrt((t(weights) %*% corrs %*% weights)[1, 1])) ``` For a combination with combined metric values $x_1, \dots, x_{10}$, the score is: $$ \text{SCORE} = \sum_{i=1}^{10} \text{score.scalars}_i \cdot (x_i - \mu_i) $$ where $\mu_i$ is that metric's population mean (`means[i]`). This is a weighted, centered sum - but the *particular* denominator in `score.scalars` isn't arbitrary. It's chosen so that `SCORE` always has variance exactly 1, no matter what weights are passed in. To see why: let $\Sigma$ be the population covariance matrix implied by `stddevs` and `corrs` (so $\Sigma_{ij} = \sigma_i \sigma_j \cdot \text{corr}_{ij}$), and let $k = \sqrt{w^\top \text{corr} \, w}$ be the normalizing term inside `score.scalars`. Then `score.scalars` $= w / (\sigma k)$ element-wise, and the variance of SCORE works out to: $$ \text{Var}(\text{SCORE}) = \text{score.scalars}^\top \Sigma \, \text{score.scalars} = \frac{1}{k^2} w^\top \text{corr} \, w = \frac{1}{k^2} \cdot k^2 = 1 $$ This puts every choice of weights on the same scale: a `SCORE` of 1.5 always means "one and a half standard deviations above the average combination, as judged by these weights", whichever weights were chosen. It doesn't make scores under *different* weights interchangeable, though - see "What this does and doesn't let you compare" below. `tests/testthat/test-scoring.R` checks this algebraic identity directly, for a battery of random weight vectors. ## Other scales: Max and Raw The standard deviation is one choice of unit for each stat. `stat.scale` offers two others: - `"max"`: each stat in units of its highest value across every armor combination (`stat.maxes`, the best set total at full upgrade). A weight then covers a stat's whole range, from nothing to the best any set can reach. - `"raw"`: no adjustment, so one point of any stat counts the same. With equal weights, stats that vary more between sets count for more: slash defense (standard deviation 33) counts about two and a half times as much as bleed resistance (13). Whatever the scale, the weights become per-point multipliers $c_i = w_i / u_i$, where $u_i$ is the stat's unit (its standard deviation, its highest value, or 1). The score is then scaled exactly as above: its variance is $c^\top \Sigma c$, so `score.scalars` $= c / \sqrt{c^\top \Sigma c}$. Under `"sd"` this is the formula above. So `SCORE` means "standard deviations above the average combination" under every scale. The scale changes how the weights trade stats off against each other, not what `SCORE`'s numbers mean. Equivalently, weights $w$ under `"raw"` give exactly the same scores as $w \cdot \sigma$ under `"sd"`, and $w$ under `"max"` the same as $w$ divided by each stat's highest value under `"raw"`. ## Weighting groups of stats The ten stats fall into three groups whose stats move together: almost perfectly for the four physical defenses (average correlation 0.98), partly for the elemental defenses (0.54), and less for the resistances (0.31). So with equal weights on standardized stats, the physical defenses, nearly one stat counted four times, drive about half of what separates scores: | Group | Share with equal stat weights | Share with equal group weights | |--------------------|-------------------------------|--------------------------------| | Physical defenses | 48% | 30% | | Elemental defenses | 32% | 39% | | Resistances | 20% | 31% | `weights.mode = "grouped"` takes one weight per group instead (`PHYSICAL_DEFENSES`, `ELEMENTAL_DEFENSES`, `RESISTANCES`). Each group counts as the average of its stats, and `stat.scale` applies to that average: - under `"sd"`, its own standard deviation, which follows from the stats' standard deviations and correlations; - under `"max"`, its highest value across every combination (`group.maxes`); - under `"raw"`, one point. Equal group weights then give each group close to a third, as in the right-hand column above (on the Std Dev scale). A grouped score is still a weighted sum of the ten stats: a group's weight $W$ becomes $W / (k \cdot u)$ per point on each of its $k$ stats, where $u$ is the group average's unit. So the search, `SCORE_QUALITY` and the trade-off curves all work exactly as before. ## Why each armor slot subtracts a quarter of the mean The four armor slots (head, chest, hands, legs) are scored independently and then added together - `optimal_armor_combinations()` sums `curr_head_OBJECTIVE + curr_chest_OBJECTIVE + curr_hands_OBJECTIVE + curr_legs_OBJECTIVE` (each piece's objective, which is its score when ranking by score); see `R/optimal-combos.R`, where each slot's contribution is computed as `score.scalars[i] * (metric_i - 0.25 * means[i])`. `means[i]` is the population mean of the *combined* metric across a full four-piece combination, not any one slot's mean. Subtracting `0.25 * means[i]` in each of the four slots, and then summing all four, subtracts exactly `means[i]` from the total - which is what actually needs to happen for the total score to be centered at 0. The even four-way split is simply the easiest way to divide that single subtraction across four separately-computed slot scores; it doesn't require (and isn't affected by) whether any individual slot's own average metric value happens to be exactly a quarter of the combo-level mean. ## From SCORE to SCORE_QUALITY `SCORE` is a number of standard deviations, which isn't very intuitive on its own. `SCORE_QUALITY` turns it into a rarity: how many of all the possible combinations at the selected upgrade levels score at least as well. A combination that 1 in every 40 combinations matches or beats reads `"Top 1 in 40"`; the single best combination at a level reads `"Top 1 in 11,930,328"` - every combination there is. Combinations in the bottom half read `"Bottom 1 in N"` instead, counting the combinations that score *at most* as well. Ties count toward the number in both cases. The comparison is against every combination at the selected `regular.level`/`twinkling.level`, regardless of the other filters (areas, starting class, equip load, minima, and so on), so it describes how good a result is among everything a player at that upgrade level could wear. ### An exact count, not an estimate There are about 11.9 million combinations at each upgrade level, but counting the ones that score at least as well as a given result doesn't require visiting them. A combination's score is just the sum of its four pieces' scores (see the previous section), so the count is a question about sums of numbers drawn from four short lists: ```r ## every head+chest+legs sum (~221 thousand values), computed once and sorted other.sums <- sort(as.vector(outer(outer(head.scores, chest.scores, `+`), legs.scores, `+`))) ## for each hands piece, a binary search counts the sums that complete a combination at or past ## the target score at.least <- 0 for(hands.score in hands.scores){ at.least <- at.least + sum(other.sums >= target - hands.score) } ``` (the package itself uses `findInterval()` for the binary search, handles every result in one pass, and allows a tiny floating-point tolerance - see `R/score-quality.R`). Looping over hands, the slot with the fewest pieces, keeps the per-result cost to about 54 binary searches; the sorted list of the other three slots' sums is built once per call. ### Why not a normal approximation? An earlier version of `SCORE_QUALITY` assumed `SCORE` was normally distributed across every combination at every upgrade level pooled together, and converted it with `pnorm()`. That turned out to be off by large factors in both directions. The score is a sum of only four independent terms (one per slot), not of many, and its tails aren't normal: the README example's top result read "1 in 318" against an exact "1 in 193", the best combination at +10/+5 read "1 in 702 million" when only about 11.9 million combinations exist at that level, and single-stat weights produced numbers like "1 in 3 x 10^27". Pooling every upgrade level also made upgraded builds look far rarer than they are among the combinations a player at that level can actually choose from. ## What this does and doesn't let you compare Because `Var(SCORE) = 1` holds for any weights, `SCORE` is directly comparable across different filter and constraint choices, including different upgrade levels, *as long as the weights, `stat.scale` and `weights.mode` are held fixed*. `SCORE_QUALITY` is comparable across filter and constraint choices at the *same* upgrade levels - `"Top 1 in 1,000"` under one set of filters really is better than `"Top 1 in 20"` under another - but is always relative to the selected upgrade levels. Scores computed with *different* weights are not directly comparable to each other in the same way, since each weight vector defines its own notion of "average" combination.