factorMerger: set of tools to support results from post hoc testing

Agnieszka Sitko

2017-06-25

Introduction

The aim of factorMerger is to provide set of tools to support results from post hoc comparisons. Post hoc testing is an analysis performed after running ANOVA to examine differences between group means (of some response numeric variable) for each pair of groups (groups are defined by a factor variable).

This project arose from the need to create a method of post hoc testing which gives the hierarchical interpretation of relations between groups means. Thereby, for a given significance level we may divide groups into nonoverlapping clusters.

Algorithm inputs

In the current version the factorMerger package supports parametric models:

Set of hypotheses that are tested during merging may be either comprehensive or limited. This gives two possibilities:

The version all-to-all considers all possible pairs of factor levels. In the successive approach factor levels are preliminarily sorted and then only consecutive groups are tested for means equality.

The factorMerger package also implements two strategies of a single iteration of the algorithm. They use one of the following:

Generating samples

To visualize functionalities of factorMerger we use samples for which response variable is generated from one of the distributions listed above and corresponding factor variable is sampled uniformly from a finite set of a size \(k\).

To do so, we may use function generateSample or generateMultivariateSample.

library(factorMerger) 
library(knitr)
library(dplyr)
randSample <- generateMultivariateSample(N = 100, k = 10, d = 3)

Merging factors

mergeFactors is a function that performs hierarchical post hoc testing. As arguments it takes:

By default (with argument abbreviate = TRUE) factor levels are abbreviated and surrounded with brackets.

Multi-dimensional Gaussian model

Computations

fmAll <- mergeFactors(randSample$response, randSample$factor)

mergeFactors outputs with information about the ‘merging history’.

mergingHistory(fmAll, showStats = TRUE) %>% 
    kable()
groupA groupB model GIC pvalVsFull pvalVsPrevious
1 -601.1520 1222.304 1.0000 1.0000
11 (D) (E) -601.2867 1220.573 0.9712 0.9712
12 (F) (H) -601.7974 1219.595 0.9788 0.8220
13 (I) (D)(E) -602.3086 1218.617 0.9898 0.8193
14 (F)(H) (J) -603.2720 1218.544 0.9856 0.6230
15 (A) (C) -604.2452 1218.490 0.9848 0.6149
16 (I)(D)(E) (G) -605.8050 1219.610 0.9693 0.4046
17 (B) (I)(D)(E)(G) -607.5798 1221.159 0.9424 0.3402
18 (B)(I)(D)(E)(G) (F)(H)(J) -612.3349 1228.670 0.6662 0.0282
19 (A)(C) (B)(I)(D)(E)(G)(F)(H)(J) -620.1979 1242.396 0.1304 0.0017

Each row of the above frame describes one step of the merging algorithm. First two columns specify which groups were merged in the iteration, columns model and GIC gather loglikelihood and Generalized Information Criterion for the model after merging. Last two columns are p-values for the Likelihood Ratio Test – against the full model (pvalVsFull) and against the previous one (pvalVsPrevious).

If we set successive = TRUE then at the beginning one dimensional response is fitted using isoMDS{MASS}. Next, in each step only groups whose means are closed are compared.

fm <- mergeFactors(randSample$response, randSample$factor, 
                   successive = TRUE, 
                   method = "hclust")

mergingHistory(fm, showStats = TRUE) %>% 
    kable()
groupA groupB model GIC pvalVsFull pvalVsPrevious
-601.1520 1222.304 1.0000 1.0000
(F) (D) -601.6740 1221.348 0.8197 0.8197
(H) (J) -602.3058 1220.612 0.9130 0.7696
(I) (F)(D) -602.9745 1219.949 0.9507 0.7505
(B) (I)(F)(D) -605.1356 1222.271 0.8417 0.2664
(H)(J) (G) -607.3748 1224.749 0.7350 0.2465
(A) (B)(I)(F)(D) -610.3871 1228.774 0.5308 0.1309
(E) (H)(J)(G) -611.1522 1228.304 0.6249 0.6948
(C) (E)(H)(J)(G) -616.5367 1237.073 0.2567 0.0163
(A)(B)(I)(F)(D) (C)(E)(H)(J)(G) -620.1979 1242.396 0.1304 0.0698

Final clusters

Algorithms implemented in the factorMerger package enable to create unequivocal partition of a factor. Below we present how to extract the partition from the mergeFactor output.

  • predict new labels for observations
cutTree(fm)
#>   [1] (I)(F)(D) (A)       (E)       (E)       (I)(F)(D) (I)(F)(D) (I)(F)(D)
#>   [8] (I)(F)(D) (I)(F)(D) (B)       (H)(J)    (H)(J)    (I)(F)(D) (I)(F)(D)
#>  [15] (G)       (B)       (A)       (H)(J)    (A)       (I)(F)(D) (H)(J)   
#>  [22] (C)       (I)(F)(D) (I)(F)(D) (C)       (I)(F)(D) (I)(F)(D) (C)      
#>  [29] (H)(J)    (I)(F)(D) (B)       (B)       (I)(F)(D) (A)       (C)      
#>  [36] (B)       (A)       (C)       (I)(F)(D) (I)(F)(D) (C)       (I)(F)(D)
#>  [43] (G)       (B)       (H)(J)    (B)       (B)       (H)(J)    (I)(F)(D)
#>  [50] (C)       (B)       (I)(F)(D) (I)(F)(D) (H)(J)    (A)       (B)      
#>  [57] (I)(F)(D) (I)(F)(D) (H)(J)    (H)(J)    (E)       (H)(J)    (I)(F)(D)
#>  [64] (G)       (I)(F)(D) (H)(J)    (I)(F)(D) (I)(F)(D) (E)       (B)      
#>  [71] (H)(J)    (I)(F)(D) (I)(F)(D) (H)(J)    (G)       (I)(F)(D) (I)(F)(D)
#>  [78] (A)       (G)       (H)(J)    (C)       (E)       (H)(J)    (I)(F)(D)
#>  [85] (A)       (H)(J)    (C)       (G)       (E)       (E)       (A)      
#>  [92] (I)(F)(D) (I)(F)(D) (C)       (I)(F)(D) (C)       (H)(J)    (B)      
#>  [99] (G)       (G)      
#> Levels: (A) (B) (I)(F)(D) (C) (E) (H)(J) (G)

By default, cutTree returns a factor split for the optimal GIC (with penalty = 2) model. However, we can specify different metrics (stat = c("loglikelihood", "p-value", "GIC") we would like to use in cutting. If loglikelihood or p-value is chosen an exact threshold must be given as a value parameter. Then cutTree returns factor for the smallest model whose statistic is higher than the threshold. If we choose GIC then value is interpreted as GIC penalty.

mH <- mergingHistory(fm, T)
thres <- mH$model[nrow(mH) / 2]
cutTree(fm, stat = "loglikelihood", value = thres)
#>   [1] (B)(I)(F)(D) (A)          (E)          (E)          (B)(I)(F)(D)
#>   [6] (B)(I)(F)(D) (B)(I)(F)(D) (B)(I)(F)(D) (B)(I)(F)(D) (B)(I)(F)(D)
#>  [11] (H)(J)       (H)(J)       (B)(I)(F)(D) (B)(I)(F)(D) (G)         
#>  [16] (B)(I)(F)(D) (A)          (H)(J)       (A)          (B)(I)(F)(D)
#>  [21] (H)(J)       (C)          (B)(I)(F)(D) (B)(I)(F)(D) (C)         
#>  [26] (B)(I)(F)(D) (B)(I)(F)(D) (C)          (H)(J)       (B)(I)(F)(D)
#>  [31] (B)(I)(F)(D) (B)(I)(F)(D) (B)(I)(F)(D) (A)          (C)         
#>  [36] (B)(I)(F)(D) (A)          (C)          (B)(I)(F)(D) (B)(I)(F)(D)
#>  [41] (C)          (B)(I)(F)(D) (G)          (B)(I)(F)(D) (H)(J)      
#>  [46] (B)(I)(F)(D) (B)(I)(F)(D) (H)(J)       (B)(I)(F)(D) (C)         
#>  [51] (B)(I)(F)(D) (B)(I)(F)(D) (B)(I)(F)(D) (H)(J)       (A)         
#>  [56] (B)(I)(F)(D) (B)(I)(F)(D) (B)(I)(F)(D) (H)(J)       (H)(J)      
#>  [61] (E)          (H)(J)       (B)(I)(F)(D) (G)          (B)(I)(F)(D)
#>  [66] (H)(J)       (B)(I)(F)(D) (B)(I)(F)(D) (E)          (B)(I)(F)(D)
#>  [71] (H)(J)       (B)(I)(F)(D) (B)(I)(F)(D) (H)(J)       (G)         
#>  [76] (B)(I)(F)(D) (B)(I)(F)(D) (A)          (G)          (H)(J)      
#>  [81] (C)          (E)          (H)(J)       (B)(I)(F)(D) (A)         
#>  [86] (H)(J)       (C)          (G)          (E)          (E)         
#>  [91] (A)          (B)(I)(F)(D) (B)(I)(F)(D) (C)          (B)(I)(F)(D)
#>  [96] (C)          (H)(J)       (B)(I)(F)(D) (G)          (G)         
#> Levels: (A) (B)(I)(F)(D) (C) (E) (H)(J) (G)

In this example data partition is created for the last model from the merging path whose loglikelihood is greater than -605.1356.

  • get final clusters and clusters dictionary
getOptimalPartition(fm)
#> [1] "(A)"       "(B)"       "(I)(F)(D)" "(C)"       "(E)"       "(H)(J)"   
#> [7] "(G)"

Function getOptimalPartition returns a vector with the final cluster names from the factorMerger object.

getOptimalPartitionDf(fm)
#>    orig      pred
#> 1   (I) (I)(F)(D)
#> 2   (A)       (A)
#> 3   (E)       (E)
#> 5   (F) (I)(F)(D)
#> 8   (D) (I)(F)(D)
#> 10  (B)       (B)
#> 11  (J)    (H)(J)
#> 15  (G)       (G)
#> 21  (H)    (H)(J)
#> 22  (C)       (C)

Function getOptimalPartitionDf returns a dictionary in a data frame format. Each row gives an original label of a factor level and its new (cluster) label.

Similarly to cutTree, functions getOptimalPartition and getOptimalPartitionDf take arguments stat and threshold.

Visualizations

We may plot results using function plot.

plot(fm, panel = "all", nodesSpacing = "equidistant", colorCluster = TRUE)

plot(fmAll, panel = "tree", statistic = "p-value", 
     nodesSpacing = "effects", colorCluster = TRUE)

plot(fm, colorCluster = TRUE, panel = "response")

The heatmap on the right shows means of all variables taken into analysis by groups.

plot(fm, colorCluster = TRUE, panel = "response", responsePanel = "profile")

In the above plots colours are connected with the group. The plot on the right shows means rankings for all variables included in the algorithm.

It is also possible to plot GIC together with the merging path plot.

plot(fm, panel = "GIC", penalty = 5)

Model with the lowest GIC is marked.

One-dimensional Gaussian model

oneDimRandSample <- generateSample(1000, 10)
oneDimFm <- mergeFactors(oneDimRandSample$response, oneDimRandSample$factor, 
                         method = "hclust")
mergingHistory(oneDimFm, showStats = TRUE) %>% 
    kable()
groupA groupB model GIC pvalVsFull pvalVsPrevious
-3243.773 6507.546 1.0000 1.0000
(B) (E) -3243.773 6505.547 0.9744 0.9744
(B)(E) (C) -3243.802 6503.604 0.9717 0.8122
(F) (H) -3243.831 6501.663 0.9898 0.8087
(A) (B)(E)(C) -3244.142 6500.284 0.9474 0.4324
(J) (G) -3244.537 6499.075 0.9112 0.3755
(I) (D) -3244.984 6497.967 0.8793 0.3461
(F)(H) (J)(G) -3247.983 6501.966 0.3022 0.0145
(A)(B)(E)(C) (I)(D) -3252.165 6508.330 0.0338 0.0039
(A)(B)(E)(C)(I)(D) (F)(H)(J)(G) -3294.754 6591.508 0.0000 0.0000 
plot(oneDimFm, palette = "Reds")

plot(oneDimFm, responsePanel = "boxplot", colorCluster = TRUE)

Binomial model

If family = "binomial" response must have to values: 0 and 1 (1 is interpreted as success).

binomRandSample <- generateSample(1000, 10, distr = "binomial")
table(binomRandSample$response, binomRandSample$factor) %>% 
    kable()
C E B H F I G D A J
0 94 95 80 88 67 27 17 14 8 4
1 4 8 14 26 28 56 93 92 102 83 
binomFm <- mergeFactors(binomRandSample$response, 
                        binomRandSample$factor, 
                        family = "binomial", 
                        successive = TRUE)
mergingHistory(binomFm, showStats = TRUE) %>% 
    kable()
groupA groupB model GIC pvalVsFull pvalVsPrevious
1 -389.1899 798.3797 1.0000 1.0000
7 (G) (D) -389.3009 796.6018 0.6374 0.6374
8 (A) (J) -389.6122 795.2244 0.6555 0.4301
4 (H) (F) -390.2114 794.4228 0.5635 0.2736
11 (C) (E) -390.8314 793.6629 0.5116 0.2655
2 (B) (H)(F) -393.1908 796.3815 0.1561 0.0298
41 (G)(D) (A)(J) -397.0998 802.1995 0.0148 0.0052
3 (I) (G)(D)(A)(J) -408.6630 823.3259 0.0000 0.0000
12 (C)(E) (B)(H)(F) -422.4052 848.8103 0.0000 0.0000
13 (C)(E)(B)(H)(F) (I)(G)(D)(A)(J) -693.0752 1388.1504 0.0000 0.0000 
plot(binomFm, colorCluster = TRUE, gicPanelColor = "red")

plot(binomFm, colorCluster = TRUE, penalty = 7)

plot(binomFm, gicPanelColor = "red")

Survival model

If family = "survival" response must be of a class Surv.

library(survival)
data(veteran)
survResponse <- Surv(time = veteran$time, 
                 event = veteran$status)
survivalFm <- mergeFactors(response = survResponse, 
                   factor = veteran$celltype, 
                   family = "survival")
mergingHistory(survivalFm, showStats = TRUE) %>% 
    kable()
groupA groupB model GIC pvalVsFull pvalVsPrevious
1 -493.0247 994.0495 1.0000 1.0000
11 (smll) (aden) -493.1951 992.3902 0.5594 0.5594
12 (sqms) (larg) -493.5304 991.0609 0.6031 0.4128
13 (sqms)(larg) (smll)(aden) -505.4491 1012.8981 0.0000 0.0000 
plot(survivalFm)

plot(survivalFm, nodesSpacing = "effects", colorCluster = TRUE)