HOXO-M Blog https://hoxo-m.github.io/blog/ Insights into R and Data Analysis quarto-1.9.38 Wed, 17 Jun 2026 13:00:00 GMT TheseusPlot 0.3.0: Visualizing the Decomposition of Differences in Rate Metrics Koji Makiyama https://hoxo-m.github.io/blog/posts/TheseusPlot-0-3-0/ TheseusPlot is an R package that decomposes differences in a rate metric between two groups into subgroup-level contributions and visualizes the results as a “Theseus Plot”.

For example, when a click-through rate, conversion rate, or retention rate differs between two time periods or groups, TheseusPlot helps answer questions such as: which subgroup contributed most to the difference?

Suppose that the click-through rate (CTR) was 6.2% in 2024 and 5.2% in 2025, a decrease of 1.0 percentage point. A Theseus Plot can show how this decrease is decomposed: in this example, 0.8 percentage points are assigned to male users and 0.2 percentage points to female users under the decomposition.

Version 0.3.0 is now available on CRAN. This release fixes a compatibility issue with waterfalls 1.1.4, improves subgroup size bar rendering, and refines several plot defaults.

What’s new in 0.3.0

Cleaner plot labels

In earlier versions, TheseusPlot automatically displayed the analyzed column name as a subtitle. However, this was not always useful, especially when the plot was already used in a document or presentation where the context was clear.

In version 0.3.0, the automatic column-name subtitle has been removed. This makes the resulting plots cleaner and easier to combine with custom titles, captions, and surrounding text.

This release also adds an xlab argument to create_ship(), so you can customize the x-axis label used by plot() and plot_flip().

For example:

ship <- create_ship(
  data_2024,
  data_2025,
  y = clicked,
  labels = c("2024", "2025"),
  xlab = "Gender",
  ylab = "CTR (%)"
)

ship$plot(gender)

This is useful when the column name in the data is short or technical, but you want a more readable label in the plot.

Better default labels

The default group labels have been changed from "Original" and "Refitted" to "Baseline" and "Comparison".

ship <- create_ship(
  data_2024,
  data_2025,
  y = clicked
)

ship$plot(gender)

The previous labels reflected the internal idea of replacing one group with another, but they were not always intuitive for users. The new defaults better match common comparison scenarios, such as year-over-year comparisons, control versus treatment, and before-and-after analyses.

Of course, you can still specify your own labels:

ship <- create_ship(
  data_Nov,
  data_Dec,
  y = on_time,
  labels = c("November", "December")
)

Simpler numeric display

The default number of displayed decimal places has been changed from 3 to 1.

In many plots, three decimal places made the labels more detailed than necessary. Since TheseusPlot is mainly intended to help users understand the structure of a metric difference, one decimal place is often enough for visual interpretation.

You can still control the precision with the digits argument when needed.

ship <- create_ship(
  data_2024,
  data_2025,
  y = clicked,
  labels = c("2024", "2025"),
  digits = 2
)

ship$plot(gender)

Plot improvements and bug fixes

Version 0.3.0 also includes several improvements and bug fixes related to plot rendering.

First, missing subgroup size bars in plot() and plot_flip() with waterfalls 1.1.4 have been fixed. Subgroup size bars are an important part of Theseus Plots because they show the sample size of each subgroup in both groups. Without them, it becomes harder to judge whether a large contribution comes from a large subgroup, a large rate difference, or both.

Second, subgroup size bar scaling has been improved. Bar heights are now computed consistently from the maximum plot score in both plot() and plot_flip(). This makes visual comparisons more stable across plot directions. The maximum height of these bars can still be controlled with the bar_max_value argument.

Third, text_size handling has been fixed when applying the current ggplot2 theme. This makes text scaling more predictable when users customize plot themes.

ship <- create_ship(
  data_2024,
  data_2025,
  y = clicked,
  labels = c("2024", "2025"),
  text_size = 1.5
)

ship$plot(gender)

Installation

You can install TheseusPlot from CRAN with:

install.packages("TheseusPlot")

Try it out

TheseusPlot is useful when you want to understand why rate metrics differ between two groups.

Typical examples include:

  • click-through rate
  • conversion rate
  • retention rate
  • success rate
  • error rate

For details, please see the package website:

]]> R https://hoxo-m.github.io/blog/posts/TheseusPlot-0-3-0/ Wed, 17 Jun 2026 13:00:00 GMT TheseusPlot 0.2.0: Visualizing Decomposition of Differences in Rate Metrics Koji Makiyama https://hoxo-m.github.io/blog/posts/TheseusPlot-0-2-0/ TheseusPlot is an R package that decomposes differences in rate metrics between two groups into contributions from individual subgroups and visualizes the results as a “Theseus Plot”.

The package is inspired by the Ship of Theseus thought experiment. It replaces subgroup data step by step, recalculates the overall metric at each step, and interprets each change as that subgroup’s contribution to the overall difference.

Suppose you notice that the click-through rate is lower in 2025 than in 2024 and want to examine how a particular attribute, such as gender, contributed to the change. If you obtain a Theseus Plot like the one below, it suggests that men contributed more to the decline in click-through rate than women.

What’s new in 0.2.0

Version 0.2.0 includes the following changes:

  • a fix for continuous-variable discretization with split = "rate", where bin boundaries for the second group could previously be computed from the first group’s data
  • a fix for the size bar of "Sum of ... other attributes", which could incorrectly use the first group’s counts for both groups
  • a fix for warnings in plot() and plot_flip() when multiple subgroups were tied for the largest absolute contribution
  • suppression of warnings generated internally by waterfalls::waterfall() during plot creation.

I would like to thank Kazuyuki Sano for reporting the first two issues and contributing to their fixes.

Installation

You can install TheseusPlot from CRAN with:

install.packages("TheseusPlot")

Try it out

TheseusPlot may be useful when you want to understand why metrics such as conversion rate, retention rate, or click-through rate changed.

For details on how to use it, please see the package website: https://hoxo-m.github.io/TheseusPlot/.

]]> R https://hoxo-m.github.io/blog/posts/TheseusPlot-0-2-0/ Fri, 10 Apr 2026 00:00:00 GMT deltatest 0.2.0: Statistical Hypothesis Testing Using the Delta Method for Online A/B Testing Koji Makiyama https://hoxo-m.github.io/blog/posts/deltatest-0-2-0/ I’m happy to share a new release of deltatest.

This release includes two main changes:

  • a new tidy() method for deltatest objects
  • a fix for p-value calculation in one-sided tests

Before looking at what changed in this release, let’s briefly revisit the purpose of deltatest.

What deltatest is for

The deltatest package provides deltatest(), a function for performing two-sample Z-tests using the delta method.

It is designed for common settings in online A/B testing where:

  • randomization is done at the user level, but
  • the metric is measured at a finer unit such as page views or sessions.

In such settings, naive tests can underestimate uncertainty—for example, standard Z-tests, chi-squared tests, or tests for differences in proportions—because observations within a user are not independent. deltatest() addresses this issue by using a delta-method-based variance estimator.

# Install the released version from CRAN
install.packages("deltatest")

# Load packages
library(dplyr)
library(deltatest)

# Generate dummy data
data <- deltatest::generate_dummy_data(2000) |>
  mutate(group = if_else(group == 0, "control", "treatment")) |>
  group_by(user_id, group) |>
  summarise(clicks = sum(metric), pageviews = n(), .groups = "drop")

# Run a test
deltatest(data, clicks / pageviews, by = group)

Typical output:

#> Two Sample Z-test Using the Delta Method
#> 
#> data:  clicks/pageviews by group
#> Z = 0.31437, p-value = 0.7532
#> alternative hypothesis: true difference in means between control and treatment is not equal to 0
#> 95 percent confidence interval:
#>  -0.01410593  0.01949536
#> sample estimates:
#>   mean in control mean in treatment        difference
#>       0.245959325       0.248654038       0.002694713

What’s new in 0.2.0

tidy() support for deltatest objects

With this release, deltatest() results can now be converted directly into a tidy tibble with broom::tidy().

deltatest() returns an htest-class object, which is convenient for printing and interactive use. But in a tidyverse workflow, it is often much easier to work with results in a tidy tibble format. This makes it easier to combine results across many experiments or metrics, and to visualize patterns in estimates, confidence intervals, or p-values with tools like ggplot2.

First, here is a simple example of converting the result to a tidy format:

library(dplyr)
library(deltatest)
library(broom)

data <- deltatest::generate_dummy_data(2000) |>
  mutate(group = if_else(group == 0, "control", "treatment")) |>
  group_by(user_id, group) |>
  summarise(clicks = sum(metric), pageviews = n(), .groups = "drop")

result <- deltatest(data, clicks / pageviews, by = group)

tidy(result)
#> # A tibble: 1 × 9
#>   estimate mean_ctrl mean_treat statistic p.value conf.low conf.high method     
#>      <dbl>     <dbl>      <dbl>     <dbl>   <dbl>    <dbl>     <dbl> <chr>      
#> 1  0.00269     0.246      0.249     0.314   0.753  -0.0141    0.0195 Two Sample…
#> # ℹ 1 more variable: alternative <chr>

Next, here is an example of using the tidy results to compare multiple experiments in a plot:

library(ggplot2)

data2 <- deltatest::generate_dummy_data(2000, xi = 0.05) |>
  mutate(group = if_else(group == 0, "control", "treatment")) |>
  group_by(user_id, group) |>
  summarise(clicks = sum(metric), pageviews = n(), .groups = "drop")

result2 <- deltatest(data2, clicks / pageviews, by = group)

result_tidy1 <- tidy(result)  |> mutate(experiment_id = "test01")
result_tidy2 <- tidy(result2) |> mutate(experiment_id = "test02")

result_tidy <- bind_rows(result_tidy1, result_tidy2)

ggplot(result_tidy, aes(experiment_id, estimate)) +
  geom_pointrange(aes(ymin = conf.low, ymax = conf.high)) +
  geom_hline(yintercept = 0, color = "red") +
  xlab(NULL) + ylab("Estimated CTR difference") +
  ggtitle("Treatment effects by experiment")

Fix for one-sided p-value calculation

This release also fixes a bug in the p-value calculation for one-sided tests. In the previous version, p-values for one-sided tests could be incorrectly calculated using the two-sided formula. That behavior has now been fixed.

I would like to thank Kazuyuki Sano for reporting this issue and contributing to the fix.

Final thoughts

I’m glad to keep improving deltatest little by little. If you use R for online A/B experiments, I hope it is useful to you.

For more details, see:

]]> R https://hoxo-m.github.io/blog/posts/deltatest-0-2-0/ Thu, 12 Mar 2026 00:00:00 GMT New R Package TheseusPlot: Visualizing Decomposition of Differences in Rate Metrics Koji Makiyama https://hoxo-m.github.io/blog/posts/TheseusPlot/

1. Overview

In data analysis, when a metric differs between two groups, we sometimes want to investigate whether a particular subgroup is driving that difference. For example, when a key metric decline is detected compared to the previous year, you may want to conduct a more detailed analysis. In this analysis, you may focus on gender among the attributes and examine whether the decline occurred among male, female, or both. However, this type of analysis is challenging when the metric is a rate, because the magnitude of each subgroup’s contribution to the rate cannot be simply calculated, unlike in the case of volume metrics.

To address this issue, we propose an approach inspired by the story of the Ship of Theseus. This approach involves gradually replacing the components of one group with those of another, recalculating the metric at each step. The change in the metric at each step can then be interpreted as the contribution of each subgroup to the overall difference.

For instance, suppose the metric was 6.2% in 2024 and decreased to 5.2% in 2025. Again, we focus on gender. We replace the male data within the 2024 dataset with the male data from 2025 and recalculate the metric. As a result, the metric would drop by 0.8 percentage points, reaching 5.4%. In this case, the contribution of the male group to the change in the metric is -0.8 percentage points. Next, we replace the female data from 2024 with that from 2025. The dataset then consists entirely of 2025 data, and the metric drops by 0.2 percentage points, reaching 5.2%. Thus, the contribution of the female group is -0.2 percentage points.

When visualized, the results appear as follows:

From this plot, we can see that the decline in the metric is primarily driven by the male group. We call this visualization the “Theseus Plot.”

The TheseusPlot package is designed to make it easy to generate Theseus Plots for various attributes.

2. Installation

You can install the TheseusPlot package from CRAN.

install.packages("TheseusPlot")

You can install the development version from GitHub with:

remotes::install_github("hoxo-m/TheseusPlot")

3. Details

3.1 Prepare Data

To create Theseus plots, you need two data frames that share common columns.

We use the 2013 New York City flight data from nycflights13 as a demo dataset. Here, we will define the rate metric as the proportion of flights that arrived on time. In December 2013, the on-time arrival rate dropped substantially compared to November. We investigate the cause using a Theseus plot.

First, we create an on_time column in the data frame to indicate whether each flight arrived on time. Next, we extract the flights for November and December into separate data frames to form two comparison groups. The on-time arrival rate was 64% in November and dropped to 47% in December.

library(dplyr)
library(nycflights13)

data <- flights |> 
  filter(!is.na(arr_delay)) |>
  mutate(on_time = arr_delay <= 0) |>  # Arrived on time
  left_join(airlines, by = "carrier") |>
  mutate(carrier = name) |>  # Convert carrier abbreviations to full names
  select(year, month, day, origin, dest, carrier, dep_delay, on_time)

data |> head()
#> # A tibble: 6 × 8
#>    year month   day origin dest  carrier                dep_delay on_time
#>   <int> <int> <int> <chr>  <chr> <chr>                      <dbl> <lgl>  
#> 1  2013     1     1 EWR    IAH   United Air Lines Inc.          2 FALSE  
#> 2  2013     1     1 LGA    IAH   United Air Lines Inc.          4 FALSE  
#> 3  2013     1     1 JFK    MIA   American Airlines Inc.         2 FALSE  
#> 4  2013     1     1 JFK    BQN   JetBlue Airways               -1 TRUE   
#> 5  2013     1     1 LGA    ATL   Delta Air Lines Inc.          -6 TRUE   
#> 6  2013     1     1 EWR    ORD   United Air Lines Inc.         -4 FALSE

data_Nov <- data |> filter(month == 11)
data_Dec <- data |> filter(month == 12)

data_Nov |> summarise(on_time_rate = mean(on_time)) |> pull(on_time_rate)
#> [1] 0.6426161
data_Dec |> summarise(on_time_rate = mean(on_time)) |> pull(on_time_rate)
#> [1] 0.4672835

3.2 Basics

Using the two prepared data frames, we first create a ship object. The ship object is an instance of the R6 class ShipOfTheseus, designed to create Theseus plots.

library(TheseusPlot)

ship <- create_ship(data_Nov, data_Dec, y = on_time, labels = c("November", "December"))

You can create a Theseus plot by passing column names to the plot method of a ship object. For example, to create a Theseus plot for the airport of origin:

ship$plot(origin)

New York City has three major airports, and Newark Liberty International Airport (EWR) accounted for the largest share of the decline in the on-time arrival rate.

Note that the number of flights at each airport matters, as a larger flight volume is expected to have a greater impact. To make this clear, the Theseus plot displays the data size for each group within each subgroup as a bar chart. From this, we see that the number of flights is similar across airports, allowing for direct comparison of contributions.

In summary, a Theseus plot consists of two components:

  • A waterfall plot showing how much each subgroup contributed to the change in the metric.
  • A bar chart representing the sample size for each group within each subgroup.

A ship object also provides the table method to inspect the exact values used in the Theseus plot.

ship$table(origin)
#> # A tibble: 3 × 8
#>   origin contrib    n1    n2    x1    x2 rate1 rate2
#>   <chr>    <dbl> <int> <int> <int> <int> <dbl> <dbl>
#> 1 EWR    -0.0831  9603  9410  6251  3901 0.651 0.415
#> 2 JFK    -0.0565  8645  8923  5702  4332 0.660 0.485
#> 3 LGA    -0.0358  8723  8687  5379  4393 0.617 0.506

3.3 Flipping the Plot

When there are many subgroups, a Theseus plot can become hard to read. In such cases, you can swap the x- and y-axes for better visualization.

ship$plot_flip(carrier)

When the number of subgroups is large, those with small contributions are automatically grouped together. By default, this happens when there are more than 10 subgroups, but the threshold can be adjusted with the n argument.

ship$plot_flip(carrier, n = 5)

From this plot, JetBlue Airways and United Air Lines appear to have the largest contributions to the decline in on-time arrival rate.

3.4 Automatic Discretization of Continuous Values

Theseus plots do not directly support continuous variables. If a continuous column is provided, it is automatically discretized. For example, we can create a Theseus plot for departure delays.

ship$plot_flip(dep_delay)

By default, continuous variables are discretized so that each subgroup has roughly equal sample sizes, with the number of bins set to 10. You can modify these settings by passing the return value of continuous_config() to the continuous argument.

ship$plot_flip(dep_delay, continuous = continuous_config(n = 5))

This result shows that both a decrease in on-time departures and an increase in delayed departures contributed to the decline in on-time arrival rate.

]]> R https://hoxo-m.github.io/blog/posts/TheseusPlot/ Sat, 30 Aug 2025 00:00:00 GMT New R Package deltatest: Statistical Hypothesis Testing Using the Delta Method for Online A/B Testing Koji Makiyama https://hoxo-m.github.io/blog/posts/deltatest/ In online A/B testing, we often face a significant practical challenge: the randomization unit differs from the analysis unit. Typically, control and treatment groups are randomly assigned at the user level, while metrics—such as click-through rate—are measured at a more granular level (e.g., per page-view). In this case, the randomization unit is user, but the analysis unit is page-view.

This discrepancy raises concerns for statistical hypothesis testing, which assumes that data points are independent and identically distributed (i.i.d.). Specifically, a single user can generate multiple page-views, and each user may have a different probability of clicking. Consequently, the data may exhibit within-user correlation, thereby violating the i.i.d. assumption.

When the standard Z-test is applied to such correlated data, the resulting p-values do not follow the expected uniform distribution under the null hypothesis. As a result, smaller p-values tend to occur more frequently even when there is no true difference, increasing the risk of falsely detecting a significant difference.

To address this problem, Deng et al. (2018) proposed a modified statistical hypothesis testing method. Their approach replaces the standard variance estimation formula in the Z-test with an approximate formula derived via the Delta method, which accounts for within-user correlation. To simplify the application of this method, the deltatest package has been developed.

To illustrate how to use this package, we prepare a data frame that includes columns for the number of clicks and page-views aggregated for each user. This data frame also contains a column indicating whether each user was assigned to the control or treatment group.

library(dplyr)

n_user <- 2000

set.seed(314)
data <- deltatest::generate_dummy_data(n_user) |> 
  mutate(group = if_else(group == 0, "control", "treatment")) |>
  group_by(user_id, group) |> 
  summarise(clicks = sum(metric), pageviews = n(), .groups = "drop")

data
#> # A tibble: 2,000 × 4
#>    user_id group     clicks pageviews
#>      <int> <chr>      <int>     <int>
#>  1       1 treatment      1         6
#>  2       2 treatment      2        11
#>  3       3 control        0        17
#>  4       4 control        4        12
#>  5       5 control        5        10
#>  6       6 control        1        15
#>  7       7 control        2         6
#>  8       8 treatment      2        11
#>  9       9 treatment      2        16
#> 10      10 control        0        17
#> # ℹ 1,990 more rows

The statistical hypothesis test using the Delta method can then be performed on this data as follows:

library(deltatest)

deltatest(data, clicks / pageviews, by = group)
#> 
#>  Two Sample Z-test Using the Delta Method
#> 
#> data:  clicks/pageviews by group
#> Z = 0.31437, p-value = 0.7532
#> alternative hypothesis: true difference in means between control and treatment is not equal to 0
#> 95 percent confidence interval:
#>  -0.01410593  0.01949536
#> sample estimates:
#>   mean in control mean in treatment        difference 
#>       0.245959325       0.248654038       0.002694713

This version of the Z-test yields p-values that follow the expected uniform distribution under the null hypothesis, even when within-user correlation is present.

For more details, refer to https://hoxo-m.github.io/deltatest/.

Installation

You can install the deltatest package from CRAN.

install.packages("deltatest")

References

]]> R https://hoxo-m.github.io/blog/posts/deltatest/ Sat, 15 Mar 2025 00:00:00 GMT MLE Adjustment for High-Dimensional Logistic Regression Koji Makiyama https://hoxo-m.github.io/blog/posts/adjustMLE/ The maximum likelihood estimator (MLE) of the logistic regression model is not an unbiased estimator. Therefore, estimates calculated with glm() contain bias. Since the MLE satisfies consistency and asymptotic normality, the bias can be disregarded when the sample size is large. However, in the analysis of high-dimensional data, the sample size is sometimes relatively small compared to the dimension of input variables.

For example, let’s consider a scenario where the number of input variables p = 300, and the sample size n = 2000. Additionally, the true parameters beta consist of

  • beta = 10 for the first 1/3,
  • beta = -10 for the next 1/3, and
  • beta = 0 for the remaining 1/3.

In such a case, the MLE returned by glm() contains a non-negligible bias.

p <- 300
n <- 2000

set.seed(314)
x <- rnorm(n * p, mean = 0, sd = sqrt(1/n))
X <- matrix(x, nrow = n, ncol = p)
beta <- matrix(rep(c(10, -10, 0), each = p/3), nrow = p, ncol = 1)
prob <- plogis(X %*% beta)
y <- rbinom(n, 1, prob)

fit <- glm(y ~ X, family = binomial, x = TRUE)

library(ggplot2)
theme_set(theme_bw())
df <- data.frame(index = seq_len(p), mle = coef(fit)[-1])
ggplot(df, aes(index, mle)) +
  geom_point(color = "blue") +
  annotate("segment", x = c(0, 100, 200), xend = c(100, 200, 300), 
           y = c(10, -10, 0), yend = c(10, -10, 0), linewidth = 1.5) +
  scale_x_continuous(breaks = c(0, 100, 200, 300)) +
  ylim(-30, 30) + xlab("Index of parameters") + ylab("MLE") +
  ggtitle("True (black line) and MLE (blue point)")

You can see that the blue points (MLE) are significantly outside the perimeter of the black line (true).

The purpose of this package is to alleviate the bias by adjusting the MLE. To achieve this, we implemented two methods:

  • “ProbeFrontier,” as proposed by Sur and Candès (2018), and
  • “SLOE,” as proposed by Yadlowsky et al. (2021).

The adjustMLE function in our package is designed to mitigate this bias.

library(adjustMLE)

fit_adj <- adjustMLE(fit)

df <- data.frame(index = seq_len(p), mle = coef(fit_adj)[-1])
ggplot(df, aes(index, mle)) +
  geom_point(color = "blue") +
  annotate("segment", x = c(0, 100, 200), xend = c(100, 200, 300), 
           y = c(10, -10, 0), yend = c(10, -10, 0), linewidth = 1.5) +
  scale_x_continuous(breaks = c(0, 100, 200, 300)) +
  ylim(-30, 30) + xlab("Index of parameters") + ylab("Adjusted MLE") +
  ggtitle("True (black line) and adjusted MLE (blue point)")

For more details, refer to https://github.com/hoxo-m/adjustMLE.

Installation

You can install the package from GitHub.

install.packages("remotes") # if you have not installed "remotes" package
remotes::install_github("hoxo-m/adjustMLE")

References

  • Sur, P., & Candès, E.J. (2018). A modern maximum-likelihood theory for high-dimensional logistic regression. Proceedings of the National Academy of Sciences of the United States of America, 116, 14516 - 14525.
  • Yadlowsky, S., Yun, T., McLean, C.Y., & D’Amour, A. (2021). SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression. Neural Information Processing Systems.
]]> R https://hoxo-m.github.io/blog/posts/adjustMLE/ Fri, 08 Nov 2024 00:00:00 GMT