Avoid numerical instability in resize

Because the layout code rounds down the dimensions of the windows
resizing would often be off by one pixel. The width/height fraction
would not exactly reflect the final computed width and so the resize
code would end up calculating things wrong.

To fix this first snap the container size fractions to the pixel grid
and only then do the resize. Also use round() instead of floor() during
layout to avoid a slightly too small width. This applies in two cases:

1. For the container we are actually resizing using floor() might result
   in being 1px too small.
2. For the other containers it might result in resizing them down by 1px
   and then if the container being resized is the last all those extra
   pixels would make the resize too large.

Fixes #4391

1 files changed, 24 insertions, 4 deletions

diff --git a/sway/commands/resize.c b/sway/commands/resize.c
index 0c9af12d..7ff4ef7b 100644
--- a/sway/commands/resize.c
+++ b/sway/commands/resize.c
@@ -172,9 +172,19 @@ void container_resize_tiled(struct sway_container *con,
                 if (prev && prev->width - sibling_amount < MIN_SANE_W) {
                         return;
                 }
+                if (con->child_total_width <= 0) {
+                        return;
+                }
+
+                // We're going to resize so snap all the width fractions to full pixels
+                // to avoid rounding issues
+                list_t *siblings = container_get_siblings(con);
+                for (int i = 0; i < siblings->length; ++i) {
+                        struct sway_container *con = siblings->items[i];
+                        con->width_fraction = con->width / con->child_total_width;
+                }
 
-                double amount_fraction =
-                        ((double)amount / con->width) * con->width_fraction;
+                double amount_fraction = (double)amount / con->child_total_width;
                 double sibling_amount_fraction =
                         prev ? amount_fraction / 2.0 : amount_fraction;
 
@@ -193,9 +203,19 @@ void container_resize_tiled(struct sway_container *con,
                 if (prev && prev->height - sibling_amount < MIN_SANE_H) {
                         return;
                 }
+                if (con->child_total_height <= 0) {
+                        return;
+                }
+
+                // We're going to resize so snap all the height fractions to full pixels
+                // to avoid rounding issues
+                list_t *siblings = container_get_siblings(con);
+                for (int i = 0; i < siblings->length; ++i) {
+                        struct sway_container *con = siblings->items[i];
+                        con->height_fraction = con->height / con->child_total_height;
+                }
 
-                double amount_fraction =
-                        ((double)amount / con->height) * con->height_fraction;
+                double amount_fraction = (double)amount / con->child_total_height;
                 double sibling_amount_fraction =
                         prev ? amount_fraction / 2.0 : amount_fraction;