Counting Baxter Matrices

Donald Knuth recently introduced the notion of a Baxter matrix, generalizing Baxter permutations. We show that for fixed number of rows, $r$, the number of Baxter matrices with $r$ rows and $k$ columns eventually satisfies a polynomial in $k$ of degree $2r-2$. We also give a proof of Knuth's conjecture that the number of 1's in a $r \times k$ Baxter matrix is less than $r+k$.


Introduction
Baxter permutations are a class of permutations that come up in numerous places. They were introduced in 1964 by Glen Baxter, who was studying fixed points of commuting continuous functions [2]. They have since been connected to Hopf Algebras [3], planar graphs [4], and tilings [5]. The number of Baxter permutations of length n is given by A001181 in the OEIS and forms a holonomic sequence of order 2 and degree 2 [6]. More recently, Donald Knuth suggested a generalization of Baxter permutations to matrices [1]. Donald Knuth defines a Baxter matrix as a matrix of 0s and 1s that satisfy 4 conditions. Before we state the conditions, we must define a pinwheel in a matrix. Definition 1. Let M be an r × k matrix. For 1 x r − 1 and 1 y k − 1, the clockwise pinwheel of index (x, y), denoted P x,y , is a specific subset of the entries of M. P x,y is divided into 4 segments.
• Segment D x,y contains the entries M[i, y], for x + 1 i r.
Similarly, for 1 x r − 1 and 1 y k − 1, the counterclockwise pinwheel of index (x, y), denoted P ′ x,y also contains four segments.
• Segment A ′ x,y contains the entries M[i, y], for 1 i x.
• Segment B ′ x,y contains the entries M[x + 1, i], for 1 i y.
• Segment C ′ x,y contains the entries M[x, i], for y + 1 i k.
• Segment D ′ x,y contains the entries M[i, y + 1], for x + 1 i r. Definition 2. A Baxter Matrix is a matrix of 0s and 1s that satisfy 4 conditions. 1. Each row contains at least one 1.
2. Each column contains at least one 1.
3. For each clockwise pinwheel, at least one of the four segments has only 0s. 4. For each counterclockwise pinwheel, at least one of the four segments has only 0s.
A pinwheel is said to be satisfied if at least one of its four segments has only 0s. In an r × k matrix, there are (r − 1)(k − 1) pinwheels of each direction that must be satisfied. As an example consider the two matrices in Figure 1. The left one is indeed a Baxter matrix, and the reader is encouraged to check that all the pinwheels are satisfied. The right one is almost a Baxter matrix; every pinwheel except P ′ 2,2 is satisfied. Note that in general a Baxter matrix could have multiple 1s in a row or column. It turns out that r×r Baxter matrices that only have a single 1 in each row and column are in bijection with Baxter permutations of length r. This can be seen by viewing the matrix as a permutation matrix. For each column i, the corresponding permutation π maps i to r minus the row in which the 1 appears in column i.
What is the maximum number of 1s that can appear in a Baxter matrix of size r × k? Knuth conjectured that this maximum is equal to r + k − 1, and exhaustively checked it for all r and k 7. In this paper we will prove that the conjecture is true, i.e. that for any r, k 1, the number of 1s in an r × k Baxter matrix is less than r + k.

A Finite State Automaton for Baxter Matrices with r rows
In this section we will describe a finite state automaton for determining whether a matrix is Baxter. We will fix the number of rows, and have the automaton read the columns of the matrix as symbols. When it is done reading the columns, it should accept or reject according to whether the matrix is Baxter. For ease of explanation, consider first the r = 2 case.

The 2 Row Case
Let's construct a finite state machine for determining whether a 0-1 matrix with 2 rows is a Baxter Matrix. The symbols that our machine should recognize as input should be the possible columns in the matrix. There are 4 possible columns in a 0-1 matrix with 2 rows, T . We can ignore the column [0,0] T because any column in a baxter matrix must not be all zeros.
As we move through the columns of our input matrix, our machine will keep track of the following information for each row: whether the row is all 0s up to this point (so that it can be used to satisfy future pinwheels) and whether the row must be all 0s in the future (because a pinwheel from earlier is depending on it). Thus each row can be in one of 4 possible states; we will refer to them as the 4 rowstates: 1. This row has a 1 in the most recent column.
2. This row only contains 0s up to now.
3. This row must only contain 0s for the rest of the columns, including the most recent one.
4. This row had a 0 in the most recent column but does not fit 2. or 3.
Our machine will have 16 states, one for each ordered pair of rowstates. We next remove the states which have all rows in rowstates 2, 3, and 4. This is because if all rows had a 0 most recently, then the corresponding column contains only 0s, and the matrix cannot be Baxter. This leaves us with 16 − 9 = 7 states. I claim that the information contained in such a state is enough to determine which columns can come next. Suppose we just read column m, and have a proposed column m + 1. For both pinwheels of index (1, m) we can mostly check whether they are satisfied with the information stored. The information about whether the vertical strips are all 0s is known because we store the exact contents of the most recent column in the state. The horizontal strip going left is available if and only if the corresponding row is in rowstate 2. The only thing we don't know yet is whether a horizontal strip going off to the right is all 0s, but if a pinwheel requires it to be so, we can set the rowstate of the corresponding row to 3, and keep track of it for later. We note some other basic constraints: • Once a row is in rowstate 3 it can never leave rowstate 3.
the electronic journal of combinatorics 29 (2022), #P00 • Once a row leaves rowstate 2 it can never come back to rowstate 2.
• A row cannot transition from 2 to 4.
• A row cannot transition from 2 to 3 directly, or else it will be all zeros. As the automaton proceeds reading columns, it checks whether each new pinwheel can be satisfied. When it encounters such a pinwheel that cannot be satisfied, it can immediately reject the sequence of columns. If a pinwheel cannot be satisfied given the first j columns, there is no Baxter matrix that begins with those first j columns. Similarly, if a pinwheel is satisfied given the first j columns, we do not need to keep track of that pinwheel any longer. It will still be satisfied in any Baxter matrix with those first j columns provided we keep track of which rows must be all 0s in the future.
We additionally add a start state that transitions to all states that have each row in either rowstate 1 or rowstate 2 (rowstates 3 and 4 cannot be reached using only a single column). We designate all states that have no rows in rowstate 2 as accept states (2 must be excluded so that no row of the final matrix is all 0s). Enforcing all of the rules we have described so far yields the automaton in Figure 2. The state label 12 indicates that the first row is in rowstate 1 and the second row is in rowstate 2. The transition label 10 indicates the column [1,0] T . We can do this same process for any fixed number of rows. There will be 2 r symbols, and 4 r − 3 r states. Let's call this automaton A r . Figure 3 contains A 3 :   Proof. Consider an arbitrary transition. The two states in the transition, let's call them s and s ′ , fully specify the contents of two consecutive columns of the matrix, let's call them m and m + 1 respectively. If a row in s is in rowstate 3, it must be in rowstate 3 in s ′ as well. Similarly if a row in s ′ is in rowstate 2, it must be in rowstate 2 in s as well. Therefore the contribution to depth from a single row cannot decrease. The total depth is computed as a sum of contributions from each row, so it follows that the total depth cannot decrease. To show that if the depth remains the same, we must have a self transition, we first note that each individual row's contribution in a depth preserving transition must not increase depth. Otherwise, if such a row did increase its contribution, then we can apply the above reasoning to the other rows, and conclude that the total depth must increase. Therefore all 2's in s must remain 2's, and all 3's in s ′ must have come from 3's in s.

Depth of States
We know that both columns must contain a 1 somewhere. Say the 1 in column m is in row i. It cannot be the case that the only 1 in column m + 1 is in row i, otherwise we would have a self-loop. Therefore there exists a row j = i so that column m + 1 contains a 1 in row j. Let's assume j < i, and note that the other case will be covered by symmetry. The clockwise pinwheel of index (j, m) indicates that row j + 1 must end up in rowstate 3. Since the depth is preserved it must have started in rowstate 3 as well. Now consider the clockwise pinwheel of index (j + 1, m). Since we cannot have row j + 1 starting in rowstate 2, we require row j + 2 to end up in rowstate 3 as well. This pattern continues all the way down to row i where we start in rowstate 1 and end in rowstate 3, which necessarily increases the depth.
In fact we have shown something stronger: whenever there exists 1s in consecutive columns that are not in the same row, the corresponding transition must increase depth. Thus a self-loop can only emerge from states that contain a single 1. It is straightforward to check that if a state has a single 1, then the corresponding self-loop does not break any rules, and is present.

Counting Baxter Matrices
Suppose we fix the number of rows, r, and want to count the number of Baxter matrices with k columns. They are in bijection with paths of length k on our state transition graph. If we fix r and ignore the self-loops, the lemma shows that there are only finitely many possible paths. Thus we can classify all Baxter matrices with r rows into finitely many classes according to the paths they take with all the self-loops removed.
How many k column matrices are there of a particular class? Let P denote a path of length l in the automaton that starts from S and avoids loops, but goes through q states that contain loops. Let N(P, k) be the number of paths of length k arising from P . To compute N(P, k), we just have to choose how many self-loops to put at each node where it is possible to put self-loops, so that the total path length is k. This is the number of ways to choose q natural numbers that sum to k − l. Using stars and bars we get that if k l: which is a polynomial in k of degree q − 1. Note that the polynomial is also correct for l + 1 − q k < l, since it correctly outputs 0 in these cases. Notice that the minimum possible depth of a state is 0 at the start state, and the maximum possible depth is 2r − 1. Thus l 2r − 1, and therefore q 2r − 1 as well. Thus for k 2r − 1, N(P, k) coincides with a polynomial in k of degree at most 2r − 2.
We can also show that the bound is tight, and that there exists a path P * with corresponding polynomial having degree exactly 2r − 2. To do this it suffices to force P * to go through exactly 2r − 1 states with self-loops, i.e. set q = 2r − 1. Let M * be the r × (2r − 1) Matrix that is all 0s except for the following locations: M * [i, i] = 1 for 1 i r, and M * [i, 2r − i] = 1 for 1 i r. Thus M * has two diagonal stripes of 1s that intersect in the middle. M * has no self-loops since it has no repeated columns, but since each column has a single 1, all of the states that is passes through have legal self-loops. It is straightforward to check that M * is indeed a Baxter matrix, and then we can set P * to be the path in A r that corresponds to M * . Now for each possible path P without self-loops, N(P, k) is eventually a polynomial in k of degree at most 2r − 2. The total number of matrices with r rows and k columns will be the sum over all these polynomials, which will be a polynomial of degree exactly 2r − 2. The only small caveat is that for small values of k we must take the maximum of each polynomial and 0, so some paths cannot contribute negatively. Summarizing: Theorem 5. Let a r,l,q be the number of paths of length l in A r that start from S, do not use self-loop transitions, but pass by q states carrying a loop. Let P r (x) be the polynomial in x defined by Then P r (x) has degree 2r − 2.
Let N r,k be the number of Baxter matrices of size r × k. Then, for k 2r − 1 we have N r,k = P r (k).
This shows that for a fixed number of rows, r, the number of Baxter matrices with r rows and k columns eventually satisfies a polynomial in k of degree 2r −2. Computing the polynomial for a fixed r is straightforward once the transition graph has been constructed. The author has Maple code that constructs the graph and computes the corresponding polynomial using the above method. The code completes instantly for r 5 and within a couple minutes for r = 6. The polynomial for r = 2 also appears in Knuth's paper, who found it using a combinatorial approach. rows formula works for 2 k 2 + 3k − 4 k 2 3 (1/3)k 4 + 3k 3 − (16/3)k 2 + 2k + 3 k (361/907200)k 10 + (403/20160)k 9 + (5177/30240)k 8 + . . . k 6 The reader will notice that the polynomials are correct for k r, which is a much better bound than k 2r − 1. Does this pattern continue for larger r? The answer is yes, and we can prove it using Lemma 7 from the next section. Recall that for any path P in A r without self-loops, the polynomial corresponding to P is correct for k l +1−q, where l is the length of P , and q is the number of states with self-loops that P passes through. Thus if we can show that for any P , l + 1 − q r, we can show that N r,k = P r (k), for k r. To compute the maximum of l + 1 − q, we note that l − q is equal to the number the electronic journal of combinatorics 29 (2022), #P00 of states without self-loops that P passes through. Using the terminology of the next section, l − q is equal the number of states with extra 1s that P passes through. By Lemma 7, P cannot pass through two states with extra 1s on consecutive depth levels. We know l 2r − 1, and since the first possible depth a state with extra 1s can occur on is depth 2, we get that l − q r − 1. Thus l + 1 − q r.

Resolving one of Knuth's conjectures
Donald Knuth conjectured that the number of 1s in any r × k Baxter matrix is fewer than (r + k). We know from the definition of Baxter matrices that each column must contain at least one 1. Then we can refer to any 1 that is not the topmost in its column as an extra 1.
Theorem 6. The number of extra 1s in a Baxter matrix with r rows is less than r.
To prove this, we will use the following lemma.
Lemma 7. The total number of extra 1s that appear in two consecutive columns is at most the change in depth of the corresponding state transition in A r .
For now let's assume the lemma is true. Suppose M is some Baxter matrix with r rows and k columns. Let p be its corresponding path in A r , and let T be the set of transitions in p. Let t * be the final state in p. If we use the fact that the start state does not contain any extra 1s and assume that t * does not contain any extra 1s, we get that (# of extra 1s in M) = 1 2 τ ∈T (# of extra 1s in the columns associated with τ ) Suppose t * does contain extra 1s. Let t ′ be the state obtained by replacing all the extra 1s in t * with rowstate 3. We now modify p to contain an extra transition from t * to t ′ , regardless of whether this transition exists in A r . Note that t ′ does not have any extra 1s by construction. This extra transition satisfies Lemma 7, because for each extra 1 in t * , the depth has increased by 1 in that row. Now we can apply Lemma 7 to the (potentially modified) p.