k-pop stack sortable permutations and 2-avoidance

We consider permutations sortable by k passes through a deterministic pop stack. We show that for any k ∈ N the set is characterised by finitely many patterns, answering a question of Claesson and Guðmundsson. Moreover, these sets of patterns are algorithmically constructible. Our characterisation demands a more precise definition than in previous literature of what it means for a permutation to avoid a set of barred and unbarred patterns. We propose a new notion called 2-avoidance. Mathematics Subject Classifications: 05A05


Introduction
A pop stack is a sorting device which operates as follows: at each step it can either push one entry from the input stream onto the top of the stack, or else pop the entire stack contents to the output stream. We consider the entries to be distinct real numbers with the usual ordering. A deterministic pop stack always performs the push move unless the entry on the top of the stack is smaller in value than the entry to be pushed from the input, or if there is no further input. See for example Figure 1. For convenience from now on we assume entries are integers and write a sequence 2, 1, 3 as 213 when entries are single digits.
Observe that by definition the stack remains ordered from smallest on top to largest on the bottom during the operation of a deterministic pop stack.
A permutation (ordered sequence of distinct real numbers) can be sorted by k passes through a deterministic pop stack if after repeating the procedure k times, the sequence  Let p 1 (α) denote the sequence obtained by passing a sequence α through a deterministic pop stack once, and define p k (α) = p k−1 (p 1 (α)). For example p 1 (41352) = 14325 and p 2 (41352) = 12345. We say α is k-pass deterministic pop stack sortable, or k-pop stack sortable for short, if p k (α) is an increasing sequence.
In 1981 Avis and Newborn characterised permutations sorted by a single pass through a pop stack [1], initiating the study of pop stack sorting. Specifically they showed a permutation can be sorted in one pass if and only if it avoids 231 and 312 in the usual sense of pattern avoidance (see Section 2 for precise definitions). Pudwell and Smith characterised permutations sorted by 2 passes, in terms of avoiding a set of six usual patterns and two special barred patterns (defined in Subsection 2.1 below), and computed a rational generating function for the number of such permutations [11]. Claesson and Guðmundsson then computed a rational generating function for permutations sorted by any finite number of passes [3], and asked whether a "useful permutation pattern characterization of the k-pop stack-sortable permutations" exists for k 3.
In this paper we provide such a characterisation. To do so, we realised the current notions of what it means to avoid a set of barred and unbarred patterns would not suffice (see Subsection 2.1). We therefore introduce a new notion called 2-avoidance, which could be of independent interest. In Appendix B we explain how 2-avoidance differs from the notion of mesh pattern avoidance.
We make the following observations. First, our present result is in contrast to the usual (nondeterministic) stacks-in-series model where in many cases no finite pattern-avoidance characterisation is possible due to the existence of infinite antichains [4,10]. Second, the operation of a pop stack is related to classical sorting: "bubble-sort" is exactly sorting by arbitrarily many passes through a pop stack of depth 2. Third, pop stacks are a natural model for genome rearrangement [11].

Preliminaries
Define a permutation to be a sequence of distinct real numbers, written as α = a 1 a 2 · · · a r (we continue the convention to write sequences without commas). The reduced form of a permutation α, denoted red(α), is the permutation obtained by replacing the ith smallest entry of α by the integer i. A permutation in reduced form is called reduced. We denote the set of all reduced permutations by S ∞ . Two permutations α = a 1 · · · a r and β = b 1 · · · b s are order-isomorphic, denoted α ∼ β, if they have the same reduced form. For example 253 and 132 are order-isomorphic. In general, we will write permutations in their reduced form.
A subpermutation of α = a 1 · · · a r is a sequence a i 1 · · · a is where 1 i 1 < . . . < i s r, while a factor is a sequence a i 1 · · · a is where i j+1 = i j + 1 for 1 j s − 1. For two permutations α and β, we say β contains α if some subpermutation of β is orderisomorphic to α. We use the notation α β if β contains α, and the notation α < subperm β if α is a subpermutation of β. For example, 423 < subperm 14253, and 312 14253 since 312 ∼ 423, but 312 < subperm 14253.
We say β avoids α if no subpermutation of β is order-isomorphic to α. For example 54123 contains 312 and avoids 231. For any set of permutations F , let Av(F ) ⊆ S ∞ denote the set of all permutations that simultaneously avoid every α ∈ F .
Knuth famously observed that a permutation can be sorted by passing it through a single stack if and only if it avoids 231 [8]. Avis and Newborn proved a permutation can be sorted by passing it through a pop stack once if and only if it avoids both 231 and 312. However, for multiple passes through a pop stack, the situation arises that some permutation cannot be sorted, while a longer permutation containing it can, so the usual notion of pattern avoidance is not useful for characterising permutations in this context (in other words, for k 2, k-pass pop stack sortable permutations are not a closed class with respect to usual pattern avoidance). As a concrete example, Figure 3 shows that 3241 is not 2-pass pop stack sortable, whereas 41352 is (as demonstrated by Figure 2), and contains 3241.
This leads us to the following notion.
Informally, we think of the set G as patterns which can potentially "save" a permutation from being forbidden by F .
A permutation 2-avoids (F, G) if it does not 2-contain (F, G). Using propositional logic we can express this as follows.
We denote the set of all permutations in S ∞ which 2-avoid (F, G) by Av 2 (F, G).
consists of all permutations in S ∞ except the permutation of length 1. This is a rather extreme example, but shows that the growth of 2-avoidance sets can be factorial.
Using this notion, we can express the result of Pudwell and Smith as follows.
Theorem 5 (Pudwell and Smith [11]). The set of 2-pass pop stack sortable permutations is equal to At this point the reader might object and say that Pudwell and Smith's use of barred patterns (see the next subsection) is more efficient to describe this set, which it is. However for 3-pop stack sortable permutations, the usual definition of avoiding sets of barred and unbarred patterns will fail.

The problem with barred patterns
A barred pattern is a permutation where certain entries (possibly none) are marked with a bar. For example, β = 43512. A barred pattern encodes two permutations, one called removebar(β) obtained by deleting entries marked with a bar, and the other called unbar(β) obtained by removing bars. For example, removebar(43512) = 452 ∼ 231, and unbar(43512) = 43512.
In [7, Definition 1.2.3] a permutation σ is said to avoid a barred pattern β if each occurrence of removebar(β) in σ (if any) is a part of an occurrence of unbar(β) in σ. There are two issues with this definition. Firstly, as written, this does not agree with the usual pattern avoidance when β itself has no bar entries. For example, σ = 21 obviously does not avoid itself in the usual sense of pattern avoidance, but if β = 21 is considered as a barred pattern then σ avoids β since it has a subpermutation, κ = 21 = removebar(β), and κ is part of an occurrence of unbar(β) = 21 in σ.
Secondly and more seriously, in applications such as [11,13] some set of permutations S is characterised by being those permutations avoiding some list of barred and unbarred patterns, where this means that each permutation in S must avoid every pattern individually. Explicitly, Tenner [12] defines, for P a collection of barred and unbarred patterns, Av(P ) to be the set of permutations simultaneously avoiding all patterns in P : However, this notion does not suffice to describe 3-pop stack sortable permutations: one may verify that 32451 is not 3-pop stack sortable while both 4631572 and 4731562 are. If we were to characterise 3-pop stack sortable permutations as those avoiding a list containing 4631572 and 4731562, then we would be mistaken since 4731562 does not avoid this list since it fails to avoid the barred pattern 4631572. Theorem 1 says that even though some permutation may contain a subpermutation which is forbidden, it can be saved if it extends to another subpermutation which appears somewhere on the list G. For the applications in [11,13] the sets of barred and unbarred patterns to be avoided have no "overlap": in the case of [11], the two barred patterns have removebar equal to 2341 and 4312 which are both different from the unbarred patterns in their list; and in the case of [13] there is only one barred pattern whose removebar is different from the unbarred pattern.
Note that our definition of 2-avoidance also differs from the notion of avoiding mesh patterns as defined in [2]. See Appendix B for a brief discussion.

Removing redundant patterns
In general, one cannot simply delete patterns from F if they contain shorter elements of F , or delete patterns from G if they are contained in longer elements of G, since 2containment involves a subtle interplay between the two sets. The following two examples demonstrate this.
Example 6. If F = {43251, 3241} and G = {41352} then we claim that the pattern 43251 ∈ F is not redundant, even though it contains a shorter element in F . Let However, we can state some general rules for removing redundant elements of F or G.  Proofs can be found in Appendix A (no other results depend on them). Certainly we believe that further lemmas could be stated and proved to remove more redundant elements. For the purpose of this paper we do not pursue this; we content ourselves to find a characterisation for k-pop stack sortable permutations in terms of finite sets only.

Blocks
Let σ be a permutation. Call a factor B i = a i,1 a i,2 · · · a i,n i of σ a block if n i > 0 and a i,j > a i,j+1 for all 1 j < n i . (Recall that factor means the entries are contiguous in σ.) A (maximal) block decomposition of σ is an expression of the form σ = B 1 B 2 B 3 · · · B m where each B i is a block and for any two adjacent blocks B i = a i,1 a i,2 · · · a i,n i and B i+1 = a i+1,1 a i+1,2 · · · a i+1,n i+1 we have a i,n i < a i+1,1 . For example σ = 87634521 has block decomposition B 1 = 8763, B 2 = 4, B 3 = 521. For convenience we indicate the block decomposition of σ by inserting | symbols to separate blocks, so for our example we write 8763 | 4 | 521.
If B i = a i,1 a i,2 · · · a i,n i is a block, let B i = a i,n i · · · a i,2 a i,1 . We have the following.
Proof. We proceed by induction. For k = 0, σ is not sortable by 0 passes since a > b. Assume the statement is true for k ∈ N, and consider σ = B 1 · · · B i+1 · · · B i+n · · · B m , a ∈ B i+1 , b ∈ B i+n with n 3 k+1 . Colour the entries a, b and all entries in B i+2 , . . . , B i+n−1 bold. Then the number of bold entries in σ is at least n 3 k+1 since each block contains at least one entry. After one pass, each block of p 1 (σ) can contain at most 3 entries, so the number of blocks with bold entries is at least 3 k+1 3 = 3 k . Since p 1 (σ) = B 1 · · · B i+1 B i+2 · · · B i+n−1 B i+n · · · B m and B i+1 , B i+n contain just one bold entry each (a, b respectively), we have that a must be in the first block with bold entries in the block decomposition of p 1 (σ), and b in the last, so by inductive hypothesis p 1 (σ) cannot be sorted by k passes. Thus σ cannot be sorted by k + 1 passes.
We remark that the bound of 3 k in the preceding statement is most likely an extreme overestimate. However for the purpose of this paper any bound suffices.

General characterisation of k-pass pop stack sortable permutations
Theorem 14. Let k ∈ N + . There exists a pair of finite sets (F k , G k ) such that the set of all k-pass pop stack sortable permutations is equal to Av 2 (F k , G k ). Moreover, the sets F k , G k can be algorithmically constructed.
Claim 1: Ω 1 and Ω 2 are both finite: both are subsets of the set of all permutations of length at most C.
Claim 2: Ω 1 and Ω 2 are algorithmically constructible: we only have finitely many τ, κ of length at most 3f max , C respectively. For each τ, κ we can check τ ∈ S k or κ ∈ S k in linear time by passing them according to the deterministic procedure, and we can check all subpermutations of κ of length at most 3f max to see whether or not at least one has reduced form in Ω 1 . To prove the forwards direction of Claim 3, suppose p 1 (σ) 2-contains (F k−1 , G k−1 ), and further assume this is because of some ζ < subperm p 1 (σ) with red(ζ) ∈ F k−1 , and there is no δ < subperm σ with ζ < subperm δ and red(δ) ∈ G k−1 . Note, there may be many choices of ζ to take, but fix one choice.
1. Mark entries corresponding to ζ in p 1 (σ) bold. Let ζ < subperm σ be such that after one pass, the entries belonging to ζ are the bold entries corresponding to ζ < subperm p 1 (σ). Mark the ζ entries bold as well. Note that |ζ | f max .
2. Next, write σ in block decomposition σ = B 1 B 2 B 3 · · · B m with B i = a i,1 · · · a i,n i . Say that B i is bold if it contains at least one bold entry (from ζ ). We wish to delete non-bold entries of σ but we do not want to merge bold blocks, so we apply the following subroutine.
-While a i,j is a non-bold letter, -if removing a i,j from τ does not cause two or more bold blocks to merge, delete a i,j from τ .
We claim that at the end of this process |τ | 3|ζ | 3f max . Let a i,j ∈ B i with 1 < i < m be a non-bold entry in τ . If at most one of B i−1 , B i , B i+1 is bold, then removing a i,j cannot merge bold blocks. Else assume at least two of B i−1 , B i , B i+1 are bold. If a i,j is not the first or last entry in B i , it can be deleted without merging blocks. This leaves at most two unbold entries in B i . For B 1 (resp. B m ) we can delete all except the last (resp. first) entry without merging bold blocks. This leaves at most two unbold entries in each block. Then in the worst case each block contains just one bold entry, with an unbold entry on either side. For example, if we get to τ = . . . | 12, 10, 8 | 975 | 642 | 31 then we cannot delete 8, 9, 5, 6, 2, 3 without merging blocks.
Thus since |τ | 3f max and τ ∈ S k , we have τ ∈ Ω 1 by definition. To finish this direction, we will show that τ is not saved by any subpermutation order-isomorphic to something in Ω 2 .
Here is how we construct κ. In σ, mark the entries corresponding to γ bold. Call a block of σ bold if it contains at least one bold entry, and otherwise call it non-bold. Starting with κ = σ, we delete non-bold entries using the following procedure, which is more careful than the subroutine used in the proof of the forward direction above. The goal is to delete non-bold entries to obtain a permutation κ with subpermutation γ such that for every block B in κ there is a block B in σ so that the entries in B are entries in B . That is, we do not allow any (bold or unbold) blocks to merge, only to be deleted entirely.
-While a i,j ∈ B i is a non-bold letter, -if removing a i,j from κ does not cause two or more blocks of any kind (bold or non-bold) in κ to merge, delete a i,j from κ, -if B i is non-bold and removing the entire block B i at once does not cause any of the remaining blocks to merge, then delete B i .
We claim that at the end of this process each block contains at most two non-bold entries, which will be the first and last entries of the block. However, since we have not deleted nonbold blocks if their removal would cause other blocks to merge, we could have arbitrarily long factors of non-bold blocks, as in the example in Suppose κ has block decomposition T 1 · · · T m , and κ contains two bold entries a, b with a ∈ T i+1 and b ∈ T i+n , where T i+2 , . . . , T i+n−1 have no bold entries. (Note that each T j is a subset of some B k in the block decomposition of σ, by construction.) If a < b then the subroutine is not complete: we could delete T i+2 , . . . , T i+n−1 completely without merging T i+1 with T i+n since the last entry of T i+1 is smaller than the first entry of T i+n . Therefore since we assume the subroutine is complete, we know that a > b. If n 3 k then, by Lemma 13, σ cannot be sorted by k passes, which is a contradiction. Thus we have |κ| is bounded above by 3|γ| + 2.3 k |γ|, where 3|γ| counts the bold blocks with a non-bold first and last entry, and 2.3 k |γ| counts the 3 k factors each containing 2 entries, in the worst case, occurring between every pair of bold entries from γ, as in Figure 4. Thus If κ cannot be sorted, then p 1 (κ) ∈ S k−1 because of some subpermutation τ with red(τ ) ∈ F k−1 , but since no block has merged in obtaining p 1 (κ), p 1 (σ) also contains τ which cannot be saved since blocks containing the entries forming τ are fixed. Thus This proves Claim 3.
Claim 3 implies that we could take F k = Ω 1 , G k = Ω 2 and the theorem is done. However, we can first apply Theorems 9 and 10 1 to obtain smaller sets with the same 2-avoidance set, so we will call the result of applying these (in some order) (F k , G k ). As remarked in Subsection 2.2, the result of applying the lemmas is not guaranteed to give a set that is minimal or unique. Note that each of these lemmas needs to check a finite set, so each is algorithmic.
Applying the construction in the proof above to the case k = 2 yields a much larger pair of sets than those appearing in Theorem 5 given by Pudwell and Smith. Explicitly, we have f max = 3 and k = 2, so C = 3.3 4 = 243. The point of our argument is that it is general; our upper bounds can certainly be lowered, and more lemmas to reduce the size of the sets F, G could be proved along the lines of Lemmas 8-10 to sharpen the result and render it more tractable for computation. See [5] for a discussion of an attempt to compute explicit avoidance sets for the case k = 3, where if we start with the sets F 2 , G 2 in Theorem 5 given by Pudwell and Smith rather than the larger sets produced by our algorithm, we have f max = 4 and C = 4.3 5 = 972 (so a brute force calculation to find Ω 1 , Ω 2 would need to examine C! = 972! permutations).

Outlook
Our notion of 2-containment opens up some interesting possibilities. Recall that by Kaiser-Klazar [6, Thm. 3.4] and Marcus-Tardos [9] the function counting the number of permutations of length n in any Av(F ) for F non-empty is either polynomial or exponential. It is conceivable that some pair of (finite or infinite) sets (F, G) could have a 2-avoidance set with a growth function strictly between polynomial and exponential or between exponential and factorial. Example 4 shows non-trivial 2-avoidance sets with factorial growth.
Generating functions for 2-avoidance sets might also exhibit interesting behaviour. For the sets (F k , G k ) in Theorem 14 we know by [3] the generating functions are rational for all k, but for general sets F, G, the set Av 2 (F, G) could have interesting enumerations.

B Mesh patterns
For the sake of disambiguation we show here that 2-avoidance is in general a different notion from that of avoiding (a set of) mesh patterns.
For n ∈ N let S n denote the set of all reduced permutations of length n, and let [a, b] be the integer interval {i ∈ Z | a i b}.
Barred patterns with a single barred entry can be viewed as mesh patterns, where R consists of a single point. For example, the two mesh patterns in Figure 5 represent the two barred patterns 41352 and 41352 respectively. Explicitly, from [2], if π(i) is the only barred letter of a given barred pattern π, then the corresponding mesh pattern is (red(removebar(π)), {(i − 1, π(i) − 1)}). However, avoiding a barred pattern β with more than one barred entry is not the same as avoiding the mesh pattern obtained in the analogous way (that is, taking (red(removebar(β)), R), with R containing a point for each barred entry of β in the appropriate location). To see this, let π = red(removebar(β)) and note that by definition τ avoids (π, R) if for every subpermutation π of τ with π ∼ π, there is at least one entry from τ inside a shaded square. By contrast, τ avoids the barred pattern β if for every subpermutation π of τ with π ∼ π, there is a subpermutation δ of τ with π < subperm δ and δ ∼ unbar(β). In other words, we would require one entry of τ in every shaded square.
One could potentially fix this by modifying the definition of avoiding a meshed pattern to encompass barred patterns with multiple barred entries. However, recall that in Subsection 2.1 we observed that 32451 is not 3-pop stack sortable while both 4631572 and 4731562 are. It is not clear how avoiding a set of mesh patterns could be defined to describe this situation. One might try to define mesh patterns (32451, R 1 ) and (32451, R 2 ) where R 1 , R 2 encode positions which must be present in any permutation containing 32451, and modify the definition of avoiding a set of mesh patterns to say that a permutation τ avoids a set of mesh patterns if for every subpermutation π of τ with π ∼ π and π appears in the first coordinate of a mesh pattern in the set, there is some R so that (π, R) is in the set, and τ contains an entry in every region coded by R. Following this kind of reasoning, we came to the definition of 2-avoidance rather than the existing notions.