Swin Transformer, two-column equations and full-width figure, PDF page 4

Scoring unit: science-swin-columns. Joined pages use the same full output.

**(a) Architecture**

Images ($H \times W \times 3$) → Patch Partition ($\frac{H}{4} \times \frac{W}{4} \times 48$) → **Stage 1**: Linear Embedding → Swin Transformer Block ×2 ($\frac{H}{4} \times \frac{W}{4} \times C$) → **Stage 2**: Patch Merging → Swin Transformer Block ×2 ($\frac{H}{8} \times \frac{W}{8} \times 2C$) → **Stage 3**: Patch Merging → Swin Transformer Block ×6 ($\frac{H}{16} \times \frac{W}{16} \times 4C$) → **Stage 4**: Patch Merging → Swin Transformer Block ×2 ($\frac{H}{32} \times \frac{W}{32} \times 8C$)

**(b) Two Successive Swin Transformer Blocks**

Block 1: $\mathbf{z}^{l-1}$ → LN → W-MSA → ⊕ → $\hat{\mathbf{z}}^{l}$ → LN → MLP → ⊕ → $\mathbf{z}^{l}$

Block 2: $\mathbf{z}^{l}$ → LN → SW-MSA → ⊕ → $\hat{\mathbf{z}}^{l+1}$ → LN → MLP → ⊕ → $\mathbf{z}^{l+1}$

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

**Swin Transformer block** Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU non-linearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

### 3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

**Self-attention in non-overlapped windows** For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains $M \times M$ patches, the computational complexity of a global MSA module and a window based one on an image of $h \times w$ patches are³:

$$\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C, \tag{1}$$
$$\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2hwC, \tag{2}$$

where the former is quadratic to patch number $hw$, and the latter is linear when $M$ is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large $hw$, while the window based self-attention is scalable.

**Shifted window partitioning in successive blocks** The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the $8 \times 8$ feature map is evenly partitioned into $2 \times 2$ windows of size $4 \times 4$ ($M = 4$). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by $(\lfloor \frac{M}{2} \rfloor, \lfloor \frac{M}{2} \rfloor)$ pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

$$
\begin{aligned}
&\hat{\mathbf{z}}^{l} = \text{W-MSA}\left(\text{LN}\left(\mathbf{z}^{l-1}\right)\right) + \mathbf{z}^{l-1},\\
&\mathbf{z}^{l} = \text{MLP}\left(\text{LN}\left(\hat{\mathbf{z}}^{l}\right)\right) + \hat{\mathbf{z}}^{l},\\
&\hat{\mathbf{z}}^{l+1} = \text{SW-MSA}\left(\text{LN}\left(\mathbf{z}^{l}\right)\right) + \mathbf{z}^{l},\\
&\mathbf{z}^{l+1} = \text{MLP}\left(\text{LN}\left(\hat{\mathbf{z}}^{l+1}\right)\right) + \hat{\mathbf{z}}^{l+1},
\end{aligned} \tag{3}
$$

where $\hat{\mathbf{z}}^{l}$ and $\mathbf{z}^{l}$ denote the output features of the (S)W-MSA module and the MLP module for block $l$, respectively;

---

³We omit SoftMax computation in determining complexity.

4

Scoring unit: science-swin-columns. Joined pages use the same full output.

\(H \times W \times 3\)

Images → Patch Partition → Linear Embedding → Swin Transformer Block ×2 → Patch Merging → Swin Transformer Block ×2 → Patch Merging → Swin Transformer Block ×6 → Patch Merging → Swin Transformer Block ×2

| | Stage 1 | Stage 2 | Stage 3 | Stage 4 |
|---|---|---|---|---|
| | \(\frac{H}{4} \times \frac{W}{4} \times 48\) → \(\frac{H}{4} \times \frac{W}{4} \times C\) | \(\frac{H}{8} \times \frac{W}{8} \times 2C\) | \(\frac{H}{16} \times \frac{W}{16} \times 4C\) | \(\frac{H}{32} \times \frac{W}{32} \times 8C\) |

(a) Architecture

\(z^{l-1}\) → LN → W-MSA → ⊕ → \(\hat{z}^{l}\) → LN → MLP → ⊕ → \(z^{l}\)

\(z^{l}\) → LN → SW-MSA → ⊕ → \(\hat{z}^{l+1}\) → LN → MLP → ⊕ → \(z^{l+1}\)

(b) Two Successive Swin Transformer Blocks

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

**Swin Transformer block** Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU non-linearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

## 3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

**Self-attention in non-overlapped windows** For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains \(M \times M\) patches, the computational complexity of a global MSA module and a window based one on an image of \(h \times w\) patches are³:

\[
\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C, \tag{1}
\]

\[
\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2hwC, \tag{2}
\]

where the former is quadratic to patch number \(hw\), and the latter is linear when \(M\) is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large \(hw\), while the window based self-attention is scalable.

**Shifted window partitioning in successive blocks** The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the \(8 \times 8\) feature map is evenly partitioned into \(2 \times 2\) windows of size \(4 \times 4\) (\(M = 4\)). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by \((\lfloor \frac{M}{2} \rfloor, \lfloor \frac{M}{2} \rfloor)\) pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

\[
\begin{aligned}
\hat{z}^{l} &= \text{W-MSA}\left(\text{LN}\left(z^{l-1}\right)\right) + z^{l-1},\\
z^{l} &= \text{MLP}\left(\text{LN}\left(\hat{z}^{l}\right)\right) + \hat{z}^{l},\\
\hat{z}^{l+1} &= \text{SW-MSA}\left(\text{LN}\left(z^{l}\right)\right) + z^{l},\\
z^{l+1} &= \text{MLP}\left(\text{LN}\left(\hat{z}^{l+1}\right)\right) + \hat{z}^{l+1},
\end{aligned}
\tag{3}
\]

where \(\hat{z}^{l}\) and \(z^{l}\) denote the output features of the (S)W-MSA module and the MLP module for block \(l\), respectively;

---

³We omit SoftMax computation in determining complexity.

4

Scoring unit: science-swin-columns. Joined pages use the same full output.

![Figure 3 diagram: (a) Architecture. Images (H×W×3) → Patch Partition → Linear Embedding (H/4 × W/4 × 48 → H/4 × W/4 × C) → Stage 1: Swin Transformer Block ×2 → Patch Merging (H/8 × W/8 × 2C) → Stage 2: Swin Transformer Block ×2 → Patch Merging (H/16 × W/16 × 4C) → Stage 3: Swin Transformer Block ×6 → Patch Merging (H/32 × W/32 × 8C) → Stage 4: Swin Transformer Block ×2. (b) Two Successive Swin Transformer Blocks: z^(l-1) → LN → W-MSA → ⊕ (ẑ^l) → LN → MLP → ⊕ (z^l); z^l → LN → SW-MSA → ⊕ (ẑ^(l+1)) → LN → MLP → ⊕ (z^(l+1)), with residual connections.]

(a) Architecture

(b) Two Successive Swin Transformer Blocks

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

**Swin Transformer block** Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU non-linearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

### 3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

**Self-attention in non-overlapped windows** For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains $M \times M$ patches, the computational complexity of a global MSA module and a window based one on an image of $h \times w$ patches are$^3$:

$$\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C, \tag{1}$$

$$\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2hwC, \tag{2}$$

where the former is quadratic to patch number $hw$, and the latter is linear when $M$ is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large $hw$, while the window based self-attention is scalable.

**Shifted window partitioning in successive blocks** The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the $8 \times 8$ feature map is evenly partitioned into $2 \times 2$ windows of size $4 \times 4$ ($M = 4$). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by $(\lfloor \frac{M}{2} \rfloor, \lfloor \frac{M}{2} \rfloor)$ pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

$$\hat{\mathbf{z}}^l = \text{W-MSA}\left(\text{LN}\left(\mathbf{z}^{l-1}\right)\right) + \mathbf{z}^{l-1},$$
$$\mathbf{z}^l = \text{MLP}\left(\text{LN}\left(\hat{\mathbf{z}}^l\right)\right) + \hat{\mathbf{z}}^l,$$
$$\hat{\mathbf{z}}^{l+1} = \text{SW-MSA}\left(\text{LN}\left(\mathbf{z}^l\right)\right) + \mathbf{z}^l,$$
$$\mathbf{z}^{l+1} = \text{MLP}\left(\text{LN}\left(\hat{\mathbf{z}}^{l+1}\right)\right) + \hat{\mathbf{z}}^{l+1}, \tag{3}$$

where $\hat{\mathbf{z}}^l$ and $\mathbf{z}^l$ denote the output features of the (S)W-MSA module and the MLP module for block $l$, respectively;

$^3$We omit SoftMax computation in determining complexity.

4

Scoring unit: science-swin-columns. Joined pages use the same full output.

![Figure 3 Architecture Diagram](image)  
(a) Architecture (b) Two Successive Swin Transformer Blocks

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

**Swin Transformer block** Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU nonlinearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

### 3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

**Self-attention in non-overlapped windows** For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains $M \times M$ patches, the computational complexity of a global MSA module and a window based one on an image of $h \times w$ patches are$^3$:

$$\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C, \tag{1}$$

$$\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2hwC, \tag{2}$$

where the former is quadratic to patch number $hw$, and the latter is linear when $M$ is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large $hw$, while the window based self-attention is scalable.

**Shifted window partitioning in successive blocks** The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the $8 \times 8$ feature map is evenly partitioned into $2 \times 2$ windows of size $4 \times 4$ ($M = 4$). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by $(\lfloor\frac{M}{2}\rfloor, \lfloor\frac{M}{2}\rfloor)$ pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

$$\begin{aligned}
\hat{\mathbf{z}}^l &= \text{W-MSA}\left(\text{LN}\left(\mathbf{z}^{l-1}\right)\right) + \mathbf{z}^{l-1}, \\
\mathbf{z}^l &= \text{MLP}\left(\text{LN}\left(\hat{\mathbf{z}}^l\right)\right) + \hat{\mathbf{z}}^l, \\
\hat{\mathbf{z}}^{l+1} &= \text{SW-MSA}\left(\text{LN}\left(\mathbf{z}^l\right)\right) + \mathbf{z}^l, \\
\mathbf{z}^{l+1} &= \text{MLP}\left(\text{LN}\left(\hat{\mathbf{z}}^{l+1}\right)\right) + \hat{\mathbf{z}}^{l+1},
\end{aligned} \tag{3}$$

where $\hat{\mathbf{z}}^l$ and $\mathbf{z}^l$ denote the output features of the (S)W-MSA module and the MLP module for block $l$, respectively;

---
$^3$We omit SoftMax computation in determining complexity.

4

Scoring unit: science-swin-columns. Joined pages use the same full output.

No output

Scoring unit: science-swin-columns. Joined pages use the same full output.

<div data-bbox="78 93 906 276" data-label="Diagram">
<img alt="Figure 3: (a) Architecture of a Swin Transformer (Swin-T) showing four stages with increasing resolution and patch merging. (b) Two successive Swin Transformer Blocks showing the internal structure of a block with a Swin Transformer Block, Patch Merging, and a residual connection."/>
<p>Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.</p>
</div>
<div data-bbox="78 280 900 308" data-label="Caption">
<p>Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.</p>
</div>
<div data-bbox="78 332 470 407" data-label="Text">
<p>with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.</p>
</div>
<div data-bbox="78 448 470 600" data-label="Text">
<p><b>Swin Transformer block</b> Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU non-linearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.</p>
</div>
<div data-bbox="78 620 398 636" data-label="Section-Header">
<h3>3.2. Shifted Window based Self-Attention</h3>
</div>
<div data-bbox="78 647 470 770" data-label="Text">
<p>The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.</p>
</div>
<div data-bbox="78 808 470 900" data-label="Text">
<p><b>Self-attention in non-overlapped windows</b> For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains <math>M \times M</math> patches, the computational complexity of a global MSA module and a window based one</p>
</div>
<div data-bbox="500 332 736 348" data-label="Text">
<p>on an image of <math>h \times w</math> patches are<sup>3</sup>:</p>
</div>
<div data-bbox="570 357 898 375" data-label="Equation-Block">
<math display="block">\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C, \quad (1)</math>
</div>
<div data-bbox="570 380 898 396" data-label="Equation-Block">
<math display="block">\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2hwC, \quad (2)</math>
</div>
<div data-bbox="500 407 898 467" data-label="Text">
<p>where the former is quadratic to patch number <math>hw</math>, and the latter is linear when <math>M</math> is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large <math>hw</math>, while the window based self-attention is scalable.</p>
</div>
<div data-bbox="500 486 898 606" data-label="Text">
<p><b>Shifted window partitioning in successive blocks</b> The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.</p>
</div>
<div data-bbox="500 606 898 713" data-label="Text">
<p>As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the <math>8 \times 8</math> feature map is evenly partitioned into <math>2 \times 2</math> windows of size <math>4 \times 4</math> (<math>M = 4</math>). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by <math>(\lfloor \frac{M}{2} \rfloor, \lfloor \frac{M}{2} \rfloor)</math> pixels from the regularly partitioned windows.</p>
</div>
<div data-bbox="500 713 898 744" data-label="Text">
<p>With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as</p>
</div>
<div data-bbox="574 753 898 833" data-label="Equation-Block">
<math display="block">\begin{aligned} \hat{z}^l &amp;= \text{W-MSA}(\text{LN}(\hat{z}^{l-1})) + \hat{z}^{l-1}, \\ z^l &amp;= \text{MLP}(\text{LN}(\hat{z}^l)) + \hat{z}^l, \\ \hat{z}^{l+1} &amp;= \text{SW-MSA}(\text{LN}(\hat{z}^l)) + \hat{z}^l, \\ z^{l+1} &amp;= \text{MLP}(\text{LN}(\hat{z}^{l+1})) + \hat{z}^{l+1}, \end{aligned} \quad (3)</math>
</div>
<div data-bbox="500 844 898 876" data-label="Text">
<p>where <math>\hat{z}^l</math> and <math>z^l</math> denote the output features of the (S)W-MSA module and the MLP module for block <math>l</math>, respectively;</p>
</div>
<div data-bbox="517 884 836 900" data-label="Footnote">
<p><sup>3</sup>We omit SoftMax computation in determining complexity.</p>
</div>
<div data-bbox="478 920 492 935" data-label="Page-Footer">
<p>4</p>
</div>

Scoring unit: science-swin-columns. Joined pages use the same full output.

<file name="science-swin-columns.pdf">
![img-0.jpeg](img-0.jpeg)
Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

Swin Transformer block Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU nonlinearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

# 3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

Self-attention in non-overlapped windows For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains  $M \times M$  patches, the computational complexity of a global MSA module and a window based one

on an image of  $h \times w$  patches are:

$$
\Omega (\mathrm {M S A}) = 4 h w C ^ {2} + 2 (h w) ^ {2} C, \tag {1}
$$

$$
\Omega (\mathrm {W} - \mathrm {M S A}) = 4 h w C ^ {2} + 2 M ^ {2} h w C, \tag {2}
$$

where the former is quadratic to patch number  $hw$ , and the latter is linear when  $M$  is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large  $hw$ , while the window based self-attention is scalable.

Shifted window partitioning in successive blocks The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the  $8 \times 8$  feature map is evenly partitioned into  $2 \times 2$  windows of size  $4 \times 4$  ( $M = 4$ ). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by  $\left(\left\lfloor \frac{M}{2} \right\rfloor, \left\lfloor \frac{M}{2} \right\rfloor\right)$  pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

$$
\hat {\mathbf {z}} ^ {l} = \mathrm {W - M S A} \left(\mathrm {L N} \left(\mathbf {z} ^ {l - 1}\right)\right) + \mathbf {z} ^ {l - 1},
$$

$$
\mathbf {z} ^ {l} = \operatorname {M L P} \left(\operatorname {L N} \left(\hat {\mathbf {z}} ^ {l}\right)\right) + \hat {\mathbf {z}} ^ {l},
$$

$$
\hat {\mathbf {z}} ^ {l + 1} = \operatorname {S W - M S A} \left(\operatorname {L N} \left(\mathbf {z} ^ {l}\right)\right) + \mathbf {z} ^ {l},
$$

$$
\mathbf {z} ^ {l + 1} = \operatorname {M L P} \left(\operatorname {L N} \left(\hat {\mathbf {z}} ^ {l + 1}\right)\right) + \hat {\mathbf {z}} ^ {l + 1}, \tag {3}
$$

where  $\hat{\mathbf{z}}^l$  and  $\mathbf{z}^l$  denote the output features of the (S)W-MSA module and the MLP module for block  $l$ , respectively;
</file>

Scoring unit: science-swin-columns. Joined pages use the same full output.

**(a) Architecture**

Images  
\(H \times W \times 3\) → Patch Partition → \(H/4 \times W/4 \times 48\) → Linear Embedding

- **Stage 1:** \(H/4 \times W/4 \times C\) → Swin Transformer Block ×2 → Patch Merging
- **Stage 2:** \(H/8 \times W/8 \times 2C\) → Swin Transformer Block ×2 → Patch Merging
- **Stage 3:** \(H/16 \times W/16 \times 4C\) → Swin Transformer Block ×6 → Patch Merging
- **Stage 4:** \(H/32 \times W/32 \times 8C\) → Swin Transformer Block ×2

**(b) Two Successive Swin Transformer Blocks**

- First block: \(z^{l-1}\) → LN → W-MSA → LN → MLP → \(z^l\)
- Second block: \(\hat{z}^{l+1}\) → LN → SW-MSA → LN → MLP → \(z^{l+1}\)

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

**Swin Transformer block**  Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU non-linearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

**3.2. Shifted Window based Self-Attention**

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

**Self-attention in non-overlapped windows**  For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains \(M \times M\) patches, the computational complexity of a global MSA module and a window based one on an image of \(h \times w\) patches are³:

\[
\Omega(\mathrm{MSA}) = 4hwC^2 + 2(hw)^2C,
\tag{1}
\]

\[
\Omega(\mathrm{W\text{-}MSA}) = 4hwC^2 + 2M^2hwC,
\tag{2}
\]

where the former is quadratic to patch number \(hw\), and the latter is linear when \(M\) is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large \(hw\), while the window based self-attention is scalable.

**Shifted window partitioning in successive blocks**  The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the \(8 \times 8\) feature map is evenly partitioned into \(2 \times 2\) windows of size \(4 \times 4\) (\(M = 4\)). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by \((\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)\) pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

\[
\hat{z}^{l} = \mathrm{W\text{-}MSA}(\mathrm{LN}(z^{l-1})) + z^{l-1},
\]

\[
z^{l} = \mathrm{MLP}(\mathrm{LN}(\hat{z}^{l})) + \hat{z}^{l},
\]

\[
\hat{z}^{l+1} = \mathrm{SW\text{-}MSA}(\mathrm{LN}(z^{l})) + z^{l},
\]

\[
z^{l+1} = \mathrm{MLP}(\mathrm{LN}(\hat{z}^{l+1})) + \hat{z}^{l+1},
\tag{3}
\]

where \(\hat{z}^{l}\) and \(z^{l}\) denote the output features of the (S)W-MSA module and the MLP module for block \(l\), respectively;

³We omit SoftMax computation in determining complexity.

4

Scoring unit: science-swin-columns. Joined pages use the same full output.

![image](86,90,900,270) Architecture and two successive Swin Transformer Blocks diagram

![caption](90,277,896,303) Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

![text](89,329,502,402) with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

![text](90,448,502,599) **Swin Transformer block** Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU non-linearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

![title](90,624,427,639) ## 3.2. Shifted Window based Self-Attention

![text](89,653,502,774) The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

![text](90,816,502,903) **Self-attention in non-overlapped windows** For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains $M \times M$ patches, the computational complexity of a global MSA module and a window based one

![text+](535,329,743,345) on an image of $h \times w$ patches are $^{3}$ :

![formula](589,355,900,375) $$
\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C, \quad (1)
$$

![formula](595,375,900,395) $$
\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2hwC, \quad (2)
$$

![text](539,407,896,465) where the former is quadratic to patch number $hw$ , and the latter is linear when $M$ is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large $hw$ , while the window based self-attention is scalable.

![text](539,486,896,603) **Shifted window partitioning in successive blocks** The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

![text](539,605,896,711) As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the $8 \times 8$ feature map is evenly partitioned into $2 \times 2$ windows of size $4 \times 4$ ( $M = 4$ ). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by $(\lfloor \frac{M}{2} \rfloor, \lfloor \frac{M}{2} \rfloor)$ pixels from the regularly partitioned windows.

![text](540,711,896,740) With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

![formula](593,750,825,770) $$
\hat{\mathbf{z}}^l = \text{W-MSA} \left( \text{LN} \left( \mathbf{z}^{l-1} \right) \right) + \mathbf{z}^{l-1},
$$

![formula](593,772,775,792) $$
\mathbf{z}^l = \text{MLP} \left( \text{LN} \left( \hat{\mathbf{z}}^l \right) \right) + \hat{\mathbf{z}}^l,
$$

![formula](593,794,820,814) $$
\hat{\mathbf{z}}^{l+1} = \text{SW-MSA} \left( \text{LN} \left( \mathbf{z}^l \right) \right) + \mathbf{z}^l,
$$

![formula](600,814,899,835) $$
\mathbf{z}^{l+1} = \text{MLP} \left( \text{LN} \left( \hat{\mathbf{z}}^{l+1} \right) \right) + \hat{\mathbf{z}}^{l+1}, \quad (3)
$$

![text](539,846,896,877) where $\hat{\mathbf{z}}^l$ and $\mathbf{z}^l$ denote the output features of the (S)W-MSA module and the MLP module for block $l$ , respectively;

![footnote](552,887,840,902) $^{3}$ We omit SoftMax computation in determining complexity.

![page_number](495,924,509,937) 4

Scoring unit: science-swin-columns. Joined pages use the same full output.

# Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

## Architecture Diagram

<table>
  <tr>
    <th>Input</th>
    <th>Stage 1</th>
    <th>Stage 2</th>
    <th>Stage 3</th>
    <th>Stage 4</th>
  </tr>
<tr>
    <td>H×W×3 Images</td>
<td>Patch Partition<br />Linear Embedding<br />Swin Transformer Block<br />x2</td>
<td>Patch Merging<br />Swin Transformer Block<br />x2</td>
<td>Patch Merging<br />Swin Transformer Block<br />x6</td>
<td>Patch Merging<br />Swin Transformer Block<br />x2</td>
  </tr>
</table>

## Two Successive Swin Transformer Blocks

<table>
  <tr>
    <th>Block 1</th>
    <th>Block 2</th>
  </tr>
<tr>
    <td>LN<br />W-MSA<br />LN<br />MLP</td>
<td>LN<br />SW-MSA<br />LN<br />MLP</td>
  </tr>
</table>

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG  [52] and ResNet  [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

## Swin Transformer block

Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU nonlinearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

## 3.2. Shifted Window based Self-Attention

The standard Transformer architecture  [64] and its adaptation for image classification  [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

### Self-attention in non-overlapped windows

For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Suppose each window contains $M \times M$ patches, the computational complexity of a global MSA module and a window based one on an image of $h \times w$ patches are<sup>3</sup>:

$$\Omega(MSA) = 4hwC^2 + 2(hw)^2C, \tag{1}$$

$$\Omega(W-MSA) = 4hwC^2 + 2M^2hwC, \tag{2}$$

where the former is quadratic to patch number $hw$, and the latter is linear when $M$ is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large $hw$, while the window based self-attention is scalable.

### Shifted window partitioning in successive blocks

The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the $8 \times 8$ feature map is evenly partitioned into $2 \times 2$ windows of size $4 \times 4$ ($M=4$). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by $(\lfloor \frac{M}{2} \rfloor, \lfloor \frac{M}{2} \rfloor)$ pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

$$\hat{z}^l = W-MSA(LN(\hat{z}^{l-1})) + \hat{z}^{l-1},$$

$$\hat{z}^l = MLP(LN(\hat{z}^l)) + \hat{z}^l,$$

$$\hat{z}^{l+1} = SW-MSA(LN(\hat{z}^l)) + \hat{z}^l,$$

$$\hat{z}^{l+1} = MLP(LN(\hat{z}^{l+1})) + \hat{z}^{l+1}, \tag{3}$$

where $\hat{z}^l$ and $\hat{z}^l$ denote the output features of the (S)W-MSA module and the MLP module for block $l$, respectively;

<sup>3</sup>We omit SoftMax computation in determining complexity.

4

Scoring unit: science-swin-columns. Joined pages use the same full output.

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

**Swin Transformer block** Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU non-linearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

## 3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

**Self-attention in non-overlapped windows** For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains *M* × *M* patches, the computational complexity of a global MSA module and a window based one

on an image of *h* × *w* patches are³:

$$ \Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C, \quad (1) $$

$$ \Omega(\text{W-MSA}) = 4hwC^2 + 2M^2hwC, \quad (2) $$

where the former is quadratic to patch number *hw*, and the latter is linear when *M* is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large *hw*, while the window based self-attention is scalable.

**Shifted window partitioning in successive blocks** The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the 8 × 8 feature map is evenly partitioned into 2 × 2 windows of size 4 × 4 (*M* = 4). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by ( ⌊ *M* / 2 ⌋, ⌊ *M* / 2 ⌋) pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

$$ \begin{align} \hat{z}^l &= \text{W-MSA}(\text{LN}(\mathbf{z}^{l-1})) + \mathbf{z}^{l-1}, \\ \mathbf{z}^l &= \text{MLP}(\text{LN}(\hat{z}^l)) + \hat{z}^l, \\ \hat{z}^{l+1} &= \text{SW-MSA}(\text{LN}(\mathbf{z}^l)) + \mathbf{z}^l, \\ \mathbf{z}^{l+1} &= \text{MLP}(\text{LN}(\hat{z}^{l+1})) + \hat{z}^{l+1}, \tag{3} \end{align} $$

where $\hat{z}^l$ and $\mathbf{z}^l$ denote the output features of the (S)W-MSA module and the MLP module for block $l$, respectively;

³We omit SoftMax computation in determining complexity.

Scoring unit: science-swin-columns. Joined pages use the same full output.

zl
H 4 x W 4 ×48 H 4 X W 4 ×C H 8 x W 8 ×2C H 16 x W 16 ×4C H 32 x W 32 x8C MLP MLP
Stage 1 Stage 2 Stage 3 Stage 4 LN LN
LH
H×Wx3 Swin Transformer Block Swin Transformer Block Patch Merging Zl
Images Linear Embedding Swin Transformer Block Swin Transformer Block W-MSA SW-MSA
Patch Partition Patch Partition
X2 x2 x6 x2 zl−1A zl

(a) Architecture

(b) Two Successive Swin Transformer Blocks

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with
Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical
convolutional networks, e.g., VGG [52] and ResNet [30].
As a result, the proposed architecture can conveniently re-
place the backbone networks in existing methods for vari-
ous vision tasks.

**Swin Transformer block** Swin Transformer is built by
replacing the standard multi-head self attention (MSA)
module in a Transformer block by a module based on
shifted windows (described in Section 3.2), with other lay-
ers kept the same. As illustrated in Figure 3(b), a Swin
Transformer block consists of a shifted window based MSA
module, followed by a 2-layer MLP with GELU non-
linearity in between. A LayerNorm (LN) layer is applied
before each MSA module and each MLP, and a residual
connection is applied after each module.

## **3.2 . Shifted Window based Self-Attention**

The standard Transformer architecture [64] and its adap-
tation for image classification [20] both conduct global self-
attention, where the relationships between a token and all
other tokens are computed. The global computation leads to
quadratic complexity with respect to the number of tokens,
making it unsuitable for many vision problems requiring an
immense set of tokens for dense prediction or to represent a
high-resolution image.

**Self-attention in non-overlapped windows** For efficient
modeling, we propose to compute self-attention within lo-
cal windows. The windows are arranged to evenly partition
the image in a non-overlapping manner. Supposing each
window contains M × M patches, the computational com-
plexity of a global MSA module and a window based one

on an image of h × w patches are<sup>3</sup>:

Ω(MSA) = 4hwC2 + 2(hw)2C,

(1)

Ω(W-MSA) = 4hwC2 + 2M2hwC,

(2)

where the former is quadratic to patch number _hw_, and the
latter is linear when M is fixed (set to 7 by default). Global
self-attention computation is generally unaffordable for a
large _hw_, while the window based self-attention is scalable.

**Shifted window partitioning in successive blocks** The
window-based self-attention module lacks connections
across windows, which limits its modeling power. To intro-
duce cross-window connections while maintaining the effi-
cient computation of non-overlapping windows, we propose
a shifted window partitioning approach which alternates be-
tween two partitioning configurations in consecutive Swin
Transformer blocks.

As illustrated in Figure 2, the first module uses a regular
window partitioning strategy which starts from the top-left
pixel, and the 8 × 8 feature map is evenly partitioned into
2 × 2 windows of size 4 × 4 (M = 4). Then, the next mod-
ule adopts a windowing configuration that is shifted from
that of the preceding layer, by displacing the windows by
( ⌊ M 2⌋, ⌊ M 2⌋) pixels from the regularly partitioned windows.

With the shifted window partitioning approach, consec-
utive Swin Transformer blocks are computed as

ˆzl = W-MSA (LN (zl−1)) + zl−1,
zl = MLP (LN (ˆzl)) + ˆzl,
ˆzl+1 = SW-MSA (LN (zl)) + zl,
zl+1 = MLP (LN (ˆzl+1)) + ˆzl+1,

(3)

where ˆzl and zl denote the output features of the (S)W-
MSA module and the MLP module for block l, respectively;

3We omit SoftMax computation in determining complexity.

4

Scoring unit: science-swin-columns. Joined pages use the same full output.

(a) Architecture

(b) Two Successive Swin Transformer Blocks

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

Swin Transformer block Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU nonlinearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

## 3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global selfattention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

Self-attention in non-overlapped windows For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains $M \times M$ patches, the computational complexity of a global MSA module and a window based one

on an image of h \times w$ patches are3:

\Omega(MSA) = 4hwC{2 +2(hw) 2C, (1)

Q(W-MSA) = 4wC 2 + 2 M 2 h, (2) where the former is quadratic to patch number hw, and the latter is linear when M$ is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large hw, while the window based self-attention is scalable.

Shifted window partitioning in successive blocks The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the 8 × 8 feature map is evenly partitioned into 2 × 2 windows of size 4 × 4 (M = 4). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by ( , ) pixels from the regularly partitioned windows. With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as {z}=W-MSA (LN\left(z{-1)+z{1}, z{}=MLP(N({x})+{z}{k,$ $11=SW-MSA (LN (z) + 21, $ 11 = MLP(LN (a 1) + a 11, (3) where $\hat{2}^{2}$ and $z^{1$ denote the output features of the (S)W- MSA module and the MLP module for block l, respectively; 3We omit SoftMax computation in determining complexity.

Scoring unit: science-swin-columns. Joined pages use the same full output.

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

<!-- image -->

with the same feature map resolutions as those of typical convolutional networks, e.g., VGG [52] and ResNet [30]. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks.

Swin Transformer block Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU nonlinearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

## 3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global selfattention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

Self-attention in non-overlapped windows For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains M × M patches, the computational complexity of a global MSA module and a window based one

on an image of h × w patches are 3 :

$$\Omega ( \text {MSA} ) = 4 h w C ^ { 2 } + 2 ( h w ) ^ { 2 } C ,$$

$$\Omega ( W \text {-MSA} ) = 4 h w C ^ { 2 } + 2 M ^ { 2 } h w C ,$$

where the former is quadratic to patch number hw , and the latter is linear when M is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large hw , while the window based self-attention is scalable.

Shifted window partitioning in successive blocks The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the 8 × 8 feature map is evenly partitioned into 2 × 2 windows of size 4 × 4 ( M = 4). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by ( ⌊ M 2 ⌋ , ⌊ M 2 ⌋ ) pixels from the regularly partitioned windows.

$$\hat { z } ^ { l } & = W \text {-MSA} \left ( \text {LN} \left ( \text {z} ^ { l - 1 } \right ) \right ) + \text {z} ^ { l - 1 } , \\ \text {z} ^ { l } & = \text {MLP} \left ( \text {LN} \left ( \hat { z } ^ { l } \right ) \right ) + \hat { z } ^ { l } , \\ \hat { z } ^ { l + 1 } & = \text {SW-MSA} \left ( \text {LN} \left ( \text {z} ^ { l } \right ) \right ) + \text {z} ^ { l } , \\ \text {z} ^ { l + 1 } & = \text {MLP} \left ( \text {LN} \left ( \hat { z } ^ { l + 1 } \right ) \right ) + \hat { z } ^ { l + 1 } ,$$

where z l and z l denote the output features of the (S)WMSA module and the MLP module for block l , respectively;

$^{3}$We omit SoftMax computation in determining complexity.

Scoring unit: science-swin-columns. Joined pages use the same full output.

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively. on an image of \(h \times w\) patches are\(^{3}\):

Swin Transformer block Swin Transformer is built by replacing the standard multi-head self attention (MSA) module in a Transformer block by a module based on shifted windows (described in Section 3.2), with other layers kept the same. As illustrated in Figure 3(b), a Swin Transformer block consists of a shifted window based MSA module, followed by a 2-layer MLP with GELU non-linearity in between. A LayerNorm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adaptation for image classification [20] both conduct global self-attention, where the relationships between a token and all other tokens are computed. The global computation leads to quadratic complexity with respect to the number of tokens, making it unsuitable for many vision problems requiring an immense set of tokens for dense prediction or to represent a high-resolution image.

Self-attention in non-overlapped windows For efficient modeling, we propose to compute self-attention within local windows. The windows are arranged to evenly partition the image in a non-overlapping manner. Supposing each window contains \(M \times M\) patches, the computational complexity of a global MSA module and a window based one on an image of \(h \times w\) patches are\(^{3}\):

\(\Omega(MSA) = 4hwC^{2} + 2(hw)^{2}C\)

(1)

\(\Omega(W-MSA) = 4hwC^{2} + 2M^{2}hwC\)

(2)

where the former is quadratic to patch number \(hw\), and the latter is linear when M is fixed (set to 7 by default). Global self-attention computation is generally unaffordable for a large \(hw\), while the window based self-attention is scalable.

Shifted window partitioning in successive blocks The window-based self-attention module lacks connections across windows, which limits its modeling power. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, we propose a shifted window partitioning approach which alternates between two partitioning configurations in consecutive Swin Transformer blocks.

As illustrated in Figure 2, the first module uses a regular window partitioning strategy which starts from the top-left pixel, and the \(8 \times 8\) feature map is evenly partitioned into \(2 \times 2\) windows of size \(4 \times 4\) (\(M = 4\)). Then, the next module adopts a windowing configuration that is shifted from that of the preceding layer, by displacing the windows by \((\frac{M}{2}, 1, \frac{M}{4}, 1)\) pixels from the regularly partitioned windows. With the shifted window partitioning approach, consecutive Swin Transformer blocks are computed as

\(\hat{z}^{l} = W\text{-}MSA \left( \ln \left( \bar{z}^{l-1} \right) \right) + \bar{z}^{l-1}\)

\(\bar{z}^{l} = \text{MLP} \left( \left( \bar{z}^{l} \right)^{\prime} \right) + \bar{z}^{l}\)

\(\bar{z}^{l+1} = SW\text{-}MSA \left( \ln \left( \bar{z}^{l} \right)^{\prime} \right) + \bar{z}^{l}\)

\(\bar{z}^{l+1} = \text{MLP} \left( \left( \bar{z}^{l+1} \right)^{\prime} \right) + \bar{z}^{l+1}\)

(3)

where \(\bar{z}^{l}\) and \(\bar{z}^{l}\) denote the output features of the (S)W-MSA module and the MLP module for block l, respectively;

\(^{3}\)We omit SoftMax computation in determining complexity.

Scoring unit: science-swin-columns. Joined pages use the same full output.

HxWx3

Swin
/Transformer|
Block

Images

Patch Merging

mi H :

‘| a H t

an ' t

mt Hl '

vot LN ' '

hai H ‘

2 ely g H :
Bi swin fp tt nya '
bP] Transformer | 1 | WeMSA 1 | |swemsa} | +
ct Block ' nyo '
zt blag '

(a) Architecture

(b) Two Successive Swin Transformer Blocks

Figure 3. (a) The architecture of a Swin Transformer (Swin-T); (b) two successive Swin Transformer Blocks (notation presented with
Eq. (3)). W-MSA and SW-MSA are multi-head self attention modules with regular and shifted windowing configurations, respectively.

with the same feature map resolutions as those of typical
convolutional networks, e.g., WGG [52] and ResNet [30].
As a result, the proposed architecture can conveniently re-
place the backbone networks in existing methods for vari-
ous vision tasks.

Swin Transformer block Swin Transformer is built by
replacing the standard multi-head self attention (MSA)
module in a Transformer block by a module based on
shifted windows (described in Section 3.2), with other lay-
ers kept the same. As illustrated in Figure 3(b), a Swin
Transformer block consists of a shifted window based MSA.
module, followed by a 2-layer MLP with GELU non-
linearity in between. A LayerNorm (LN) layer is applied
before each MSA module and each MLP, and a residual
connection is applied after each module.

3.2. Shifted Window based Self-Attention

The standard Transformer architecture [64] and its adap-
tation for image classification [20] both conduct global self-
attention, where the relationships between a token and all
other tokens are computed. The global computation leads to
quadratic complexity with respect to the number of tokens,
making it unsuitable for many vision problems requiring an
immense set of tokens for dense prediction or to represent a
high-resolution image.

Self-attention in non-overlapped windows For efficient
modeling, we propose to compute self-attention within lo-
cal windows. The windows are arranged to evenly partition
the image in a non-overlapping manner. Supposing each
window contains M x M patches, the computational com-
plexity of a global MSA module and a window based one

on an image of h x w patches are’:

Q(MSA) = 4hwCO? + 2(hw)?C, ()
OQ(W-MSA) = 4hwC? + 2M? hwC, (2)

where the former is quadratic to patch number hw, and the
latter is linear when M is fixed (set to 7 by default). Global
self-attention computation is generally unaffordable for a
large hw, while the window based self-attention is scalable.

Shifted window partitioning in successive blocks The
window-based _self-attention module lacks connections
across windows, which limits its modeling power. To intro-
duce cross-window connections while maintaining the effi-
cient computation of non-overlapping windows, we propose
a shifted window partitioning approach which alternates be-
tween two partitioning configurations in consecutive Swin
Transformer blocks.

As illustrated in Figure 2, the first module uses a regular
window partitioning strategy which starts from the top-left
pixel, and the 8 x 8 feature map is evenly partitioned into
2 x 2 windows of size 4 x 4 (M = 4). Then, the next mod-
ule adopts a windowing configuration that is shifted from
that of the preceding layer, by displacing the windows by
({4J, [4)) pixels from the regularly partitioned windows.
With the shifted window partitioning approach, consec-
utive Swin Transformer blocks are computed as

= W-MSA (LN (z'~!)) + 2/1,

z' = MLP (LN (2')) +2!,

a+ = SW-MSA (LN (z')) +2!,

zi*} = MLP (LN (a'*!)) + 2!*1, (3)
where 2! and z! denote the output features of the (S)W-
MSA module and the MLP module for block /, respectively;

3We omit SoftMax computation in determining complexity.