Visitar URL original
[inductor] pad_mm: benchmark padded operands in the layout the graph will use instead of contiguous by LuFinch · Pull Request #204782 · pytorch/pytorch · GitHub
Skip to content

[inductor] pad_mm: benchmark padded operands in the layout the graph will use instead of contiguous - #204782

Open
LuFinch wants to merge 1 commit into
pytorch:mainfrom
LuFinch:lfq/fix_padmm_layout
Open

[inductor] pad_mm: benchmark padded operands in the layout the graph will use instead of contiguous#204782
LuFinch wants to merge 1 commit into
pytorch:mainfrom
LuFinch:lfq/fix_padmm_layout

Conversation

@LuFinch

@LuFinch LuFinch commented Nov 13, 2026 •
edited by pytorch-bot Bot
Loading

Copy link
Copy Markdown
Contributor

should_pad_bench decides whether shape padding is profitable by timing an unpadded
mm against a padded one. The padded operands it times can have a different memory
layout from the ones the compiled graph actually produces, so the decision can be
based on a kernel that is never run.

For example, aten.constant_pad_nd always materializes a contiguous output. Inductor lowers
constant_pad_nd as a pointwise op and preserves the stride order of its input.
Padding a transposed operand therefore gives a contiguous tensor in eager and a
transposed one in the compiled graph.

The mm behind an nn.Linear has mat2 = weight.t(),
which is column major. Whenever such an mm is a padding candidate, the benchmark
times a contiguous weight while the graph runs a transposed one.

Why it matters

The two layouts are not interchangeable in the GEMM backends. On an Intel Arc Pro
B60 with oneDNN 3.14, benchdnn --matmul --dt=f16:f16:f16 98304x200:200x200:

weight layout time
wtag=ab (contiguous, what the benchmark times) 0.324 ms
wtag=ba (transposed, what the graph runs) 0.781 ms

A 2.4x gap, and the benchmark only ever sees the fast side of it.

timm/gmlp_s16_224 f16 inference trips on this. Traced decision for the one mm it
benchmarks, mat1=[98304, 196] stride=(196, 1), mat2=[196, 196] stride=(1, 196):

ori=0.4049  pad=0.3342  ratio=1.211  ->  pad

The threshold is 1.1, so padding wins by measuring the wrong layout. The benchmark has always
measured the padded operand in the wrong layout; it only changes the outcome once
the two layouts' timings diverge far enough to cross the 1.1 threshold.

Fix

Pad through a permuted view in the benchmark so the timed operand keeps the stride
order the graph will produce:

def _constant_pad_nd_preserving_stride_order(t: Tensor, pad_arg: list[int]) -> Tensor:
    ndim = t.dim()
    perm = sorted(range(ndim), key=lambda d: -t.stride(d))
    if perm == list(range(ndim)):
        return aten.constant_pad_nd(t, pad_arg)
    ...

The permutation is derived from the strides rather than special cased for 2D, so
bmm operands and arbitrary stride orders are handled too. When the input is
already contiguous the helper returns aten.constant_pad_nd unchanged, so models
whose padded operands are contiguous see no behavior change at all.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @jataylo

@pytorch-bot

pytorch-bot Bot commented Nov 13, 2026 •
edited
Loading

Copy link
Copy Markdown

PR Status: missing actionable issue

Auto PR Triage did not find either of the following when it checked this PR:

  • a linked issue labeled actionable (Fixes #<issue> or Part of #<issue>); or
  • a maintainer named in the description who agreed to the change (Supported by @<maintainer>).

Such PRs are a low priority for maintainers to review and might not get a response given the very high volume of PRs we receive. Please focus on actionable issues to ensure a maintainer will be available to review your PR.

If one of these applies, add the matching line to the PR description. If there is no actionable issue for this change yet, you can open one (or comment on an existing one) and wait for a maintainer to label it actionable. Adding actionable to an already-linked issue automatically requests re-triage of open, non-draft PRs with open source and missing actionable issue, unless automated triage is disabled. Adding a valid actionable-issue link or Supported by line to the description also requests re-triage automatically. Marking the PR ready, reopening it, or re-enabling automated triage rechecks existing links and support. If re-triage is still needed, comment @pytorchbot triage to run Auto PR Triage again.

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/204782

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 5 Awaiting Approval

As of commit 79b2531 with merge base 94d7447 (image):

AWAITING APPROVAL - The following workflows need approval before CI can run:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot

pytorch-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:

  • ciflow/inductor
  • ciflow/torchtitan

Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows.

1 similar comment
@pytorch-bot

pytorch-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:

  • ciflow/inductor
  • ciflow/torchtitan

Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows.

@pytorch-bot

pytorch-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@LuFinch LuFinch mentioned this pull request Nov 13, 2026
Open
@github-actions github-actions Bot added bot-triaged This is a label only to be used by the auto triage bot missing actionable issue A PR without a linked actionable issue or maintainer sponsor triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Nov 13, 2026

This branch was successfully deployed

1 active deployment
bedrock — 79b25316 Deployed Nov 13, 2026 by pytorchbot via analyze #30959
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

bot-triaged This is a label only to be used by the auto triage bot ciflow/inductor ciflow/torchtitan Run TorchTitan integration tests missing actionable issue A PR without a linked actionable issue or maintainer sponsor module: inductor open source triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@LuFinch @pytorchbot