Visitar URL original
slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." · Issue #1883 · dotnet/dnceng · GitHub
Skip to content

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

New issue
New issue
Open
Open
slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."#1883
Labels
First ResponderKnown Build Error

Description

@jeffschwMSFT
jeffschwMSFT
opened on Jan 24, 2024

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

  • Feature changes/additions
  • Bug fixes
  • Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build Repository Step Name Console log Pull Request
3135910 dotnet-sdk Darwin_AoT_Tests Log #68155
3135821 dotnet-sdk Darwin Log #68755
1680655 dotnet/maui Build Log dotnet/maui#40117
3135648 dotnet-sdk Darwin Log #68155
3135608 dotnet-sdk AoT: macOS (x64) Log #67569
3135437 dotnet-runtime osx-arm64 release CrossAOT_Mono crossaot Log
1680330 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#134855
1680303 dotnet/runtime osx-x64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137505
3135432 dotnet-efcore macOS Log
3135300 dotnet-efcore macOS Log
1679720 dotnet/maui Build Log
3135080 dotnet-maui RunOniOS_MauiReleaseTrimFull ARM64 Log
1679729 dotnet/maui Controls Shell2 Log dotnet/maui#40223
1679240 dotnet/runtime osx-x64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137459
1679166 dotnet/maui Controls Shell2 Log
3134852 dotnet-sdk TestBuild: macOS (x64) Log #68454
3134766 dotnet-sdk AoT: macOS (x64) Log #68454
3134765 dotnet-sdk AoT: macOS (x64) Log #67569
3134757 dotnet-sdk Darwin_AoT_Tests Log #68676
3134643 dotnet-runtime Performance micro coreclr JIT osx arm64 M4 Log
3134370 dotnet-runtime tvossimulator-arm64 release Mono Log
1678310 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log
1678235 dotnet/runtime osx-x64 release CoreCLR_AllSubsets Log dotnet/runtime#136259
3134371 dotnet-runtime osx-arm64 release NativeAOT Log
1678146 dotnet/runtime maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#136770
1678166 dotnet/aspnetcore Test: macOS Log
1678116 dotnet/runtime osx-x64 release CoreCLR_AllSubsets Log dotnet/runtime#137441
1678097 dotnet/runtime maccatalyst-arm64 Release AllSubsets_NativeAOT_Smoke Log dotnet/runtime#136163
1678105 dotnet/runtime ios-arm64 Release AllSubsets_NativeAOT_Smoke Log
1678103 dotnet/runtime osx-arm64 checked Log
1678054 dotnet/runtime ios-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#134787
1678027 dotnet/runtime maccatalyst-arm64 Release AllSubsets_NativeAOT_Smoke Log dotnet/runtime#137437
1677925 dotnet/maui Build Log dotnet/maui#40183
1678005 dotnet/runtime ios-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#137347
1677833 dotnet/maui Build Log dotnet/maui#40144
3134204 dotnet-efcore macOS Log
1677817 dotnet/runtime maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#137431
1677662 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#136900
3134059 dotnet-runtime osx-x64 release CrossAOT_Mono crossaot Log
1677644 dotnet/runtime maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#137251
1676977 dotnet/runtime ios-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#137404
1676867 dotnet/runtime maccatalyst-arm64 Release AllSubsets_NativeAOT Log
1676755 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137114
1676773 dotnet/runtime osx-arm64 Debug CoreCLR_AllSubsets dotnet/runtime#137400
1676704 dotnet/runtime maccatalyst-arm64 Release AllSubsets_Mono Log
1676663 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#136259
1676613 dotnet/runtime osx-arm64 release CoreCLR_Libraries Log dotnet/runtime#137396
1676586 dotnet/runtime maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#136163
3133566 dotnet-runtime tvossimulator-arm64 release NativeAOT Log
1676527 dotnet/runtime osx-arm64 release CrossAOT_Mono crossaot Log dotnet/runtime#135446
1676480 dotnet/runtime osx-arm64 release CrossAOT_Mono crossaot Log dotnet/runtime#137389
1676310 dotnet/runtime osx-x64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137381
1676232 dotnet/runtime iossimulator-arm64 Release AllSubsets_Mono Log
3133388 dotnet-runtime tvos-arm64 release Mono Log
1676100 dotnet/runtime osx-arm64 Debug CoreCLR_AllSubsets Log dotnet/runtime#137119
1675862 dotnet/runtime osx-x64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137369
1675474 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137344
1675761 dotnet/runtime maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#137251
3133141 dotnet-efcore macOS Log
3133137 dotnet-efcore macOS Log
1675602 dotnet/runtime osx-arm64 Release NativeAOT_Libraries Log dotnet/runtime#136364
1675584 dotnet/runtime osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter Log dotnet/runtime#136864
1675546 dotnet/runtime osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit Log dotnet/runtime#136752
1675543 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#136893
1675533 dotnet/runtime maccatalyst-arm64 Release AllSubsets_Mono Log
1675518 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137347
1675501 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#136872
1675179 dotnet/machinelearning MacOS_arm64 Release_Build Log dotnet/machinelearning#7771
1675207 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137340
1675194 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137077
1675036 dotnet/runtime maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke Log dotnet/runtime#137145
3132570 dotnet-sdk TestBuild: macOS (x64) Log #67570
3132593 dotnet-sdk Darwin Log #68155
3132569 dotnet-sdk AoT: macOS (x64) Log #67569
1674273 dotnet/runtime osx-x64 Debug PALTests Log
1674226 dotnet/runtime osx-x64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137144
1674224 dotnet/aspnetcore Test: macOS Log dotnet/aspnetcore#70485
1673975 dotnet/runtime osx-x64 Debug Libraries_CheckedCoreCLR Log dotnet/runtime#137292
3132343 dotnet-runtime iossimulator-x64 release NativeAOT Log
1673816 dotnet/runtime osx-arm64 checked Log
1673791 dotnet/runtime osx-x64 Release NativeAOT_Libs Log
1673568 dotnet/runtime osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter Log dotnet/runtime#137105
1673497 dotnet/maui Controls Shell2 Log
1673550 dotnet/runtime iossimulator-arm64 Release AllSubsets_Mono Log
1673491 dotnet/runtime ios-arm64 Release AllSubsets_NativeAOT_Smoke Log dotnet/runtime#137014
3132102 dotnet-runtime tvossimulator-x64 release NativeAOT Log
1673343 dotnet/runtime osx-x64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137265
1672580 dotnet/runtime osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit Log dotnet/runtime#137188
1672480 dotnet/runtime osx-arm64 checked Libraries_CheckedCoreCLR Log dotnet/runtime#137231
1672474 dotnet/runtime osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter Log dotnet/runtime#136605
1672437 dotnet/runtime maccatalyst-arm64 Release AllSubsets_NativeAOT_Smoke Log
3131033 dotnet-sdk Darwin Log #68155
3130766 dotnet-sdk TestBuild: macOS (x64) Log #68454
1671647 dotnet/runtime osx-arm64 Debug Libraries_CheckedCoreCLR Log dotnet/runtime#137214
3130861 dotnet-sdk Darwin Log #68155
3130767 dotnet-sdk TestBuild: macOS (x64) Log #67569
1671313 dotnet/runtime osx-x64 Debug Libraries_CheckedCoreCLR Log dotnet/runtime#137204
3130354 dotnet-runtime tvos-arm64 Release AllSubsets_NativeAOT Log #68422
1670883 dotnet/maui Controls Layout Log dotnet/maui#40073
3130388 dotnet-runtime tvossimulator-x64 release NativeAOT Log
Displaying 100 of 497 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
14 103 497

Activity

  1. jeffschwMSFT
    added
    First Responder
    Known Build Error
    on Jan 24, 2024

  2. lewing commented on Feb 9, 2024

    @lewing
    lewing
    on Feb 9, 2024
    Member

    🤔

  3. nagilson commented on Oct 3, 2024

    @nagilson
    nagilson
    on Oct 3, 2024
    Member

    @dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

  4. ivanpovazan commented on Jan 27, 2025

    @ivanpovazan
    ivanpovazan
    on Jan 27, 2025
    Member

    @dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

    More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

    • On success: Run tests in Helix step reports:
    Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

    ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

    • While on failure - timeout the same step gets stuck with:
    Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

    ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

    NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

  5. garath commented on Jan 27, 2025

    @garath
    garath
    on Jan 27, 2025
    Member

    Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

  6. garath
    self-assigned this
    on Jan 27, 2025

  7. garath commented on Jan 28, 2025

    @garath
    garath
    on Jan 28, 2025
    Member

    Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

  8. garath
    removed their assignment
    on Jan 29, 2025

  9. ivanpovazan commented on Feb 13, 2025

    @ivanpovazan
    ivanpovazan
    on Feb 13, 2025
    Member

    Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

    We are still experiencing the problem.

    Should we try to change the queue to osx.13.arm64?

    @dotnet/dnceng

  10. ilyas1974 commented on Mar 14, 2025

    @ilyas1974
    ilyas1974
    on Mar 14, 2025

    Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.

  11. missymessa commented on Sep 26, 2025

    @missymessa
    missymessa
    on Sep 26, 2025
    Member

    @jeffschwMSFT is this still an issue?

  12. lewing commented on Sep 26, 2025

    @lewing
    lewing
    on Sep 26, 2025
    Member

    @jeffschwMSFT is this still an issue?

    @missymessa does it make sense to ask this question about issues that have active hit from today/this week?

  13. missymessa commented on Sep 26, 2025

    @missymessa
    missymessa
    on Sep 26, 2025
    Member

    @lewing given that these issues are over a year old, I'm checking to make sure they're still relevant for folks in the event an error happened with Build Analysis that would prevent the issue from updating on hit counts :)

  14. lewing commented on Sep 26, 2025

    @lewing
    lewing
    on Sep 26, 2025
    Member

    @lewing given that these issues are over a year old, I'm checking to make sure they're still relevant for folks in the event an error happened with Build Analysis that would prevent the issue from updating on hit counts :)

    Ah! Thanks for the explanation. One way to check if the issue is still happening is to see if the recent failed builds listed in the description are actually recent or alternatively, have already been deleted)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    First ResponderKnown Build Error

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions