sccache consistently errors on CI since Rust nightly-2019-11-07

https://community-tc.services.mozilla.com/tasks/NvN6U22bQ-C626hsVGu4fA

error: failed to execute compile
caused by: error reading compile response from server
caused by: Failed to read response header
caused by: failed to fill whole buffer
error: could not compile `script`.

The error message is the same as in #24510, but setting SCCACHE_IDLE_TIMEOUT to zero does not help.

Merges in the regression range of nightly-2019-11-06 to nightly-2019-11-07 rust-lang/rust@1423bec...3804876:

Auto merge of Fix promotion in a const when projections are present rust-lang/rust#65728 - ecstatic-morse:promotion-const-proj
Auto merge of improper_ctypes: extern "C" fns rust-lang/rust#65134 - davidtwco:issue-19834-improper-ctypes-in-extern-C-fn
Auto merge of Use ident.span instead of def_span in dead-code pass rust-lang/rust#65830 - Quantumplation:master
Auto merge of Rollup of 9 pull requests rust-lang/rust#66143 - Centril:rollup-qmzuex0
Rollup merge of use American spelling for pluralize! rust-lang/rust#66139 - euclio:pluralize
Rollup merge of Update the bundled wasi-libc repository rust-lang/rust#66133 - alexcrichton:update-wasi-libc
Rollup merge of Add target thumbv7neon-unknown-linux-musleabihf rust-lang/rust#66103 - smaeul:patch/thumb-musl
Rollup merge of Use KERN_ARND syscall for random numbers on NetBSD, same as FreeBSD. rust-lang/rust#66092 - niacat:master
Rollup merge of bump smallvec to 1.0 rust-lang/rust#66086 - RalfJung:smallvec
Rollup merge of Configure LLVM module PIC level rust-lang/rust#66062 - smaeul:patch/pic-level
Rollup merge of librustc_lexer: Refactor the module rust-lang/rust#66015 - popzxc:refactor-librustc_parser
Rollup merge of caller_location: point to macro invocation sites, like file!/line!, and use in core::panic!. rust-lang/rust#65973 - eddyb:caller-location-panic
Rollup merge of Rename LocalInternedString and more rust-lang/rust#65776 - nnethercote:rename-LocalInternedString-and-more
Auto merge of Rollup of 11 pull requests rust-lang/rust#66141 - Centril:rollup-n2fcvp9
Rollup merge of rustc: remove "GlobalMetaData" dead code from hir::map::definitions. rust-lang/rust#66115 - eddyb:global-meta-what
Rollup merge of Fix typo in explanation of E0080 rust-lang/rust#66106 - JOE1994:master
Rollup merge of Tweak type mismatch caused by break on tail expr rust-lang/rust#66101 - estebank:break-tail-e0308
Rollup merge of Detect :: -> : typo when involving turbofish rust-lang/rust#66098 - estebank:path-asciption-typo
Rollup merge of Do not ICE with a precision flag in formatting str and no format arguments rust-lang/rust#66093 - estebank:fmt-ice
Rollup merge of let caller of check_ptr_access_align control the error message rust-lang/rust#66081 - RalfJung:ptr-offset
Rollup merge of use silent emitter for rustdoc highlighting pass rust-lang/rust#66068 - euclio:null-emitter
Rollup merge of syntax: Avoid span arithmetic for delimiter tokens rust-lang/rust#66054 - petrochenkov:delspan
Rollup merge of Move has_panic_handler to query rust-lang/rust#66027 - Mark-Simulacrum:panic-handler-query
Rollup merge of Show type parameter name and definition in type mismatch error messages rust-lang/rust#66014 - dkadashev:47319-type-param-def-location
Rollup merge of Remove PartialEq and Eq from the SpecialDerives. rust-lang/rust#65892 - pnkfelix:trim-special-derives

What's the best way to tackle this? Try to reproduce this locally?

I haven’t managed to reproduce when building on my machine with the same version of sccache.

The output shows that this errors occurs when read_exact fails to read 4 bytes from a BufReader<TcpStream>.

https://github.com/mozilla/sccache/blob/f1322dca19577310cfcfb40059ff954d2140598b/src/client.rs#L56-L59

This happens when libc::recv returns 0, which indicates that the TCP connection has been closed.

Is it worth seeing if there's any interesting compiler output on CI if we don't use sccache?

Good idea, I’ve started https://community-tc.services.mozilla.com/tasks/Wxja6BOTQ6OYbp-79FhFtA to try that.

By the way, my workflow for this kind of "try build" without a PR is:

Make a local branch
Make changes to try
In the try-taskcluster list in the by_branch_name dict in etc/taskcluster/decision_task.py, add tasks to be run
Commit
git push origin -f HEAD:try-taskcluster
Navigate to https://github.com/servo/servo/commits/try-taskcluster
Click the orange dot on the latest commit (which should be the one you just pushed)
Click "Community-TC (push)" to view the task group on Taskcluster

It’s mostly out of habit, I don’t know if it’s really better or worse than using try* Homu commands in a PR that is not destined to be merged.

if we don't use sccache?

https://community-tc.services.mozilla.com/tasks/Wxja6BOTQ6OYbp-79FhFtA succeeded. The only thing of note is that it generated more than 300k stderr lines worth of warnings, most of them from #24684. And rust-lang/rust#65134 is in the regression range.

I’ve also started https://community-tc.services.mozilla.com/tasks/BgPRR43VTKWDVM8nY6A5Ew with RUST_LOG=trace into a log file for the sccache server.

https://community-tc.services.mozilla.com/tasks/BgPRR43VTKWDVM8nY6A5Ew/runs/0/logs/https%3A%2F%2Fcommunity-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FBgPRR43VTKWDVM8nY6A5Ew%2Fruns%2F0%2Fartifacts%2Fpublic%2Fsccache.log#L321219

ERROR 2019-11-12T15:31:54Z: sccache::server: frame size too big

https://github.com/tokio-rs/tokio/blob/master/tokio-util/src/codec/length_delimited.rs

            // Default max frame length of 8MB
            max_frame_len: 8 * 1_024 * 1_024,

mozilla/sccache#329 / https://github.com/mozilla/sccache/blob/master/src/server.rs

        if let Ok(max_frame_length_str) = env::var("SCCACHE_MAX_FRAME_LENGTH") {
            if let Ok(max_frame_length) = max_frame_length_str.parse::<usize>() {
                builder.max_frame_length(max_frame_length);

Fix sccache failure when rustc generates hundreds of thousands of war…

https://community-tc.services.mozilla.com/tasks/LpLx0VqfTaqiCXEmEo-Qfw is green with SCCACHE_MAX_FRAME_LENGTH set to 100 MB.

Fix sccache failure when rustc generates hundreds of thousands of war…

Auto merge of #24717 - servo:fraaaaaaaaaaaaame, r=jdm

Navigation Menu

Repository navigation

sccache consistently errors on CI since Rust nightly-2019-11-07 #24714

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

sccache consistently errors on CI since Rust nightly-2019-11-07 #24714

Description

Activity

jdm commented on Nov 12, 2019

SimonSapin commented on Nov 12, 2019

jdm commented on Nov 12, 2019

SimonSapin commented on Nov 12, 2019

SimonSapin commented on Nov 12, 2019

SimonSapin commented on Nov 12, 2019

SimonSapin commented on Nov 12, 2019

SimonSapin commented on Nov 12, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions