TST: Collect coverage from subprocesses - #32366

I thought we were collecting from subprocess already (we went through a big thing with coverage when we moved the sub-process tests to be import-like rather than "giant-string-via--c like) but looking at the coverage results that clearly was not the case and this fixes it!

I thought we were collecting from subprocess already (we went through a big thing with coverage when we moved the sub-process tests to be import-like rather than "giant-string-via--c like) but looking at the coverage results that clearly was not the case and this fixes it!

I had the same recollection, so I'm not sure what we previously fixed or what has happened since. 🤷

When I was researching this earlier, the docs state:

You will also need the parallel option to collect separate data for each process, and the coverage combine command to combine them together before reporting.

Do we need to do this?

I don't think so. This is handled by codecoverage I believe, each of the runners uploads to codecoverage independently and gets merged upstream there by them (hence how we can have windows, macos, linux coverage all additive in the final result)

From my (limited) understanding, there needs to be two merges:

1. Each subprocess generates a data file. This gets merged into a combined data file for the runner.
2. The combined data file is sent upstream and then merged by codecoverage.

coverage.py's parallel and combine affect Merge 1. After more digging, per the docs:

The subprocess patch sets parallel = True and will require combining data files before reporting. See Combining data files: coverage combine for more details.

However, combining isn't automatic until 7.14:

As of version 7.14.0, files are combined by the reporting commands, so there is less need to use an explicit combine command.

It sounds like 7.10-7.13 required an explicit combine command; however, the 🤖 told me that pytest-cov is doing this for us.

I think all of my concerns are alleviated now.

@meeseeksdev backport to v3.11.x