Fix provider test acessing importlib-resources - #18228

(retriggering CI)

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

Oops sorry I didn’t see the re-trigger above.

I have reverted the importlib-resources changes - #18250. Can you add those changes in this PR again so we test it with the change.

I’ve applied the importlib-resources version bump. My change is different from #18209 in several ways:

1. I bumped to 5.2 instead of 5.0. This is because the mock behavioural change is introduced in 5.2.
2. I removed the importlib-resources entry from setup.py. This package is already installed unconditionally (in install_requires) and does not need to be listed in the devel extra.
3. I added the python_version < '3.7' marker. We only import importlib_resources for Python 3.6 (in provider_manager.py) so having it installed on 3.7+ is useless.

tests/providers/amazon/aws/hooks/test_kinesis.py::TestAwsFirehoseHook::test_insert_batch_records_kinesis_firehose:
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the
CreateDeliveryStream operation: The security token included in the request is invalid.

Uhhhggrrr

WTH this seems to break mock_kinesis somehow?

WTH this seems to break mock_kinesis somehow?

That looks an unrelated failure that affects main branch too:

https://github.com/apache/airflow/runs/3607014793#step:6:8000

WTH this seems to break mock_kinesis somehow?

This change touches setup.py so it upgrades dependencies to latest.

Seems that Botocore patchlevel was released yesterday and it breaks the test apparently. https://pypi.org/project/botocore/1.21.42/

This is our early-warning for new dependencies breaking Airflow at its best. We should take a look at this failed kinesis test and fix it (either fix the test or limit botocore). Only PRs that are updating setup.py (and main) are affected. All other PRs are not, which limits the impact of such dependencies breaking our build.

This can be reproduced locally by building the image with newer dependencies ./breeze --upgrade-to-newer-dependencies .
Anyone cares to fix it :) ?

WTH this seems to break mock_kinesis somehow?

This change touches setup.py so it upgrades dependencies to latest.

Seems that Botocore patchlevel was released yesterday and it breaks the test apparently. https://pypi.org/project/botocore/1.21.42/

This is our early-warning for new dependencies breaking Airflow at its best. We should take a look at this failed kinesis test and fix it (either fix the test or limit botocore). Only PRs that are updating setup.py (and main) are affected. All other PRs are not, which limits the impact of such dependencies breaking our build.

This can be reproduced locally by building the image with newer dependencies ./breeze --upgrade-to-newer-dependencies .
Anyone cares to fix it :) ?

Don't think so, we have pinned boto3 to <1.18 (

Which pins botocore version (botocore>=1.20.112,<1.21.0): https://github.com/boto/boto3/blob/577672298d619e59b0806aad0606d2974803c561/setup.py#L17

And from CI (https://github.com/apache/airflow/pull/18228/checks?check_run_id=3606187405#step:8:1157):

  Collecting botocore<1.21.0,>=1.20.112
    Downloading botocore-1.20.112-py2.py3-none-any.whl (7.7 MB)

Don't think so, we have pinned boto3 to <1.18 (

Could be also another dependency :).

Don't think so, we have pinned boto3 to <1.18 (

Could be also another dependency :).

When you look at the logs of the failed tests - this is the exact command that we can run to reproduce this exact environment (inspect the versions installed/rerun tests and try to find out the diff between the installed/constraints).
Maybe also in the future we should add the diff between "latest" and "updated" constraints - this would help in the future to "guess" which packages were updated in this case.

./breeze --github-image-id be3fc5475849e6ed737c0a4531ab212b9a87c7d1 --backend sqlite --python 3.9 --db-reset --skip-mounting-local-sources --test-type Providers shell

Interesting, we had a dev version of moto

Yay my guess is correct. The dev version was pulled because previously we had moto~=2.2, >=2.2.1.dev9. According to PEP 440, if the version range specifier contains a prerelease, prereleases will be considered when pulling in dependencies, which is why a dev version was pulled with --upgrade-strategy=eager.

Moto likely made some backward incompatibility changes recently to make mock_kinesis not work properly, but I have absolutely zero knowledge on moto to discern what exactly broke things (and didn’t bother to do a bisect).

Unfortunately pip does not provide a way to express greater than or equal to 2.2.1.dev9, but only allow prereleases from 2.2.1, not 2.2.2 onwards, so the best we can do is to bump the specifier to ~=2.2, >=2.2.1.

Yay my guess is correct. The dev version was pulled because previously we had moto~=2.2, >=2.2.1.dev9. According to PEP 440, if the version range specifier contains a prerelease, prereleases will be considered when pulling in dependencies, which is why a dev version was pulled with --upgrade-strategy=eager.

TIL!

So it was close guess. boto and moto - only one letter different ;). I will add a feature to show diff vs. "fixed" constraints whenever we get a failure on "eager upgrade" build and make it quite prominent with some description like:

If you see an unexpected test failures here, it might be because of upgraded dependencies (this build was run with eager upgrade on dependencies due to changed setup.* files)

Please check if the test also fails in main branch and if so, it's likely that the reason is one of those: