Visitar URL original
fix(post-processing): keep the caller's index when parsing geography columns by SEPURI-SAI-KRISHNA · Pull Request #45086 · apache/superset · GitHub
Skip to content

fix(post-processing): keep the caller's index when parsing geography columns - #45086

Merged
rusackas merged 1 commit into
apache:masterfrom
SEPURI-SAI-KRISHNA:fix/geography-keeps-caller-index
Oct 13, 2026
Merged

rusackas merged 1 commit into
apache:masterfrom
SEPURI-SAI-KRISHNA:fix/geography-keeps-caller-index

Conversation

@SEPURI-SAI-KRISHNA

Copy link
Copy Markdown
Contributor

SUMMARY

geohash_decode and geodetic_parse build their parsed frame as a bare DataFrame(), so it gets a fresh RangeIndex. _append_columns aligns on the index, so when the caller's frame is indexed by anything other than 0..n-1 nothing matches: the real rows get no coordinates and the parsed values arrive as extra rows.

Both now build the frame on the caller's index:

lonlat_df = DataFrame(index=df.index)

That makes the alignment a no-op instead of a mismatch. It is correct for either branch of _append_columns, the overwrite one and the append one, so it does not depend on #45083.

geohash_encode is left alone deliberately: it slices df[[latitude, longitude]], so it already inherits the caller's index. A test pins that it stays correct.

The index has never been carried here. Before #19116 _append_columns was a plain assign, which also aligns on the index, so this surfaced as silently null coordinates with the right row count; #19116's pd.concat(axis="columns") unions the index instead and turned that into extra rows as well. So this fixes a null-coordinate bug that is older than the row-count one, and both come from the same bare DataFrame().

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A: backend only.

A pivot followed by a geodetic_parse, which is the shortest chain that hands these operations a non-RangeIndex frame:

pivoted = pivot(df=df, index=["region"], columns=[],
                aggregates={"geodetic": {"operator": "max"}})
geodetic_parse(df=pivoted, geodetic="geodetic", latitude="lat", longitude="lon")

before, 2 rows in and 4 out:

        geodetic   lat   lon
east  41.1 -73.5   NaN   NaN
west  40.7 -74.0   NaN   NaN
0            NaN  41.1 -73.5
1            NaN  40.7 -74.0

after:

        geodetic   lat   lon
east  41.1 -73.5  41.1 -73.5
west  40.7 -74.0  40.7 -74.0
operation builds its own frame affected
geohash_decode yes, DataFrame() fixed
geodetic_parse yes, DataFrame() fixed
geohash_encode no, slices df not affected, pinned by a new test

A frame that already has a 0-based index is unchanged, which is why the existing tests pass untouched: they all build from lonlat_df, which has one.

TESTING INSTRUCTIONS

pytest tests/unit_tests/pandas_postprocessing/test_geography.py

Five cases are added, four of which fail on master:

test covers on master
test_geography_keeps_the_index_it_was_given[geohash_decode] a city-indexed frame keeps its rows and gets its coordinates fails
...[geodetic_parse] the same for the geodetic parser fails
...[geodetic_parse-with-altitude] the three-column parse, where the altitude is also mapped fails
test_geodetic_parse_after_pivot_keeps_one_row_per_group the pivot chain above, end to end through two real operations fails
test_geohash_encode_keeps_the_index_it_was_given that the one operation needing no change stays correct passes, guards it

The file goes from 5 tests to 10, and the five existing ones are unchanged. The post-processing suite is 191 passed, up from 186. tests/unit_tests/{pandas_postprocessing,charts,queries} run clean at 584 passed, 2 xfailed (both pre-existing). ruff check, ruff format --check and pre-commit run mypy are clean, and pylint --rcfile=.pylintrc rates the changed file 10.00/10.

I also ran this against #45083's branch, since both touch this area: 194 passed with the two changes together, so they compose.

ADDITIONAL INFORMATION

@codeant-ai-for-open-source

Copy link
Copy Markdown
Contributor

CodeAnt PR Risk: Low Risk

  • The PR appears safe to merge; geography parsers now preserve the caller's index when building coordinate frames.
  • Tests cover non-RangeIndex inputs, pivoted data, altitude parsing, and geohash encoding.

Assessed commit: 20d6a817452b

@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.62%. Comparing base (100f512) to head (20d6a81).

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #45086   +/-   ##
=======================================
  Coverage   82.62%   82.62%           
=======================================
  Files        3006     3006           
  Lines      187140   187140           
  Branches    43374    43374           
=======================================
  Hits       154632   154632           
  Misses      29742    29742           
  Partials     2766     2766           
Flag Coverage Δ
hive 36.18% <0.00%> (ø)
mysql 55.09% <0.00%> (ø)
postgres 55.10% <0.00%> (+<0.01%) ⬆️
presto 38.04% <0.00%> (ø)
python 86.42% <100.00%> (ø)
sqlite 54.83% <0.00%> (ø)
unit 79.56% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sadpandajoe
sadpandajoe self-requested a review October 8, 2026 17:13

@rusackas rusackas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, carrying the caller's index through is the right fix and geohash_encode staying untouched (with a test pinning it) is a nice touch.

@rusackas
rusackas merged commit 0cd13e9 into apache:master Oct 13, 2026
76 checks passed
villebro pushed a commit that referenced this pull request Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

geohash_decode and geodetic_parse drop the caller's index, doubling the row count

3 participants