Visitar URL original
BigQuery: Autofetch table schema on load if not provided by plamut · Pull Request #9108 · googleapis/google-cloud-python · GitHub
Skip to content

BigQuery: Autofetch table schema on load if not provided - #9108

Merged
plamut merged 7 commits into
googleapis:masterfrom
plamut:iss-8142
Sep 4, 2019
Merged

BigQuery: Autofetch table schema on load if not provided#9108
plamut merged 7 commits into
googleapis:masterfrom
plamut:iss-8142

Conversation

@plamut

@plamut plamut commented Aug 27, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #8142.

This PR adds automatic table schema fetching to the load_table_from_dataframe() method to improve automatic schema detection.

No additional system tests, because there already exists a "schema autodetect" test.

How to test

Check that the code is in-line with ticket specs.

@plamut plamut added the api: bigquery Issues related to the BigQuery API. label Aug 27, 2019
@plamut
plamut requested a review from a team August 27, 2019 09:22
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 27, 2019
@plamut plamut added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 27, 2019
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 27, 2019
@plamut

plamut commented Aug 27, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Unit tests passing locally, but failing on Kokoro, investigating...

Update: Found a non-essential lint issue (unused import), but unit tests also fail on the latest master. Even if going quite a long way back in the commit history.

Update 2: Submitted a fix in #9112.

tswast
tswast requested changes Aug 27, 2019
View reviewed changes
Comment thread bigquery/google/cloud/bigquery/client.py Resolved
Comment thread bigquery/google/cloud/bigquery/client.py Resolved
plamut added 3 commits August 28, 2019 13:16
@plamut
Merge branch 'master' into iss-8142
0b9188d
@plamut
Avoid fetching table schema if WRITE_TRUNCATE job
8ad6c56
@plamut
Skip dataframe columns list check
2ea14ee 
A similar check is already performed on the server, and server-side
errors are preferred to client errors.
@plamut
plamut requested a review from tswast August 28, 2019 13:34
tswast
tswast reviewed Aug 28, 2019
View reviewed changes
Comment thread bigquery/tests/unit/test_client.py Resolved
Comment thread bigquery/tests/unit/test_client.py Resolved
@plamut
Raise table NotFound in auto Pandas schema tests
2e1d137 
A mock should raise this error instead of returning a table to
trigger schema generation from Pandas dtypes.
@plamut
plamut requested a review from tswast August 29, 2019 12:00
@plamut plamut added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 29, 2019
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 29, 2019
@plamut

plamut commented Aug 29, 2019

Copy link
Copy Markdown
Contributor Author

Restarted due to failed snippets tests (500 - internal server error) - Kokoro log.

@plamut plamut added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 29, 2019
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 29, 2019
@plamut

plamut commented Aug 30, 2019

Copy link
Copy Markdown
Contributor Author

The test_copy_table_cmek snippets test fails on the latest master, too (500 server error).

@plamut plamut added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 2, 2019
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 2, 2019
@plamut

plamut commented Sep 4, 2019

Copy link
Copy Markdown
Contributor Author

@tswast Is there anything to add here? The PR has been unblocked now that we merged #9156.

tswast
tswast requested changes Sep 4, 2019
View reviewed changes
Comment thread bigquery/google/cloud/bigquery/client.py Resolved
@plamut
plamut requested a review from tswast September 4, 2019 19:13
@plamut
plamut merged commit dce1326 into googleapis:master Sep 4, 2019
@plamut
plamut deleted the iss-8142 branch September 4, 2019 20:12
emar-kar pushed a commit to MaxxleLLC/google-cloud-python that referenced this pull request Sep 11, 2019
@plamut @emar-kar
BigQuery: Autofetch table schema on load if not provided (googleapis#…
fc10302 
…9108)

* Autofetch table schema on load if not provided

* Avoid fetching table schema if WRITE_TRUNCATE job

* Skip dataframe columns list check

A similar check is already performed on the server, and server-side
errors are preferred to client errors.

* Raise table NotFound in auto Pandas schema tests

A mock should raise this error instead of returning a table to
trigger schema generation from Pandas dtypes.

* Use list_columns_and_indexes() for names list
emar-kar pushed a commit to MaxxleLLC/google-cloud-python that referenced this pull request Sep 18, 2019
@plamut @emar-kar
BigQuery: Autofetch table schema on load if not provided (googleapis#…
6031f06 
…9108)

* Autofetch table schema on load if not provided

* Avoid fetching table schema if WRITE_TRUNCATE job

* Skip dataframe columns list check

A similar check is already performed on the server, and server-side
errors are preferred to client errors.

* Raise table NotFound in auto Pandas schema tests

A mock should raise this error instead of returning a table to
trigger schema generation from Pandas dtypes.

* Use list_columns_and_indexes() for names list
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tswast tswast tswast approved these changes

Assignees

No one assigned

Labels

api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

BigQuery: get table schema if not supplied (and have pyarrow) in load_table_from_dataframe

4 participants

@plamut @tswast @googlebot @yoshi-kokoro