Repository navigation
Support INTERVAL data type in DB-API, arrow, and pandas connectors #15739
Description
Follow-up to #15740 since pandas and Arrow do not yet have a structured type that aligns with INTERVAL. The existing Timedelta support would work for INTERVALS with only a time component, but it is not calendar aware, so supporting year, month, and day intervals would require some mapping to timedelta, which is not ideal.
Why is a new data type needed?
- YEAR: Leap years are a thing. Not every year is 365 days long.
- MONTH: Not every month is the same length.
- DAY: Daylight savings is a thing. Not every day is 24 hours long.
Note: DB-API support is included here because it uses the BigQuery Storage API, where we use the Arrow wire format.
TODO:
- Auto-detect data type in DB-API query parameters
- Might be possible to do this before reading INTERVAL columns is supported.
- Row data is converted to relevant type in DB-API
- Row data is converted to relevant type in
to_dataframe- Might need to be
object, sincetimedelta64doesn't have years/months.
- Might need to be
- Check if
to_arrowtype is expected datatype - Convert data type in
insert_rows_from_dataframe - Convert data type in
load_rows_from_dataframe(CSV) - Convert data type in
load_rows_from_dataframe(Parquet)
Reacted by Tyler BoddySpargo
Activity
tswast commentedon Jul 29, 2021
Update: Maybe pandas does have the right data type? https://pandas.pydata.org/docs/reference/api/pandas.tseries.offsets.DateOffset.html
tswast commentedon Feb 25, 2022
Note: There is currently a bug with the interval type in pyarrow. https://issues.apache.org/jira/projects/ARROW/issues/ARROW-15783
Sounds like we need to call
pa.array([1])to initialize some pandas code before attempting to convert an interval column to pandas.parthea commentedon Jan 12, 2026
Transferring to google-cloud-python as the code for python-bigquery will be migrated there in 1-3 weeks.
parthea commentedon Jan 12, 2026
Transferring this issue back to python-bigquery.
chalmerlowe commentedon Mar 2, 2026
This issue was moved from the following repo as part of the monorepo migration: googleapis/python-bigquery