Spec: Add stats-only fields to table metadata - #18405

The only issue here is the type used to build the content stats schema.

Right now, data-type is not required for partition-value because the type may be promoted (int -> long) when the source column is widened and I didn't want to include metadata that could be stale.

However, by not having data-type for all cases, the requirement to preserve existing stats becomes harder because the output type for a new stats-only field type may not be known.

It might be a good idea to require data-type for all types except partition-value. But if we add a sort-value later we would want to do the same thing.

Could we require data-type unless the stats-only field type defines how to derive its current result

A stats-only field stores its current result type in `data-type` 
unless its `type` defines how to derive that result type. 
`data-type` must be a primitive or variant. 
`partition-value` derives its result type from the partition field's 
result type and must omit `data-type`. The result type 
may change only according to the type promotion rules above.

How should we handle schema evolution which drops source columns for the expression?

Maybe:
If an expression *cannot be bound* or produces a different output type when bound, a writer *must* not produce stats.

@stevenzwu, the problem with what you suggest is that it is only possible when the reader understands the stats-only field type. We can only assume that at format version boundaries where we specify what all readers must understand.

The only real solution that I can see is to say that readers and writers may omit stats-only fields with an unknown type unless data-type is present.

@pvary, I agree with updating the last sentence to cover that case.

Could we require data-type unless the stats-only field type defines how to derive its current result

A stats-only field stores its current result type in `data-type` 
unless its `type` defines how to derive that result type. 
`data-type` must be a primitive or variant. 
`partition-value` derives its result type from the partition field's 
result type and must omit `data-type`. The result type 
may change only according to the type promotion rules above.

Do we want to codify the restrictions of the expr discussed on the dev thread, like deterministic functions, ID based fields.

nit: link should be relative

Why relative? Then it doesn't work when viewing the spec in github.

Updated to specify ID references are required. The discussion about deterministic functions was that the functions should not have hidden inputs, which is not a concern here.

Why relative? Then it doesn't work when viewing the spec in github.

If github link is a concern the we should fix this in expression-spec.md ([bucket-ref], [truncate-ref]), and in spec-md ([Puffin file format])

Did some checking and this one works too in github, and on the site as well as MkDocs changes them:

[value expression](expressions-spec.md#value-expressions)

But cleaning up this could be another PR. Don't waste loops on it here. Opened #18441.

Is the primary function of partition-value to tie the partition-field-id to a "stats" field-id? Would this only be used for bucket/multi-arg transforms?

I assume we wouldn't track identity columns this way.

That's correct. This is to be able to store the output of a non-monotonic function. I would not expect anyone to use this for identity because the easier path is to store the identity value in the field's stats directly.

It would be good to clarify if we intend to track ALL partition fields this way or just ones that require a non-identity transform.

I'm updating this to add "This may be used in v4 to filter by bucket partition values".

I'm reluctant to over-specify what it can be used for because this should be mostly independent of the partition transforms that we add over time, and we don't typically put recommendations in the spec (like "don't use this for monotonic functions"). Hopefully this clarification / example works.

This may be used in v4 to filter by bucket partition values.

do we need to mention "v4" here? the stats-only fields are allowed in v3. if v3 writer populates these partition stats fields, do we need to restrict v3 readers from using them (although they don't need to).

if we keep it, maybe v4 can be renamed to v4+ for future proof?

produces a different output type when bound

nit: this reads a bit awkward to me. it is one of the 3 conditions for when. maybe sth like

binding produces a different output type