TypedDelimited is confusing: needs both sinkFields and fields overrides

[johnynek]

If you forget to override sinkFields, you get a cryptic error if you use it to write.

Solution: there should be a final override def sinkFields = fields in TypedDelimited, probably.

[reconditesea]

Sounds good.
One small comment. Looks like sinkFields is only needed for TypedSink. Traits like DelimitedSchema don't need it and can still be used as Sink. For the purpose of reduce confusion, does it make sense to rename it to typedSinkFields?

[ianoc]

renaming fields is a big pain for compatibility, so -1 on that. Also not sure we want to call that out, ideally most sources are setup to be used as typed sources/sink's.

[reconditesea]

Good point. But in this sense, making sinkFields a final is not back-compatible either? It's overridden in all user classes that extends TypedSInk.

[ianoc]

Its localized to TypeDelimited where i guess the inference of this ticket is that people shouldn't be changing it?

[reconditesea]

How to reproduce the issue described in the ticket? Extending TypeDelimited trait but forget to override sinkFields? Or extend TypeDelimited and override sinkFields to something wrong?

Are we sure there is no case where people extend TypeDelimited and override sinkFields in their own code? If yes, this sounds great and should be a very straightforward change.

[reconditesea]

If we believe no user code is overriding sinkFileds when extending TypedDelimited, I think the change here is really simple as in #1032.