GH-1035: Support custom_metadata on RecordBatch

[ennuite]

What's Changed

Closes #1035.

[rustyconover]

Hi @ennuite,

This PR seemed reasonable, can you tell me why you're giving up on it? Are you starting a new one?

I do use custom_metadata quite a bit and have gotten it implemented in a number of Arrow implementations.

Rusty

[ennuite]

@rustyconover Hi! I want to continue with this PR eventually, I'm finishing some other work before I continue with this.

Maybe closing this is a lack of understanding of Github etiquette: I don't have time to finish it before 20.0.0, I tried removing the milestone but I couldn't so I closed it to let maintainers know I couldn't finish it on time for the release. I'm targeting this for the next release (21.0.0).

In terms of the state of this PR: I think the low level plumbing is mostly done, but to have feature parity with your Rust PR there is still quite a bit of work:

• figure out how to represent per-batch metadata in Java’s higher-level APIs so that slicing and other transformations preserve it;
• expose metadata through ArrowStreamReader and ArrowFileReader;
• allow ArrowStreamWriter and ArrowFileWriter to write it;
• propagate it through Flight without conflating it with Flight application metadata;
• add public-API, multi-batch, and (potentially) PyArrow interoperability tests.

Right now, metadata round-trips through the low-level ArrowRecordBatch/MessageSerializer APIs, but it is discarded when a batch is loaded into a VectorSchemaRoot, and the standard writer path has no way to supply it.

Do you agree with this summary?

I don't have the bandwidth to tackle all of this right now and I don't think what I have implemented is useful as its own PR. As I'm not very familiar with Arrow IPC, it might take me some time to go through all the things even once I do grab this :)

[rustyconover]

I can help you! I hope that was the only impression I was trying to make.

Don't worry about finishing it. I know you can't engage right now but I'll turn some gears regarding it. I guess I'd like to keep working on it since now I have

https://github.com/Query-farm/vgi-rpc-java and
https://github.com/Query-farm/vgi-java

That uses custom_metadata quite a bit, my code just did some hacks to make it work for me. Maybe combined with this PR and my bad hacks we can get somewhere where we are both happy.

Rusty

[ennuite]

@rustyconover this is pretty cool! I thought no one was using this in Java, that was part of why I deprioritized it. I'm happy there is community interest, that changes my perspective 🚀 I'm very much interested in collaborating with you.

I'll start looking into this again tomorrow and refreshing my brain as today I'm trying to unblock some other PRs I have for this release. I don't think we'll get it custom_metadata for 20.0.0 Are you ok with aiming for the next one?

[rustyconover]

Sure thing! I know the gears of Arrow releases turn slowly.