Flatbuffer investigations collector

[adrianschultz]

Please add your learnings and results here about your flatbuffer investigations, so that we can have a detailed discussion in the performance and packaging WG.

Pros:

• faster than protobuffers -> especially on sensor model side (de-serialization)

Cons:

• more complex code especially for serialization -> Are the improvement gains worth the refactoring effort for existing code?

General remarks:

• Are there other possibilties to acchieve better performance?
• Flexibility vs performance: Do we want to keep OSI as flexible as it is right now and life with performance limitations or not?

[ThomasNaderBMW]

Investigations on our side regarding the traffic participant called "OSI-Pedestrian".
Thanks to @FabianPfeufferCAP and @tbleher !!

OsiPedestrian-Flatbuffer-Protobuf-Comparison.pdf

[jdsika]

For the sake of completeness I am adding further links to reports here:

1. AVL report
2. Persival report

[PhRosenberger]

In addition, check out our Application report

[jdsika]

Everyone could give a "thumbs up" on this issue to raise awareness for this bug google/flatbuffers#7109

[jdsika]

@pmai should we just create a PR that puts nested ENUM definitions at the beginning of the message definition? I do not think that this would break backwards compatibility?

[jdsika]

@PhRosenberger I think we should open a feature request in the Flatbuffer GitHub and ask for the feature? What do you think?

Flutbuffers does not come with a deep copy function, so there is no easy way to copy table or entire buffers to a different buffer. With OSI it is currently common practice to copy the SensorView input of a sensor model to the SensorData output. As there is no standard function to do this with Flatbuffers, one has to be custom written.

[PhRosenberger]

@PhRosenberger I think we should open a feature request in the Flatbuffer GitHub and ask for the feature? What do you think?

Flutbuffers does not come with a deep copy function, so there is no easy way to copy table or entire buffers to a different buffer. With OSI it is currently common practice to copy the SensorView input of a sensor model to the SensorData output. As there is no standard function to do this with Flatbuffers, one has to be custom written.

Yes we can do that, but I am a bit disappointment by the response to our issue over there, so I am not sure if we can expect it to actually happen. If we get some feedback, it would be nice to discuss such a feature with the google team behind FlatBuffers.

[jdsika]

@PhRosenberger I think we should open a feature request in the Flatbuffer GitHub and ask for the feature? What do you think?
Flutbuffers does not come with a deep copy function, so there is no easy way to copy table or entire buffers to a different buffer. With OSI it is currently common practice to copy the SensorView input of a sensor model to the SensorData output. As there is no standard function to do this with Flatbuffers, one has to be custom written.

Yes we can do that, but I am a bit disappointment by the response to our issue over there, so I am not sure if we can expect it to actually happen. If we get some feedback, it would be nice to discuss such a feature with the google team behind FlatBuffers.

Yes, I can understand this feeling. Let's just keep going and do the feature request. If we all start commenting on it we might get things moving :)

[PhRosenberger]

Yes, I can understand this feeling. Let's just keep going and do the feature request. If we all start commenting on it we might get things moving :)

I just created such an issue, so we will know their thoughts on this soon, hopefully: google/flatbuffers#7798

[PhRosenberger]

We should discuss the concept of nested flatbuffers, e.g. nesting SensorView and then nesting GlobalGroundTruth inside of the SensorView, as suggested by google/flatbuffers#7798 (comment)

[PhRosenberger]

Copying of stuff (e.g. GlobalGroundTruth or SensorViewConfiguration) is now moved to #703

[thomassedlmayer]

Benchmark results/presentation slides - Flatbuf struct vs. table vs. protobuf based on https://github.com/OpenSimulationInterface/osi-sensor-model-packaging/tree/feature/flatbuffers_examples

presentation-flatbuf-comparison.pdf