Add support for manual inspection of the responses and expected answers

[clincolnoz]

It would make manual validation of the of benchmark results easier, if a convenient to visualise table was provided.
Perhaps two tables is the best solution, where

1. the first table has query, response, expected response (incl. not answer ..., yes and no) and foreign keys to
2. a second table that contains the context(s)

[johann-petrak]

This has been implemented slightly differently as program/command ragability_2tsv

The output is a single tsv which contains any fields with array values in the form e.g. a.b_2 for what originally was the element with index 2 of the key b within a dict value of key a.

In other words, nested dictionary names get flattened by concatenating the keys with a dot, and any array valued fields get converted to columns with the element index after an underscore.
The tsv file contains for each array-valued key as many columns as the maximum number of elements in the array in any of the input rows.

This has been choses because:

• there is only a small number of elements in each of the arrays
• there may be any number of array-valued fields, which would require to have several, not just two different output files which will make manual inspection complex. Note that there are at least two different array valued fields already (facts and checks) which would have required at least 3 files to get generated making manual inspection a pain

[johann-petrak]

Reopened for testing feedback.

[grossst]

Feedback on the resulting table:

• pid occurs twice in the table ( as 'pid' and as 'pid_0'); todo: del pids_0
• 'llm': rename, e.g. to 'queried_llm', so that it does not get confused with the checker llm and add another column with the name of the checker llm
• 'facts':
  • 'facts' only contains the context, if the prompt contains 1 context (either context_1 or context_2)
  • 'fact_0' - 'fact_3': only contains an entry, if the amount of contexts in the prompt is > 1.
  • todo: either move the content of 'facts' to 'facts_0' or add all contexts to one column
• c1xq is added as the 4th column, c2xq as the 13th column: add them next to each other (e.g. next to context1 / context2) or remove them from the table
• 'costs' can probably be removed from this table

[johann-petrak]

It seems the wrong code is checked in an the version I have tested has somehow disappeared ... investigating ...

[johann-petrak]

OK, checked in a new implementation for this which is less generic / more ad-hoc but probably also more user-friendly.

Omitted the fields cost, c1xq and c2xq

[clincolnoz]

Hi Johann, if you are happy with the current version can you please update the docs to include the new ragability_2tsv?

[grossst]

I now added ragability_2tsv to the docs in ragability_experiments.

[clincolnoz]

Good day,

thanks for the updates it definitely useful to have this output. A few observations after checking the new output:

1. regarding documentation, I am still missing ragability_2tsv in the pythondoc.
2. the 'check.query' column is always empty in the output
3. And, if i am being a bit precious, it would be nice to split the short and long answers (check_for) into separate columns

[johann-petrak]

Thanks for checking!

The reason why "check.query" is included is because "query" is a standard field of a "check" block. It just so happens that for our particular corpus, this is never needed since for all checks, everything is already in the prompt template. However, for other corpora, this would be the actual, instance-specific query to include into the prompt template for checking the response the base LLM generated.

The "short/long" answer is simply the content of that field in JSON and the actual content injected into the prompt template so it is semantically a single field and something that is also just used in this specific corpus, I do not think it should get split up as it is the actual verbatim text that gets presented to the LLM

[johann-petrak]

OK, I have no updated 2tsv this way:

By default, columns where the same value occurs in all rows (e.g. all empty) are not included
in the tsv but the value of these fields is shown in the console output.
The option --all can be used to also include those fields in the tsv.

Hope this is a good solution for 2) !