Split out of #80, which fixed the numerator.
rhet.em-dash reports a density: dashes per 100 words. #80 stopped the numerator counting markup. The denominator — context.Statistics.WordCount — still counts every word inside fenced code blocks, table cells and frontmatter.
For a technical document that understates the density, so the rule is now too lenient there rather than too harsh. Concretely, adding a code block to a text with three dashes in its prose lowers the reported density and the finding's weight, without a word of prose changing. EmDashCountingTests.Markup_does_not_change_how_many_dashes_are_reported pins the count and deliberately does not pin the weight, for exactly this reason.
Why it was not fixed there
The word count is shared. stat.burstiness, lexical diversity, every humanRatePer1000 gate and the whole calibration read it. Changing what counts as a word moves the published number, which is a different kind of change and needs the corpus re-measured and a committee.
The shape of the work
The engine has no prose-only view of a document at all — TextDocument exposes Raw and tokenises from it. The question is whether to add one (TextDocument.Prose, markup stripped) and which analyzers should read it. That is a design decision, not a patch:
- a rule about punctuation or rhythm should read prose
- a rule about characters that typing cannot produce should read raw, because the point is what is actually in the file
Both readings are already right for their own rule, which is why this needs deciding rather than sweeping.
Not to be done
Stripping markup inside the calibration tool only. The corpus is plain text with almost no markup (10 texts of 308 contain -- at all), so it would change nothing there and quietly leave real users on the other behaviour.
Split out of #80, which fixed the numerator.
rhet.em-dashreports a density: dashes per 100 words. #80 stopped the numerator counting markup. The denominator —context.Statistics.WordCount— still counts every word inside fenced code blocks, table cells and frontmatter.For a technical document that understates the density, so the rule is now too lenient there rather than too harsh. Concretely, adding a code block to a text with three dashes in its prose lowers the reported density and the finding's weight, without a word of prose changing.
EmDashCountingTests.Markup_does_not_change_how_many_dashes_are_reportedpins the count and deliberately does not pin the weight, for exactly this reason.Why it was not fixed there
The word count is shared.
stat.burstiness, lexical diversity, everyhumanRatePer1000gate and the whole calibration read it. Changing what counts as a word moves the published number, which is a different kind of change and needs the corpus re-measured and a committee.The shape of the work
The engine has no prose-only view of a document at all —
TextDocumentexposesRawand tokenises from it. The question is whether to add one (TextDocument.Prose, markup stripped) and which analyzers should read it. That is a design decision, not a patch:Both readings are already right for their own rule, which is why this needs deciding rather than sweeping.
Not to be done
Stripping markup inside the calibration tool only. The corpus is plain text with almost no markup (10 texts of 308 contain
--at all), so it would change nothing there and quietly leave real users on the other behaviour.