Skip to content

Fix \\G matching for interpolated /x patterns used by String::Formatter #1364

Description

@fglock

Summary

Fix a PerlOnJava regex-engine mismatch that prevents String::Formatter from
matching its \G-anchored, interpolated format-token pattern. This causes
String::Errf to return format strings literally instead of expanding them.

CPAN failure

CPAN compatibility run 20260910-101908-8428:

  • Distribution: String-Errf v0.009
  • Target module: String::Errf
  • Result: 3 of 6 subtests failed; 1 of 3 test programs failed
  • Failing test: t/basic.t
  • Dependency: String::Formatter v1.235, pure Perl and successfully built

The failures cover integer, float, timestamp, numbered, string, and undef
formatting paths. The common symptom is that tokens such as %{x}i remain
unchanged instead of being expanded.

Minimal reproducer

The upstream formatter eventually uses this pattern structure:

my $regex = qr/
 (%
  (?:{
    ([^;]*?)
    (?: ; ([^\}]*?) )?
  })
  ($|.)
 )
/xi;

my $s = '%{x}i';
while ($s =~ m{\G(.*?)$regex}gs) {
    print "matched=[$1][$2][$3][$4][$5]\n";
}

Expected under system Perl:

matched=[][%{x}i][x][][i]

Observed under PerlOnJava:

(no match)

The smallest user-visible reproducer is:

use String::Errf errf => ();
print errf('%{x}i', { x => 10 }), "\n";

Expected output:

10

PerlOnJava output:

%{x}i

The same result occurs on the JVM and interpreter backends.

System-Perl comparison

The complete String::Errf upstream test suite passes under system Perl. The
only non-ok output is the distribution's existing TODO-marked local-time
expectation, caused by the test host timezone; the test harness exits
successfully.

The pure-Perl String::Formatter dependency is therefore not the cause. Its
formatter works under system Perl and fails under PerlOnJava at the regex
matching stage.

Ownership and root-cause hypothesis

Primary ownership is the regex engine, specifically the interaction among:

  • a compiled qr// object interpolated into a larger pattern;
  • /x whitespace/comments and nested captures;
  • \G as the independent search-position anchor;
  • /g execution and zero-width/progression state.

This is not a String::Errf or String::Formatter implementation defect.
The existing \G control check with a simple literal pattern succeeds, so the
investigation should preserve the distinction between basic \G support and
the failing interpolated/capture pattern path.

Related but distinct: #1265 concerns unescaped literal-brace handling and
warning publication in a substitution. It may share regex compiler or matcher
infrastructure, but this issue must independently preserve the format-token
match behavior.

Investigation requirements

Trace the first divergence through these stages:

  1. Confirm raw and cooked pattern source, including the interpolated qr//
    identity and /x modifier.
  2. Confirm compiled Joni options, capture numbering, and any optimizer/start
    metadata.
  3. Record search start, region bounds, \G position, and every /g result.
  4. Compare the same pattern with and without interpolation, /x, captures, and
    \G to isolate the smallest failing feature combination.
  5. Test JVM and interpreter paths and direct Joni behavior where applicable.

Do not fix this with a String::Formatter-specific source rewrite, a
java.util.regex fallback, or a special case recognizing %{x}i. Preserve
general Perl matching semantics.

Requested fix

  • Make the failing pattern match with system-Perl-compatible captures and
    progression.
  • Preserve correct \G and /g behavior for adjacent and repeated tokens.
  • Verify interpolated qr// patterns under /x, including empty-prefix and
    non-empty-prefix cases.
  • Retain correct behavior for escaped braces and malformed format patterns.
  • Keep JVM and interpreter results identical.

Regression coverage

Add a tracked project-owned regression test independent of the downloaded CPAN
build tree. It must:

  • pass unchanged under system Perl;
  • fail on the unfixed PerlOnJava parent for the expected mismatch;
  • run on JVM and interpreter after the fix;
  • include the String::Formatter-style minimal pattern;
  • include direct String::Errf formatting for strings, integers, floats,
    numbered values, timestamps, and undef handlers;
  • include a nearby escaped-brace control;
  • record the complete /g match sequence and pos values;
  • cover both interpolated and non-interpolated forms.

After the focused fix, rerun String::Errf v0.009, adjacent regex tests, and
the related String::Formatter compatibility paths. Scan for regressions in
captures, \G, /g, /x, and qr// interpolation.

Labels: bug, area:regex, area:cpan-port, area:backend.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:backendJVM interpreter or execution-backend behaviorarea:cpan-portCPAN compatibility ports and providersarea:regexRegex/Joni/parser matchingbugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions