GH-50925: [C++] Allow CSV reader to pad rows with missing trailing fields

[HuaHuaY]

Rationale for this change

Some CSV files omit trailing optional fields. The C++ CSV reader currently rejects these short rows, requiring callers to preprocess or discard them.

What changes are included in this PR?

Add an option pad_short_rows in CSV struct ParseOptions that pads missing trailing fields with nulls.

Are these changes tested?

Yes.

Are there any user-facing changes?

Add an option pad_short_rows in CSV struct ParseOptions.

• GitHub Issue: [C++] Allow CSV reader to pad rows with missing trailing fields #50925

[HuaHuaY]

I'll temporarily switch to a draft and reorganize my PR.

[Copilot]

Pull request overview

Adds a new C++ CSV ParseOptions::pad_short_rows option to accept rows with fewer trailing fields than the expected column count, padding the missing trailing fields as nulls (instead of rejecting the row). This aligns the CSV reader behavior with common “optional trailing column” CSV inputs while keeping the default strict behavior unchanged.

Changes:

• Introduce ParseOptions::pad_short_rows (default false) to enable padding of short rows with nulls.
• Extend the CSV parser/visitor path to propagate “missing (padded)” cells to converters so they can reliably append nulls.
• Add parser- and reader-level tests covering both string and typed/dictionary conversions.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
cpp/src/arrow/dataset/file_csv.cc	Includes pad_short_rows in CsvFileFormat::Equals so formats with different padding behavior are not considered equal.
cpp/src/arrow/csv/options.h	Adds the public ParseOptions::pad_short_rows option and documents its intent.
cpp/src/arrow/csv/parser.cc	Implements padding behavior for short rows by appending empty field descriptors and recording missing ranges.
cpp/src/arrow/csv/parser.h	Threads a missing flag through VisitColumn/VisitLastRow (backward-compatible visitor signature support).
cpp/src/arrow/csv/converter.cc	Ensures padded “missing” values convert to nulls across supported converter paths (primitive + dictionary).
cpp/src/arrow/csv/parser_test.cc	Adds unit coverage verifying padding and the missing signal in visitor callbacks.
cpp/src/arrow/csv/reader_test.cc	Adds integration coverage for padded short rows for string columns and typed/dictionary columns.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[HuaHuaY]

@pitrou Please take a look.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cpp/src/arrow/csv/parser.h:240

• BlockParser::VisitColumn / VisitLastRow are public template APIs in a public header; changing the visitor signature to require an extra bool missing argument is a source-breaking change for downstream callers. Consider keeping backward compatibility by accepting both 3-arg and 4-arg visitors (e.g., use std::is_invocable_r_v<Status, Visitor, const uint8_t*, uint32_t, bool, bool> / if constexpr to dispatch), and document precisely what missing means (padded field due to pad_short_rows, not an empty field).

  /// \brief Visit parsed values in a column
  ///
  /// The signature of the visitor is
  /// Status(const uint8_t* data, uint32_t size, bool quoted, bool missing)
  template <typename Visitor>
  Status VisitColumn(int32_t col_index, Visitor&& visit) const {
    return parsed_batch().VisitColumn(col_index, first_row_num(),
                                      std::forward<Visitor>(visit));
  }

  template <typename Visitor>
  Status VisitLastRow(Visitor&& visit) const {
    return parsed_batch().VisitLastRow(std::forward<Visitor>(visit));
  }

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

cpp/src/arrow/dataset/file_csv.cc:196

• GetOrderedColumnNames() ignores the new missing flag from VisitLastRow(). If pad_short_rows is enabled and earlier parsed rows have more columns than the header row (e.g. due to skip_rows), this will currently append empty strings as column names for missing header fields, potentially creating empty/duplicate column names and confusing downstream errors. Consider failing fast when missing is true (or generating names explicitly).

      [&](const uint8_t* data, uint32_t size, bool quoted, bool missing) -> Status {
        std::string_view view{reinterpret_cast<const char*>(data), size};
        column_names.emplace_back(view);
        return Status::OK();
      }));

[wgtmac]

+1

[HuaHuaY]

Rebase and add one more commit. Not allow to pad header row when using dataset reader.

In the situation, the number of columns is not determined by the number of headers in the file, which seems strange. However, changing this would result in behavior inconsistent with the existing code (even though the inconsistency would only arise with files that previously caused errors) and need more modifications. Therefore, I opted for the simple solution of temporarily disabling this newly added flag.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

[pitrou]

Therefore, I opted for the simple solution of temporarily disabling this newly added flag.

Wouldn't it be better to return Status::NotImplemented to signal that the option is not supported?

[HuaHuaY]

Therefore, I opted for the simple solution of temporarily disabling this newly added flag.

Wouldn't it be better to return Status::NotImplemented to signal that the option is not supported?

I intend for the newly added flag in this situation to still be usable for filling in missing columns in the data; it simply shouldn't fill in the header row of the CSV.

In the CSV reader, when the user does not explicitly provide column names, the number of columns is determined by the first row after skipping skip_rows (meaning that if the file contains a header row, the count reflects the number of column names).

However, in the dataset reader, skip_rows is not processed upfront, causing the column count to be determined before the file's header row is parsed (I think the practice of determining the column count based on skipped rows is strange and likely indicative of an imperfect earlier implementation).

[pitrou]

I'm not sure I understand: if I enable pad_short_rows with the dataset reader, is the flag ignored or not?

[HuaHuaY]

I'm not sure I understand: if I enable pad_short_rows with the dataset reader, is the flag ignored or not?

In this PR, it is ignored when parsing column names from the file, but remains effective when parsing the data.

[pitrou]

LGTM, just one nit

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.