Skip to content

fix(extraction): retain errors for empty LLM responses - #2228

Open
Kuang-xianxin wants to merge 1 commit into
unclecode:developfrom
Kuang-xianxin:codex/report-empty-llm-extraction-errors
Open

fix(extraction): retain errors for empty LLM responses#2228
Kuang-xianxin wants to merge 1 commit into
unclecode:developfrom
Kuang-xianxin:codex/report-empty-llm-extraction-errors

Conversation

@Kuang-xianxin

@Kuang-xianxin Kuang-xianxin commented Sep 5, 2026

Copy link
Copy Markdown

Summary

Keep empty LLM completions marked as extraction errors in both LLMExtractionStrategy.extract and aextract.

When a provider returns content=None or "" (for example with finish_reason="length" or "content_filter"), the existing code creates an error block and then overwrites its error flag with False in the successful-block loop. Whitespace-only content instead disappears as an empty list. Downstream code filtering successful records by error can therefore accept diagnostics as extracted data or miss the failure entirely.

Return the empty-response diagnostic before successful-result normalization, retain the actual section index and finish reason, and treat whitespace-only content as empty. Usage accounting still runs before this return. A valid model response containing [] continues to mean successful extraction with no matching records.

Files changed

  • crawl4ai/extraction_strategy.py: the same narrow empty-content guard in sync and async extraction.
  • tests/unit/test_llm_extraction_empty_response.py: parameterized regressions using real LiteLLM ModelResponse objects with mocked completion boundaries; no model API calls.

Validation

  • Before the fix: 36 regression cases fail, while eight valid-response controls pass.
  • After the fix: all 44 new cases pass, covering sync/async, JSON/XML-block parsing, None/empty/whitespace content, stop/length/content_filter reasons, section indices, usage preservation and valid empty result lists.
  • Full tests/unit suite: 121 passed on Python 3.13.14 after installing the optional pypdf dependency needed by existing PDF tests.
  • Test-file Black check (--target-version py310), compilation of both changed Python files, and git diff --check pass. The existing source file was not reformatted wholesale.

Command: LITELLM_LOCAL_MODEL_COST_MAP=True uv run --no-sync pytest tests/unit -q.

Based on and targeting develop. Found by source inspection; no existing issue is claimed as resolved. Older #1450 addressed alternative response fields rather than this error-flag overwrite.

No live browser crawl or model-provider validation is claimed.

Return empty-response diagnostics before successful-block normalization in both
sync and async extraction, preserving section indices and token accounting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant