fix(security): reject per-IP TLS pending overflow instead of evicting peers

[cursor]

Security fix

Fixes #269

Finding

Severity: Medium
Location: src/server/mod.rs

When max_pending_tls_per_addr was reached, queue_pending_tls evicted the first pending TLS handshake from the same source IP to admit a new stalled connection. An attacker sharing a NAT/carrier egress IP could repeatedly open stalled RTMPS connects and drop legitimate peers' in-flight handshakes before they complete.

Impact

Cross-connection DoS: RTMPS clients behind the same public IP could not complete TLS while an attacker saturated the per-IP pending cap.

Fix

Reject the incoming stalled handshake when the per-IP cap is full, mirroring the existing global-cap behavior that already rejects instead of evicting unrelated peers.

Test

Added pending_tls_per_addr_cap_rejects_new_handshake_instead_of_evicting_oldest regression test.

  

---

Note

Medium Risk
Changes TLS admission under load in a security-sensitive path; behavior is intentionally stricter (reject vs evict) with a targeted regression test.

Overview
Security fix for per-IP pending TLS handshake admission when max_pending_tls_per_addr is saturated.

queue_pending_tls no longer removes the oldest pending handshake from the same source IP to admit a new stalled RTMPS connection. It now drops the incoming handshake and returns, matching how the global pending-TLS cap already behaves. That closes a NAT/shared-egress DoS where one co-located attacker could repeatedly open stalled connects and evict legitimate peers’ in-flight handshakes.

Adds regression test pending_tls_per_addr_cap_rejects_new_handshake_instead_of_evicting_oldest to assert the queue length and oldest entry stay unchanged when a sixth connect arrives at the default per-IP cap.

^{Reviewed by Cursor Bugbot for commit 1df3250. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 2e9d1a2f-6aa3-4c12-ad2a-1ae6bb1063ae

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_b40b4f6f-0c60-4810-83af-e3f0485ce09b)

[AlexanderWagnerDev]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 1df3250f3f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".