CAS: configurable mount lease timing

[k-morozov]

Added settings to configure the CAS mount lease TTL and renewal period. The lightweight delete hard-restart test kills ClickHouse 100 times, and the default 30-second lease causes about 36.5 seconds of recovery waiting after every kill, making the test run for more than an hour. Test configurations can now use shorter lease timings to reduce CI duration. These settings are also useful in production because they control the balance between recovery time, object-storage delays, and lease renewal frequency.

Changelog category (leave one):

• Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added settings to configure the CAS mount lease TTL and renewal period.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [e424083]

[ilejn]

LGTM

[filimonov]

Thanks, the change itself is fine and it does control the hard-restart pause: the wait is ttl + ttl/20 + renew_period/2 (36.5 s with defaults), and both values now come from these settings.

A few points:

1. Please document two rules in configuration.md. Both settings must be the same on all servers that share a pool. The observation threshold is computed from the local config only, so a server or GC leader with a shorter TTL can treat a healthy peer with a longer TTL as dead. Also, lowering the TTL is only safe when all members of the pool are stopped first (or do shutdown / restart gracefully). A restart with a lower TTL waits less than the old lease of the previous process.

2. A shorter TTL / renew period increases the risk of losing the lease. Any object-storage delay or scheduling hiccup longer than the margin will cause a lease loss and a remount. This should be stated in the setting description.

3. I have a draft plan (not implemented yet, there are some difficulties) to make lease recovery fast and cheap in the happy path (short connection loss). But it will not help this test: the test kills the process, so there is no lease to recover, only a slot to reclaim.

4. Maybe a better tool for this test is a separate setting like cas_unsafe_remount_no_delay (name is open). It would allow a node to reclaim a mount that belongs to its own server_uuid without the observation wait. It is unsafe in production (a stalled previous process could still be writing), so the name should say so. Then the test does not need to touch the lease timings at all.

5. It would be good to add a test: a config with these two settings reaches PoolConfig, and a kill/restart shows the expected waiting ~N ms (token-stability observation) value in the log.