First steps towards PostgreSQL integration - #651
Conversation
Tests are yet to be implemented, so the code remains in an early stage, completely untested even for basic functionality
There was a problem hiding this comment.
🟡 Changes recommended
There are multiple confirmed correctness/build issues (schema DDL syntax error, incorrect SELECT result handling via affected_rows(), missing test source file in CMake, and credential logging risk) that must be fixed before it can be safely validated.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Introduces an initial PostgreSQL-backed implementation of the FaultStorage backend for ros2_medkit_fault_manager, wiring it into FaultManagerNode via a new storage_type=postgres option and a database_url parameter.
Changes:
- Add
PgFaultStorage(libpqxx-based) with schema initialization and implementations for fault/events, snapshots, near-misses, and rosbag retention APIs. - Extend
FaultManagerNodeto select PostgreSQL storage via parameters. - Update build/package dependencies to include PostgreSQL/libpqxx and add a placeholder GTest target for PostgreSQL storage.
File summaries
| File | Description |
|---|---|
| src/ros2_medkit_fault_manager/src/postgres_fault_storage.cpp | New PostgreSQL FaultStorage implementation and schema creation logic. |
| src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/postgres_fault_storage.hpp | Public header for PgFaultStorage implementing the FaultStorage interface. |
| src/ros2_medkit_fault_manager/src/fault_manager_node.cpp | Adds database_url param and selects PostgreSQL storage when storage_type=postgres. |
| src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp | Stores new database_url_ member. |
| src/ros2_medkit_fault_manager/CMakeLists.txt | Adds PostgreSQL dependency linkage and a PostgreSQL test target (currently missing source). |
| src/ros2_medkit_fault_manager/package.xml | Declares libpqxx dependency. |
Review details
Suppressed comments (4)
src/ros2_medkit_fault_manager/src/postgres_fault_storage.cpp:547
get_fault()usesaffected_rows()on a SELECT result; this can incorrectly return nullopt even when a fault exists. Useres.empty()to test whether the query returned rows.
auto res = tx.exec_params(
"SELECT fault_code, severity, description, first_occurred_ns, last_occurred_ns, occurrence_count, status, "
"reporting_sources, last_passed_ns FROM faults WHERE fault_code = $1",
fault_code);
if (res.affected_rows() == 0) {
src/ros2_medkit_fault_manager/src/postgres_fault_storage.cpp:653
contains()usesaffected_rows()on a SELECT result; that can incorrectly reportfalseeven when the row exists. For SELECT queries, checkres.empty()instead.
auto res = tx.exec_params("SELECT 1 FROM faults WHERE fault_code = $1 LIMIT 1", fault_code);
tx.commit();
return res.affected_rows() > 0;
src/ros2_medkit_fault_manager/src/postgres_fault_storage.cpp:748
- The newest-capture query aliases the column as
max_capture_idbut then readsmax_capture, and also usesaffected_rows()on a SELECT. This can throw at runtime and/or disable trimming. Prefer COALESCE +res.empty()and read the correct alias.
auto res =
tx.exec_params("SELECT MAX(capture_id) AS max_capture_id FROM snapshots WHERE fault_code = $1", fault_code);
int64_t newest_capture = 0;
if (res.affected_rows() > 0) {
newest_capture = res[0]["max_capture"].as<int64_t>();
src/ros2_medkit_fault_manager/src/postgres_fault_storage.cpp:753
- The snapshot-trimming loop uses
count_res.affected_rows() == 0on aSELECT COUNT(*)query; for SELECTs this is not a valid emptiness check and can short-circuit trimming unexpectedly. Usecount_res.empty()(or just read the first row) instead.
auto count_res = tx.exec_params("SELECT COUNT(*) AS sz FROM snapshots WHERE fault_code = $1", fault_code);
if (count_res.affected_rows() == 0 || count_res[0]["sz"].as<size_t>() <= max_snapshots_per_fault_) {
break;
- Files reviewed: 6/6 changed files
- Comments generated: 6
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # PostgreSQL storage tests | ||
| medkit_add_gtest(test_postgres_storage test/test_postgres_storage.cpp) | ||
| target_link_libraries(test_postgres_storage fault_manager_lib) | ||
| medkit_target_dependencies(test_postgres_storage rclcpp ros2_medkit_msgs) | ||
|
|
| } | ||
|
|
||
| if (storage_type_ == "postgres") { | ||
| RCLCPP_INFO(get_logger(), "Using PostgreSQL fault storage: %s", database_url_.c_str()); |
| CREATE INDEX IF NOT EXISTS idx_snapshots_fault_code ON snapshots(fault_code); | ||
| CREATE INDEX IF NOT EXISTS idx_snapshots_fault_topic ON snapshots(fault_code, topic))"); |
| "first_occurred_ns FROM faults WHERE fault_code = $1", | ||
| fault_code); | ||
|
|
||
| if (res.affected_rows() > 0) { |
| auto res = tx.exec_params("SELECT COUNT(*) FROM rosbag_files WHERE file_path = $1", file_path); | ||
| tx.commit(); | ||
| return res.affected_rows(); |
| for (const auto & r : res) { | ||
| paths.insert(r["file_path"].as<std::string>()); | ||
| } | ||
| removed = res.affected_rows() > 0; |
|
Oh, wow, Copilot came in aggressively! I will consider the LLM's remarks as I am implementing the tests. |
No worries, he is always like that. Use your own judgement, some of his comments are not worth fixing. Happy to take a look. |
|
Hello, I am currently finishing up the unit tests. I plan to update this PR with the new changes tomorrow. Apart from the integration and testing code, I also want to update the documentation. Can you please point me to the required documentation file(s) that need to be updated? Thanks! |
Hi @gstavrinos, two files to update:
That should be it, but if I find anything else, I'll let you know. |
Improved SELECT result handling based on best practices, fixed typos, fixed potential password leaks from printing the database url and other minor issues
|
I have updated the source code, added the unit tests, a docker compose file and relevant documentation points. Some remarks:
I will be waiting for your feedback on the provided material and the next steps towards merging. Thanks again! |
bburda
left a comment
bburda
left a comment
There was a problem hiding this comment.
Thanks for this contribution :)
There are also two bigger things: the PostgreSQL tests do not run in CI, and the fault manager stops on the next fault report after the database restarts. We can open separate issues for them, or you can fix them in this PR. Please also make sure that CI passes.
| find_package(rclcpp REQUIRED) | ||
| find_package(ros2_medkit_msgs REQUIRED) | ||
| find_package(ros2_medkit_serialization REQUIRED) | ||
| find_package(PostgreSQL REQUIRED) |
There was a problem hiding this comment.
Could you put the PostgreSQL backend behind a CMake option that is OFF by default? Right now libpqxx is required for every build, and that breaks builds that never use PostgreSQL:
- Humble has libpqxx 6.4. The new file does not compile there:
connection::close()is protected, astd::vector<std::string>cannot be a query parameter (list_faults()), and<filesystem>is not included. - On Lyrical, 22 of 24 fault_manager tests fail. For example,
test_capture_thread_poolpasses all 12 tests and then aborts at exit withfree(): double free detected in tcache 2. The cause is the libpqxx 7.10.0 package in Ubuntu 26.04. A program with only#include <pqxx/pqxx>andint main() { return 0; }aborts in the same way. If I link libpqxx and leave out the header, it exits normally. - The runtime stage of the Dockerfile in this repository does not install libpqxx.
fault_manager_nodenow needslibpqxx-7.8.so, so the Docker image cannot start the fault manager.
With the option OFF, please build nothing PostgreSQL-related: no find_package, no postgres_fault_storage.cpp, no pqxx link and no test target. In such a build, create_storage() should stop with a clear error when storage_type is postgres. The #include of postgres_fault_storage.hpp in fault_manager_node.cpp needs the same guard. Please also remove <depend>libpqxx-dev</depend> from package.xml, and write the option name and the libpqxx-dev package in the README, because rosdep does not install it without the <depend>.
| // The SQLite backend has to opt into a transaction for this (BEGIN IMMEDIATE) | ||
| // In the PostgreSQL implementation the lock used is a transaction lock, so no extra handling is required apart from | ||
| // updating the tables properly | ||
| pqxx::work tx(*db_conn_); |
There was a problem hiding this comment.
I built this branch and ran the fault manager against PostgreSQL 16. I reported one fault, restarted the database container and reported a second fault. The node stopped on the second report:
terminate called after throwing an instance of 'pqxx::broken_connection'
what(): Lost connection to the database server.
The object opens one pqxx::connection and never reconnects, and the service handlers do not catch storage exceptions. So every database restart or network drop stops the fault manager. If you want to fix this in the PR, two changes are needed:
- In
PgFaultStorage: when a transaction cannot start because the connection is broken, open a new connection and try the call once more. Do not retry onpqxx::in_doubt_error, because the commit may already be applied. - In the node: catch exceptions from
storage_in the service and timer callbacks and return an error response, so the process keeps running while the database is down.
| for (const auto & path : unique_paths) { | ||
| if (path_referenced(path)) { | ||
| continue; | ||
| } | ||
| std::error_code ec; | ||
| std::filesystem::remove_all(path, ec); | ||
| } |
There was a problem hiding this comment.
Could you make sure this loop cannot throw? The rows are already committed at this point. path_referenced() starts a new transaction, so it throws when the connection is lost. RosbagCapture treats any exception from store_rosbag_files() as "nothing was stored" and deletes the new bag, and the committed rows then point to a bag that no longer exists. With the change below, the worst case is an orphaned directory, which the other comments in this file already accept.
| for (const auto & path : unique_paths) { | |
| if (path_referenced(path)) { | |
| continue; | |
| } | |
| std::error_code ec; | |
| std::filesystem::remove_all(path, ec); | |
| } | |
| for (const auto & path : unique_paths) { | |
| try { | |
| if (path_referenced(path)) { | |
| continue; | |
| } | |
| } catch (const std::exception &) { | |
| continue; // The rows are already committed, so keep the directory. | |
| } | |
| std::error_code ec; | |
| std::filesystem::remove_all(path, ec); | |
| } |
| try { | ||
| db_conn_ = std::make_unique<pqxx::connection>(base_conn_info()); | ||
| } catch (const std::exception & e) { | ||
| GTEST_SKIP() << "No PostgreSQL server reachable (set ROS2_MEDKIT_TEST_PG_CONN): " << e.what(); |
There was a problem hiding this comment.
All 98 tests in this file are skipped in CI, because CI has no PostgreSQL server. The Jazzy test log shows [ SKIPPED ] for each of them. If you want to fix this in the PR, the jazzy-test job in .github/workflows/ci.yml needs:
- a
postgresservice at the minimum version you support libpqxx-devinstalled and a build with the new option ONROS2_MEDKIT_TEST_PG_CONNset topostgresql://user:password@postgres:5432/<db>
jazzy-test runs in a container, so the service name works as the host name. After that job is in place, please change GTEST_SKIP() to FAIL() here, so a missing server is reported as a failure. With the option OFF the test target is not built, so the other jobs are not affected.
| auto res = tx.exec_params( | ||
| "DELETE FROM near_misses WHERE id IN (" | ||
| "SELECT id FROM (SELECT id, ROW_NUMBER() OVER " | ||
| "(PARTITION BY fault_code ORDER BY id DESC) AS rn FROM near_misses) " |
There was a problem hiding this comment.
Could you add an alias to this subquery? PostgreSQL accepts a subquery in FROM without an alias only since version 16. I ran the fault manager from this branch against postgres:15 with default parameters. It stops at startup, because near_miss.max_per_fault defaults to 200 and the constructor runs this statement:
set_max_near_misses_per_fault PostgreSQL error: ERROR: subquery in FROM must have an alias
With the alias below, the statement works on PostgreSQL 15. Ubuntu 22.04 still ships PostgreSQL 14. Please also write the minimum PostgreSQL version in the docs and run the tests against that version. The compose file uses postgres:18, so the tests could not catch this.
| "(PARTITION BY fault_code ORDER BY id DESC) AS rn FROM near_misses) " | |
| "(PARTITION BY fault_code ORDER BY id DESC) AS rn FROM near_misses) AS ranked " |
| TEST_F(PgFaultStorageTest, ReopenRepairsLastOccurredInflatedByOldPassedBug) { | ||
| // Rows written by releases that advanced last_occurred_ns on PASSED events are | ||
| // inflated, and a latched CONFIRMED fault that never fails again would keep the | ||
| // wrong timestamp forever. Opening the storage must repair them from | ||
| // last_failed_ns. | ||
| // NOTE: This should not be possible in the PostgreSQL implementation since it was introduced after the fix | ||
| const rclcpp::Time failed_at(1000, 0, RCL_SYSTEM_TIME); | ||
| storage_->report_fault_event("FAULT_MIG", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "Test", "/node1", | ||
| failed_at, default_config()); | ||
| storage_.reset(); | ||
|
|
||
| // Simulate the old bug: a PASSED event at t=9000s re-dated the row. | ||
| exec_raw("UPDATE faults SET last_occurred_ns = 9000000000000 WHERE fault_code = 'FAULT_MIG'"); | ||
|
|
||
| storage_ = std::make_unique<PgFaultStorage>(conn_info_); | ||
|
|
||
| auto fault = storage_->get_fault("FAULT_MIG"); | ||
| ASSERT_TRUE(fault.has_value()); | ||
| EXPECT_EQ(rclcpp::Time(fault->last_occurred).nanoseconds(), failed_at.nanoseconds()); | ||
| } | ||
|
|
There was a problem hiding this comment.
Could you remove this test together with the migration in initialize_schema()? Without that UPDATE this test fails.
| TEST_F(PgFaultStorageTest, ReopenRepairsLastOccurredInflatedByOldPassedBug) { | |
| // Rows written by releases that advanced last_occurred_ns on PASSED events are | |
| // inflated, and a latched CONFIRMED fault that never fails again would keep the | |
| // wrong timestamp forever. Opening the storage must repair them from | |
| // last_failed_ns. | |
| // NOTE: This should not be possible in the PostgreSQL implementation since it was introduced after the fix | |
| const rclcpp::Time failed_at(1000, 0, RCL_SYSTEM_TIME); | |
| storage_->report_fault_event("FAULT_MIG", ReportFault::Request::EVENT_FAILED, Fault::SEVERITY_ERROR, "Test", "/node1", | |
| failed_at, default_config()); | |
| storage_.reset(); | |
| // Simulate the old bug: a PASSED event at t=9000s re-dated the row. | |
| exec_raw("UPDATE faults SET last_occurred_ns = 9000000000000 WHERE fault_code = 'FAULT_MIG'"); | |
| storage_ = std::make_unique<PgFaultStorage>(conn_info_); | |
| auto fault = storage_->get_fault("FAULT_MIG"); | |
| ASSERT_TRUE(fault.has_value()); | |
| EXPECT_EQ(rclcpp::Time(fault->last_occurred).nanoseconds(), failed_at.nanoseconds()); | |
| } |
| - ``""`` | ||
| - Where the audit database lives. Empty puts it beside the fault database, | ||
| or in memory when the fault store is itself in memory or not SQLite. | ||
| or in memory when the fault store is itself in memory or unknown storage type. |
There was a problem hiding this comment.
With storage_type: postgres, create_audit_log() writes the audit log to fault_audit.db next to database_path. By default that is /var/lib/ros2_medkit/, a path that PostgreSQL users do not use otherwise. Could you write that here, so users know the audit trail stays on the robot? The complete example at the end of this page also lists only storage_type and database_path. Please add database_url there.
| or in memory when the fault store is itself in memory or unknown storage type. | |
| or in memory when the fault store is itself in memory or unknown storage type. | |
| With ``storage_type: postgres`` it is a local SQLite file ``fault_audit.db`` next to | |
| ``database_path``, so the audit trail stays on the robot. |
|
|
||
| **Memory**: Faults are stored in memory only. Useful for testing or when persistence is not required. | ||
|
|
||
| **PostgreSQL**: Faults are persisted to disk and survive node restarts. Needs an external PostgreSQL server. |
There was a problem hiding this comment.
Could you update the Features list above too? It still says Persistent storage: SQLite backend ensures faults survive node restarts. Please also change this line so it says where the faults and the audit log are stored:
| **PostgreSQL**: Faults are persisted to disk and survive node restarts. Needs an external PostgreSQL server. | |
| **PostgreSQL**: Faults are stored in an external PostgreSQL server and survive node restarts. The audit log stays a local SQLite file next to `database_path`. |
| medkit_add_gtest(test_sqlite_storage test/test_sqlite_storage.cpp) | ||
| target_link_libraries(test_sqlite_storage fault_manager_lib) | ||
| medkit_target_dependencies(test_sqlite_storage rclcpp ros2_medkit_msgs) | ||
|
|
There was a problem hiding this comment.
Could you remove the trailing whitespace on this line? lint_cmake in the format-lint job fails on it.
| { | ||
| std::ofstream(bag_dir / "payload.mcap") << "data"; | ||
| } |
There was a problem hiding this comment.
Could you format this block as clang-format expects? The format-lint job fails on it. pre-commit run --all-files from the repository root catches this and the CMake whitespace before CI does.
| { | |
| std::ofstream(bag_dir / "payload.mcap") << "data"; | |
| } | |
| { std::ofstream(bag_dir / "payload.mcap") << "data"; } |
|
Hey Bartosz, Thank you for your input. I will look into your suggestions and return with fixes and/or questions. I am a little bit surprised that the tests do not pass, since I run them on my (jazzy) setup. Seems like I will have to test against humble and lyrical too since Ubuntu uses different PostgreSQL versions. Give me some time for all that and I will be back. |
Pull Request
Summary
As discussed in #649, this is an early implementation of the PostgreSQL integration. Keep in mind that currently the code does not compile because the testing suite is not included in this PR.
Issue
Link the related issue (required):
Type
Testing
Tests are yet to be implemented, so the code remains in an early stage, completely untested even for basic functionality
Checklist
TODOs (based on your checklist, will tick the list as improvements come along)
As this is still a WIP, feel free to offer suggestions, recommendations or problems you might think of.