Output: Don't replace dmp files by default

manual/sphinx/user_docs/adios2.rst

@@ -34,6 +34,10 @@ or on the BOUT++ command line set ``output:type=adios``. The default
 prefix is "BOUT.dmp" so the ADIOS file will be called "BOUT.dmp.bp". To change this,
 set the ``output:prefix`` option.
 
+As with NetCDF output, BOUT++ will by default refuse to overwrite an
+existing dump file. Use ``append`` to add new output to an existing
+ADIOS dataset, or ``replace`` to overwrite it.
+
 Restart files
 -------------
 
@@ -43,3 +47,4 @@ so to read and write restarts from an ADIOS dataset, put in BOUT.inp::
   [restart_files]
   type = adios
 
+Restart files still overwrite existing files by default.

manual/sphinx/user_docs/bout_options.rst

@@ -568,7 +568,9 @@ in table :numref:`tab-outputopts`.
    | Option               | Description                             | Default value  |
    +======================+=========================================+================+
    | ``append``           | Append to existing file if true,        | ``false``      |
-   |                      | otherwise overwrite                     |                |
+   |                      | otherwise create a new file             |                |
+   +----------------------+-----------------------------------------+----------------+
+   | ``replace``          | Overwrite an existing dump file if true | ``false``      |
    +----------------------+-----------------------------------------+----------------+
    | ``enabled``          | Writing is enabled                      | ``true``       |
    +----------------------+-----------------------------------------+----------------+
@@ -587,6 +589,9 @@ in table :numref:`tab-outputopts`.
 
 - ``enabled`` is useful mainly for doing performance or scaling tests, where you
   want to exclude I/O from the timings.
+- By default BOUT++ will not overwrite existing dump files. Set ``append=true``
+  to add new time points to an existing file, or ``replace=true`` to overwrite it.
+  Restart files still overwrite existing files by default.
 - If you find that IO is taking more and more time as your simulation goes on,
   try setting ``flush_frequency`` to a larger value such as ``10``. This can
   workaround an issue with NetCDF where subsequent writes take longer and
@@ -882,6 +887,10 @@ NetCDF file::
 
   bout::OptionsIO::create("settings.nc")->write(Options::root());
 
+This form creates or overwrites ``settings.nc`` if it already exists.
+To prevent overwriting, pass ``replace=false`` explicitly when creating
+the file.
+
 and to read it in again::
 
   Options data = bout::OptionsIO::create("settings.nc")->read();
@@ -954,6 +963,11 @@ automatically set the ``"time_dimension"`` attribute::
   // Append data to file
   bout::OptionsIO::create({{"file", "time.nc"}, {"append", true}})->write(data);
 
+If you want to overwrite an existing file instead of appending, use
+``replace=true``. If neither ``append`` nor ``replace`` is set, then
+writing to an existing dump/output file will fail with an error rather
+than silently overwriting data.
+
 .. note:: By default, `bout::OptionsIO::write` will only write variables
           with a ``"time_dimension"`` of ``"t"``. You can write
           variables with a different time dimension by passing it as

manual/sphinx/user_docs/running_bout.rst

@@ -96,12 +96,20 @@ Restart files allow the run to be restarted from where they left off::
 
      $ mpirun -np 2 ./conduction restart
 
-This will delete the output data ``BOUT.dmp.*.nc`` files, and start
-again. If you want to keep the output from the first run, add “append”::
+By default, this will keep the existing output data ``BOUT.dmp.*.nc``
+files and then stop with an error rather than overwrite them. To
+continue from the restart files and add new time points to the existing
+output, add ``append``::
 
      $ mpirun -np 2 ./conduction restart append
 
-which will then append any new outputs to the end of the old data files.
+To continue from the restart files and overwrite the existing output,
+add ``replace``::
+
+     $ mpirun -np 2 ./conduction restart replace
+
+which will replace the old dump files with new output from the restarted
+run. Restart files are still overwritten by default.
 For more information on restarting, see :ref:`sec-restarting`.
 
 To see some of the other command-line options try "-h"::
@@ -538,14 +546,24 @@ of the command, for example::
 
      $ mpirun -np 2 ./conduction restart
 
-Equivalently, put “restart=true” near the top of the BOUT.inp input
-file. Note that this will overwrite the existing data in the
-``BOUT.dmp.\*.nc`` files. If you want to append to them instead then add
-the keyword append to the command, for example::
+Equivalently, put ``restart=true`` near the top of the ``BOUT.inp`` input
+file. By default, BOUT++ will refuse to overwrite existing
+``BOUT.dmp.\*.nc`` files and will stop with an error instead. If you want
+to append to them then add the keyword ``append`` to the command, for example::
 
      $ mpirun -np 2 ./conduction restart append
 
-or also put ``append=true`` near the top of the BOUT.inp input file.
+or put ``append=true`` near the top of the ``BOUT.inp`` input file.
+
+If you want to overwrite the existing dump files, add ``replace`` to the
+command::
+
+     $ mpirun -np 2 ./conduction restart replace
+
+or put ``replace=true`` near the top of the ``BOUT.inp`` input file.
+
+Restart files are not affected by this change: they are still overwritten
+by default so that restarting continues to update the saved restart state.
 
 When restarting simulations BOUT++ will by default output the initial
 state, unless appending to existing data files when it will not output

src/sys/options/options_adios.cxx

@@ -24,9 +24,11 @@
 #include <cstddef>
 #include <cstdint>
 #include <exception>
-#include <stdexcept>
+#include <filesystem>
 #include <string>
 
+namespace fs = std::filesystem;
+
 namespace {
 std::size_t getLocalNDims(adios2::IO& io, const std::string& name,
                           const adios2::Dims& shape) {
@@ -233,6 +235,9 @@ OptionsADIOS::OptionsADIOS(Options& options) : OptionsIO(options) {
                   ? adios2::Mode::Append
                   : adios2::Mode::Write;
 
+  replace_existing_file =
+      options["replace"].doc("Replace existing file?").withDefault<bool>(false);
+
   singleWriteFile = options["singleWriteFile"].withDefault<bool>(false);
 }
 
@@ -404,6 +409,17 @@ namespace bout {
 void OptionsADIOS::write(const Options& options, const std::string& time_dim) {
   const Timer timer("io");
 
+  if (file_mode == adios2::Mode::Write and fs::exists(filename)
+      and !replace_existing_file) {
+    throw BoutException(
+        "Output file '{:s}' already exists. Aborting rather than overwriting data.\n"
+        "Set 'append=true' to append, or 'replace=true' to overwrite.",
+        filename);
+  }
+  // Write gets called multiple times.
+  // Only prevent overwrite the first time.
+  replace_existing_file = true;
+
   // ADIOSStream is just a BOUT++ object, it does not create anything inside ADIOS
   ADIOSStream& stream = ADIOSStream::ADIOSGetStream(filename, file_mode);
 

src/sys/options/options_adios.hxx

@@ -67,6 +67,7 @@ private:
   std::string filename;
   /// How to open the file for writing
   adios2::Mode file_mode{adios2::Mode::Write};
+  bool replace_existing_file; ///< Replace existing file if it exists?
   bool singleWriteFile = false;
 };
 

src/sys/options/options_io.cxx

@@ -2,11 +2,15 @@
 #include "bout/bout.hxx"
 #include "bout/globals.hxx"
 #include "bout/mesh.hxx"
+#include "bout/options.hxx"
 #include "bout/version.hxx"
 
 #include "options_adios.hxx"
 #include "options_netcdf.hxx"
 
+#include <memory>
+#include <string>
+
 namespace bout {
 std::unique_ptr<OptionsIO> OptionsIO::create(const std::string& file) {
   return OptionsIOFactory::getInstance().createFile(file);
@@ -25,6 +29,7 @@ OptionsIOFactory::ReturnType OptionsIOFactory::createRestart(Options* optionsptr
       Options::root()["datadir"].withDefault<std::string>("data"));
   options["prefix"].overrideDefault("BOUT.restart");
   options["append"].overrideDefault(false);
+  options["replace"].overrideDefault(true); // Restart files are overwritten each output
   options["singleWriteFile"].overrideDefault(true);
   return create(getType(&options), options);
 }
@@ -39,11 +44,15 @@ OptionsIOFactory::ReturnType OptionsIOFactory::createOutput(Options* optionsptr)
   options["append"].overrideDefault(Options::root()["append"]
                                         .doc("Add output data to existing (dump) files?")
                                         .withDefault<bool>(false));
+  options["replace"].overrideDefault(
+      Options::root()["replace"]
+          .doc("Replace output (dump) files if they exist and not appending?")
+          .withDefault<bool>(false));
   return create(getType(&options), options);
 }
 
 OptionsIOFactory::ReturnType OptionsIOFactory::createFile(const std::string& file) const {
-  Options options{{"file", file}};
+  Options options{{"file", file}, {"replace", true}};
   return create(getDefaultType(), options);
 }
 

src/sys/options/options_netcdf.cxx

@@ -4,17 +4,30 @@
 
 #include "options_netcdf.hxx"
 
+#include "bout/array.hxx"
+#include "bout/assert.hxx"
 #include "bout/bout.hxx"
 #include "bout/bout_types.hxx"
+#include "bout/boutexception.hxx"
 #include "bout/mesh.hxx"
+#include "bout/options_io.hxx"
 #include "bout/sys/timer.hxx"
+#include "bout/sys/variant.hxx"
 #include "bout/traits.hxx"
+#include "bout/unused.hxx"
+#include "bout/utils.hxx"
 
 #include <fmt/format.h>
 
 #include <climits>
 #include <exception>
+#include <fstream>
+#include <functional>
+#include <map>
+#include <memory>
 #include <netcdf>
+#include <string>
+#include <utility>
 #include <vector>
 
 using namespace netCDF;
@@ -206,7 +219,7 @@ void readGroup(const std::string& filename, const NcGroup& group, Options& resul
 namespace bout {
 
 Options OptionsNetCDF::read(bool lazy) {
-  Timer timer("io");
+  const Timer timer("io");
 
   // Open file
   auto read_file = std::make_shared<netCDF::NcFile>(filename, NcFile::read);
@@ -399,7 +412,7 @@ struct NcPutVarVisitor {
 
 template <>
 void NcPutVarVisitor::operator()<bool>(const bool& value) {
-  int int_val = value ? 1 : 0;
+  const int int_val = value ? 1 : 0;
   var.putVar(&int_val);
 }
 
@@ -494,7 +507,7 @@ struct NcPutAttVisitor {
 
 template <>
 void NcPutAttVisitor::operator()(const bool& value) {
-  int ival = value ? 1 : 0;
+  const int ival = value ? 1 : 0;
   var.putAtt(name, ncInt, ival);
 }
 template <>
@@ -751,9 +764,17 @@ OptionsNetCDF::OptionsNetCDF(Options& options) : OptionsIO(options) {
                            options["prefix"].as<std::string>(), BoutComm::rank());
   }
 
-  file_mode = (options["append"].doc("Append to existing file?").withDefault<bool>(false))
-                  ? FileMode::append
-                  : FileMode::replace;
+  const bool appending =
+      options["append"].doc("Append to existing file?").withDefault<bool>(false);
+  const bool replacing =
+      options["replace"].doc("Replace existing file?").withDefault<bool>(false);
+  if (appending) {
+    file_mode = FileMode::append;
+  } else if (replacing) {
+    file_mode = FileMode::replace;
+  } else {
+    file_mode = FileMode::newFile;
+  }
 }
 
 void OptionsNetCDF::verifyTimesteps() const {
@@ -778,17 +799,34 @@ void OptionsNetCDF::verifyTimesteps() const {
 
 /// Write options to file
 void OptionsNetCDF::write(const Options& options, const std::string& time_dim) {
-  Timer timer("io");
+  const Timer timer("io");
 
   // Check the file mode to use
   auto ncmode = NcFile::replace;
-  if (file_mode == FileMode::append) {
+  switch (file_mode) {
+  case FileMode::append: {
     // NetCDF doesn't have a "read-write, create if exists" mode, so
     // we need to check ourselves if the file already exists; if it
     // doesn't, tell NetCDF to create it
-    std::ifstream file(filename);
+    const std::ifstream file(filename);
     ncmode = file.good() ? NcFile::FileMode::write : NcFile::FileMode::newFile;
+    break;
   }
+  case FileMode::replace: {
+    // Replacing the file if it exists
+    break;
+  }
+  case FileMode::newFile:
+    // Creating a new file. Don't overwrite existing data.
+    const std::ifstream file(filename);
+    if (file.good()) {
+      throw BoutException(
+          "Output file '{:s}' already exists. Aborting rather than overwriting data.\n"
+          "Set 'append=true' to append, or 'replace=true' to overwrite.",
+          filename);
+    }
+    break;
+  };
 
   if (not data_file) {
     data_file = std::make_unique<netCDF::NcFile>(filename, ncmode);

src/sys/options/options_netcdf.hxx

@@ -63,6 +63,7 @@ public:
 
 private:
   enum class FileMode {
+    newFile, ///< Create a new file. Do not overwrite or append
     replace, ///< Overwrite file when writing
     append   ///< Append to file when writing
   };