CASSSIDECAR-508: Add local job execution and storage support for coordinated cluster-wide operations

[andresbeckruiz]

CASSSIDECAR-508

Updates operational job storage and tracking so that cluster-wide operational jobs can be coordinated in CASSSIDECAR-378. Also adds LocalJob and LocalJobManager, which the local coordinator uses to run individual node jobs that are part of a larger cluster-wide operation.

The local job coordinator (StorageBackedLocalJobCoordinator) that uses these changes is in this CASSSIDECAR-378 stacked PR.

Changes

• OperationalJob and OperationalJobRecord get three new optional fields for cluster-wide operations: target datacenter, node execution order, and operation metadata
• CassandraStorageProvider resolves the local datacenter from the managed instance's node settings
• DurableOperationalJobTracker now adds a CREATED node status row for each node in the execution order when it persists a coordinated job, and marks the job RUNNING. Support for creating jobs without immediate execution will be added in CASSSIDECAR-379
• OperationalJobTracker adds a new method inflightJobs. When durable tracking is used, it merges active operations from storage with in-process jobs, so a coordinated job stays visible in GET /operational-jobs while it runs. ListOperationalJobsHandler now calls it off the event loop
• LocalJob represents a job that runs on a Cassandra node locally managed by Sidecar. LocalJobManager manages the lifecycle of these jobs. These classes parallel OperationalJob and OperationalJobManager

CI: https://app.circleci.com/pipelines/github/andresbeckruiz/cassandra-sidecar/136

Future Work

• Future CEP-53 work:
  • CASSSIDECAR-378: Add support for creation and coordination of local Sidecar jobs
  • CASSSIDECAR-381: Create rolling restart operational job
  • CASSSIDECAR-382: Add POST /restart API and rolling restart handler