Skip to content

CSTACKEX-207: Automation for snapshot and revert for VM and CS-volume #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sathvikaragi wants to merge 4 commits into
base: main
Choose a base branch
from
Open

CSTACKEX-207: Automation for snapshot and revert for VM and CS-volume #122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
823c7a6
CSTACKEX-207: Automation for snapshot and revert for VM and CS-volume
sathvikaragi Oct 14, 2026
095c1d5
CSTACKEX-207: resolving comments
sathvikaragi Oct 20, 2026
1af6ca0
CSTACKEX-207: build failure
sathvikaragi Oct 20, 2026
7d9ff17
CSTACKEX-207: fixing build failure
sathvikaragi Oct 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion private-cicd/benchmark/ontap/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Scale matrix: [ONTAP Plugin - CloudStack Operations, Scale & Parallel Test Matri
Test ids written to the CSVs and report map to the page sections as follows:

| Test id | Page section | What it measures |
|---|---|---|
| --- | --- | --- |
| `pool_sequential_create` | 5.1.1 | Sequential create (1 pool at a time, cumulative to N) |
| `pool_sequential_delete` | 5.1.2 | Sequential delete (1 pool at a time, from N remaining) |
| `pool_concurrent_create` | 6.1.1 | Parallel pool creation |
Expand Down
20 changes: 10 additions & 10 deletions private-cicd/benchmark/ontap/USAGE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ this file is deliberately just a lookup table / cheat sheet.
## 1. Prerequisites

| # | Requirement | Notes |
|---|---|---|
| --- | --- | --- |
| 1 | Python 3.9+ | No other version-specific features used; anything 3.9-3.12 should work. |
| 2 | Network access to a CloudStack management server's API (`/client/api`) | Admin credentials with permission to create/delete storage pools, service/disk offerings, and deploy/destroy VMs. |
| 3 | Network access (HTTPS/443) from **your machine** to the ONTAP cluster's management LIF | Only needed if you run the storage-pool scripts' `--cleanup-only` helper functions that talk to ONTAP directly for orphan checks; the benchmark scripts themselves only ever call the CloudStack API, never ONTAP directly. |
Expand Down Expand Up @@ -70,7 +70,7 @@ cluster IDs, and ONTAP connection details (see inline comments in
## 2. Shared library modules (not run directly)

| File | Purpose |
|---|---|
| --- | --- |
| `benchmark_support.py` | Shared run ids, resource names, `mean_seconds`, and the one log format used by every script. |
| `cloudstack_client.py` | Minimal CloudStack HTTP/REST client (session-key login, generic `call()`, automatic `queryAsyncJobResult` polling). Used by every benchmark script. |
| `storage_pool_common.py` | Shared helpers for the storage-pool scripts: `createStoragePool`/`deleteStoragePool` request building, CSV logging (`RawLogger`, `append_summary_csv`), cleanup-by-filter, config loading. |
Expand All @@ -82,15 +82,15 @@ Reproduces Confluence sections 5.1 (sequential) and 6.1 (concurrency) -
`createStoragePool` / `deleteStoragePool` timing at scale.

| Script | What it does | Typical command |
|---|---|---|
| --- | --- | --- |
| `run.py` | Entry point. Loads `config.yaml` and runs the test named by `--test`. | `python3 run.py --test storage-pool-sequential --config config.yaml` |
| `benchmark_storage_pool_sequential.py` | Creates storage pools one at a time (per protocol) up to N=30, then deletes them one at a time, logging every call and reporting checkpoint totals at N = 1, 5, 10, 20, 30. | `python3 run.py --test storage-pool-sequential --config config.yaml` |
| `benchmark_storage_pool_concurrency.py` | For each concurrency level C (default 2, 5, 10, 20, 30), creates C pools in parallel via a thread pool, records wall-clock time for the whole batch, then deletes the same C pools in parallel. | `python3 run.py --test storage-pool-concurrency --config config.yaml` |

Common flags (both scripts):

| Flag | Meaning |
|---|---|
| --- | --- |
| `--config PATH` | Config YAML to use (default `config.yaml`). Protocols, checkpoints, and concurrency levels are read from here. |
| `--run-id ID` | Reuse a run id. Auto-generated otherwise as `RUN_<UTC timestamp>_<4 hex chars>`. Letters, digits, and underscores only — a hyphen is rewritten to `_` because storage-pool names (and the ONTAP volume behind them) disallow `-`. Use the **same id** across sequential and concurrency to merge one report. |
| `--dry-run` | Simulate timings with no real API calls - use this first to sanity-check your config. |
Expand All @@ -100,7 +100,7 @@ Common flags (both scripts):
`run.py` also requires `--test`. Protocols, checkpoints, and concurrency levels are not flags; change them in `config.yaml`.

| Flag | Meaning |
|---|---|
| --- | --- |
| `--test NAME` | Required. `storage-pool-sequential`, `storage-pool-concurrency`, `vm-instance-sequential`, `vm-instance-concurrency`, or `vm-instance-combined`. |

## 4. VM-instance benchmark scripts
Expand All @@ -126,7 +126,7 @@ VM getting both a root disk and a data disk on the same tagged pool.
### Scripts

| Script | What it does | Typical command |
|---|---|---|
| --- | --- | --- |
| `benchmark_vm_instance_sequential.py` | Deploys VMs one at a time (per protocol) up to N=30, then destroys them one at a time, reporting checkpoint totals at N = 1, 2, 5, 10, 20, 30. | `python3 run.py --test vm-instance-sequential --config config.yaml` |
| `benchmark_vm_instance_concurrency.py` | For each concurrency level C, deploys C VMs in parallel, records wall-clock time, then destroys the same C VMs in parallel. | `python3 run.py --test vm-instance-concurrency --config config.yaml` |
| `benchmark_vm_instance_combined.py` | Runs the sequential matrix immediately followed by the concurrency matrix, under one shared run id, in a single invocation. Use once you already trust your config/environment. | `python3 run.py --test vm-instance-combined --config config.yaml` |
Expand All @@ -143,7 +143,7 @@ with `--cleanup-only` once done.
## 5. Report rendering

| Script | What it does | Typical command |
|---|---|---|
| --- | --- | --- |
| `render_report.py` | Turns a run's `raw_ops_<run_id>.csv` + `summary_<run_id>.csv` into markdown tables (section layout plus Min/Avg/P95/Max rows for the Results Log). Writes to stdout and to `results/report_<run_id>.md`. | `python3 render_report.py --run-id RUN_20260927_081500_a1b2 --cloudstack-build 4.23.0.0-SNAPSHOT --ontap-version 9.17.1` |

For VM-instance runs (which write `raw_ops_vm_<run_id>.csv` /
Expand All @@ -156,7 +156,7 @@ python3 render_report.py --run-id RUN_20260927_081500_a1b2 \
```

| Flag | Meaning |
|---|---|
| --- | --- |
| `--run-id ID` | **Required.** The run id used by the benchmark script(s). |
| `--output-dir DIR` | Where the CSVs live / the report gets written (default `results`). |
| `--cloudstack-build STR` | Label only, stamped into the Results Log rows. |
Expand All @@ -169,7 +169,7 @@ python3 render_report.py --run-id RUN_20260927_081500_a1b2 \
## 6. Config files at a glance

| File | Committed? | Purpose |
|---|---|---|
| --- | --- | --- |
| `config.example.yaml` | Yes | Sanitized template - copy to `config.yaml` and fill in your lab's real values. |
| `config.yaml` | **No** (gitignored) | Your real, lab-specific config (real IPs/credentials) - never commit this. |
| `config_sanity.yaml` | **No** (gitignored) | Optional scratch config some contributors keep locally for quick one-off dry-runs; not part of the checked-in tooling. |
Expand All @@ -180,7 +180,7 @@ All written under `results/` (gitignored - these are run artifacts, not
source):

| Pattern | Produced by | Contents |
|---|---|---|
| --- | --- | --- |
| `raw_ops_<run_id>.csv` | storage-pool scripts | One row per individual API call (timestamps, duration, success/failure, error). |
| `raw_ops_vm_<run_id>.csv` | VM-instance scripts | Same, for VM deploy/destroy calls. |
| `summary_<run_id>.csv` / `summary_vm_<run_id>.csv` | both | One row per checkpoint/concurrency level (totals, averages, success/failure counts). |
Expand Down
48 changes: 42 additions & 6 deletions test/integration/plugins/ontap/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,10 @@ for the deployment, health-gate, and artifact contracts.
test/integration/plugins/ontap/
├── ontap.cfg # Environment config (IPs, credentials, zone info)
├── ontap_test_base.py # Shared base class and ONTAP REST client
├── TEST_CASES.md # Full test case reference table (119 tests)
├── guest_ssh.py # Guest SSH (paramiko) for the VM workflow suites
├── setup_env.sh # venv: Marvin plus requirements.txt
├── requirements.txt # setuptools, pyvmomi, paramiko pins
├── TEST_CASES.md # Full test case reference table (151 tests)
├── README.md # This file
│
├── nfs3/
Expand All @@ -54,7 +57,7 @@ test/integration/plugins/ontap/
│ ├── volume/
│ │ └── test_volume_lifecycle.py # Volume create/delete/negative-delete
│ ├── instance/
│ │ └── test_vm_volume_attach.py # Pool + volume + VM + attach/resize/detach + template cache
│ │ └── test_vm_volume_attach.py # Pool + volume + VM + resize, snapshots, attach/detach + template cache
│ └── template/
│ └── test_template_cache_negative.py # Template-cache boundary cases
│
Expand All @@ -66,7 +69,7 @@ test/integration/plugins/ontap/
├── volume/
│ └── test_volume_lifecycle.py # LUN create/delete/negative-delete
├── instance/
│ └── test_vm_volume_attach.py # Pool + LUN + VM + attach/resize/LUN-map lifecycle + template cache
│ └── test_vm_volume_attach.py # Pool + LUN + VM + resize, snapshots, LUN-map lifecycle + template cache
└── template/
└── test_template_cache_negative.py # Template-cache boundary cases
```
Expand Down Expand Up @@ -122,6 +125,38 @@ python3 -m pip install --user \
python3 -c "import marvin; print('Marvin OK')"
```

The VM workflow suites also need a venv with pinned dependencies. From the repo root:

```bash
bash test/integration/plugins/ontap/setup_env.sh
```

The script is safe to re-run. It creates `test/integration/plugins/ontap/.venv` when that directory is missing, then installs into it:

- Marvin (from `MARVIN_ARCHIVE` when set, otherwise from `tools/marvin` after generating `commands.xml`)
- the pins in `requirements.txt`: `setuptools>=40.3.0,<81`, `pyvmomi==8.0.2.0.1`, and `paramiko>=3.4,<4`

`run_tests.sh` uses that venv automatically.

#### Paramiko (guest SSH)

`nfs3/instance/test_vm_volume_attach.py` and `iscsi/instance/test_vm_volume_attach.py` log into the guest through `guest_ssh.py`. The lab template is CentOS 5.5, and its sshd only offers SHA-1 key exchange (`diffie-hellman-group14-sha1`) and `ssh-rsa` host keys. Paramiko 4.x removed SHA-1, so those logins fail unless the venv has Paramiko 3.x.

`setup_env.sh` installs the pin from `requirements.txt`:

```bash
test/integration/plugins/ontap/.venv/bin/pip install -r test/integration/plugins/ontap/requirements.txt
```

`guest_ssh.check_paramiko()` checks for `diffie-hellman-group14-sha1` before the first guest connection and tells you to re-run `setup_env.sh` when the active interpreter has Paramiko 4. Guest login itself comes from `ontap.cfg` (`cloudstack.templateUsername` / `cloudstack.templatePassword`).

Confirm the venv before a VM workflow run:

```bash
test/integration/plugins/ontap/.venv/bin/python -c \
"import paramiko; print(paramiko.__version__); assert 'diffie-hellman-group14-sha1' in paramiko.Transport._kex_info"
```

---

## Configuration — `ontap.cfg`
Expand Down Expand Up @@ -347,13 +382,13 @@ another ONTAP pool is present.
| NFS3 Pool with Volumes | `nfs3/pool/test_pool_with_volumes.py` | 12 | Lifecycle with a live volume, deletion with a pre-deleted FlexVol or export policy, and host cases for an inactive libvirt pool, a read-only NFS mount, and a deleted mount point |
| NFS3 Zone-Scoped Pool | `nfs3/pool/test_zone_scoped_pool.py` | 5 | Zone-scoped pool create, grow and shrink with CloudStack/ONTAP/KVM verification, disable, enable, and delete |
| NFS3 Volume Lifecycle | `nfs3/volume/test_volume_lifecycle.py` | 5 | Volume is metadata-only; FlexVol unchanged on delete |
| NFS3 VM + Volume Attach | `nfs3/instance/test_vm_volume_attach.py` | 13 | Full VM lifecycle with stopped-VM qcow2 grow; ROOT on tagged pool seeds/reuses template cache, which survives VM delete |
| NFS3 VM + Volume Attach | `nfs3/instance/test_vm_volume_attach.py` | 29 | Full VM lifecycle with stopped-VM qcow2 grow, guest SSH, volume and VM snapshots; ROOT on tagged pool seeds/reuses template cache, which survives VM delete |
| NFS3 Template Cache Negative | `nfs3/template/test_template_cache_negative.py` | 3 | Tag mismatch; undersized pool; out-of-band cache delete |
| iSCSI Pool Lifecycle | `iscsi/pool/test_pool_lifecycle.py` | 21 | Create rejects for aggregate space, invalid capacitybytes, and a duplicate FlexVol name; grow to 300 TiB, reject grow past 300 TiB, safe shrink, disable, resize while disabled, enable, maintenance, grow and shrink in maintenance, delete + igroups, then re-create with a volume (LUN), reject shrink below used capacity, deploy a VM and attach the volume, cleanup, plus empty-pool deletion with a pre-deleted FlexVol or igroups |
| iSCSI Pool with Volumes | `iscsi/pool/test_pool_with_volumes.py` | 13 | Lifecycle with a live LUN, deletion with a pre-deleted FlexVol or igroups, maintenance with pre-deleted LUN maps, and host cases for a logged-out iSCSI session, an existing session during volume delete, and a replaced by-path symlink |
| iSCSI Zone-Scoped Pool | `iscsi/pool/test_zone_scoped_pool.py` | 5 | Zone-scoped pool create, grow and shrink with CloudStack/ONTAP verification, disable, enable, and delete |
| iSCSI Volume Lifecycle | `iscsi/volume/test_volume_lifecycle.py` | 5 | LUN created per CS volume; LUN removed on delete |
| iSCSI VM + Volume Attach | `iscsi/instance/test_vm_volume_attach.py` | 13 | Full VM lifecycle; stopped-VM LUN grow and LUN-map checks; ROOT on tagged pool seeds/reuses `cs_tmpl_*` LUN cache |
| iSCSI VM + Volume Attach | `iscsi/instance/test_vm_volume_attach.py` | 29 | Full VM lifecycle; stopped-VM LUN grow, LUN-map checks, guest SSH, volume and VM snapshots; ROOT on tagged pool seeds/reuses `cs_tmpl_*` LUN cache |
| iSCSI Template Cache Negative | `iscsi/template/test_template_cache_negative.py` | 3 | Tag mismatch; undersized pool; out-of-band cache delete |

For the goal, dependencies, and exact success criteria of every individual test, see [TEST_CASES.md](TEST_CASES.md).
Expand All @@ -364,12 +399,13 @@ For the goal, dependencies, and exact success criteria of every individual test,

| Symptom | Likely cause | Fix |
| --------- | ------------- | ----- |
| `paramiko … has no SHA-1 key exchange` / guest SSH fails on CentOS 5.5 | Paramiko 4.x is installed; the guest sshd only offers SHA-1 | Re-run `bash test/integration/plugins/ontap/setup_env.sh` and run tests with `test/integration/plugins/ontap/.venv` (`paramiko>=3.4,<4`) |
| `ModuleNotFoundError: No module named 'ontap_test_base'` | Missing `PYTHONPATH` prefix | Prefix every run with `PYTHONPATH=test/integration/plugins/ontap` |
| `Marvin Init Failed` | CloudStack API unreachable | Check `mgtSvrIp:8096` is reachable; restart `cloudstack-management` |
| `Lost connection to MySQL` | MySQL not accepting remote connections | Enable remote MySQL access (see Prerequisites §3) |
| `sh: python: command not found` (repeated) | Marvin internal call — harmless on macOS | Ignore; Marvin Init still succeeds |
| Pool state never reaches `Maintenance` | KVM agent not responding | Check `cloudstack-agent` on KVM host; verify host is connected in CloudStack UI |
| iSCSI `test_09` error 530 | KVM guest does not ACK SCSI hot-unplug | Known environment limitation — see TEST_CASES.md Suite 10 note |
| iSCSI `test_24` error 530 | KVM guest does not ACK SCSI hot-unplug | Detach of the data LUN from a running VM; see TEST_CASES.md Suite 10 `test_24_detach_volume_from_vm` |
| ONTAP REST `401 Unauthorized` | Wrong credentials in `ontap.cfg` | Verify `username`/`password` under `ontap` section |
| `ISCSI protocol is not enabled on SVM` / many pool tests EXCEPTION then FAIL | SVM iSCSI service off or no `data_iscsi` LIF | Enable iSCSI on the SVM and assign `default-data-iscsi` (or `default-data-blocks`) to at least one data LIF; `run_tests.sh` now fails this check before Marvin starts |
| `No ready KVM user template available` | Template still downloading | Re-run `setup_zone` (step 12 waits for template readiness); or wait in CloudStack UI |
Expand Down
Loading
Loading