Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
148 commits
Select commit Hold shift + click to select a range
95d36e9
chore: bootstrap batch-5 from batch-4
timcadman Apr 15, 2026
d1cf587
refactor: batch-5 matrix client functions
timcadman Apr 15, 2026
1f23991
chore: update dsBase tar with batch-5
timcadman Apr 15, 2026
981ac60
chore: bootstrap batch-6 from batch-5
timcadman Apr 15, 2026
501abec
refactor: batch-6 factor and recoding client functions
timcadman Apr 15, 2026
12f85f7
chore: update dsBase tar with batch-6
timcadman Apr 15, 2026
efe187e
chore: bootstrap batch-7 from batch-6
timcadman Apr 15, 2026
c69f7c5
refactor: batch-7 first-pass modelling client functions
timcadman Apr 15, 2026
e94d3d2
chore: update dsBase tar with batch-7 first pass
timcadman Apr 15, 2026
dd686cd
chore: update dsBase tar with batch-7 second pass
timcadman Apr 15, 2026
11aaaa5
chore: bootstrap batch-8 from batch-7
timcadman Apr 16, 2026
c74ad81
refactor: batch-8 client-side RNG and sampling functions
timcadman Apr 16, 2026
d837b22
chore: update dsBase tar for CI
timcadman Apr 16, 2026
196a343
test: add performance tests for batch 5 functions
timcadman Apr 16, 2026
fda3ead
test: add performance tests for batch 6 functions
timcadman Apr 16, 2026
ee0a2ce
test: add performance tests for batch 7 functions
timcadman Apr 16, 2026
f93ab86
test: add performance tests for batch 8 functions
timcadman Apr 16, 2026
5dff8ad
fix: remove third-party copyright from new tests
timcadman Apr 16, 2026
a904692
fix: remove third-party copyright from new tests
timcadman Apr 16, 2026
8738039
fix: remove third-party copyright from new tests
timcadman Apr 16, 2026
4fce13b
fix: remove third-party copyright from new tests
timcadman Apr 16, 2026
41189a0
fix: escaped dollar signs in perf tests
timcadman Apr 16, 2026
672802a
fix: escaped dollar signs in perf tests
timcadman Apr 16, 2026
6a632e5
fix: correct perf test setup for batch 5 matrix functions
timcadman Apr 16, 2026
37ba40c
fix: correct perf test function calls for batch 6
timcadman Apr 16, 2026
99586cf
fix: correct perf test setup for batch 7 (gamlss/glm still need model…
timcadman Apr 16, 2026
323a4c9
fix: correct perf test setup for glmPredict, glmSummary, gamlss
timcadman Apr 16, 2026
df8ab68
docs: sync REFACTOR_GUIDE.md from batch-9
timcadman Apr 16, 2026
f4cf555
docs: sync REFACTOR_GUIDE.md from batch-9
timcadman Apr 16, 2026
1709640
docs: sync REFACTOR_GUIDE.md from batch-9
timcadman Apr 16, 2026
67f1d35
docs: sync REFACTOR_GUIDE.md from batch-9
timcadman Apr 16, 2026
9632f19
docs: updated authorship
timcadman Apr 16, 2026
330d1a8
docs: redocumented
timcadman Apr 16, 2026
f0d64f6
docs: sync REFACTOR_GUIDE.md from batch-10
timcadman Apr 16, 2026
937d3e1
docs: sync REFACTOR_GUIDE.md from batch-10
timcadman Apr 16, 2026
9d0f8ec
docs: sync REFACTOR_GUIDE.md from batch-10
timcadman Apr 16, 2026
1555327
docs: sync REFACTOR_GUIDE.md from batch-10
timcadman Apr 16, 2026
fdbb0ad
fix: align asFactor and changeRefGroup perf tests with smoke tests
timcadman Apr 19, 2026
4ac9eb9
fix: align perf test datasets with smoke tests
timcadman Apr 19, 2026
735da5a
fix: align perf test datasets with smoke tests
timcadman Apr 19, 2026
1b42aa0
docs: sync REFACTOR_GUIDE.md
timcadman Apr 19, 2026
30528a1
docs: sync REFACTOR_GUIDE.md
timcadman Apr 19, 2026
7c90832
docs: sync REFACTOR_GUIDE.md
timcadman Apr 19, 2026
c404362
docs: sync REFACTOR_GUIDE.md
timcadman Apr 19, 2026
0de07a7
Fix: Armadillo 'hp-laptop-quay' Perf Profile
StuartWheater Jun 17, 2026
6fcda4b
Fix formatting issues in performance profile CSV
StuartWheater Jun 18, 2026
7a3562a
Update performance metrics in armadillo CSV file
StuartWheater Jun 19, 2026
fcbdd9e
Fixed 'armadillo hp-laptop-quay' perf profile
StuartWheater Jun 19, 2026
bc1e6cc
Merge branch 'datashield:v7.0-dev' into v7.0-dev
StuartWheater Jul 14, 2026
b2c00a5
Merge branch 'datashield:v7.0-dev' into v7.0-dev
StuartWheater Jul 16, 2026
d37d2ea
Update docs and man
StuartWheater Jul 20, 2026
a6658e0
Setting 'useFancyQuotes' option to 'FALSE'
StuartWheater Jul 22, 2026
8fea89e
merged in 7.0 dev
timcadman Jul 23, 2026
2b2a481
ci: install R deps as Posit binary packages
timcadman Jul 23, 2026
49820fd
Profile image change and performance update
StuartWheater Jul 26, 2026
b4e55df
try: force CI
timcadman Jul 27, 2026
2f19c68
Update 'docker-compose' analysis images
StuartWheater Jul 27, 2026
fa13e63
Merge branch 'datashield:v7.0-dev' into v7.0-dev
StuartWheater Jul 27, 2026
8ddbfa0
Update profile image
StuartWheater Jul 29, 2026
b7515b1
Update Roxygen and NAMESPACE
StuartWheater Aug 10, 2026
581c9a2
Change profile image to 'datashield/rock_citest-permissive:latest'
StuartWheater Aug 17, 2026
36cf744
Update to roxygen2 8.1.0
StuartWheater Aug 18, 2026
7a06aa4
Updated perf profile
StuartWheater Aug 20, 2026
a3d2c4a
try: update dsBase tar
timcadman Aug 25, 2026
239f81e
Merge pull request #678 from datashield/ci/speedup-dependency-install
timcadman Aug 27, 2026
cd02e06
Merge branch 'datashield:v7.0-dev' into v7.0-dev
StuartWheater Aug 27, 2026
3f8720b
Merge pull request #691 from datashield/refactor/perf-batch-5
timcadman Aug 27, 2026
bac1712
Merge branch 'v7.0-dev' into v7.0-dev
StuartWheater Aug 27, 2026
f66cb78
Merge pull request #74 from datashield/v7.0-dev
StuartWheater Aug 27, 2026
a11c3ac
Merge branch 'v7.0-dev_perf-profile-update' into v7.0-dev
StuartWheater Aug 27, 2026
cefa52b
Switch to armadillo dev
StuartWheater Aug 27, 2026
d681c2f
Fix typo in perf profile
StuartWheater Aug 28, 2026
0be5023
Switch to armadillo back citest:latest - 5.17.2
StuartWheater Aug 28, 2026
8072e0a
Add composite actions for Opal/Armadillo setup with dsBase
timcadman Sep 1, 2026
7e67f64
Add shared PR-comment and JUnit-summary helper scripts
timcadman Sep 1, 2026
c488dfe
Add Check workflow (doc sync, R CMD check)
timcadman Sep 1, 2026
ff95fc4
Add Lint workflow (lintr + SARIF)
timcadman Sep 1, 2026
7e43432
Add sharded Opal/Armadillo test suite with coverage and PR reporting
timcadman Sep 1, 2026
caef9e0
fix missing dependency
timcadman Sep 2, 2026
b6574c4
merged in dev7.0
timcadman Sep 3, 2026
2ebc12b
added performance profiles for local testing
timcadman Sep 3, 2026
6bb2b11
removed now duplicate code
timcadman Sep 3, 2026
ecef498
chore: updated tar for CI
timcadman Sep 3, 2026
bd48415
simplify fetching latest armadillo release
timcadman Sep 3, 2026
bf5cfb5
remove old code left over from Azure
timcadman Sep 3, 2026
78ba192
further simplifying code
timcadman Sep 3, 2026
50515d7
try: build armadillo using docker image
timcadman Sep 3, 2026
7202628
replace curl call with armadillo.install_pkg
timcadman Sep 3, 2026
dd61a75
fixed path error
timcadman Sep 3, 2026
ed9357f
add helper file
timcadman Sep 3, 2026
75fca46
fix path again
timcadman Sep 3, 2026
548de7c
Upgrade DSI to 1.8.0
StuartWheater Sep 4, 2026
574cbf4
Fix typo in perf profile
StuartWheater Sep 4, 2026
a39b0d7
Regenerate version
StuartWheater Sep 4, 2026
5df5428
Extend perf profile for armadillo
StuartWheater Sep 4, 2026
5c6f0cd
Emprovements to perf profile
StuartWheater Sep 4, 2026
ce9492a
Additional profile
StuartWheater Sep 6, 2026
44baf61
Align 'default' and 'opal' pipeline
StuartWheater Sep 6, 2026
98b38e3
Additional profile
StuartWheater Sep 6, 2026
d600e42
Merge pull request #699 from StuartWheater/refactor/perf-batch-6
StuartWheater Sep 8, 2026
5868a46
Merge pull request #696 from datashield/refactor/perf-batch-6
timcadman Sep 8, 2026
1f83c0d
Merge branch 'v7.0-dev' of github.com:datashield/dsBaseClient into re…
timcadman Sep 8, 2026
a7e8867
Update 'tar files'
StuartWheater Sep 8, 2026
de6e6a5
complete batch 7 refactor
timcadman Sep 8, 2026
085eb20
remove stray refactor guide
timcadman Sep 8, 2026
f2f2c73
updated dsBase tar
timcadman Sep 8, 2026
3e3772c
Fixed typo in 'connects...'
StuartWheater Sep 8, 2026
dc2cb6f
test: fix performance test errors
timcadman Sep 8, 2026
cdd2317
fix tests that now rely on serverside messages
timcadman Sep 8, 2026
d979f0c
Update dsBase tars
StuartWheater Sep 9, 2026
3f0fb1c
Merge branch 'refactor/perf-batch-7' into refactor/perf-batch-7
StuartWheater Sep 9, 2026
ab464e6
refactor: make intent of functions clearer
timcadman Sep 10, 2026
d697924
Restructure/reorder of importFrom
StuartWheater Sep 10, 2026
ca8b512
Add performance data for new functions in CSV
StuartWheater Sep 10, 2026
bdbceed
Add performance metrics for new functions
StuartWheater Sep 10, 2026
c214ce1
Merge pull request #701 from StuartWheater/refactor/perf-batch-7
StuartWheater Sep 11, 2026
fb91f17
Merge branch 'v7.0-dev' into v7.0-dev
StuartWheater Sep 11, 2026
e163f1e
run on push to PR, master or dev branch
timcadman Sep 14, 2026
10e39db
Merge pull request #700 from datashield/refactor/perf-batch-7
timcadman Sep 14, 2026
4332c37
make dsBase version settable, build Armadillo tar in CI
timcadman Sep 14, 2026
38256f5
Merge branch 'v7.0-dev' of github.com:datashield/dsBaseClient into re…
timcadman Sep 14, 2026
5bd5a31
fixed tests and docs
timcadman Sep 14, 2026
fe815ab
pin version of armadillo
timcadman Sep 14, 2026
84c4018
bump versions to stop warnings
timcadman Sep 14, 2026
94a3046
docs: correct ds.rNorm vector example comment
timcadman Sep 14, 2026
9937412
Merge pull request #698 from datashield/chore/githib-actions-ci
timcadman Sep 14, 2026
3e86521
fix: match docs to allow different values to be sent per study
timcadman Sep 14, 2026
83143b4
Merge branch 'v7.0-dev' of github.com:datashield/dsBaseClient into re…
timcadman Sep 14, 2026
c2539e2
fix lint errors
timcadman Sep 14, 2026
461af37
ci: show full failure text in test summary
timcadman Sep 14, 2026
52965ed
separate docker compose for azure and gitaction, fix lint, show full …
timcadman Sep 14, 2026
7b749b0
simplify CI by using one docker compose
timcadman Sep 14, 2026
46b5c9e
removed now undeeded file
timcadman Sep 14, 2026
bfaeac8
display correct code coverage
timcadman Sep 14, 2026
d187106
fix codecov summary line, scroll to correct place for test failures
timcadman Sep 14, 2026
5b6e58a
try again to fix coverage report
timcadman Sep 14, 2026
92099fc
try: show failure in individual shard when test fails
timcadman Sep 15, 2026
bb78bc0
Update of 'ds.Boole' perf results
StuartWheater Sep 15, 2026
facf882
show better info on test failure
timcadman Sep 15, 2026
36e9a55
try: break test to check reporting mechanism
timcadman Sep 15, 2026
eaa0364
fix path failure
timcadman Sep 15, 2026
df60665
Guard against Codecov returning a null patch object
timcadman Sep 15, 2026
6b2f213
Merge pull request #705 from StuartWheater/refactor/perf-batch-8
StuartWheater Sep 15, 2026
23c0d4a
revert temp test fail, clean up messaging
timcadman Sep 15, 2026
e3221f5
Merge pull request #703 from datashield/chore/fix-ci
timcadman Sep 15, 2026
a7dc35f
Merge branch 'v7.0-dev' of github.com:datashield/dsBaseClient into re…
timcadman Sep 15, 2026
7cecae2
Merge branch 'refactor/perf-batch-8' of github.com:datashield/dsBaseC…
timcadman Sep 15, 2026
943fe26
Merge pull request #702 from datashield/refactor/perf-batch-8
timcadman Sep 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .Rbuildignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@
^\.circleci$
^\.circleci/config\.yml$
^\.github$
^REFACTOR_GUIDE\.md$
^man-roxygen$
^codecov\.yml$
118 changes: 118 additions & 0 deletions .github/actions/setup-armadillo-with-dsbase/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
name: Setup Armadillo with dsBase
description: >
Starts a fresh Armadillo instance via docker-compose, with the "default"
Rock profile as a static sibling container. Installs R and system
dependencies, waits for readiness, uploads test datasets, and installs
dsBase. Assumes the caller has already checked out dsBaseClient at
./dsBaseClient. Shared by the Armadillo dsbase shard workflow and the
Armadillo dsdanger job, so this setup only needs fixing in one place.

inputs:
dsbase-ref:
description: dsBase GitHub ref to build and install
required: true

runs:
using: composite
steps:
- name: Start Armadillo docker-compose
shell: bash
run: docker compose -f docker-compose_armadillo.yml up -d
working-directory: dsBaseClient

- uses: ./dsBaseClient/.github/actions/setup-r-with-dsbase-deps
with:
additional-packages: cran::covr

- name: Wait for Armadillo to be ready
shell: bash
run: |
for i in $(seq 1 60); do
if curl -fsS http://localhost:8080/actuator/health 2>/dev/null | grep -q '"status":"UP"'; then
echo "Armadillo is ready"
exit 0
fi
sleep 5
done
echo "Armadillo did not become ready in time"
docker compose -f docker-compose_armadillo.yml logs armadillo || true
exit 1
working-directory: dsBaseClient

- name: Install test datasets to Armadillo
shell: bash
run: R -q -f "molgenis_armadillo-upload_testing_datasets.R"
working-directory: dsBaseClient/tests/testthat/data_files

- name: Check out dsBase
uses: actions/checkout@v5
with:
repository: datashield/dsBase
ref: ${{ inputs.dsbase-ref }}
path: dsBase

- name: Build dsBase tarball
id: build-dsbase
shell: bash
run: |
R CMD build dsBase
echo "tarball=$GITHUB_WORKSPACE/$(ls dsBase_*.tar.gz)" >> "$GITHUB_OUTPUT"

- name: Install dsBase to Armadillo
shell: bash
run: |
installed=false
for i in $(seq 1 30); do
if Rscript -e "
library(MolgenisArmadillo)
armadillo.login_basic('http://localhost:8080', 'admin', 'admin')
armadillo.install_packages('${{ steps.build-dsbase.outputs.tarball }}', profile = 'default')
"; then
installed=true
break
fi
sleep 10
done
if [ "$installed" != "true" ]; then
echo "dsBase install via armadillo.install_packages() failed (default profile may not have started in time)"
exit 1
fi
working-directory: dsBaseClient

- name: Verify installed dsBase version
shell: bash
run: |
expected_version=$(sed -n 's/^Version: //p' "$GITHUB_WORKSPACE/dsBase/DESCRIPTION" | tr -d '\r')
if [ -z "$expected_version" ]; then
echo "Could not determine expected dsBase version from dsBase@${{ inputs.dsbase-ref }}"
exit 1
fi
echo "Expected dsBase version: $expected_version"

actual_version=""
for i in $(seq 1 30); do
packages_json=$(curl -sf -u admin:admin -X GET http://localhost:8080/packages || true)
actual_version=$(echo "$packages_json" | jq -r '.[] | select(.name == "dsBase") | .version' 2>/dev/null || true)
if [ "$actual_version" == "$expected_version" ]; then
break
fi
sleep 10
done
echo "Installed dsBase version: $actual_version"
if [ "$actual_version" != "$expected_version" ]; then
echo "dsBase version mismatch: expected $expected_version, found '$actual_version'"
exit 1
fi
echo "$actual_version" > dsbase_version.txt
working-directory: dsBaseClient

- name: Whitelist dsBase
shell: bash
run: curl -u admin:admin -X POST http://localhost:8080/whitelist/dsBase
working-directory: dsBaseClient

- name: Dump Armadillo server log
if: failure()
shell: bash
run: docker compose -f docker-compose_armadillo.yml logs armadillo | grep "Caused by:" | sort -u || true
working-directory: dsBaseClient
82 changes: 82 additions & 0 deletions .github/actions/setup-opal-with-dsbase/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
name: Setup Opal with dsBase
description: >
Starts a fresh Opal instance via docker-compose, installs R and system
dependencies, uploads test datasets, and installs dsBase with disclosure
test options. Assumes the caller has already checked out dsBaseClient at
./dsBaseClient. Shared by the Opal dsbase shard workflow and the Opal
dsdanger job, so this setup only needs fixing in one place.

inputs:
dsbase-ref:
description: dsBase GitHub ref to install
required: true

runs:
using: composite
steps:
- name: Start Opal docker-compose
shell: bash
run: docker compose -f docker-compose_opal.yml up -d --build
working-directory: dsBaseClient

- uses: ./dsBaseClient/.github/actions/setup-r-with-dsbase-deps

- name: Install test datasets to Opal
shell: bash
run: |
for i in $(seq 1 18); do
if curl -sf -o /dev/null http://localhost:8080/; then
echo "Opal is responding"
break
fi
sleep 5
done
R -q -f "obiba_opal-upload_testing_datasets.R"
working-directory: dsBaseClient/tests/testthat/data_files

- name: Install dsBase to Opal
shell: bash
run: |
R -q -e "library(opalr); opal <- opal.login(username = 'administrator', password = 'datashield_test&', url = 'http://localhost:8080/'); opal.put(opal, 'system', 'conf', 'general', '_rPackage'); opal.logout(opal)"
R -q -e "library(opalr); opal <- opal.login('administrator','datashield_test&', url='http://localhost:8080/'); dsadmin.install_github_package(opal, 'dsBase', username = 'datashield', ref = '${{ inputs.dsbase-ref }}'); opal.logout(opal)"
working-directory: dsBaseClient/tests/testthat/data_files

- name: Verify installed dsBase version
shell: bash
run: |
expected_version=$(curl -sf "https://raw.githubusercontent.com/datashield/dsBase/${{ inputs.dsbase-ref }}/DESCRIPTION" | sed -n 's/^Version: //p' | tr -d '\r')
if [ -z "$expected_version" ]; then
echo "Could not determine expected dsBase version from GitHub"
exit 1
fi
echo "Expected dsBase version: $expected_version"
echo "$expected_version" > dsbase_version.txt

ok=false
for i in $(seq 1 18); do
if Rscript -e "
library(opalr)
opal <- opal.login('administrator', 'datashield_test&', url = 'http://localhost:8080/')
desc <- dsadmin.package_description(opal, 'dsBase')
opal.logout(opal)
installed <- desc[['Version']]
cat('Installed dsBase version:', if (is.null(installed)) '(none yet)' else installed, '\n')
quit(status = if (!is.null(installed) && installed == '$expected_version') 0 else 1)
"; then
ok=true
break
fi
sleep 10
done
if [ "$ok" != "true" ]; then
echo "dsBase version mismatch or install not detected after polling: expected $expected_version"
exit 1
fi
working-directory: dsBaseClient/tests/testthat/data_files

- name: Initialize default profile and set disclosure test options
shell: bash
run: |
R -q -e "library(opalr); opal <- opal.login('administrator', 'datashield_test&', url='http://localhost:8080/'); dsadmin.profile_init(opal, name = 'default', packages = c('dsBase', 'dsTidyverse', 'resourcer')); opal.logout(opal)"
R -q -e "library(opalr); opal <- opal.login('administrator', 'datashield_test&', url='http://localhost:8080/'); dsadmin.set_option(opal, 'default.datashield.privacyControlLevel', 'permissive'); opal.logout(opal)"
working-directory: dsBaseClient/tests/testthat/data_files
61 changes: 61 additions & 0 deletions .github/actions/setup-r-with-dsbase-deps/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Setup R with dsBaseClient dependencies
description: >
Installs the system libraries, R, and R package dependencies shared by the
Opal and Armadillo setup actions. Assumes the caller has already checked
out dsBaseClient at ./dsBaseClient.

inputs:
additional-packages:
description: Extra pak package specs (one per line) to install beyond the common set
required: false
default: ''

runs:
using: composite
steps:
- name: Install system libraries
shell: bash
env:
SYSTEM_LIBRARIES: >-
libxml2-dev libcurl4-openssl-dev libssl-dev libgsl-dev libgit2-dev
libharfbuzz-dev libfribidi-dev libmagick++-dev libudunits2-dev libuv1-dev
run: |
sudo apt-get update -qq
sudo apt-get install -qq $SYSTEM_LIBRARIES -y

- uses: r-lib/actions/setup-r@v2
with:
r-version: release
http-user-agent: release
use-public-rspm: true

- uses: r-lib/actions/setup-r-dependencies@v2
env:
# RSPM doesn't always have a binary for every package/R-version combo; when it
# falls back to source, pak needs LinkingTo deps present to compile it.
PKG_INCLUDE_LINKINGTO: true
R_PACKAGES: |
cran::devtools
cran::fields
cran::meta
cran::metafor
cran::ggplot2
cran::gridExtra
cran::data.table
cran::DSI
cran::opalr
cran::DSOpal
cran::DSLite
cran::MolgenisAuth
cran::MolgenisArmadillo
cran::DSMolgenisArmadillo
cran::DescTools
cran::e1071
github::datashield/dsDangerClient
with:
working-directory: dsBaseClient
dependencies: 'c("Depends", "Imports", "LinkingTo")'
extra-packages: |
${{ env.R_PACKAGES }}
${{ inputs.additional-packages }}
needs: check
Loading
Loading