Skip to content

Move install to separate step and give more leeway to benchmarks #712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hansott merged 1 commit into from
Sep 4, 2026
Merged

Move install to separate step and give more leeway to benchmarks #712

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,12 @@ jobs:
run: |
python -m pip install --upgrade pip
make install && make build
- name: Install flask-mysql
working-directory: ./sample-apps/flask-mysql
run: make install
- name: Start flask-mysql
working-directory: ./sample-apps/flask-mysql
run: nohup make runBenchmark & nohup make runZenDisabled &
run: nohup make startBenchmark & nohup make startZenDisabled &

- name: Check app health (max 3 attempts)
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
Expand Down Expand Up @@ -113,9 +116,12 @@ jobs:
run: |
python -m pip install --upgrade pip
make install && make build
- name: Install app
working-directory: ./sample-apps/${{ matrix.app }}
run: make install
- name: Start app
working-directory: ./sample-apps/${{ matrix.app }}
run: nohup make runBenchmark & nohup make runZenDisabled &
run: nohup make startBenchmark & nohup make startZenDisabled &

- name: Check app health (max 3 attempts)
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
Expand Down
16 changes: 8 additions & 8 deletions benchmarks/flask-mysql-benchmarks.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,42 +10,42 @@ export const options = {
vus: 1, // Number of virtual users
thresholds: {
test_40mb_payload: [{
threshold: "avg<15", // This is a higher threshold due to the data being processed
threshold: "avg<25", // This is a higher threshold due to the data being processed
abortOnFail: true,
delayAbortEval: '10s',
}],
test_multiple_queries: [{
threshold: "avg<6",
threshold: "avg<10",
abortOnFail: true,
delayAbortEval: '10s',
}],
test_multiple_queries_with_big_body: [{
threshold: "avg<6",
threshold: "avg<10",
abortOnFail: true,
delayAbortEval: '10s',
}],
test_create_with_big_body: [{
threshold: "avg<6",
threshold: "avg<10",
abortOnFail: true,
delayAbortEval: '10s',
}],
test_normal_route: [{
threshold: "avg<6",
threshold: "avg<10",
abortOnFail: true,
delayAbortEval: '10s',
}],
test_id_route: [{
threshold: "avg<6",
threshold: "avg<10",
abortOnFail: true,
delayAbortEval: '10s',
}],
test_open_file: [{
threshold: "avg<6",
threshold: "avg<10",
abortOnFail: true,
delayAbortEval: '10s',
}],
test_execute_shell: [{
threshold: "avg<10",
threshold: "avg<15",
abortOnFail: true,
delayAbortEval: '10s',
}],
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/wrk_benchmark/flask_mysql.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
"http://localhost:8086/create",
"http://localhost:8087/create",
"",
percentage_limit=40
percentage_limit=50
)
4 changes: 2 additions & 2 deletions benchmarks/wrk_benchmark/flask_mysql_uwsgi.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@
"http://localhost:8088/benchmark",
"http://localhost:8089/benchmark",
"a non empty route which makes a simulated request to a database",
percentage_limit=40
percentage_limit=50
)


run_benchmark(
"http://localhost:8088/benchmark_io",
"http://localhost:8089/benchmark_io",
"a route that makes multiple I/O calls",
percentage_limit=35
percentage_limit=45
)
2 changes: 1 addition & 1 deletion benchmarks/wrk_benchmark/starlette_postgres_uvicorn.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
"http://localhost:8102/benchmark",
"http://localhost:8103/benchmark",
"a non empty route which makes a simulated request to a database",
percentage_limit=40
percentage_limit=50
)
4 changes: 1 addition & 3 deletions sample-apps/common.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,7 @@ AIKIDO_ENV_DISABLED = \
# Common target definitions
.PHONY: install
install:
# ignore poetry install failures, these can be due to locking issues when
# 2 poetry installs are run at the same time.
poetry install || true;
poetry install

@aikido-pr-checks aikido-pr-checks Bot Sep 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium - Removing the install failure suppression re-breaks concurrent sample-app startup in end-to-end CI

The shared install target now returns the raw poetry install exit status even though the repository still starts two targets from the same sample-app directory in parallel during end-to-end tests. In .github/workflows/end2end.yml, make runZenDisabled is backgrounded and then make run is started shortly after; both targets depend on install, so they can hit the same Poetry lock/contention that the removed || true was explicitly masking. When one of those installs exits non-zero, the corresponding app process never comes up and the matrix job fails or flakes across many sample apps.

Suggested change
poetry install
poetry install || true

More info - Reply on this comment to give feedback or ignore the issue.


.PHONY: health-check
health-check:
Expand Down
10 changes: 6 additions & 4 deletions sample-apps/flask-mysql-uwsgi/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,16 @@ run: install
$(AIKIDO_ENV_COMMON) \
poetry run uwsgi --ini uwsgi.ini

.PHONY: runBenchmark
runBenchmark: install
.PHONY: runBenchmark startBenchmark
runBenchmark: install startBenchmark
startBenchmark:
@echo "Running sample app flask-mysql-uwsgi with Zen on port $(PORT)"
$(AIKIDO_ENV_BENCHMARK) \
poetry run uwsgi --single-interpreter --ini uwsgi.ini

.PHONY: runZenDisabled
runZenDisabled: install
.PHONY: runZenDisabled startZenDisabled
runZenDisabled: install startZenDisabled
startZenDisabled:
@echo "Running sample app flask-mysql-uwsgi without Zen on port $(PORT_DISABLED)"
$(AIKIDO_ENV_DISABLED) \
poetry run uwsgi --single-interpreter --ini uwsgi2.ini
10 changes: 6 additions & 4 deletions sample-apps/flask-mysql/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,16 @@ run: install
$(AIKIDO_ENV_COMMON) \
poetry run flask --app app.py run --host=0.0.0.0 --port=$(PORT) --no-reload

.PHONY: runBenchmark
runBenchmark: install
.PHONY: runBenchmark startBenchmark
runBenchmark: install startBenchmark
startBenchmark:
@echo "Running sample app flask-mysql with Zen (benchmark mode) on port $(PORT)"
$(AIKIDO_ENV_BENCHMARK) \
poetry run flask --app app.py run --host=0.0.0.0 --port=$(PORT) --no-reload

.PHONY: runZenDisabled
runZenDisabled: install
.PHONY: runZenDisabled startZenDisabled
runZenDisabled: install startZenDisabled
startZenDisabled:
@echo "Running sample app flask-mysql without Zen on port $(PORT_DISABLED)"
$(AIKIDO_ENV_DISABLED) \
poetry run flask --app app.py run --host=0.0.0.0 --port=$(PORT_DISABLED) --no-reload
10 changes: 6 additions & 4 deletions sample-apps/starlette-postgres-uvicorn/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,14 @@ run: install
@echo "Running sample app starlette-postgres-uvicorn with Zen on port $(PORT)"
$(AIKIDO_ENV_COMMON) poetry run uvicorn app:app --host 0.0.0.0 --port $(PORT) --workers 4

.PHONY: runBenchmark
runBenchmark: install
.PHONY: runBenchmark startBenchmark
runBenchmark: install startBenchmark
startBenchmark:
@echo "Running sample app starlette-postgres-uvicorn with Zen (benchmark mode) on port $(PORT)"
$(AIKIDO_ENV_BENCHMARK) poetry run uvicorn app:app --host 0.0.0.0 --port $(PORT) --workers 4

.PHONY: runZenDisabled
runZenDisabled: install
.PHONY: runZenDisabled startZenDisabled
runZenDisabled: install startZenDisabled
startZenDisabled:
@echo "Running sample app starlette-postgres-uvicorn without Zen on port $(PORT_DISABLED)"
$(AIKIDO_ENV_DISABLED) poetry run uvicorn app:app --host 0.0.0.0 --port $(PORT_DISABLED) --workers 4
Loading