feat(tests): Add host creation performance test - #231
Conversation
Measures host creation throughput via hammer CLI at configurable concurrency levels (default: 1, 5, 10, 20). Creates 100 hosts per level, collects per-host timing and server-side metrics (CPU, memory, PostgreSQL connections), and generates a comparison report with avg/median/p90/p95/throughput/failure rate. Tested on Satellite 6.19 (katello-4.20.0). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe new Ansible playbook benchmarks Satellite host creation with concurrency levels 1, 5, 10, and 20. It creates 100 hosts per run, collects system and PostgreSQL metrics, calculates performance statistics, generates a report, and removes helper scripts. ChangesHost creation benchmark
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant AnsiblePlaybook
participant HostCreationScripts
participant SatelliteAPI
participant MetricsCollector
AnsiblePlaybook->>MetricsCollector: Start system and PostgreSQL metric collection
AnsiblePlaybook->>HostCreationScripts: Launch concurrent host creation
HostCreationScripts->>SatelliteAPI: Create test host
SatelliteAPI-->>HostCreationScripts: Return success or failure
HostCreationScripts-->>AnsiblePlaybook: Record result and latency
MetricsCollector-->>AnsiblePlaybook: Record system metrics
AnsiblePlaybook->>HostCreationScripts: Delete test hosts
AnsiblePlaybook->>AnsiblePlaybook: Generate final report
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@playbooks/tests/host-create.yaml`:
- Around line 245-273: Restrict report generation in the run-processing loop to
directories created by the current playbook invocation, rather than every
persistent run_c* directory under results_dir. Create and use an
invocation-specific parent directory, or capture the current run directory list
before execution and iterate only over those paths so the report contains no
stale or duplicate concurrency results.
- Around line 39-40: Configure Hammer credentials for the remote user in
~/.hammer/cli.modules.d/foreman.yml with mode 0600, then remove sat_pass from
every Hammer invocation and generated helper script in
playbooks/tests/host-create.yaml:39-40, 49-50, 66-67, 92-100, and 117-123,
including both cleanup calls; each affected site requires this change.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 00461baa-cb49-4b10-820e-f7b9e189a12a
📒 Files selected for processing (1)
playbooks/tests/host-create.yaml
| for DIR in $(ls -d {{ results_dir }}/run_c* 2>/dev/null | sort -t'c' -k2 -n); do | ||
| CSV="${DIR}/results.csv" | ||
| [ ! -f "${CSV}" ] && continue | ||
|
|
||
| CONC=$(basename "${DIR}" | sed 's/run_c\([0-9]*\)_.*/\1/') | ||
| DATA=$(tail -n +2 "${CSV}") | ||
| TOTAL_N=$(echo "${DATA}" | grep -c . 2>/dev/null || echo 0) | ||
| SUCCESSES=$(echo "${DATA}" | awk -F',' '$6==0' | grep -c . 2>/dev/null || echo 0) | ||
| FAILURES=$(echo "${DATA}" | awk -F',' '$6!=0' | grep -c . 2>/dev/null || echo 0) | ||
|
|
||
| if [ "${SUCCESSES}" -gt 0 ]; then | ||
| AVG=$(echo "${DATA}" | awk -F',' '$6==0{sum+=$5;n++} END{printf "%.2f",sum/n}') | ||
| MEDIAN=$(echo "${DATA}" | awk -F',' '$6==0{print $5}' | sort -n | awk '{a[NR]=$1} END{if(NR%2==1) print a[int(NR/2)+1]; else printf "%.2f",(a[NR/2]+a[NR/2+1])/2}') | ||
| P90=$(echo "${DATA}" | awk -F',' '$6==0{print $5}' | sort -n | awk '{a[NR]=$1} END{idx=int(NR*0.9)+1; if(idx>NR) idx=NR; print a[idx]}') | ||
| P95=$(echo "${DATA}" | awk -F',' '$6==0{print $5}' | sort -n | awk '{a[NR]=$1} END{idx=int(NR*0.95)+1; if(idx>NR) idx=NR; print a[idx]}') | ||
| MAX=$(echo "${DATA}" | awk -F',' '$6==0{print $5}' | sort -n | tail -1) | ||
| FIRST_START=$(echo "${DATA}" | awk -F',' 'NR==1{print $3}') | ||
| LAST_END=$(echo "${DATA}" | awk -F',' '{print $4}' | sort -n | tail -1) | ||
| WALL_SEC=$(awk "BEGIN{printf \"%d\", (${LAST_END} - ${FIRST_START}) / 1000}") | ||
| [ "${WALL_SEC}" -eq 0 ] && WALL_SEC=1 | ||
| THROUGHPUT=$(awk "BEGIN{printf \"%.3f\", ${SUCCESSES} / ${WALL_SEC}}") | ||
| else | ||
| AVG="N/A"; MEDIAN="N/A"; P90="N/A"; P95="N/A"; MAX="N/A"; THROUGHPUT="0" | ||
| fi | ||
| FAIL_PCT=$(awk "BEGIN{printf \"%.1f\", (${TOTAL_N} > 0 ? ${FAILURES} * 100 / ${TOTAL_N} : 0)}") | ||
|
|
||
| printf "%-12s %-8s %-8s %-8s %-8s %-8s %-10s %-6s\n" \ | ||
| "${CONC}" "${AVG}" "${MEDIAN}" "${P90}" "${P95}" "${MAX}" "${THROUGHPUT}" "${FAIL_PCT}%" >> "${REPORT}" | ||
| done |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Generate the report from only this invocation's runs.
results_dir persists run directories. The run_c* glob includes results from previous playbook executions. A later report can contain duplicate concurrency rows and stale throughput, latency, and failure values.
Create an invocation-specific parent directory, or retain the current run directory list and report only those paths.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@playbooks/tests/host-create.yaml` around lines 245 - 273, Restrict report
generation in the run-processing loop to directories created by the current
playbook invocation, rather than every persistent run_c* directory under
results_dir. Create and use an invocation-specific parent directory, or capture
the current run directory list before execution and iterate only over those
paths so the report contains no stale or duplicate concurrency results.
There was a problem hiding this comment.
Yep, I agree that dir should be cleaned on every run. Possibly just delete it before Create results directory task?
There was a problem hiding this comment.
@jhutar, yes. Remove {{ results_dir }} with state: absent immediately before the existing Create results directory task. Then recreate it as the task already does.
This limits the run_c* report scan to the current invocation. It also removes prior reports and helper scripts, which is appropriate if results_dir is only a temporary benchmark workspace.
🐇
You are interacting with an AI system.
| loop_control: | ||
| label: "concurrency={{ item.concurrency }}" | ||
|
|
||
| - name: "Generate final report" |
There was a problem hiding this comment.
I think we might also want to create our usual JSON files per concurrency level so we can process the results with our usual workflow, but definitely that can come in follow-up PR.
| done | ||
| echo "Cleanup complete." | ||
|
|
||
| - name: collect_metrics.sh |
There was a problem hiding this comment.
We are already collecting monitoring data in our monitoring stack, so this is a duplication, but IMO does not hurt if you want it here.
2 participants
Measures host creation throughput via hammer CLI at configurable concurrency levels (default: 1, 5, 10, 20). Creates 100 hosts per level, collects per-host timing and server-side metrics (CPU, memory, PostgreSQL connections), and generates a comparison report with avg/median/p90/p95/throughput/failure rate.
Tested on Satellite 6.19 (katello-4.20.0).