# CI Analysis Tools

Tools for analyzing GitHub Actions CI performance and queue status.

## Scripts

| Script                          | Purpose                                                                   |
| ------------------------------- | ------------------------------------------------------------------------- |
| `analyze-ci-parallelization.py` | Analyze a completed CI run's parallelization and bottlenecks              |
| `ci-queue-status.py`            | Live snapshot of the current CI queue, runner utilization, and wait times |

---

# analyze-ci-parallelization.py

Analyze a completed GitHub Actions workflow run and identify parallelization bottlenecks.

## Features

- **Workflow Metrics**: Total duration, CPU time, parallelization efficiency
- **Critical Path Analysis**: Identifies longest-running jobs
- **Sequential Chain Detection**: Finds build→test dependencies and wait times
- **Runner Utilization**: Shows how efficiently runners are being used
- **Actionable Recommendations**: Provides specific optimization suggestions

## Installation

No installation required! Just download the script and run it with Python 3.7+.

```bash
# Make it executable
chmod +x analyze-ci-parallelization.py
```

## Usage

### Basic Usage (Recommended)

Pipe GitHub Actions job data directly from `gh` CLI:

```bash
gh api --paginate repos/OWNER/REPO/actions/runs/RUN_ID/jobs | python3 analyze-ci-parallelization.py
```

**Example:**

```bash
gh api --paginate repos/shader-slang/slang/actions/runs/21618930965/jobs | python3 analyze-ci-parallelization.py
```

### Alternative: Save to File First

```bash
# Save jobs data
gh api --paginate repos/shader-slang/slang/actions/runs/21618930965/jobs > jobs.json

# Analyze the saved file
python3 analyze-ci-parallelization.py jobs.json
```

### Finding the Run ID

1. Go to your GitHub Actions page: `https://github.com/OWNER/REPO/actions`
2. Click on a workflow run
3. The URL will be: `https://github.com/OWNER/REPO/actions/runs/RUN_ID`
4. Copy the `RUN_ID` number

Or use `gh` CLI to find recent runs:

```bash
# List recent runs
gh run list --repo OWNER/REPO

# Get the latest run ID
gh run list --repo OWNER/REPO --limit 1 --json databaseId -q '.[0].databaseId'
```

## Output Sections

### 1. Workflow Overview

- Total duration and CPU time
- Number of jobs
- Parallelization efficiency (how many jobs run concurrently on average)
- Failed job detection

### 2. Longest Running Jobs

- Top 10 jobs by duration
- Helps identify optimization targets

### 3. Sequential Job Chains

- Groups related jobs (e.g., build → test)
- Shows wait time gaps
- Identifies platforms with sequential bottlenecks

### 4. Runner Utilization

- Shows how each runner is used
- Identifies runners handling too many sequential jobs
- Calculates utilization percentage and idle time

### 5. Optimization Recommendations

- Overall efficiency rating (🔴 low, 🟡 moderate, 🟢 good)
- Lists long-running jobs (>20 minutes)
- Identifies runner bottlenecks
- Provides actionable suggestions

## Interpreting Results

### Parallelization Efficiency

```
Parallelization Efficiency: 3.4x
```

This means on average, 3.4 jobs run in parallel. For a workflow using 15+ runners, this indicates low efficiency.

**Good**: 10x+ (most runners actively working)
**Moderate**: 5-10x (some optimization possible)
**Poor**: <5x (significant sequential bottlenecks)

### Wait Time Gaps

```
test-linux-debug-gcc-x86_64:
  Jobs: 3, Total work: 29.6 min, Elapsed: 39.0 min
  🔍 Gap: 9.4 min (potential wait time)
```

The gap indicates time where jobs could have run in parallel but didn't. Investigate job dependencies.

### Runner Bottlenecks

```
slang-win-gcp-t4-runner-4:
  Jobs: 4, Total work: 45.3 min, Idle: 5.7 min, Utilization: 89%
  ⚠️  Runner handles 4 sequential jobs - potential bottleneck
```

This runner is overloaded. Consider:

- Adding more runners of this type
- Allowing some jobs to run in parallel instead of sequentially

## Common Optimization Strategies

### 1. Parallelize Test Jobs

**Before:**

```yaml
test-suite-1:
  needs: build
test-suite-2:
  needs: test-suite-1 # ❌ Unnecessary dependency
```

**After:**

```yaml
test-suite-1:
  needs: build
test-suite-2:
  needs: build # ✅ Both can run in parallel
```

### 2. Add More Runners

If specific runner types are bottlenecks:

- Add more self-hosted runners
- Use matrix strategies to distribute work
- Consider using faster runner types (GitHub-hosted runners with more CPU)

### 3. Split Long-Running Jobs

If a single job takes >20 minutes:

- Use matrix strategy to split tests
- Parallelize build steps
- Use caching to reduce rebuild time

### 4. Optimize Job Dependencies

Review `needs:` clauses in your workflow:

- Remove unnecessary dependencies
- Allow independent jobs to run in parallel
- Consider if all tests really need all builds

## Example Analysis

```bash
$ gh api --paginate repos/shader-slang/slang/actions/runs/21618930965/jobs | python3 analyze-ci-parallelization.py

====================================================================================================
CI RUN PARALLELIZATION ANALYSIS
====================================================================================================

Workflow Duration: 67.7 minutes
Total Jobs: 22
Total CPU Time: 228.9 minutes
Parallelization Efficiency: 3.4x

====================================================================================================
OPTIMIZATION RECOMMENDATIONS
====================================================================================================

🔴 LOW PARALLELIZATION EFFICIENCY
   Current: 3.4x
   Many jobs are running sequentially. Consider:
   - Reviewing job dependencies to allow more parallel execution
   - Adding more runners to handle parallel workloads

⏱️  LONG-RUNNING JOBS (3 jobs over 20 minutes)
   - build-macos-release-clang-aarch64 / build: 28.3 min
   - test-windows-debug-cl-x86_64-gpu / test-slang: 22.2 min

🔧 RUNNER BOTTLENECKS (2 runners with 3+ sequential jobs)
   - slang-win-gcp-t4-runner-4: 4 jobs, 45.3 min
   Consider adding more runners or redistributing work
```

## Requirements

- Python 3.7 or later
- GitHub CLI (`gh`) for fetching workflow data
- No additional Python packages required (uses only stdlib)

---

# ci-queue-status.py

Live snapshot of the current CI queue. Shows what's waiting, what's running, and where
the self-hosted runner bottlenecks are. Uses the same GitHub API endpoints that
[CREMA/KEDA's GitHub runner scaler](https://keda.sh/docs/2.17/scalers/github-runner/)
uses for auto-scaling decisions.

## Usage

```bash
# Check queue for shader-slang/slang (default)
python3 ci-queue-status.py

# Check a different repo
python3 ci-queue-status.py --repo OWNER/REPO

# Show more waiting jobs
python3 ci-queue-status.py --top-waiting 30
```

## Output Sections

1. **Snapshot Summary** - Counts of queued/in-progress runs and jobs
2. **Queue Depth by Runner Group** - Jobs queued vs running per runner label group, with
   available self-hosted runner counts
3. **Longest-Waiting Queued Jobs** - Top N jobs sorted by wait time (only jobs truly waiting
   for a runner, not blocked by workflow dependencies)
4. **In-Progress Runs** - Active runs with branch, workflow, age, and actor
5. **Self-Hosted Runner Status** - Each runner's online/busy state and current job

## Runner Label Groups

The script classifies jobs into groups based on their `runs-on` labels. The groups are
configured as a constant at the top of the script (`LABEL_GROUPS`) and can be adjusted
for other repositories.

## Permissions Note

The self-hosted runner status section requires admin access to the repository's runners
endpoint (`/repos/{repo}/actions/runners`). If access is denied, the section is skipped
with a warning. All other sections work without admin access.

---

## Requirements (both scripts)

- Python 3.7 or later
- GitHub CLI (`gh`) for fetching workflow data
- No additional Python packages required (uses only stdlib)
