Self-Healing Triage Dataset #

Complete audit of all triage decisions made by the sisakuintel-agent self-healing system, published for independent verification and community challenge.

Data Sources

• sisakulint Issues/PRs by sisakuintel-agent — False positive reports and fix PRs (45 items)
• sisakuintel-worker Scan Reports — Full triage results including true positives (~220 reports, #300–#555)

Last Updated: 2026-03-06

How to Read This Dataset #

The self-healing architecture routes findings through two repositories:

1. sisakuintel-worker: The scan-orchestrator creates [Scan Report] issues for each scanned repository. A triage comment classifies every finding as TP or FP with reasoning.
2. sisakulint: When the triage agent identifies a false positive, it creates a bug report Issue and a corresponding fix PR in the scanner’s own repository. Only FP-related items appear here.

The “45 triaged items” referenced in the evaluation metrics counts 27 Finding Issues + 18 Fix PRs created by sisakuintel-agent[bot] in the sisakulint repository. TP findings are documented in sisakuintel-worker Scan Report comments.

Classification Criteria #

A finding is classified as:

• True Positive (TP): The flagged pattern is exploitable given the workflow’s trigger type, permission scope, and step dependency graph.
• False Positive (FP): Any contextual factor (trigger restrictions, permission scope, step dependencies, safe data types) eliminates exploitability.

Ground truth was determined by the author (Atsushi Sada). This dataset publishes the complete classification with per-finding rationale to enable independent re-classification by any reviewer.

Part A: sisakulint Repository — FP Reports + Fix PRs (45 Items) #

Legend #

Week 1 (2026-01-06 – 2026-01-11) — 12 items #

Week 3–4 (2026-01-24 – 2026-02-01) — 6 items #

Note: #308 was fixed by #315 (by ultra-supara, introducing JobTriggerAnalyzer). Not a bot PR, not counted in 45.

Week 5–6 (2026-02-04 – 2026-02-17) — 10 items #

Week 6–7 (2026-02-18 – 2026-02-27) — 10 items #

Week 8–9 (2026-03-01 – 2026-03-03) — 8 items #

Visualizations #

Scan Report Classification Distribution #

pie title Scan Report Classification (n≈220)
    "All TP" : 130
    "Mixed (TP+FP)" : 50
    "All/Majority FP" : 20
    "No triage / Non-scan" : 20

FP Corrections by Rule Category #

xychart-beta
    title "False Positive Corrections by Rule Category"
    x-axis ["impostor-commit", "parser", "cond/expr/code-inj", "credential", "artifact-poison", "cache-poison", "permissions", "toctou", "commit-sha"]
    y-axis "Number of Corrections" 0 --> 7
    bar [6, 4, 3, 3, 2, 2, 2, 1, 1]

Self-Healing Timeline (Cumulative FP Corrections) #

xychart-beta
    title "Cumulative FP Corrections Over Time"
    x-axis ["W1 (Jan 6)", "W2", "W3-4 (Jan 24)", "W5-6 (Feb 4)", "W6-7 (Feb 18)", "W8-9 (Mar 1)"]
    y-axis "Cumulative Corrections" 0 --> 27
    line [8, 8, 13, 19, 24, 27]
    bar [8, 0, 5, 6, 5, 3]

Bar = new FP findings per period. Line = cumulative total.

Self-Healing Pipeline Flow #

flowchart LR
    A["sisakuintel-worker\nscans trending repo"] --> B{"Triage\ncomment"}
    B -->|All TP| C["TP findings\ndocumented in\nScan Report"]
    B -->|FP detected| D["sisakuintel-agent\ncreates Issue\nin sisakulint"]
    D --> E["sisakuintel-agent\ncreates Fix PR"]
    E --> F{"Human review"}
    F -->|Approve| G["Merged\n(15 bot PRs)"]
    F -->|Reject/Revise| H["Manual fix\n(2 human PRs)"]
    G --> I["Scanner improved\nFP eliminated"]
    H --> I
    I -.->|"Next scan"| A

    style A fill:#4a90d9,color:#fff
    style B fill:#f5a623,color:#fff
    style C fill:#7ed321,color:#fff
    style D fill:#d0021b,color:#fff
    style E fill:#d0021b,color:#fff
    style G fill:#7ed321,color:#fff
    style I fill:#7ed321,color:#fff

Fix PR Outcome Distribution #

pie title Fix PR Outcomes (18 total)
    "Bot PR Merged" : 15
    "Bot PR Open" : 1
    "Human PR (outside count)" : 2

Part B: sisakuintel-worker Repository — Scan Report Triage (TP + FP) #

Each [Scan Report] issue in sisakuintel-worker contains a triage comment with per-finding TP/FP classification and reasoning. This section summarizes all triaged reports from #300 to #555.

Aggregate Classification (#300–#555, ~220 Scan Reports) #

Representative HIGH/CRITICAL Severity TP Findings #

Representative All-TP Scan Reports #

Representative Mixed (TP + FP) Scan Reports #

Reports where the triage correctly separated TP from FP within the same scan:

FP-to-Fix Traceability #

Complete chain from FP discovery in worker to scanner fix:

Statistics #

Fix PR Status #

Rule Category Breakdown #

Verification Guide #

To independently verify any triage decision:

1. Pick a Scan Report from sisakuintel-worker issues (e.g., #543)
2. Read the triage comment — it contains the full reasoning for each finding
3. Clone the target repository listed in the Scan Report
4. Inspect the workflow file at the specified path and line number
5. Assess exploitability using the classification criteria above
6. Compare your classification with the published determination

For FP findings that led to scanner fixes:

1. Read the sisakulint Issue (e.g., #377) for the FP report with root cause analysis
2. Read the Fix PR (e.g., #378) for the code change and regression tests
3. Verify the fix does not reduce detection of true positives by checking CI results

If you disagree with any TP/FP determination, open an issue in the sisakulint repository with your analysis.

Limitations #

1. Single annotator: Ground truth was classified by the author. Per-finding rationale and triage reasoning are published for independent re-classification. GHSA/GHSL advisory outcomes provide partial external validation for the advisory-confirmed subset, though advisories confirm vulnerability existence at the repository level, not individual rule-level TP/FP status.

2. FP-only in sisakulint: The sisakulint repository contains only false positive reports and fixes because the architecture routes only FPs to the scanner for correction. TP evidence is in sisakuintel-worker Scan Report comments.

3. Evolving dataset: The self-healing system continues to operate. New findings, triage decisions, and fixes are added continuously and are publicly visible in real time.