{
  "title": "Self-Improvement Agent Harness: A Deterministic SIA Exemplar",
  "version": "0.1.2",
  "doi": "10.5281/zenodo.20453879",
  "doi_url": "https://doi.org/10.5281/zenodo.20453879",
  "zenodo_record": "https://zenodo.org/records/20453879",
  "record_id": "20932066",
  "publication_date": "2026",
  "resource_type": {
    "title": "Journal article",
    "type": "publication",
    "subtype": "article"
  },
  "creators": [
    {
      "name": "Daniel Ari Friedman",
      "affiliation": "Active Inference Institute",
      "orcid": "0000-0001-6232-9096"
    }
  ],
  "description": "This exemplar documents template_sia, a deterministic implementation of the Self-Improvement Agent (SIA) harness contract described in the Self-Improvement Agents specification (Hexo AI, 2026, arXiv:2605.27276). The default pipeline replays fixture-backed generations for the mini_classify task; opt-in live mode runs bounded target subprocesses and optional Ollama-backed meta/feedback steps.\n\nRun snapshot. Task mini_classify, run 1, 3 generation(s), live=false. Final accuracy=0.8333 over 6 held-out samples. Values are injected by scripts/z_generate_manuscript_variables.py after analysis.\n\nKeywords: self-improvement agents, benchmark harness, reproducible evaluation, agent loops\n\n---\nAssociated artifacts\nGitHub release: v0.1.2 (https://github.com/docxology/template_sia/releases/tag/v0.1.2)\nDOI: https://doi.org/10.5281/zenodo.20453879\nZenodo: https://zenodo.org/records/20453879\nPDF SHA-256: 6e6d19d04182628bb825471cf8094b5c32d2c491d2c646652ec7e2439ba80773",
  "keywords": [
    "self-improvement agents",
    "benchmark harness",
    "reproducible research",
    "agent evaluation"
  ],
  "files": [
    {
      "name": "Friedman_2026_Selfimprovement_6e6d19d0.pdf",
      "size_bytes": 223019,
      "checksum": "md5:f8042650bd618719fb4eab9c40563527",
      "download_url": "https://zenodo.org/api/records/20932066/files/Friedman_2026_Selfimprovement_6e6d19d0.pdf/content"
    }
  ],
  "related_resources": [
    {
      "type": "repository",
      "url": "https://github.com/docxology/template_sia"
    }
  ],
  "github_repo": "docxology/template_sia",
  "source": "zenodo-only",
  "checked_at": "2026-09-24T18:29:54Z",
  "accepted_pairings": [
    {
      "doi": "10.5281/zenodo.20453879",
      "record_title": "Self-Improvement Agent Harness: A Deterministic SIA Exemplar",
      "zenodo_record_url": "https://zenodo.org/records/20453880",
      "github_repo": "docxology/template",
      "github_release_url": "https://github.com/docxology/template/releases/tag/0.1",
      "release_tag": "0.1",
      "release_name": "0.1",
      "resource_type": {
        "title": "Journal article",
        "type": "publication",
        "subtype": "article"
      },
      "raw_confidence": "needs_review",
      "evidence": [
        "github_repo_self_linked"
      ],
      "review_decision": "accept",
      "reviewed_at": "2026-06-04T22:32:27Z",
      "review_source": "user: Accept for all",
      "source_report": "reports/paired_publications_2026-06-04.json"
    },
    {
      "doi": "10.5281/zenodo.20453879",
      "record_title": "Self-Improvement Agent Harness: A Deterministic SIA Exemplar",
      "zenodo_record_url": "https://zenodo.org/records/20453880",
      "github_repo": "docxology/template",
      "github_release_url": "https://github.com/docxology/template/releases/tag/0.2",
      "release_tag": "0.2",
      "release_name": "0.2",
      "resource_type": {
        "title": "Journal article",
        "type": "publication",
        "subtype": "article"
      },
      "raw_confidence": "needs_review",
      "evidence": [
        "github_repo_self_linked"
      ],
      "review_decision": "accept",
      "reviewed_at": "2026-06-04T22:32:27Z",
      "review_source": "user: Accept for all",
      "source_report": "reports/paired_publications_2026-06-04.json"
    },
    {
      "doi": "10.5281/zenodo.20453879",
      "record_title": "Self-Improvement Agent Harness: A Deterministic SIA Exemplar",
      "zenodo_record_url": "https://zenodo.org/records/20453880",
      "github_repo": "docxology/template",
      "github_release_url": "https://github.com/docxology/template/releases/tag/0.3",
      "release_tag": "0.3",
      "release_name": "0.3",
      "resource_type": {
        "title": "Journal article",
        "type": "publication",
        "subtype": "article"
      },
      "raw_confidence": "needs_review",
      "evidence": [
        "github_repo_self_linked"
      ],
      "review_decision": "accept",
      "reviewed_at": "2026-06-04T22:32:27Z",
      "review_source": "user: Accept for all",
      "source_report": "reports/paired_publications_2026-06-04.json"
    },
    {
      "doi": "10.5281/zenodo.20453879",
      "record_title": "Self-Improvement Agent Harness: A Deterministic SIA Exemplar",
      "zenodo_record_url": "https://zenodo.org/records/20453880",
      "github_repo": "docxology/template",
      "github_release_url": "https://github.com/docxology/template/releases/tag/0.4",
      "release_tag": "0.4",
      "release_name": "0.4",
      "resource_type": {
        "title": "Journal article",
        "type": "publication",
        "subtype": "article"
      },
      "raw_confidence": "needs_review",
      "evidence": [
        "github_repo_self_linked"
      ],
      "review_decision": "accept",
      "reviewed_at": "2026-06-04T22:32:27Z",
      "review_source": "user: Accept for all",
      "source_report": "reports/paired_publications_2026-06-04.json"
    },
    {
      "doi": "10.5281/zenodo.20453879",
      "record_title": "Self-Improvement Agent Harness: A Deterministic SIA Exemplar",
      "zenodo_record_url": "https://zenodo.org/records/20453880",
      "github_repo": "docxology/template",
      "github_release_url": "https://github.com/docxology/template/releases/tag/0.5",
      "release_tag": "0.5",
      "release_name": "Multiproject",
      "resource_type": {
        "title": "Journal article",
        "type": "publication",
        "subtype": "article"
      },
      "raw_confidence": "needs_review",
      "evidence": [
        "github_repo_self_linked"
      ],
      "review_decision": "accept",
      "reviewed_at": "2026-06-04T22:32:27Z",
      "review_source": "user: Accept for all",
      "source_report": "reports/paired_publications_2026-06-04.json"
    },
    {
      "doi": "10.5281/zenodo.20453879",
      "record_title": "Self-Improvement Agent Harness: A Deterministic SIA Exemplar",
      "zenodo_record_url": "https://zenodo.org/records/20453880",
      "github_repo": "docxology/template",
      "github_release_url": "https://github.com/docxology/template/releases/tag/v3.0.0",
      "release_tag": "v3.0.0",
      "release_name": "Research Project Template v3.0.0",
      "resource_type": {
        "title": "Journal article",
        "type": "publication",
        "subtype": "article"
      },
      "raw_confidence": "needs_review",
      "evidence": [
        "github_repo_self_linked"
      ],
      "review_decision": "accept",
      "reviewed_at": "2026-06-04T22:32:27Z",
      "review_source": "user: Accept for all",
      "source_report": "reports/paired_publications_2026-06-04.json"
    }
  ],
  "manual_pairing_review": {
    "decision": "accept",
    "reviewed_at": "2026-06-04T22:32:27Z",
    "source": "data/paired-publication-decisions.json"
  },
  "github_release_url": "https://github.com/docxology/template_sia/releases/tag/v0.1.2",
  "release_tag": "v0.1.2",
  "release_name": "Self-Improvement Agent Harness: A Deterministic SIA Exemplar (v0.1.2)",
  "pdf_sha256": "6e6d19d04182628bb825471cf8094b5c32d2c491d2c646652ec7e2439ba80773",
  "pairing_confidence": "strong",
  "pairing_evidence": [
    "zenodo_related_identifier_mentions_release",
    "github_repo_self_linked",
    "title_overlap"
  ],
  "domain": "Computational",
  "type": "Paper",
  "methods": [
    {
      "name": "Meta -> Target -> Feedback three-agent SIA loop",
      "description": "The harness cycles a meta agent that seeds a target agent, the target run on public data, and a feedback agent that reads private metrics to propose the next generation.",
      "evidence": "1. Meta — proposes or seeds a target agent for generation 𝑛. 2. Target — runs against public task data and writes agent_execution.json. 3. Feedback — reads private evaluation metrics and proposes improvements for generation 𝑛+ 1."
    },
    {
      "name": "Public/private task split with deterministic reference baseline",
      "description": "Each task separates agent-visible data and instructions from held-out evaluation labels, plus a deterministic reference target agent.",
      "evidence": "data/public/ — agent-visible instructions and data (task.md, train.csv, evaluate.py). • data/private/ — held-out labels for evaluation only. • reference/reference_target_agent.py — deterministic baseline."
    },
    {
      "name": "Fixture-replay determinism contract with opt-in live mode",
      "description": "By default generations replay recorded fixtures so CI never runs generated code or calls LLM APIs; a flag enables bounded subprocess execution with optional Ollama feedback.",
      "evidence": "When sia.live: false (default), generations replay recorded fixtures from src/fixtures/recorded_generation s/. CI never executes generated agent code or calls external LLM APIs."
    },
    {
      "name": "mini_classify single-feature threshold classifier task",
      "description": "The bundled exemplar task is a threshold classifier on one feature column, evaluated over 3 generations on 6 held-out samples.",
      "evidence": "Scope. The bundled mini_classify task is a threshold classifier on one feature column. Results demonstrate harness wiring and artifact contracts—not state-of-the-art accuracy."
    }
  ],
  "key_findings": [
    "In the bundled fixture-replay run, final accuracy on mini_classify was 0.8333 over 6 held-out samples after 3 generations.",
    "Accuracy rose from the first to the final generation by a metric delta of 0.3333 in the fixture replay.",
    "template_sia shows the SIA harness contract can be embedded in the Research Project Template without vendoring upstream orchestration code, split into an infrastructure layer and a project layer.",
    "The author states the fixture-replay metrics validate wiring only and are not evidence of state-of-the-art self-improvement."
  ],
  "related_papers": [
    "2023_NSFReporting",
    "2023_NaturalAIBased",
    "2025_AuBI"
  ],
  "related_software": [
    "docxology/template_sia"
  ],
  "key_findings_evidence": [
    "Run snapshot. Task mini_classify, run 1, 3 generation(s), live=false. Final accuracy=0.8333 over 6 held-out samples.",
    "Metric delta (final −first generation): 0.3333.",
    "template_sia demonstrates how to embed the SIA harness contract in the Research Project Template without vendoring upstream orchestration code.",
    "Fixture-replay metrics (final accuracy=0.8333) validate wiring only—they are not evidence of state-of-the-art self-improvement."
  ],
  "summary_provenance": {
    "source": "full_text.md",
    "date": "2026-10-01",
    "work_kind": "software/methods",
    "method": "Evidence-grounded extraction from full_text.md: every method and finding carries a verbatim quote from the paper text, mechanically verified to occur in full_text.md (whitespace/ligature-normalized) before merge.",
    "notes": "Short harness/documentation exemplar; reported metrics are from recorded fixtures, explicitly not a benchmark.",
    "reviewed": "2026-10-01 independent review of every item against its quote and context; no corrections needed"
  }
}
