Test-Effectiveness Audit for Zebra: an independent regression witness for consensus-critical code

1. Executive summary and decision snapshot

The request is for $45,000 to fund an independent two-person audit of Zebra’s test effectiveness on consensus-critical code paths. The proposed work uses targeted code mutations to determine whether existing tests actually detect realistic defect shapes, followed by regression tests for confirmed gaps and optional upstream patches. [1][3]

The project lead is jiehuo100net, supported by zhenyuan for mutation-engine and build infrastructure work. The team is unaffiliated with Zebra’s maintainers and states that it has no prior grant-delivery record. [1][3]

The strongest value proposition is a security-focused, independent assessment of whether passing tests provide meaningful regression protection, rather than merely adding test count. The principal delivery questions are whether the undisclosed full milestone and budget details are sufficiently measurable, whether the two-person team can execute the tooling and adjudication workload, and how confidential findings will be validated and routed to maintainers. [1][3]

Inference: The proposal’s value depends less on volume of mutations run than on credible selection of defect models, rigorous treatment of surviving mutants, and a disclosure process trusted by Zebra maintainers.

2. Applicant and team track record

The lead states that they are an independent systems and Rust engineer with three merged Zebra fixes on consensus-critical paths and one additional open fix. The application identifies fixes involving inbound peer-address normalization, a network-upgrade boundary check in subsidy metadata, and address-book cleanup for banned peers. [1]

The applicant also states that two of the three merged fixes affected subsystems proposed for audit, and that the proposed negative-control method was used publicly in one earlier pull request. [1]

The second team member, zhenyuan, is described as a long-standing collaborator with C/C++, build-system, and cross-language toolchain experience. Their proposed responsibilities include mutation generation, reproducible builds, incremental reruns after code changes, mutation-point relocation across refactors, and independent review of adjudication decisions. [1]

The applicant explicitly reports no Zebra write access, no maintainer role, no organizational affiliation, and no previous grant delivery. [3]

Evidence limitation: The supplied evidence documents the team’s claimed Zebra contributions but does not independently verify their quality, the team’s previous delivery of comparable security tooling, or their capacity to complete a $45,000 project.

3. Proposal scope, milestones, budget, technical approach, and dependencies

Scope and approach

• Audit existing Zebra tests rather than primarily create new functionality or tests. [3]
• Introduce isolated, targeted mutations derived from realistic defect shapes on selected consensus-critical code paths. [1][3]
• Run Zebra’s existing tests against each mutation; a mutation that leaves tests passing is treated as a potential test-coverage gap. [3]
• Hand-adjudicate surviving mutants as equivalent changes or confirmed gaps, then create regression tests for confirmed gaps that fail when the defect is present. [1]
• Offer resulting patches upstream as optional contributions. [3]

The team proposes a reproducibility package containing the baseline commit, toolchain version, operator list, and seeds. It also proposes syntax- and function-signature-based mutation anchors and reports on anchor drift after refactors. [1]

The proposal states that Zebra source and CI would not be modified or used, that milestones would not depend on upstream merges, and that maintainers should receive no required requests under the project’s success metric. [3]

Milestones and budget

• Total request: $45,000. [1]
• No grounded evidence found for the milestone count, payment allocation, schedule, per-milestone acceptance criteria, or line-item budget in the supplied material.

Dependencies

• The technical audit is designed not to depend on upstream code merges or Zebra CI access. [3]
• Safe handling of material findings depends on a workable private disclosure channel to maintainers. [3]

4. Community discussion, arguments, responses, and resolution

Strongest arguments supporting the request

• The applicant argues that test presence and passing CI do not establish that a regression test exercises the actual defect mechanism. They cite prior recurring Zebra advisories, claiming prior fixes had dedicated tests but defects reappeared. [3]
• The audit seeks independently derived evidence of test effectiveness in consensus-critical paths, potentially addressing security gaps that conventional test-count metrics would miss. [3]
• The work is structured to minimize maintainer operational burden by avoiding edits to Zebra source and use of Zebra CI. [3]

Strongest arguments against or concerns

• Public disclosure of detailed weak-test locations could reveal information useful to attackers, especially for consensus divergence, funds-safety, or remote-crash risks. [3]
• The work requires subjective judgment in selecting mutations and distinguishing equivalent mutants from true coverage gaps. [1]
• No grounded evidence found for substantive community objections, maintainer feedback, or technical peer review of the method.

Applicant clarification and resolution

• The applicant proposes withholding findings that point toward consensus divergence, funds safety, or remote crash; those findings would be disclosed privately to maintainers. Public outputs would be limited to methodology, tooling, aggregate figures, and already-closed gaps. [3]
• The application proposes blind re-adjudication and recorded disagreements between the two team members as an internal check on subjective decisions. [1]
• Resolved in design: Avoiding mandatory upstream merge or CI dependency. [3]
• Unresolved: Whether maintainers accept the disclosure model, whether private findings can be independently validated for grant acceptance, and whether the proposed audit scope is achievable within the undisclosed milestones and budget.

5. Relevant precedents and documented outcomes

Zebra Regtest CI Kit (ZRK). This is the closest precedent because it concerned Zebra testing and CI infrastructure. It was approved, with three milestones recorded as complete, but was later cancelled before completion. [8] This demonstrates that Zebra-focused testing work has been considered fundable, while also showing that partial milestone completion does not establish final delivery or sustained value. No documented reason for cancellation is supplied. [8]

Zebra Mempool Adversarial Resilience Suite. This proposal concerned security-oriented test infrastructure for Zebra and was declined. Committee members cited unclear applicant expertise, proposal vagueness, lack of track record, and the need for involvement by expert teams developing the technology to support adoption. [10] It is relevant because the current request likewise proposes specialized security-testing tooling for core node code, although the present applicant offers a stated record of Zebra contributions. [1][10]

Zebra State Snapshot & Fast Sync Infrastructure. This proposal was declined because committee members assessed security and centralization risks as outweighing the expected operational benefit, and questioned likely broad adoption. [9] The relevance is decision principle rather than technical similarity: Zebra infrastructure proposals may need to demonstrate that security risks and adoption dependencies are actively mitigated.

6. Material risks and execution considerations

• Security-disclosure risk: Publishing granular test weaknesses could create attacker-relevant information. The proposed private-routing model mitigates this, but its practical acceptance and escalation path are unverified. [3]
• Acceptance risk for confidential outputs: If the most consequential findings cannot be public, committee evaluation may require a defined confidential validation mechanism. No grounded evidence found for that mechanism.
• Methodological risk: Mutation survivors can represent either genuine test gaps or equivalent mutations. The proposed cross-review reduces but does not eliminate this interpretive risk. [1]
• Capacity risk: A two-person unaffiliated team must build and operate mutation tooling, manage reproducibility, adjudicate results, and handle responsible disclosure. [1]
• Value-measurement risk: No grounded evidence found for quantitative target coverage, expected number of audited paths, expected mutation volume, or threshold defining a successful audit.

7. Material gaps and questions for the applicant

1. What are the exact milestones, payment amounts, schedule, and acceptance criteria for the $45,000 request?
2. Which Zebra subsystems and defect classes are in scope, and what work is explicitly out of scope?
3. What quantitative outputs will demonstrate useful progress: audited paths, mutation operators, reproducible runs, confirmed gaps, or validated regression tests?
4. Who will receive confidential findings, what response timeframe is expected, and how can ZCG validate completion when publication is unsafe?
5. What will be delivered if no confirmed test gaps are found: a reproducible methodology, negative results, toolchain, aggregate report, or all of these?
6. How will the team demonstrate that mutation operators are realistic representations of prior or plausible Zebra defects rather than arbitrary code changes?
7. What compute budget, reproducibility controls, and contingency plan support the claimed ability to rebuild and test the selected workspace per mutation? [1]

8. Neutral decision considerations

Committee members may weigh the request as a bounded security-research investment rather than ordinary feature development. The proposal has a focused objective, a stated low-maintainer-burden approach, and a lead who claims prior accepted Zebra contributions. [1][3]

Funding confidence would be strengthened by converting the currently unsupported milestone and budget structure into measurable deliverables, particularly for confidential findings and negative results. The committee may also consider whether an external maintainer or qualified security reviewer should confirm the disclosure channel and acceptance method before work begins.

Inference: A staged structure that first validates the mutation framework, scope selection, reproducibility, and disclosure protocol could reduce uncertainty before funding the full audit effort.

9. Numbered source list

• [1] Grant Application -Test-Effectiveness Audit for Zebra: an independent regression witness for consensus-critical code — https://github.com/ZcashCommunityGrants/zcashcommunitygrants/issues/389
• [3] Test-Effectiveness Audit for Zebra: an independent regression witness for consensus-critical code — posts 1–5 — https://forum.zcashcommunity.com/t/test-effectiveness-audit-for-zebra-an-independent-regression-witness-for-consensus-critical-code/56989/1
• [8] Zebra Regtest CI Kit (ZRK) — comparison summary — https://github.com/ZcashCommunityGrants/zcashcommunitygrants/issues/105
• [10] Zebra Mempool Adversarial Resilience Suite — comparison summary — https://github.com/ZcashCommunityGrants/zcashcommunitygrants/issues/362
• [9] Grant Application – Zebra State Snapshot & Fast Sync Infrastructure — comparison summary — https://github.com/ZcashCommunityGrants/zcashcommunitygrants/issues/187