Skip to main content

121. Artifact-Based Reporting for Validation

Status: Accepted Date: 2025-07-06

Context​

The Maat module's primary output is a report of "Bad Signal" (BS) candidates for human review (adr://human-in-the-loop-validation). We need to decide on the format and location for these reports.

The reports need to be:

  • Persistent: They should be stored permanently for historical analysis.
  • Accessible: Reviewers need an easy way to view them, preferably via a web interface.
  • Richly Formatted: They should support formatting like headers, code blocks (for JSON context), and links.
  • Appendable: It should be easy to add new candidates to an existing report (e.g., the "daily BS report").

Building a custom UI for this from scratch would be a significant amount of work.

Decision​

The Maat module will use GitHub Gists as the primary medium for its reports.

When a new BS candidate is collected (either via an event or a collector service), it will be appended to a specific Gist file. The filename within the Gist can be used to categorize the candidates (e.g., dike_comparison_reviews.json, morpheus_losing_trades.json).

The BSStoreService will use the ArtifactsService (adr://artifact-integration), which will contain the logic for interacting with the GitHub API to create and update these Gists. The daily summary report will simply contain links to the relevant Gists that were updated during that 24-hour period.

Consequences​

Positive:

  • Leverages Existing Infrastructure: We get a robust, accessible, version-controlled platform for our reports for free, with zero UI development effort.
  • Excellent Developer Experience: Reviewing reports in the Gist UI is a familiar and effective experience for developers. It supports syntax highlighting for JSON, commenting for discussion, and shows a full revision history.
  • Programmatic Access: Gists have a simple and powerful API, making them easy to work with from our backend services.
  • Persistence and History: Every change to a report is automatically versioned by Git, giving us a perfect audit trail of when candidates were added.

Negative:

  • Dependency on GitHub: Creates a hard dependency on the GitHub Gist service for a core part of our QA workflow.
  • Less Structured than a Database: A Gist is fundamentally a text file. While we will store structured JSON, it's not as queryable as a proper database. We can't easily run a query like "show me all BS candidates from the Dike module with a confidence score below 0.6".
  • Potential for Large Files: If not managed properly, the Gist files could become very large, although Gists handle this reasonably well.

Mitigation:

  • Abstraction via ArtifactsService: The rest of the system does not interact with Gists directly. It interacts with our ArtifactsService. This means if we ever needed to migrate away from Gists to a different solution, we would only need to change the implementation of the ArtifactsService, not every module that uses it.
  • Reporting, Not Analytics: The primary purpose of these reports is for human review, not for complex, automated querying. For that kind of analytics, we would stream the events to a proper data warehouse or logging system.
  • Log Rotation: We will implement a "log rotation" strategy. A new Gist will be created for each day or week, preventing any single file from growing indefinitely. The daily summary report will link to the currently active Gists.