Skip to main content

95. Standardized Structured Output from AI

Status: Accepted Date: 2025-07-06

Context​

When interacting with Large Language Models (LLMs) for analysis (adr://ai-first-analysis), the output can be highly variable. By default, LLMs produce free-form natural language text. While this is useful for human consumption, it is very difficult and brittle to parse programmatically. Relying on regular expressions or keyword searching to extract information from a block of text is unreliable and prone to breaking whenever the model's phrasing changes slightly. To use the AI's output reliably in our application, we need it in a predictable, machine-readable format.

Decision​

All LLM-driven analysis features, starting with the Apollo portfolio analysis, will be required to produce their output in a standardized, structured JSON format.

This will be achieved by:

  1. Explicit Prompting: The prompts sent to the LLM will contain explicit instructions to format its entire response as a JSON object that conforms to a specific schema.
  2. Schema Definition in Prompt: The prompt will include a description or even a TypeScript interface definition of the desired JSON schema, giving the model a clear template to follow.
  3. Strict Validation and Parsing: The raw string response from the LLM will be parsed into a JavaScript object. This object will then be validated against a predefined schema using a library like Zod. If the validation fails, the response is considered invalid and will be rejected.

The output of any AI analysis service will not be a string, but a strongly-typed TypeScript object (e.g., PortfolioAnalysis) that has been successfully parsed and validated.

Consequences​

Positive:

  • Reliability and Robustness: By enforcing a strict schema, we make the integration with the AI service much more robust. We are no longer guessing or searching for information in a block of text. If the AI's output doesn't conform to the schema, it fails loudly and predictably.
  • Type Safety: The validated output is a strongly-typed object. This allows us to use the analysis results in the rest of our TypeScript codebase with full type safety and autocompletion.
  • Separation of Concerns: The "AI interaction" logic is cleanly separated from the business logic. The Morpheus AI gateway is responsible for getting a valid, structured object from the AI; the rest of the application can then work with this trusted object without worrying about the messy details of parsing natural language.

Negative:

  • Increased Prompt Complexity: Forcing the model to generate valid JSON can make the prompts more complex and may slightly reduce the "creativity" of the model's text generation.
  • Model Capability Dependent: This approach depends on using a modern LLM that is good at following complex instructions and generating structured data. Older or smaller models may struggle to produce valid JSON consistently.

Mitigation:

  • Use of Modern LLMs: We will use capable, modern LLMs (like those available through Ollama, such as Llama 3 or Mixtral) that have been specifically trained to be good at following instructions and generating structured data like JSON.
  • Iterative Prompt Development: We will use a "Chain of Thought" approach in our prompts, asking the model to "think" about the analysis first, and then format its final answer into the required JSON structure. This improves the reliability of the output.
  • Error Handling and Retries: If the JSON parsing or Zod validation fails, our AI gateway service can automatically retry the request, potentially with a modified prompt that reminds the model of the required format. If it repeatedly fails, a clear error can be logged and returned.