You may not care for those site designs, but that’s not the point. The point is that they don’t look like so many other generated sites and impeccable did achieve the looks I was after.

From Impeccable.stlye:

Impeccable strips the slop from AI-generated interfaces, gives you precise commands to steer, and iterates visual variants live in your product. Then the dice: pushed out of distribution, your model designs ambitious visual worlds it could never reach alone.

Recommended.

Building a presentation thingy.

Creating a gem that writes your missing tests

No new project survives contact with the real world unscathed. We built TestGenAI, ran it on itself, and it worked well. Then we ran it on another codebase, and two things broke immediately. The fixes turned out to just as interesting as the original build.

This is a walkthrough of how the tool works and what we learned when we took it outside the greenhouse.

The code here is from TestGenAI, a working Ruby CLI gem you can install and run against your own codebase.

The pipeline

The pipeline has five stages:

1. Scan your codebase to find classes and methods without test coverage
2. Build context for each untested method
3. Generate tests using an LLM with the mechanically curated context
4. Validate that the generated tests run and pass
5. Collect the results

Each stage needs to be reliable enough that you can walk away and trust the process to complete. That means handling errors gracefully, providing clear output about what happened, and making it easy to pick up where things left off if something breaks.

Finding untested code

Before you can generate tests, you need to know what needs testing. The right approach depends on whether SimpleCov is available in the project.

If SimpleCov is set up, TestGenAI runs your test suite with COVERAGE=true, reads the resulting coverage/.resultset.json, and uses AST parsing to find methods where every executable line has zero hits. This scanner handles partially-tested files correctly. It reports individual methods that were never exercised, even if other methods in the same file have full coverage.

If SimpleCov isn’t available, the scanner falls back to checking whether a spec or test file exists for each source file. This approach is less accurate. A file tested only through integration tests or through specs for its subclasses will appear fully untested even if its methods are exercised constantly. The SimpleCov scanner is worth setting up.

Both scanners share the same underlying logic for locating methods in source files, which brings up something worth explaining.

Walking the AST

To locate methods, TestGenAI parses each Ruby source file into an abstract syntax tree and walks it recursively. The walker looks for :def and :defs nodes (instance and class methods), tracks the current class/module namespace, and records each method’s file, class, name, and line range.

That line range matters. The SimpleCov scanner uses it to check whether any executable lines in the method had zero hits. A nil in SimpleCov’s coverage array means a line isn’t executable, like a blank line, a comment, or an end. The scanner filters those out before checking for zeros, so it only flags methods where runnable code was never touched.

The parser compatibility problem

To parse Ruby, the gem relies on the parser gem. In older versions, you’d call Parser::CurrentRuby.parse(source) and get back an AST. This worked fine until Ruby 3.4, which switched its internal default parser to prism. Using Parser::CurrentRuby with Ruby 3.4 produces warnings, and in some configurations it fails entirely.

The prism project ships a compatibility shim, Prism::Translation::ParserCurrent, that produces the same AST node types as the old parser gem. The AST-walking code works unchanged. The only question is which one to load.

The solution is a small file that runs at load time and sets a constant:

module Testgenai
  if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3.4")
    require "prism"
    CurrentParser = Prism::Translation::ParserCurrent
  else
    require "parser/current"
    CurrentParser = Parser::CurrentRuby
  end
end

The rest of the codebase calls CurrentParser.parse(source) and never thinks about which parser is underneath. The pattern of version check at load time, constant as the abstraction is a clean way to handle the same kind of compatibility gap you’ll run into whenever Ruby ships a significant internal change.

Context, generation, and validation

When you ask an LLM to write tests for a method, you can’t just paste in the method body. It needs the full class, the dependencies that file requires, examples of how the method is called elsewhere in the codebase, and existing test files it can match in style. Context quality is where quick-and-dirty AI test generators fall apart, too little and the tests don’t compile, too much and you hit token limits.

The generator builds a prompt from all of that, sends it to the LLM via the ruby_llm gem (which keeps the generator code provider-agnostic), and strips any markdown fences from the response before passing it to the validator.

The validator writes the code to a temp file, runs bundle exec rspec or the Minitest equivalent, and distinguishes between three outcomes: the file failed to load (syntax errors, undefined constants), the tests ran but failed, or the tests passed. Each outcome needs different handling. A file that doesn’t load gets deleted immediately because it’s useless. A file that runs but fails gets its error output fed back to the LLM for a retry.

The pipeline retries up to three times, passing failure details back each time. LLMs are reasonably good at fixing specific errors when told what went wrong. Undefined constants and wrong require paths almost always resolve in one retry. More complex failures, like incorrect behavior assumptions, may not, and those end up in a failed bucket for manual review.

Then we ran it on a real project

The first external test run revealed two problems, both on the same day.

The first: generated tests were syntactically valid, ran, and passed — but they looked nothing like the rest of the project’s test suite. Wrong authentication setup, wrong factory usage, helpers that weren’t available. Tests that technically pass but violate project conventions create a maintenance burden.

The second: the tool was silently destroying existing tests. When a spec file already existed at the output path, the pipeline would overwrite it with the newly generated content. Any tests already in that file were gone.

Both problems make complete sense in retrospect. The tool had only ever run on its own codebase, where it was always generating new files and where the conventions were deeply familiar to the model from the context it was seeing. A different project broke both assumptions.

Fixing the conventions gap

The core problem is that the LLM knows what your method does, but it doesn’t know how your team writes tests. It doesn’t know that you authenticate in before blocks a certain way, or that you have specific factory traits available, or that you’re not using rails-controller-testing so assigns isn’t an option.

The fix is a conventions system with two parts.

ConventionsExtractor scans your existing test files and pulls out mechanical facts: the most common authentication setup pattern, available factory traits from your factories directory, frequently stubbed objects, whether transactional fixtures are disabled and how cleanup is handled, and whether specific helpers are unavailable based on what’s in your Gemfile. These aren’t judgments, they’re observations extracted directly from the code.

ConventionsSynthesizer takes those raw facts and sends them to the LLM with a prompt asking it to write a concise conventions guide explaining the rule behind each pattern. The result is plain prose. Something like “authentication is handled in before blocks using session[:user_id] = rather than Devise helpers; use this pattern consistently.” That text gets prepended to every generation prompt.

The synthesized guide is cached to spec/conventions.md and invalidated automatically when spec files or Gemfiles change. Regenerating it costs one LLM call.

Enable it with the --conventions flag:

testgenai generate --provider anthropic --model claude-opus-4-7 --conventions

Add spec/conventions.md to your .gitignore. It’s a derived artifact and probably not something to check in.

Fixing the overwrite problem

The overwrite bug was straightforward to diagnose and subtle to fix correctly.

The naive fix would be to skip generation if a spec file already exists. That’s less than ideal. The point is to add tests for untested methods, and partially-tested files are the most common case.

Better behavior is to inject the generated tests into the existing file. The pipeline now reads existing content before doing anything else. It combines existing content with the newly generated code and validates the combined file. If it passes, the combined content is written to the spec file.

If it fails, the pipeline restores the original file exactly as it was and writes the generated-only code to a fallback path. A method in lib/payments/processor.rb that already has a spec/payments/processor_spec.rb would get its fallback at spec/payments/processor_context_spec.rb (using the method name to scope the filename). The failure output tells you where to find it:

  ✗ Payments::Processor#context failed after 3 attempt(s)
    → Generated tests saved to spec/payments/processor_context_spec.rb for manual review

You end up with your original tests intact and the generated attempt sitting somewhere you can look at it and decide what to do.

Running it

Install the gem and point it at your project:

gem install testgenai
cd your_project
testgenai generate --provider anthropic --model claude-opus-4-7

Or add it to your Gemfile in the development group and use bundle exec. Configuration can also come from environment variables:

export TESTGENAI_PROVIDER=anthropic
export TESTGENAI_MODEL=claude-opus-4-7
export ANTHROPIC_API_KEY=your_api_key
testgenai generate --conventions

Three diagnostic commands are available before you commit to a full run:

testgenai scan      # find untested methods without making any API calls
testgenai context   # show what context would be sent to the LLM for each method

scan gives you a picture of your coverage gaps. context is useful for understanding what the LLM will see before spending API credits.

The goal isn’t to replace the developer who understands the code and makes decisions about testing. It’s to handle the mechanical work: setting up describe blocks, wiring test data, writing happy-path coverage. Then you can spend your time on the parts that actually need your judgment. The second project taught us that “mechanical” is more context-dependent than it looks.

Building a gem.

Building MockOpenAI: a weekend MVP story

Last weekend I built and published a Ruby gem. From idea to published thing in about four days. Here’s how it went, including the part where I had to reconsider whether I’d built something useful at all.

Friday: 20 ideas, one bet

I’m between jobs right now. Good position to be in if you like building things, terrible position to be in if you like eating. So I’ve been running a little experiment: each weekend, pick one small idea and see if I can ship it.

Friday’s job was to generate and pick an idea. I sat down with an AI and brainstormed 20 candidates. Developer tools. Content products. Micro-SaaS. I narrowed it down to one: a local mock server for testing OpenAI-compatible APIs.

The pitch to myself was simple. I write a lot of Ruby. I write a lot of tests. Testing code that talks to LLMs is a bit annoying because, while the happy path is easy to mock, some of the failure modes and edge cases can be more of a pain. There had to be a better way.

By end of day Friday I had a repo, a gemspec, and a clear plan.

Saturday: build day

The core idea for MockOpenAI is that it’s a real HTTP server running on localhost, not a mock object or a stub. Your application code talks to it exactly the way it would talk to the LLM provider in production. You just point your client at http://localhost:4000 instead of the usaul API endpoint.

That distinction makes a difference, I think. With a real HTTP server you can test things that object-level mocking can’t touch: actual TCP timeouts, truncated streaming responses, retry headers. The kind of failure modes that bite you in production but never show up in your test suite because you stubbed them away.

The architecture is deliberately simple. A Rack server reads a shared JSON state file on every request. Your tests write rules to that file. The server is stateless. No client wrapping, no monkey-patching, no magic.

Here’s what using it looks like:

it "handles a rate limit", :mock_openai_rate_limit do
  expect { MyService.call_llm("Hello") }.to raise_error(RubyLLM::RateLimitError)
end

it "handles mixed outcomes", :mock_openai do
  MockOpenAI.set_responses([
    { match: "Step 1", response: "OK" },
    { match: "Step 2", failure_mode: :timeout },
    { match: "Step 3", response: "Done" }
  ])

  expect(MyService.step1).to eq("OK")
  expect { MyService.step2 }.to raise_error(Timeout::Error)
  expect(MyService.step3).to eq("Done")
end

The failure modes are:

You can also mix success and failure in a single test:

it "handles mixed outcomes", :mock_openai do
  MockOpenAI.set_responses([
    { match: "Step 1", response: "OK" },
    { match: "Step 2", failure_mode: :timeout },
    { match: "Step 3", response: "Done" }
  ])
end

Saturday was productive. By end of day I had all the core classes written TDD-style: Config, State, Matcher, ResponseBuilder, TemplateRenderer, all five failure mode classes. The code was written, the tests passed, and life was good.

Sunday: documentation and shipping

Sunday was docs day. I set up a Jekyll site and wrote the README. I added an Anthropic endpoint too, because my personal projects use both.

I also migrated the first personal project to use MockOpenAI. That went smoothly. The HTTP-level fidelity made a few tests a little more honest than they’d been with simple stubs at the client level.

Monday: the uncomfortable question

Monday I migrated a second personal project. This one used a helper module I’d written a while back to stub LLM calls. Just a few lines of code. It worked fine for that project.

I stared at that code for a while. Here it is:

module RubyLLMMocks
  def mock_ruby_llm_chat(content: nil, error: nil)
    if error
      allow(RubyLLM).to receive(:chat).and_raise(error)
    else
      mock_response = instance_double(
        RubyLLM::Message,
        content: content,
        inspect: "RubyLLM::Message(content: #{content.inspect})"
      )
      mock_chat = instance_double(RubyLLM::Chat, ask: mock_response)
      allow(RubyLLM).to receive(:chat).and_return(mock_chat)
    end
  end
end

# Example:
it "handles general ruby_llm errors gracefully" do
  error = RubyLLM::Error.new(nil, "Unexpected error")
  mock_ruby_llm_chat(error: error)

  generator = described_class.new(options)

  expect { generator.generate }
    .to output(/Error.*Unexpected error/m).to_stdout
    .and raise_error(SystemExit)
end

That’s it. Fifteen lines, no gem dependency, works perfectly for a project that uses RubyLLM as a wrapper. The error case is handled with and_raise. Clean.

So the question had to be asked: did I just build a solution in search of a problem?

After sitting with it, I don’t think so. But I did have to sharpen my thinking about when MockOpenAI actually earns its place versus when a helper method is the right call.

The short version: if you’re using a wrapper library like RubyLLM for all your LLM calls, and you only need happy-path responses and exception simulation in unit tests, the 15-line helper is probably the right answer. It’s less to maintain, has no extra dependencies, and does the job.

MockOpenAI is the right call when you need the actual HTTP layer in the picture. When you’re using the raw OpenAI or Anthropic client directly. When you’re running integration or system tests that make real HTTP connections. When you need to test what happens when TCP actually times out, or when a streaming response gets cut off halfway through, or when your retry logic processes a Retry-After header.

Those are real problems. They’re just not every project’s problems. I added a when not to use page to the docs to make the tradeoffs explicit.

What I’d do differently

One thing I’d change: I’d research the problem space a bit more to make sure I had a better understanding of problem scope and existing solution. (Especially ones I wrote myself!) The tool is solid, but I made some assumptions about the breadth of problems it would solve for. That’s a classic weekend MVP trap I suppose. You’re so focused on building that you skip the a bit of due diligence you think you don’t need.

The gem is published, the docs are live, and it works. The scope is narrower than I originally thought, but the use case is real. That feels like an honest result for a long weekend.

MockOpenAI is a Ruby gem for testing OpenAI-compatible and Anthropic APIs locally. References:

• Source: github.com/grymoire7/mockopenai.
• Docs: grymoire7.github.io/mockopenai.
• Landing page: tracyatteberry.com/mockopenai.

Documentation search: When your Rails app needs which approach

Imagine spending an afternoon watching a developer tear out a perfectly functional search feature. They replace their solid Postgres full-text search with a vector database and RAG pipeline because, well, that’s what you’re supposed to do now, right? The new system is slower, cost them $200 a month in OpenAI API calls, and returned worse results for their specific use case.

This keeps happening. The AI hype cycle has convinced developers that every search problem needs embeddings, vector databases, and agentic loops. Sometimes that’s true. Often it’s not.

Let’s build the same feature four different ways and see what each approach actually costs you.

The use case: searching Ruby gem documentation

We’re building a search feature for a documentation site that indexes about 5,000 Ruby gems. Each gem has README content, API documentation, and code examples. Users ask questions like “How do I upload files to S3?” or “What’s the best gem for handling webhooks?”

This is a realistic scale for most Rails apps. Not Google-sized, not trivial. Just normal business software that needs to help users find information.

I’ll show you four implementations, each adding a layer of complexity. We’ll look at the code, measure the actual costs, and figure out when the added complexity pays for itself.

Approach 1: Traditional search with AI summarization

Start with what works. Postgres full-text search has been solving search problems since before your junior devs were born.

class Documentation < ApplicationRecord
  include PgSearch::Model
  
  pg_search_scope :search_content,
    against: {
      title: 'A',
      content: 'B',
      code_examples: 'C'
    },
    using: {
      tsearch: { prefix: true },
      trigram: { threshold: 0.3 }
    }
end

class DocumentationSearcher
  def initialize(query)
    @query = query
  end
  
  def search
    results = Documentation.search_content(@query).limit(10)
    
    {
      results: results,
      summary: summarize_results(results)
    }
  end
  
  private
  
  def summarize_results(results)
    return nil if results.empty?
    
    prompt = <<~PROMPT
      User question: #{@query}
      
      Here are the top search results:
      #{format_results(results)}
      
      Provide a concise answer to the user's question based on these results.
      If the results don't contain relevant information, say so.
    PROMPT
    
    client = OpenAI::Client.new
    client = OpenAI::Client.new
    response = client.chat(
      parameters: {
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: prompt }],
        temperature: 0.3
      }
    )

    response.dig(:choices, 0, :message, :content)
  end
  
  def format_results(results)
    results.map.with_index do |doc, i|
      "#{i + 1}. #{doc.title}\n#{doc.content.truncate(500)}"
    end.join("\n\n")
  end
end

This approach does one database query and one API call. The search uses proven Postgres features: full-text search with ranking, trigram matching for typos, and weighted fields. Then we send the top results to GPT-4 to generate a summary.

Cost per query:

• Database: ~5ms
• OpenAI API: ~$0.002 (about 1,000 input tokens, 200 output tokens)
• Total latency: ~800ms

When this fails:

• User queries are conceptually different from how docs are written (“async jobs” versus “background processing”)
• Important information is buried in the middle of long documents
• You need to combine information from multiple sources

The failure mode is subtle. Traditional search ranks by keyword matching and field weights. When users phrase questions differently than your documentation uses terminology, they get poor results. You can’t fix this with better prompt engineering because the LLM never sees the relevant documents.

Approach 2: Basic RAG with vector embeddings

RAG (Retrieval Augmented Generation) means embedding your documents as vectors, embedding the user’s query as a vector, and finding documents with similar embeddings. This solves the terminology mismatch problem.

# We need to store embeddings
class AddEmbeddingsToDocumentation < ActiveRecord::Migration[7.1]
  def change
    add_column :documentations, :embedding, :vector, limit: 1536
    add_index :documentations, :embedding, using: :hnsw, opclass: :vector_cosine_ops
  end
end

class Documentation < ApplicationRecord
  has_neighbors :embedding
  
  after_save :generate_embedding, if: :content_changed?
  
  private
  
  def generate_embedding
    text = "#{title}\n\n#{content}"
    
    client = OpenAI::Client.new
    response = client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    self.update_column(:embedding, response.dig(:data, 0, :embedding))
  end
end

class RagSearcher
  def initialize(query, client: OpenAI::Client.new)
    @query = query
    @client = client
  end

  def search
    query_embedding = generate_embedding(@query)
    results = Documentation.nearest_neighbors(:embedding, query_embedding, distance: "cosine").limit(5)

    {
      results: results,
      answer: generate_answer(results)
    }
  end

  private

  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end

  def generate_answer(results)
    context = results.map { |doc| "#{doc.title}\n#{doc.content}" }.join("\n\n---\n\n")

    prompt = <<~PROMPT
      Answer the user's question based only on the following documentation:

      #{context}

      Question: #{@query}

      If you cannot answer based on the provided documentation, say so clearly.
    PROMPT

    response = @client.chat(
      parameters: {
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: prompt }],
        temperature: 0.3
      }
    )

    response.dig(:choices, 0, :message, :content)
  end
end

Now we’re making two API calls per search: one to embed the query, one to generate the answer. We’re also using pgvector with HNSW indexing for fast similarity search.

Cost per query:

• Database: ~15ms (vector similarity search)
• OpenAI embeddings API: ~$0.00001 (negligible)
• OpenAI chat API: ~$0.003
• Total latency: ~1,200ms

When this works better: The semantic matching is noticeably better. A query about “background jobs” will match documents about “async processing” and “delayed tasks” because the concepts are similar in vector space. This is real improvement over keyword search.

When this still fails:

• Complex questions requiring information from many documents
• Multi-step reasoning (“compare these two approaches”)
• Questions where the first retrieval doesn’t get the right context

Here’s a concrete failure case I hit: A user asks “What’s the difference between Sidekiq and Good Job?” The vector search retrieves five documents, but three are about Sidekiq and two are about Good Job. The LLM tries to compare them but doesn’t have complete information about both systems. It hedges and gives a vague answer.

Approach 3: Agentic RAG with adaptive retrieval

This is where we let the LLM decide if it needs more information. Instead of one retrieve-then-generate pass, we give the LLM tools to search again, rephrase queries, or combine results.

class AgenticRagSearcher
  MAX_ITERATIONS = 3

  def initialize(query, client: OpenAI::Client.new)
    @query = query
    @client = client
    @conversation_history = []
    @retrieved_docs = []
  end
  
  def search
    initial_prompt = <<~PROMPT
      You are a helpful assistant that searches Ruby gem documentation.
      
      User question: #{@query}
      
      You have access to these tools:
      - search_docs(query): Search documentation with a semantic query
      - get_related(doc_id): Get documents related to a specific document
      
      Think step by step. You can search multiple times with different queries
      to gather complete information before answering.
    PROMPT
    
    @conversation_history << { role: "user", content: initial_prompt }
    
    MAX_ITERATIONS.times do
      response = call_llm_with_tools
      
      break if response[:finish_reason] == "stop"
      
      if response[:tool_calls]
        handle_tool_calls(response[:tool_calls])
      end
    end
    
    {
      results: @retrieved_docs.uniq,
      answer: @conversation_history.last[:content],
      iterations: @conversation_history.length
    }
  end
  
  private
  
  def call_llm_with_tools
    response = @client.chat(
      parameters: {
        model: "gpt-4o",
        messages: @conversation_history,
        tools: tool_definitions,
        temperature: 0.3
      }
    )

    message = response.dig(:choices, 0, :message)
    @conversation_history << message

    {
      finish_reason: response.dig(:choices, 0, :finish_reason),
      tool_calls: message[:tool_calls]
    }
  end
  
  def tool_definitions
    [
      {
        type: "function",
        function: {
          name: "search_docs",
          description: "Search documentation using semantic search",
          parameters: {
            type: "object",
            properties: {
              query: {
                type: "string",
                description: "The search query"
              }
            },
            required: ["query"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "get_related",
          description: "Get documents related to a specific document",
          parameters: {
            type: "object",
            properties: {
              doc_id: {
                type: "integer",
                description: "The ID of the document"
              }
            },
            required: ["doc_id"]
          }
        }
      }
    ]
  end
  
  def handle_tool_calls(tool_calls)
    results = tool_calls.map do |tool_call|
      function_name = tool_call.dig(:function, :name)
      arguments = JSON.parse(tool_call.dig(:function, :arguments) || "{}")

      result = case function_name
      when "search_docs"
        search_docs(arguments["query"])
      when "get_related"
        get_related_docs(arguments["doc_id"])
      end

      @retrieved_docs.concat(result)

      {
        role: "tool",
        tool_call_id: tool_call[:id],
        content: format_docs_for_llm(result)
      }
    end

    @conversation_history.concat(results)
  end
  
  def search_docs(query)
    embedding = generate_embedding(query)
    Documentation.nearest_neighbors(:embedding, embedding, distance: "cosine").limit(3)
  end
  
  def get_related_docs(doc_id)
    doc = Documentation.find(doc_id)
    Documentation
      .nearest_neighbors(:embedding, doc.embedding, distance: "cosine")
      .where.not(id: doc_id)
      .limit(3)
  end
  
  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end
  
  def format_docs_for_llm(docs)
    docs.map do |doc|
      {
        id: doc.id,
        title: doc.title,
        content: doc.content.truncate(1000)
      }
    end.to_json
  end
end

This is a real step up in complexity. We’re now orchestrating multiple LLM calls with tool use. The LLM can search multiple times, explore related documents, and build up context before answering.

Cost per query:

• Database: 30-90ms (multiple vector searches)
• OpenAI embeddings API: $0.00002-0.00006 (2-6 embedding calls)
• OpenAI chat API: $0.015-0.045 (3-5 LLM calls with larger context)
• Total latency: 3-8 seconds

Notice the variance. Some queries get answered in one iteration. Complex ones might do three searches with different phrasings, explore related documents, and make five total LLM calls.

When this works better: That comparison query from before (“What’s the difference between Sidekiq and Good Job?”) now works great. The LLM searches for “Sidekiq background jobs”, gets those docs, then searches for “Good Job background jobs”, gets those docs, then synthesizes a real comparison.

Multi-part questions work too. “How do I set up Stripe payments and handle webhooks?” triggers two separate searches that gather comprehensive information.

When this gets expensive: Every query where the LLM decides it needs more information costs you 3-5x more. If your users ask a lot of complex questions, your API bill climbs fast.

The latency is also noticeable. Eight seconds feels slow in a web UI. You need to stream responses or show progress indicators.

Approach 4: Full conversational agent with external tools

Now we’re building a real agent that can search your documentation, browse external sites, and maintain conversation context across multiple turns.

class DocumentationAgent
  def initialize(session_id, client: OpenAI::Client.new)
    @session_id = session_id
    @client = client
    @conversation_history = load_conversation_history
  end
  
  def chat(message)
    @conversation_history << { role: "user", content: message }
    
    loop do
      response = call_llm_with_tools
      
      break if response[:finish_reason] == "stop"
      
      if response[:tool_calls]
        handle_tool_calls(response[:tool_calls])
      else
        break
      end
    end
    
    save_conversation_history
    
    {
      response: @conversation_history.last[:content],
      sources: extract_sources
    }
  end
  
  private

  def call_llm_with_tools
    response = @client.chat(
      parameters: {
        model: "gpt-4o",
        messages: @conversation_history,
        tools: tool_definitions,
        temperature: 0.3
      }
    )

    message = response.dig(:choices, 0, :message)
    @conversation_history << message

    {
      finish_reason: response.dig(:choices, 0, :finish_reason),
      tool_calls: message[:tool_calls]
    }
  end

  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end

  def tool_definitions
    [
      {
        type: "function",
        function: {
          name: "search_internal_docs",
          description: "Search our Ruby gem documentation",
          parameters: {
            type: "object",
            properties: {
              query: { type: "string" }
            },
            required: ["query"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "fetch_external_url",
          description: "Fetch content from an external URL like GitHub or RubyGems.org",
          parameters: {
            type: "object",
            properties: {
              url: { type: "string" }
            },
            required: ["url"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "search_github",
          description: "Search GitHub repositories and code",
          parameters: {
            type: "object",
            properties: {
              query: { type: "string" }
            },
            required: ["query"]
          }
        }
      }
    ]
  end
  
  def handle_tool_calls(tool_calls)
    results = tool_calls.map do |tool_call|
      function_name = tool_call.dig(:function, :name)
      arguments = JSON.parse(tool_call.dig(:function, :arguments) || "{}")

      result = case function_name
      when "search_internal_docs"
        search_internal_docs(arguments["query"])
      when "fetch_external_url"
        fetch_external_url(arguments["url"])
      when "search_github"
        search_github(arguments["query"])
      end

      {
        role: "tool",
        tool_call_id: tool_call[:id],
        content: result.to_json
      }
    end

    @conversation_history.concat(results)
  end
  
  def search_internal_docs(query)
    embedding = generate_embedding(query)
    docs = Documentation.nearest_neighbors(:embedding, embedding, distance: "cosine").limit(5)

    docs.map { |d| { title: d.title, content: d.content.truncate(800), source: "internal", id: d.id } }
  end
  
  def fetch_external_url(url)
    # In production, use a proper HTTP client with timeouts and error handling
    response = HTTP.timeout(5).get(url)
    
    {
      url: url,
      content: extract_main_content(response.body.to_s).truncate(2000),
      source: "external"
    }
  rescue HTTP::Error => e
    { error: "Failed to fetch URL: #{e.message}" }
  end
  
  def search_github(query)
    # Use Octokit or similar
    client = Octokit::Client.new(access_token: ENV['GITHUB_TOKEN'])
    results = client.search_code(query, language: "ruby")
    
    results.items.first(3).map do |item|
      {
        name: item.name,
        repo: item.repository.full_name,
        url: item.html_url,
        source: "github"
      }
    end
  rescue Octokit::Error => e
    { error: "GitHub search failed: #{e.message}" }
  end
  
  def load_conversation_history
    cache_key = "agent_conversation:#{@session_id}"
    JSON.parse(Rails.cache.read(cache_key) || "[]")
  end
  
  def save_conversation_history
    cache_key = "agent_conversation:#{@session_id}"
    # Keep last 10 messages to control context size
    trimmed_history = @conversation_history.last(10)
    Rails.cache.write(cache_key, trimmed_history.to_json, expires_in: 1.hour)
  end
  
  def extract_sources
    @conversation_history
      .select { |msg| msg[:role] == "tool" }
      .flat_map { |msg| JSON.parse(msg[:content] || "[]") }
      .select { |item| item.is_a?(Hash) && item["source"] }
      .uniq { |item| item["id"] || item["url"] || item["title"] }
  end
end

This is a different beast. We’re maintaining conversation state, hitting external APIs, and letting the LLM orchestrate complex research tasks.

Cost per conversation turn:

• Database: 15-50ms
• External API calls: 200-2000ms (GitHub, external sites)
• OpenAI embeddings: $0.00001-0.00005
• OpenAI chat: $0.02-0.15 (larger context windows, multiple turns)
• Total latency: 5-15 seconds

When this is worth it: You’re building a research assistant or technical support bot where users have complex, multi-turn conversations. They ask follow-up questions, need you to check external sources, and expect the system to remember context.

A user might ask “What’s the best gem for image processing?”, then follow up with “Show me examples from the ImageMagick wrapper”, then “Is there a more modern alternative?” The agent maintains context and can search different sources for each question.

When this is overkill: Most search features. If users are doing one-off queries and moving on, you’re paying for conversational capabilities they don’t need.

Choosing your approach

I’ve built all four of these systems in production. Here’s how I decide which to use.

Start with enhanced traditional search if:

• You have well-written documentation with consistent terminology
• Queries are mostly straightforward lookup tasks
• You need predictable costs and latency
• Your document corpus is under 10,000 items

The cost difference matters. At 1,000 queries per day, enhanced traditional search costs you $2/day. Basic RAG costs $3/day. Agentic RAG costs $15-45/day. A full agent costs $100-300/day.

Move to basic RAG when:

• Users phrase questions differently than your docs
• Keyword search returns poor matches for valid queries
• You have good quality source documents
• Your corpus is large enough that keyword search becomes unwieldy (50,000+ documents)

You’ll know you need this when users complain that search doesn’t work, and you look at their queries and think “we have docs about that, but they’re using different words.”

Move to agentic RAG when:

• Users ask complex questions requiring multiple sources
• You see patterns of users doing multiple searches in sequence
• Simple RAG returns incomplete answers
• You have budget for 3-5x higher API costs

Watch your analytics. If users do three searches in a row and then give up, they’re manually doing what an agentic system would do automatically.

Build a full agent when:

• You’re building a product feature, not just search
• Users need multi-turn conversations with context
• You need to integrate external data sources
• You have engineering resources for proper tool integration and error handling

The engineering complexity here is significant. You need proper timeout handling, retry logic, conversation state management, and graceful degradation when external APIs fail. This is a feature, not a quick enhancement.

The implementation details that matter

Some practical considerations that aren’t obvious from the code samples.

Chunking strategy for vector search: Don’t just embed entire documents. Break them into logical chunks. For documentation, consider chunking by section with overlap:

class DocumentationChunker
  CHUNK_SIZE = 1000 # characters
  OVERLAP = 200
  
  def chunk(document)
    sections = document.content.split(/^##\s+/)
    
    sections.flat_map do |section|
      break_into_overlapping_chunks(section)
    end
  end
  
  private
  
  def break_into_overlapping_chunks(text)
    chunks = []
    start = 0
    
    while start < text.length
      chunk_end = start + CHUNK_SIZE
      chunks << text[start...chunk_end]
      start += CHUNK_SIZE - OVERLAP
    end
    
    chunks
  end
end

This means each document generates multiple rows in your database with different embeddings. Your vector search returns chunks, not whole documents.

Hybrid search combines the best of both:

def hybrid_search(query)
  # Keyword search results
  keyword_results = Documentation.search_content(query).limit(20)
  
  # Vector search results
  query_embedding = generate_embedding(query)
  vector_results = Documentation.nearest_neighbors(:embedding, query_embedding, distance: "cosine").limit(20)
  
  # Combine with reciprocal rank fusion
  combine_results(keyword_results, vector_results)
end

def combine_results(keyword_results, vector_results)
  scores = Hash.new(0)
  
  keyword_results.each_with_index do |doc, i|
    scores[doc.id] += 1.0 / (i + 60)
  end
  
  vector_results.each_with_index do |doc, i|
    scores[doc.id] += 1.0 / (i + 60)
  end
  
  Documentation.where(id: scores.keys).sort_by { |doc| -scores[doc.id] }
end

This gives you semantic understanding from vectors plus precise keyword matching. The reciprocal rank fusion formula is from research on combining search results. It works better than naive score addition.

Caching saves you money:

class CachedRagSearcher
  def search(query)
    cache_key = "rag_search:#{Digest::MD5.hexdigest(query)}"
    
    Rails.cache.fetch(cache_key, expires_in: 1.hour) do
      perform_search(query)
    end
  end
end

Popular queries get asked repeatedly. Cache the embeddings and the LLM responses. This cuts your API costs dramatically for common questions.

The above caching method uses the raw query as the cache key. If users ask the same question with slightly different wording, they won’t hit the cache. You might want to normalize queries before hashing them for the cache key. For example, you could lowercase the query, remove stop words, etc.

Monitor your failure modes:

class SearchMetrics
  def self.track(query, approach, results)
    SearchLog.create!(
      query: query,
      approach: approach,
      result_count: results.length,
      latency_ms: results[:latency],
      cost_cents: calculate_cost(results),
      user_clicked: false # updated when user clicks a result
    )
  end
end

Track which results users actually click. If they click the first result, your search works. If they reformulate their query three times, it doesn’t. This data tells you whether to upgrade your approach.

What I actually recommend

Build the simplest thing first. Most Rails apps should start with Postgres full-text search plus GPT summarization. It costs almost nothing, has predictable latency, and works fine for straightforward queries.

Add instrumentation immediately. Track user behavior, measure latency, and watch your API costs. You need this data to know if upgrading is worth it.

When you see concrete evidence that simple search fails for your use case, add vector embeddings. This is a real improvement for semantic search. The pgvector extension makes this straightforward in Postgres. You don’t need a separate vector database until you have millions of documents.

Only add agentic features when you can point to specific query patterns that need them. “Users ask comparison questions and we don’t have comparison docs” is a good reason. “Agents are cool and I want to try them” is not.

Save full conversational agents for when you’re building a product feature that needs conversations. This is engineering work, not just adding a library. Budget for it appropriately.

The hype cycle pushes developers toward complexity. Resist it. Your users don’t care about your architecture. They care about getting answers quickly and cheaply. Often the simplest approach gives them exactly that.

But does the code work? See this repo for an implementation of all four approaches.

Disclaimer: The code samples in this post are simplified for clarity. In production, you need proper error handling, timeouts, retries, and security considerations (e.g., sanitizing user input before embedding). The costs and latencies mentioned are estimates based on recent OpenAI pricing and typical response times; your actual costs may vary based on usage patterns and model choices. Always monitor your API usage and costs when deploying AI features. Void where prohibited.

The debate about how much credit to give AI tools for their contributions to software development has been going on for a while now. I only recently, however, became aware of how academic publishers have been handling this issue in the context of research papers.

A number of academic publishers have policies stating that 1) use of AI tools must be disclosed, and 2) AI tools cannot be listed as authors. Here’s an example of the latter from Elsevier Books:

Authors should not list generative AI and AI-assisted technologies as an author or co-author, nor cite AI as an author. Authorship implies responsibilities and tasks that can only be attributed to and performed by humans…[1]

There are good arguments on both sides of the debate, but I tend to agree with the spirit of this policy and would extend it to software development as well.

If I submit code, I am responsible for that code. Full stop. The AI is not an author, it’s not a co-author, and it doesn’t get a byline. I don’t want to be able to say “well, the AI wrote that part, so it’s not really my fault if it has a bug.” If I use an AI tool to generate code, I am responsible for reviewing that code, making sure it meets the coding standards, and being able to defend and explain it to my teammates. If I vibe code a commit and send it off to be reviewed by my team, I am now shifting all of the responsibility for code quality onto them. That creates cognitive debt and burns trust.

At the same time, I think it’s important to be transparent about the use of AI tools. It’s not something to be ashamed of or hide, any more than I’d hide the fact that I used a code generator or a linter. These days it’s a good bet that AI is involved at some point, and I honestly may forget to point that out every time. Maybe I should do better in that area, but I don’t want to pedantic about it. I just want to make sure that when I do use AI, I’m taking responsibility for the output.

Used well, AI can clear space for better judgment. Used poorly, it can move more work onto others and/or create more cognitive debt.

Note: I usually have an AI look over my writing or even brainstorm the initial draft, but I didn’t do that this time. However, I will admit that Copilot was on in my editor and I spent a lot of time dismissing bad auto-complete suggestions before I finally turned it off. Okay, hopefully that’s enough transparency for this short post.

[1] https://libguides.tamusa.edu/AI_for_lit_research/journal_requirements

I was excited to try the new Cowork feature in Claude. The idea of a persistent, safe workspace where I can collaborate with an AI over time sounded great. I set up my space, pointed it at my screenshots directory, and asked Claude to help me organize it. It actually worked pretty well. Claude understood my files, suggested a new folder structure, and moved everything around for me. Cool. What next?

I noticed a sample task offering to help build a background creator based on biological principles. That sounded fun, so I gave it a shot. Claude got started and then… nothing. Some of the presets just didn’t work at all.

I asked Claude to fix it. It tried, and failed, repeatedly. The problem was it had no way to see what the output actually looked like, so I tried to give it a tool to help. Unfortunately, it couldn’t install the tool — no network access — and it also couldn’t reach a locally installed version outside the Cowork VM.

The pattern that emerged was pretty clear. For tasks where Claude can get feedback from the files in its workspace, it works pretty well. For tasks like the background creator, where it needs to see rendered output, it falls flat. I get the safety rationale behind the sandboxing, but I hope they can improve the tooling enough to at least make their own examples work.

When “just return JSON” isn’t enough

If you’re building a tool that calls an AI API and needs structured data back, at some point you’ll write a prompt that ends with something like: “Return a JSON array. No prose, no explanation, just the JSON.”

And most of the time, that works. But “most of the time” isn’t good enough when a parse error crashes your command mid-run.

I ran into this while building Jojo, a CLI tool that generates job application materials (tailored resumes, cover letters, and a custom landing page) using AI. Several of its commands need structured data back from the model: an array of annotations, an array of FAQ entries, an array of integer indices. Plain text won’t do; the output feeds directly into the next stage of the pipeline.

The problem with prompt instructions

Language models are trained to be helpful. Part of being helpful, from the model’s perspective, is adding context: explaining what it did, noting any assumptions, formatting output to be readable. A prompt instruction to “return only JSON” fights against that training.

The failure modes are predictable:

Here is the JSON array you requested:

```json
[{"question": "What is your experience with Ruby?", "answer": "..."}]
```

I've formatted this as valid JSON with the required fields.

The model followed the spirit of the instruction (it returned valid JSON) but wrapped it in markdown fences and a helpful introductory sentence. JSON.parse on that string raises a ParserError.

The original code defended against this with a quick fence-strip:

cleaned_json = json_string.strip
  .gsub(/\A```(?:json)?\n?/, "")
  .gsub(/\n?```\z/, "")
JSON.parse(cleaned_json, symbolize_names: true)

That handles the fence case. But what about the introductory sentence? What about a model that adds a trailing note after the JSON? The regex grows, cases multiply, and you end up with the same defensive code copy-pasted across three different generators.

JsonExtractor

Instead, we pull this logic into a single place and make it more robust by trying multiple strategies to just find the JSON in whatever the model returned.

module JsonExtractor
  def self.call(content, symbolize_names: false)
    return content if content.is_a?(Hash) || content.is_a?(Array)

    stripped = content.gsub(/\A```\w*\s*|\s*```\z/m, "").strip

    try_parse(content, symbolize_names: symbolize_names) ||
      try_parse(stripped, symbolize_names: symbolize_names) ||
      extract_first_structure(content, symbolize_names: symbolize_names) ||
      raise(JSON::ParserError, "No JSON object found in response")
  end
end

It tries three strategies in order:

1. Parse the raw response directly. If the model actually followed instructions, done.
2. Strip markdown fences and try again. Handles the most common failure mode.
3. Find the first { or [ in the response and parse the complete structure from there, using a depth-tracking parser to handle nested objects correctly. This handles the “Here is the JSON: […]” case regardless of what prose surrounds it.

Each generator now calls JsonExtractor.call(response) instead of rolling its own fence-stripping. One test suite covers all the edge cases.

The cleaner alternative

If you’re using RubyLLM, a Ruby library that wraps multiple AI providers behind a common interface, there’s a more architecturally elegant solution: structured output.

Instead of asking the model to “return JSON” and then parsing whatever it sends back, you define a schema and ask the model to conform to it:

class FaqSchema < RubyLLM::Schema
  array :faqs do
    string :question
    string :answer
  end
end

response = chat.with_schema(FaqSchema).ask(prompt)
# response.faqs is an array of objects with .question and .answer

The model is constrained at the API level, not the prompt level. You get back a parsed object that already matches your schema. No JSON.parse, no fence-stripping, no JsonExtractor. The schema also serves as documentation for what the command expects.

This is genuinely cleaner. So why not use it?

The provider support problem

Structured output is a feature that providers implement, and not all models support it. The major providers (Anthropic, OpenAI, Gemini) support it for their recent models. But older models, like GPT-3.5, Gemini 1.0, Codex, etc., don’t. Some providers, like Hugging Face, don’t support it at all.

Jojo is designed to work with whatever model and provider the user configures. It dynamically picks up provider credentials from environment variables and routes requests through RubyLLM. A user might be running Claude Haiku, GPT-4o, or a locally-hosted model through an OpenAI-compatible endpoint.

RubyLLM’s structured output interface is uniform across providers: chat.with_schema works the same way regardless of which model you’re talking to. But when the underlying model doesn’t support structured output, you don’t get graceful degradation. You get a provider error.

That’s the thing:

• JsonExtractor works on any string. If the model returns prose with JSON buried in it, that’s handled. If the model returns clean JSON, that’s handled. It never throws a provider error because it never makes a structured output request.
• Structured output fails loudly on unsupported models. There’s no fallback. You have to catch the error and decide what to do with it.

For a tool where wide model support is a goal, JsonExtractor is more defensive. The cost is that you’re still at the mercy of whatever the model decides to return, but in practice, every model returns something that contains valid JSON, even if it’s surrounded by other content. The extraction strategy covers the real-world failure modes.

Okay, you also give up strict JSON schema guarantees (if supported) but that’s a tradeoff I’m willing to make for broader compatibility.

When structured output is the right call

None of this is an argument against structured output in general. If you have good control over exactly which model and provider your code will run against, structured output is the better choice. The schema and be explicit, the contract is enforced at the API level, and you’re unlikely to have a parse error at runtime.

You might wonder whether a hybrid approach could get the best of both worlds: check whether the model supports structured output, use it if so, and fall back to JsonExtractor if not. RubyLLM does expose this information. Model objects carry a capabilities array that includes 'structured_output' when the feature is supported. So the check is technically feasible.

The problem is that it doesn’t simplify anything. JsonExtractor still has to exist for the fallback branch, so you haven’t reduced maintenance burden. And now you have two code paths returning different shapes: structured output returns a typed schema object, while JsonExtractor returns a Hash. Every caller has to handle both, or you need a normalization layer on top. You’ve added complexity without removing any.

The tradeoff is essentially: how much do you want to trade portability for guarantees? JsonExtractor is portable and handles the real-world messiness of model output. Structured output gives you stronger guarantees but requires a supported model.

For a tool like Jojo, where users configure their own API credentials and model choices, JsonExtractor keeps things working across the widest range of setups. The “clean” architecture turned out to be less pragmatic once you considered who was actually going to run it. That person is me, btw, and I want it to work with whatever model I’m testing with at the moment, without having to worry about structured output support.

Building Jojo: turning job applications into marketing campaigns

When you apply for a job, you’re competing against hundreds of other candidates. Most of them submit a resume and a cover letter. The ambitious ones tailor those documents to the role. And then everyone waits.

No matter how good your resume is, it’s still a PDF in a pile of PDFs. You’re asking a hiring manager to do the work of figuring out why you’re a fit. What if you did that work for them?

That’s the idea behind Jojo, a Ruby CLI I built to transform job applications into personalized marketing campaigns. Instead of sending documents, you send a package: a tailored resume, a cover letter informed by company research, and a dedicated landing page that shows exactly why you’re a match for the role.

The landing page is the centerpiece. It’s a mini marketing site with an annotated job description that maps your experience to their requirements, portfolio projects selected for relevance to their tech stack, a branding statement written for their company, LinkedIn recommendations, an FAQ section, and a call-to-action to schedule a conversation. It turns a passive application into an active pitch.

Think of it as treating each job application like a product launch. You’re the product. The company you’re applying to is the only customer. Jojo builds the marketing campaign.

How it works

The workflow starts with two inputs: your resume data (a structured YAML file) and a job description (a file or URL). From there, Jojo runs a pipeline of AI-powered generation steps.

# Create a new application workspace
jojo new --slug acme-senior-dev --job posting.txt

# Generate everything
jojo generate --slug acme-senior-dev

The generate command kicks off a sequence:

1. Research — AI analyzes the job description and (optionally) searches the web to build a research document about the company, the role, and how to position yourself.
2. Resume — Your structured resume data is curated and rendered into a tailored resume, emphasizing the most relevant experience.
3. Branding — AI writes a personal branding statement specific to the company and role.
4. Cover letter — Generated from the research and tailored resume, so it references specific things about the company rather than generic platitudes.
5. Annotations — The job description is analyzed requirement by requirement, with each one mapped to your matching experience.
6. FAQ — AI generates role-specific questions and answers based on your background and the job requirements.
7. Website — Everything comes together in a self-contained landing page.
8. PDF — Resume and cover letter are converted to PDF via Pandoc.

Each step feeds into the next. The research informs the resume tailoring. The resume informs the cover letter. The annotations and FAQ feed into the website. It’s a pipeline, not a collection of independent scripts.

Every application gets its own workspace directory organized by slug:

  applications/acme-senior-dev/
  ├── job_description.md
  ├── job_details.yml
  ├── research.md
  ├── resume.md
  ├── cover_letter.md
  ├── branding.md
  ├── faq.json
  ├── job_description_annotations.json
  ├── status.log
  └── website/
      └── index.html

For daily use, there’s also an interactive TUI mode. Running jojo with no arguments launches a dashboard that shows all your applications, tracks which steps are complete, detects when artifacts are stale (because you regenerated a dependency), and lets you generate or regenerate individual steps with a keypress. The staleness detection uses file modification times. If you regenerate your research, the dashboard knows your resume is now stale because it was built from the old research.

  ┌─ Jojo ────────────────────────────────────────────┐
  │  Active: acme-senior-dev                          │
  │  Company: Acme Corp  •  Role: Senior Developer    │
  ├───────────────────────────────────────────────────┤
  │  Workflow                           Status        │
  │  1. Job Description            $   ✓ Generated    │
  │  2. Research                   $   ✓ Generated    │
  │  3. Resume                     $   * Stale        │
  │  4. Cover Letter               $   ○ Ready        │
  │  ...                                              │
  ├───────────────────────────────────────────────────┤
  │  [1-9] Generate item    [a] All ready    [q] Quit │
  └───────────────────────────────────────────────────┘

The $ indicator shows which steps call paid APIs, so you know if an action will cost something before you press the key. Steps that just combine existing artifacts (like website generation) are free.

Architecture: the command pipeline

Jojo is over 5K lines of Ruby source across ~50 source files. Most CLI commands follow the same three-file pattern:

  lib/jojo/commands/{command_name}/
  ├── command.rb    — Orchestration: validates inputs, manages file I/O
  ├── generator.rb  — Content generation: builds context, calls AI
  └── prompt.rb     — AI prompts: system and user prompt templates

So when I need to add a new command, I can create these three files, follow the pattern from the existing commands, and it (hopefully/usually) works. I don’t have to modify a central router or understand the internals of unrelated commands. The pattern helps make the codebase predictable. If you’ve read one command, you understand the shape of all of them. That helps the human and the AI assistant.

This wasn’t the original architecture. The CLI started as a monolith in cli.rb. Thor command definitions were mixed with validation logic, file handling, and generation orchestration. It worked fine for the first few commands, but soon things got messy. Adding a new feature meant navigating a growing code heap and hoping your changes didn’t break something unrelated.

The refactor extracted each command into its own module with a shared base class that provides common behavior (slug resolution, config loading, AI client setup). The CLI file shrank to a thin router with about 150 lines of small methods that delegate to command classes. Interactive mode, which breifly had a circular dependency calling back into the CLI class (eww) now calls command classes directly through a simple adapter.

Dual AI models

Jojo configures two AI models. There’s a reasoning model for complex tasks and a text generation model for simpler ones.

Company research and resume tailoring need the strongest reasoning capabilities as they’re analyzing job requirements, cross-referencing your experience, and making judgment calls about relevance. But extracting metadata from a job description (company name, location, job title) is easier. Using a powerful model for that is like hiring a senior architect to hang shelves.

The reasoning model handles research, resume curation, and cover letter writing. The text generation model handles job description processing, annotations, FAQ generation, and branding statements. Both models are configurable per provider, so you can use a frontier model for reasoning and a faster model for text generation, or whatever suits your budget and quality needs.

Even with the right model architecture, the AI still has a fundamental trustworthiness problem when it comes to factual content (welcome to AI).

Solving the hallucination problem

This was a technical decision that came from a hard fail.

The original resume generation would take the user’s resume data (stored as structured YAML), combine it with the job description and research, and ask the AI to generate a tailored resume in markdown. The prompt included extensive instructions about not fabricating information. It said things like “only include skills the candidate actually has” and “do not add technologies not present in the source data.”

The AI ignored these instructions way too often. I’d review a generated resume and find “Kubernetes” listed in my skills because the AI noticed I mentioned Docker and helpfully inferred I must know Kubernetes too. Or it would embellish a job description with responsibilities I never had. For a resume, this is not good.

The first instinct was to add more guardrails to the prompt. More emphatic instructions. More examples of what not to do. This helped a little, but it didn’t solve the problem. The AI still had the ability to modify anything, and language-level instructions are suggestions, not constraints.

The insight: different fields have different risk profiles

A professional summary should be rewritten for each role. That’s the whole point, but a list of programming languages must not be modified. The years you worked at a company are facts. Your name is your name.

The problem was that “AI shouldn’t have the same permissions everywhere.” Some fields need smart tailoring. Others need strict preservation. And still others might be removed or reordered. The idea was to define a permission system that specifies what the AI can for different kinds of content.

Permission-based curation

The solution was a permission system embedded directly in the resume data:

name: "Bob Denver"               # default: read-only
email: "bob@example.com"         # default: read-only

summary: |                       # permission: rewrite
  Polyglot developer who enjoys solving problems
  with software...

skills:                          # permission: remove, reorder
  - software engineering
  - full stack development
  - AI assisted development

languages:                       # permission: reorder
  - Ruby
  - Java
  - Python
  - Go

experience:                      # permission: reorder
  - company: "Island Adventures Inc."
    role: "Senior Software Engineer"
    start_date: "2020-07"        # read-only (nested)
    description: |               # permission: rewrite
      Full-stack developer delivering a SaaS platform...
    technologies:                # permission: remove, reorder
      - Ruby on Rails
      - Vue
      - Python
      - Docker

Four permission levels:

• read-only (default) — AI cannot modify, delete, add, or reorder. Contact info, dates, company names.
• remove — AI can exclude irrelevant items but can’t modify the ones it keeps. A database skill list can drop SQLite if the role is all PostgreSQL.
• reorder — AI can prioritize by relevance but can’t remove or modify. Your programming languages list stays complete but puts the most relevant ones first.
• rewrite — AI can generate new content using the original as a factual baseline. Professional summary, job descriptions.

In particular, the AI should never add items that aren’t in the source data. Though there is still a risk of hallucination in rewrite fields, the presence of original content in smaller chunks provides a grounding that makes it less likely.

Two-pass pipeline

The curation happens in two passes:

Pass 1: Filter and reorder. The AI receives the full resume data and the job description. It returns a filtered, reordered version that respects the permissions on each field. Skills marked remove, reorder get filtered to ~70% of the most relevant items and sorted by relevance. Lists marked reorder get sorted but all items are preserved.

Pass 2: Rewrite fields. The AI receives the filtered data and generates new content for fields marked rewrite. For example, the professional summary and experience descriptions. It uses the original content as a factual baseline.

Then an ERB template renders the final markdown. The template handles structure and formatting. The AI never touches the output templating.

What makes this work as an engineering solution is that the Ruby code enforces the permissions where possible. If the AI returns a reordered list that’s shorter than the original for a field that only has reorder permission, the Transformer class raises a PermissionViolation error:

unless can_remove
  if indices.length != original_count
    raise PermissionViolation,
      "LLM removed items from reorder-only field: #{field_path}"
  end
end

The permissions are no longer buried in prompt instructions that the AI might ignore. They’re enforced in code. The AI provides suggestions for how to curate the data, and the Ruby code validates those suggestions against the permission rules before applying them. If the AI tries to exceed its permissions, the operation fails rather than silently producing a resume with fabricated content.

The result is a skills section always contains skills I actually have. My job dates are always accurate. But my professional summary is tailored for each role, emphasizing the experience most relevant to that specific position.

What structured data enables

In order to make the permission system work, we had to switch from an unstructured markdown resume to a structured YAML format. This was a significant architectural change and it required reworking the entire resume generation pipeline. However, it was necessary to address the hallucination problem.

The permission system is the most visible benefit of using structured data, but there are other advantages:

• Narrower AI focus — With structured data, the AI can focus on curating specific fields rather than trying to parse and understand a free-form markdown document. This leads to better quality and more consistent results.
• Better output control — The ERB template handles formatting and structure, so the AI only generates smaller pieces of content. This reduces the chances of formatting errors or hallucinated sections and increases the human control over the final output.
• Easier testing — Structured data is easier to work with in tests. You can create synthetic resume data with specific permissions and verify that the output respects those permissions. With unstructured markdown, it’s harder to assert that the AI didn’t add or modify content it shouldn’t have.

Testing as a development discipline

A permission system that enforces constraints in code is only trustworthy if you actually test the enforcement. Jojo has 530 tests across two tiers, with 84% code coverage. Getting there was an intentional investment.

AI coding assistants are enthusiastic about writing features. They’re less enthusiastic about writing tests. This mirrors human tendencies. Tests aren’t as exciting as shipping the next feature, but with AI-assisted development the gap is amplified.

When the first large refactor was needed I noticed that test coverage was sitting at 31%. The code worked, but I had no safety net for refactoring. The push to 84% was a conscious decision to invest in change enablement.

Three kinds of tests

Jojo has three kinds of tests:

All tests run on every ./bin/test (or rake test:all) invocation and in CI.

Testing API-dependent code

The trickiest part of testing Jojo is that a lot of interesting work involves AI and Search API calls. You can’t run those in CI without spending money on every test run, but you also want tests that exercise real response parsing.

The solution was the VCR gem. VCR records real HTTP interactions the first time a test runs and saves them as “cassettes.” On subsequent runs, it replays the recorded responses instead of making real API calls. You get fast, deterministic tests that still exercise the full response-parsing pipeline.

Fixture discipline

One rule that has saved me more than once is that tests (and AI) never touch the inputs/ directory — no matter how much AI would like to. That directory contains real resume data from the user. Tests use test/fixtures/ exclusively, with synthetic data designed for testability.

This is codified in the project’s AI guidelines, which was previously prone to such mistakes. The instructions are explicit, emphatic, and took a few iterations to be effective. This testing discipline was part of the broader experience of building with AI.

Building with AI

There’s a meta quality to this project: it’s a tool that uses AI to generate content, and it was built with AI assistance. Both Claude and Z helped with development.

AI is pretty good at generating boilerplate, brainstorming design alternatives, and automating the tedious parts of refactoring (like updating 50 files when you rename a class).

But the decisions this post is about — the curation system, the architecture, the decision to refactor and when, the test organization — those were human decisions (as was the choice to use em-dashes just then). AI helped implement them a bit faster, but it didn’t tell me they were needed.

One nice thing about AI-assisted development was the ability to explore approaches quickly. When I was designing the permission system, I could describe different architectures, brainstorm, and get working prototypes, all in fairly short order. That kind of rapid experimentation is really helpful. The design thinking, however, still has to be yours.

One not-so-nice thing was needing to prod the AI to write tests for the features it’s helping to build. Also, let’s be honest, there’s a temptation to let the AI go a little too long before reviewing its output. Left to its own devices, an AI assistant will happily build feature after feature, with no test coverage and growing technical debt. Just like a human developer on a deadline, it needs someone to say “we’re not adding anything else until we address the technical debt, and that includes tests.”

What I learned and what’s next

A few things I’d do differently if I started over:

Start with structured data sooner. The original design used a free-form markdown resume as input. This was a frightful battle of prompt engineering from the beginning. The switch to structured YAML data (resume_data.yml) was the right call, but it required reworking the entire resume generation pipeline. If I’d started with structured data, the permission system would have been a natural extension rather than a redesign.

Build the interactive mode earlier. The TUI dashboard made the tool dramatically more usable, but it came in Phase 6 out of 7. Earlier access to the dependency graph and staleness detection would have improved my own workflow during development.

Force TDD from the start, or very near it. I had a test suite from the beginning, of course, but it wasn’t until I hit a major refactor that I made a conscious decision to invest in better test coverage. If I had enforced TDD from the start.

Basically, I would have spent a lot more time up front on planning the architecture and testing strategy, which would have made the development process smoother and more maintainable. AI assistance can be great, but it’s also really good at seducing you into bad habits.

What’s next

A few potential things for the roadmap:

• Interview prep generation — STAR-method examples drawn from your resume data, tailored to the specific role
• More and better theming options — The landing page is Jojo’s UVP, but the current design is pretty basic. More themes and customization options would let users create a landing page that better reflects their personal brand.
• Application tracking — Status tracking across all applications with dates, notes, and follow-up reminders
• Full SaaS product — A Rails app version of Jojo with a user-friendly interface and full job search management features — this would be a much bigger project but could help a wider audience.

Try it out

Jojo is open source and available on GitHub with a documentation site. It’s a Ruby CLI that requires AI and Search provider API keys. Setup takes just a few minutes.

If you’re interested in the code, the architecture, or just want to talk about AI-assisted development, I’d enjoy hearing from you. You can find me on LinkedIn or Mastodon.