Why I built Speechwave in Elixir (as a Rails developer)

I’ve been developing with Ruby on Rails for a long time. If you had asked me to bet on what I’d build my next web app with, I would have bet on Rails.

Then I started building Speechwave, a live emoji reaction app for conference talks, and I wrote it in Elixir instead

This isn’t a “Rails is dead” post. It’s still a great choice for lots of things. This is about what this particular app needed, why Elixir turned out to a great choice, and why the hosting bill is about as small as a bill can get. If you’re an Elixir developer, some of this may be familiar.

What Speechwave does

Attendees scan a QR code at a talk and are sent to a page of emoji buttons. They tap one. A Chrome extension on the speaker’s laptop receives that reaction and floats the emoji over the speaker’s Google Slides in real time. After the talk, the speaker can see which slides got which reactions.

Look at that from a server’s point of view. It’s a lot of small, short-lived, simultaneous things:

• Dozens (or hundreds) of phones holding open WebSocket connections
• Each tap needs to be rate limited, optionally saved, and fanned out to everyone who cares
• The speaker’s extension holds one more long-lived connection, and it isn’t even a web page
• The whole thing is quiet for days, then very busy for 45 minutes

That last point will matter quite a bit when we get to the hosting section.

Why it works: the three things I would have had to do in Rails

I don’t want to oversell this. Rails can do real-time. Action Cable exists, and Rails keeps improving. But in Rails I would have been assembling pieces. In Phoenix, the pieces come in the box and they were designed together.

1. A message bus that doesn’t care who’s listening

Phoenix PubSub lets any process broadcast to a topic and any other process subscribe to it. The attendee’s LiveView doesn’t know the Chrome extension exists. After the rate limit check (and saving the reaction, if a session is recording), it just broadcasts. Here’s the handler from talk_live.ex, with the persistence code trimmed out:

def handle_event("react", %{"emoji" => emoji}, socket) do
  if RateLimiter.allow?(socket.assigns.session_id) do
    # ...save the reaction if a session is recording...

    SpeechwaveWeb.Endpoint.broadcast!(
      "reactions:#{socket.assigns.talk.slug}",
      "new_reaction",
      %{emoji: emoji}
    )
  end

  {:noreply, socket}
end

The extension’s channel is subscribed to the same topic, so it gets the message. If it add a third kind of listener tomorrow, say a live dashboard, nothing in that function changes.

2. LiveView for the attendee page

The attendee page is a LiveView, so we got WebSocket-backed interactivity without building a single-page app and without maintaining a separate JavaScript frontend. The UI is a function of server state, and Phoenix handles keeping the browser in sync. For an app where nearly every interesting event is “something happened on the server, tell the clients,” that model is a very good match.

3. Cheap, isolated processes for everything else

The rate limiter is a GenServer that owns an ETS table (an in-memory key/value store built into the runtime). Reads don’t queue behind the GenServer, so a room full of people tapping at once doesn’t turn into a traffic jam. It’s a few dozen lines and there is no Redis anywhere.

Presence, which tracks how many extensions are connected to a talk so we can enforce plan limits, is also built in. So is the supervision tree that restarts any piece that crashes without taking down the rest. Here’s the top level of the tree from application.ex, trimmed a bit for space:

children =
  [
    Speechwave.Repo,
    {Phoenix.PubSub, name: Speechwave.PubSub},
    Speechwave.RateLimiter,
    Speechwave.AuthThrottle,
    # ...a few more throttles and reapers...
    SpeechwaveWeb.Endpoint,
    SpeechwaveWeb.Presence
  ] ++ backup_children()

opts = [strategy: :one_for_one, name: Speechwave.Supervisor]
Supervisor.start_link(children, opts)

I wrote down what happens when each child crashes (the rate limiter loses its cooldown state, Presence resets its count, and so on) and for this app every answer is “acceptable, by design.” Thinking about failure that explicitly is something the platform nudges you toward. In a Rails apps, I mostly find out what a crash does when it happens.

The hosting bill

This is the part I was the most excited about, and why I’m writing this post in the first place.

Speechwave runs on Fly.io, and here are the essential bits configuration:

[[vm]]
  memory = '512mb'
  cpu_kind = 'shared'
  cpus = 1

[http_service]
  auto_stop_machines = 'suspend'
  auto_start_machines = true
  min_machines_running = 0

That’s one shared CPU and 512 MB of memory. When nobody is using the app, the machine suspends. When someone scans a QR code, it wakes up.

Three things make that work, and Elixir is part of each of them.

The runtime is frugal. Because the BEAM runs lightweight processes instead of OS threads, one small machine can hold a lot of open connections. I’ve configured the app to allow up to 1,000 concurrent connections on that single 512 MB box. For context, my realistic target is a few conference rooms, not a stadium.

One process serves everything. The reaction fan-out, the LiveView pages, the Channel for the extension, the rate limiter, the presence tracking, and the scheduled backups all run inside a single Elixir application. I don’t need a separate Redis for pub/sub, a separate worker process for background jobs, or a separate WebSocket server. Fewer moving parts means fewer things to pay for and fewer things to keep alive.

SQLite is enough. The first version of this app used Fly’s managed Postgres. I’ve since moved to SQLite on a Fly volume, with Ecto’s SQLite adapter and a pool size of 2. Reactions are small writes, the data volume is tiny, and a few talk-shaped workloads doesn’t need a database server.

That does mean I own backups. A GenServer takes a snapshot every hour using SQLite’s VACUUM INTO (which gives a consistent copy of a live database, unlike copying the file) and uploads it to an S3-compatible object storage. It’s about a hundred lines of Elixir, it’s supervised like everything else, and a failed backup logs an error instead of crashing anything. If you want the details, the codebase explainer walks through it.

Put those together and the cost is tiny. Fly waives invoices under $5, and only last month did my bill creep above that to $5.49, just a little over the line. Usage of the S3 object store at Tigris has stayed within its free allowances.

I want to be careful about what I’m claiming. That number reflects low traffic on a young app. I haven’t load tested a 5,000-person keynote, and I’m not claiming this configuration handles one. What I can say is that the floor is very low, and the architecture doesn’t force me to pay for a fleet of services before I’ve earned a paying user.

Could I have done this in Rails?

Probably, yes. Rails 8 made SQLite a first-class citizen, and Solid Cable removes the Redis requirement for Action Cable. I’m not going to pretend that Rails can’t get to a similar place. A fair comparison, though, is about how much I had to assemble and tune to get there, and about how comfortable the result is at idle and at peak.

For this specific shape of app, many long-lived connections, constant broadcast, tiny writes, spiky traffic, Phoenix gave me the right design without much negotiation. In Rails I would have been making deliberate choices to approximate it.

I’ll also say what I did not get for free. Rails has an enormous ecosystem of gems that solve the boring parts of an app. In Elixir I wrote more of the glue myself. Authentication is a good example: magic links and OAuth took more work (Phoenix’s generator got me started, and a library called Assent handles the OAuth side). If this had been a plain CRUD app with admin screens, Rails may have been faster to build, and I may have shipped it sooner.

What learning Elixir was like for a Ruby developer

Easier than I expected, honestly. The syntax is friendly to Rubyists, the tooling (mix, iex, ExUnit) feels familiar, and the community writes really good documentation. The adjustments that took some effort were more conceptual:

• Data is immutable, so you transform values instead of mutating objects.
• Pattern matching shows up everywhere, and it replaces a surprising amount of if and case logic.
• You think in processes, which is a new mental model, but a pleasant one once it clicks.

If you’ve used Ruby 3’s case/in, pattern matching won’t be a brand new idea. You can destructure hashes and arrays, bind variables, and pin values with ^:

case response
in {status: 200, body:}
  render(body)
in {status: 404}
  not_found
end

The difference is how deep it goes. In Ruby, pattern matching is a feature you opt into with case/in, and most Ruby code still uses if, case/when, and guard clauses. In Elixir it’s foundational. Even = is a match operator, not just assignment. Most importantly, you can match right in a function’s arguments, so each clause of a function handles one shape of input. Speechwave’s Channel has two join functions, for example (from reaction_channel.ex, with the first one’s body trimmed):

def join("reactions:" <> slug, %{"api_key" => api_key}, socket) do
  # ...check the talk, the user, and the plan limits...
end

def join("reactions:" <> _slug, _params, _socket) do
  {:error, %{reason: "unauthorized"}}
end

The first clause only matches when the topic starts with "reactions:" and the params include an api_key. Anything else falls through to the second clause, which rejects it. No if required.

Ruby methods have one body, so you branch inside it. Elixir functions can have many clauses, and the branching happens before your code even runs. That’s the part that changed how code was structured, and it’s why pattern matching feels more like a way of working than a convenient syntax.

Here’s an example. The Channel join for the Chrome extension is a chain of four checks, and each failure has its own reason. This is the body of that first join clause, minus a couple of setup lines inside the do block:

with {:talk, %Talks.Talk{} = talk} <- {:talk, Talks.get_talk_by_slug(slug)},
     {:user, %Accounts.User{} = user} <- {:user, Accounts.get_user_by_api_key(api_key)},
     {:owner, true} <- {:owner, talk.user_id == user.id},
     {:capacity, :ok} <-
       {:capacity,
        Plans.check(
          :max_participants,
          user.plan,
          Presence.list("reactions:#{slug}") |> map_size()
        )} do
  # ...subscribe to disconnect events, schedule presence tracking...
  {:ok, join_payload(user), assign(socket, talk: talk, user: user)}
else
  {:talk, nil} -> {:error, %{reason: "not_found"}}
  {:user, nil} -> {:error, %{reason: "unauthorized"}}
  {:owner, false} -> {:error, %{reason: "unauthorized"}}
  {:capacity, {:error, :limit_reached}} -> {:error, %{reason: "capacity_reached"}}
end

I might have written that as a pile of guard clauses in Ruby. This version reads top to bottom as a list of requirements, and the else block is a table of what goes wrong.

As an aid to learning I had AI help me write a short course for Ruby developers, drawing the examples from the Speechwave codebase. If you’re coming from the Rails side, that’s an interesting place to start.

What I’d tell another Rails developer

If your app is mostly forms, tables, and background jobs, stay with Rails. You’ll move faster and you’ll have more gems to lean on.

If your app is mostly “lots of people connected at once, and something happening to all of them in real time,” look hard at Phoenix. The runtime and the framework were built for exactly that, and you may find, as I did, that the infrastructure shrinks down to a single small machine.

And if you’re already an Elixir developer: I’d love your feedback on the codebase. It’s open, it’s about 6,800 lines of application code, and I’m sure there are idioms I’m still getting wrong.

• Try Speechwave: speechwave.live
• Read the code: github.com/speechwave-live/speechwave
• Tour the internals: codebase explainer
• Peek at the AI-assisted short course for Ruby developers

Building a presentation thingy.

Creating a gem that writes your missing tests

No new project survives contact with the real world unscathed. We built TestGenAI, ran it on itself, and it worked well. Then we ran it on another codebase, and two things broke immediately. The fixes turned out to just as interesting as the original build.

This is a walkthrough of how the tool works and what we learned when we took it outside the greenhouse.

The code here is from TestGenAI, a working Ruby CLI gem you can install and run against your own codebase.

The pipeline

The pipeline has five stages:

1. Scan your codebase to find classes and methods without test coverage
2. Build context for each untested method
3. Generate tests using an LLM with the mechanically curated context
4. Validate that the generated tests run and pass
5. Collect the results

Each stage needs to be reliable enough that you can walk away and trust the process to complete. That means handling errors gracefully, providing clear output about what happened, and making it easy to pick up where things left off if something breaks.

Finding untested code

Before you can generate tests, you need to know what needs testing. The right approach depends on whether SimpleCov is available in the project.

If SimpleCov is set up, TestGenAI runs your test suite with COVERAGE=true, reads the resulting coverage/.resultset.json, and uses AST parsing to find methods where every executable line has zero hits. This scanner handles partially-tested files correctly. It reports individual methods that were never exercised, even if other methods in the same file have full coverage.

If SimpleCov isn’t available, the scanner falls back to checking whether a spec or test file exists for each source file. This approach is less accurate. A file tested only through integration tests or through specs for its subclasses will appear fully untested even if its methods are exercised constantly. The SimpleCov scanner is worth setting up.

Both scanners share the same underlying logic for locating methods in source files, which brings up something worth explaining.

Walking the AST

To locate methods, TestGenAI parses each Ruby source file into an abstract syntax tree and walks it recursively. The walker looks for :def and :defs nodes (instance and class methods), tracks the current class/module namespace, and records each method’s file, class, name, and line range.

That line range matters. The SimpleCov scanner uses it to check whether any executable lines in the method had zero hits. A nil in SimpleCov’s coverage array means a line isn’t executable, like a blank line, a comment, or an end. The scanner filters those out before checking for zeros, so it only flags methods where runnable code was never touched.

The parser compatibility problem

To parse Ruby, the gem relies on the parser gem. In older versions, you’d call Parser::CurrentRuby.parse(source) and get back an AST. This worked fine until Ruby 3.4, which switched its internal default parser to prism. Using Parser::CurrentRuby with Ruby 3.4 produces warnings, and in some configurations it fails entirely.

The prism project ships a compatibility shim, Prism::Translation::ParserCurrent, that produces the same AST node types as the old parser gem. The AST-walking code works unchanged. The only question is which one to load.

The solution is a small file that runs at load time and sets a constant:

module Testgenai
  if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3.4")
    require "prism"
    CurrentParser = Prism::Translation::ParserCurrent
  else
    require "parser/current"
    CurrentParser = Parser::CurrentRuby
  end
end

The rest of the codebase calls CurrentParser.parse(source) and never thinks about which parser is underneath. The pattern of version check at load time, constant as the abstraction is a clean way to handle the same kind of compatibility gap you’ll run into whenever Ruby ships a significant internal change.

Context, generation, and validation

When you ask an LLM to write tests for a method, you can’t just paste in the method body. It needs the full class, the dependencies that file requires, examples of how the method is called elsewhere in the codebase, and existing test files it can match in style. Context quality is where quick-and-dirty AI test generators fall apart, too little and the tests don’t compile, too much and you hit token limits.

The generator builds a prompt from all of that, sends it to the LLM via the ruby_llm gem (which keeps the generator code provider-agnostic), and strips any markdown fences from the response before passing it to the validator.

The validator writes the code to a temp file, runs bundle exec rspec or the Minitest equivalent, and distinguishes between three outcomes: the file failed to load (syntax errors, undefined constants), the tests ran but failed, or the tests passed. Each outcome needs different handling. A file that doesn’t load gets deleted immediately because it’s useless. A file that runs but fails gets its error output fed back to the LLM for a retry.

The pipeline retries up to three times, passing failure details back each time. LLMs are reasonably good at fixing specific errors when told what went wrong. Undefined constants and wrong require paths almost always resolve in one retry. More complex failures, like incorrect behavior assumptions, may not, and those end up in a failed bucket for manual review.

Then we ran it on a real project

The first external test run revealed two problems, both on the same day.

The first: generated tests were syntactically valid, ran, and passed — but they looked nothing like the rest of the project’s test suite. Wrong authentication setup, wrong factory usage, helpers that weren’t available. Tests that technically pass but violate project conventions create a maintenance burden.

The second: the tool was silently destroying existing tests. When a spec file already existed at the output path, the pipeline would overwrite it with the newly generated content. Any tests already in that file were gone.

Both problems make complete sense in retrospect. The tool had only ever run on its own codebase, where it was always generating new files and where the conventions were deeply familiar to the model from the context it was seeing. A different project broke both assumptions.

Fixing the conventions gap

The core problem is that the LLM knows what your method does, but it doesn’t know how your team writes tests. It doesn’t know that you authenticate in before blocks a certain way, or that you have specific factory traits available, or that you’re not using rails-controller-testing so assigns isn’t an option.

The fix is a conventions system with two parts.

ConventionsExtractor scans your existing test files and pulls out mechanical facts: the most common authentication setup pattern, available factory traits from your factories directory, frequently stubbed objects, whether transactional fixtures are disabled and how cleanup is handled, and whether specific helpers are unavailable based on what’s in your Gemfile. These aren’t judgments, they’re observations extracted directly from the code.

ConventionsSynthesizer takes those raw facts and sends them to the LLM with a prompt asking it to write a concise conventions guide explaining the rule behind each pattern. The result is plain prose. Something like “authentication is handled in before blocks using session[:user_id] = rather than Devise helpers; use this pattern consistently.” That text gets prepended to every generation prompt.

The synthesized guide is cached to spec/conventions.md and invalidated automatically when spec files or Gemfiles change. Regenerating it costs one LLM call.

Enable it with the --conventions flag:

testgenai generate --provider anthropic --model claude-opus-4-7 --conventions

Add spec/conventions.md to your .gitignore. It’s a derived artifact and probably not something to check in.

Fixing the overwrite problem

The overwrite bug was straightforward to diagnose and subtle to fix correctly.

The naive fix would be to skip generation if a spec file already exists. That’s less than ideal. The point is to add tests for untested methods, and partially-tested files are the most common case.

Better behavior is to inject the generated tests into the existing file. The pipeline now reads existing content before doing anything else. It combines existing content with the newly generated code and validates the combined file. If it passes, the combined content is written to the spec file.

If it fails, the pipeline restores the original file exactly as it was and writes the generated-only code to a fallback path. A method in lib/payments/processor.rb that already has a spec/payments/processor_spec.rb would get its fallback at spec/payments/processor_context_spec.rb (using the method name to scope the filename). The failure output tells you where to find it:

  ✗ Payments::Processor#context failed after 3 attempt(s)
    → Generated tests saved to spec/payments/processor_context_spec.rb for manual review

You end up with your original tests intact and the generated attempt sitting somewhere you can look at it and decide what to do.

Running it

Install the gem and point it at your project:

gem install testgenai
cd your_project
testgenai generate --provider anthropic --model claude-opus-4-7

Or add it to your Gemfile in the development group and use bundle exec. Configuration can also come from environment variables:

export TESTGENAI_PROVIDER=anthropic
export TESTGENAI_MODEL=claude-opus-4-7
export ANTHROPIC_API_KEY=your_api_key
testgenai generate --conventions

Three diagnostic commands are available before you commit to a full run:

testgenai scan      # find untested methods without making any API calls
testgenai context   # show what context would be sent to the LLM for each method

scan gives you a picture of your coverage gaps. context is useful for understanding what the LLM will see before spending API credits.

The goal isn’t to replace the developer who understands the code and makes decisions about testing. It’s to handle the mechanical work: setting up describe blocks, wiring test data, writing happy-path coverage. Then you can spend your time on the parts that actually need your judgment. The second project taught us that “mechanical” is more context-dependent than it looks.

Building a gem.

When “just return JSON” isn’t enough

If you’re building a tool that calls an AI API and needs structured data back, at some point you’ll write a prompt that ends with something like: “Return a JSON array. No prose, no explanation, just the JSON.”

And most of the time, that works. But “most of the time” isn’t good enough when a parse error crashes your command mid-run.

I ran into this while building Jojo, a CLI tool that generates job application materials (tailored resumes, cover letters, and a custom landing page) using AI. Several of its commands need structured data back from the model: an array of annotations, an array of FAQ entries, an array of integer indices. Plain text won’t do; the output feeds directly into the next stage of the pipeline.

The problem with prompt instructions

Language models are trained to be helpful. Part of being helpful, from the model’s perspective, is adding context: explaining what it did, noting any assumptions, formatting output to be readable. A prompt instruction to “return only JSON” fights against that training.

The failure modes are predictable:

Here is the JSON array you requested:

```json
[{"question": "What is your experience with Ruby?", "answer": "..."}]
```

I've formatted this as valid JSON with the required fields.

The model followed the spirit of the instruction (it returned valid JSON) but wrapped it in markdown fences and a helpful introductory sentence. JSON.parse on that string raises a ParserError.

The original code defended against this with a quick fence-strip:

cleaned_json = json_string.strip
  .gsub(/\A```(?:json)?\n?/, "")
  .gsub(/\n?```\z/, "")
JSON.parse(cleaned_json, symbolize_names: true)

That handles the fence case. But what about the introductory sentence? What about a model that adds a trailing note after the JSON? The regex grows, cases multiply, and you end up with the same defensive code copy-pasted across three different generators.

JsonExtractor

Instead, we pull this logic into a single place and make it more robust by trying multiple strategies to just find the JSON in whatever the model returned.

module JsonExtractor
  def self.call(content, symbolize_names: false)
    return content if content.is_a?(Hash) || content.is_a?(Array)

    stripped = content.gsub(/\A```\w*\s*|\s*```\z/m, "").strip

    try_parse(content, symbolize_names: symbolize_names) ||
      try_parse(stripped, symbolize_names: symbolize_names) ||
      extract_first_structure(content, symbolize_names: symbolize_names) ||
      raise(JSON::ParserError, "No JSON object found in response")
  end
end

It tries three strategies in order:

1. Parse the raw response directly. If the model actually followed instructions, done.
2. Strip markdown fences and try again. Handles the most common failure mode.
3. Find the first { or [ in the response and parse the complete structure from there, using a depth-tracking parser to handle nested objects correctly. This handles the “Here is the JSON: […]” case regardless of what prose surrounds it.

Each generator now calls JsonExtractor.call(response) instead of rolling its own fence-stripping. One test suite covers all the edge cases.

The cleaner alternative

If you’re using RubyLLM, a Ruby library that wraps multiple AI providers behind a common interface, there’s a more architecturally elegant solution: structured output.

Instead of asking the model to “return JSON” and then parsing whatever it sends back, you define a schema and ask the model to conform to it:

class FaqSchema < RubyLLM::Schema
  array :faqs do
    string :question
    string :answer
  end
end

response = chat.with_schema(FaqSchema).ask(prompt)
# response.faqs is an array of objects with .question and .answer

The model is constrained at the API level, not the prompt level. You get back a parsed object that already matches your schema. No JSON.parse, no fence-stripping, no JsonExtractor. The schema also serves as documentation for what the command expects.

This is genuinely cleaner. So why not use it?

The provider support problem

Structured output is a feature that providers implement, and not all models support it. The major providers (Anthropic, OpenAI, Gemini) support it for their recent models. But older models, like GPT-3.5, Gemini 1.0, Codex, etc., don’t. Some providers, like Hugging Face, don’t support it at all.

Jojo is designed to work with whatever model and provider the user configures. It dynamically picks up provider credentials from environment variables and routes requests through RubyLLM. A user might be running Claude Haiku, GPT-4o, or a locally-hosted model through an OpenAI-compatible endpoint.

RubyLLM’s structured output interface is uniform across providers: chat.with_schema works the same way regardless of which model you’re talking to. But when the underlying model doesn’t support structured output, you don’t get graceful degradation. You get a provider error.

That’s the thing:

• JsonExtractor works on any string. If the model returns prose with JSON buried in it, that’s handled. If the model returns clean JSON, that’s handled. It never throws a provider error because it never makes a structured output request.
• Structured output fails loudly on unsupported models. There’s no fallback. You have to catch the error and decide what to do with it.

For a tool where wide model support is a goal, JsonExtractor is more defensive. The cost is that you’re still at the mercy of whatever the model decides to return, but in practice, every model returns something that contains valid JSON, even if it’s surrounded by other content. The extraction strategy covers the real-world failure modes.

Okay, you also give up strict JSON schema guarantees (if supported) but that’s a tradeoff I’m willing to make for broader compatibility.

When structured output is the right call

None of this is an argument against structured output in general. If you have good control over exactly which model and provider your code will run against, structured output is the better choice. The schema and be explicit, the contract is enforced at the API level, and you’re unlikely to have a parse error at runtime.

You might wonder whether a hybrid approach could get the best of both worlds: check whether the model supports structured output, use it if so, and fall back to JsonExtractor if not. RubyLLM does expose this information. Model objects carry a capabilities array that includes 'structured_output' when the feature is supported. So the check is technically feasible.

The problem is that it doesn’t simplify anything. JsonExtractor still has to exist for the fallback branch, so you haven’t reduced maintenance burden. And now you have two code paths returning different shapes: structured output returns a typed schema object, while JsonExtractor returns a Hash. Every caller has to handle both, or you need a normalization layer on top. You’ve added complexity without removing any.

The tradeoff is essentially: how much do you want to trade portability for guarantees? JsonExtractor is portable and handles the real-world messiness of model output. Structured output gives you stronger guarantees but requires a supported model.

For a tool like Jojo, where users configure their own API credentials and model choices, JsonExtractor keeps things working across the widest range of setups. The “clean” architecture turned out to be less pragmatic once you considered who was actually going to run it. That person is me, btw, and I want it to work with whatever model I’m testing with at the moment, without having to worry about structured output support.

Building Jojo: turning job applications into marketing campaigns

When you apply for a job, you’re competing against hundreds of other candidates. Most of them submit a resume and a cover letter. The ambitious ones tailor those documents to the role. And then everyone waits.

No matter how good your resume is, it’s still a PDF in a pile of PDFs. You’re asking a hiring manager to do the work of figuring out why you’re a fit. What if you did that work for them?

That’s the idea behind Jojo, a Ruby CLI I built to transform job applications into personalized marketing campaigns. Instead of sending documents, you send a package: a tailored resume, a cover letter informed by company research, and a dedicated landing page that shows exactly why you’re a match for the role.

The landing page is the centerpiece. It’s a mini marketing site with an annotated job description that maps your experience to their requirements, portfolio projects selected for relevance to their tech stack, a branding statement written for their company, LinkedIn recommendations, an FAQ section, and a call-to-action to schedule a conversation. It turns a passive application into an active pitch.

Think of it as treating each job application like a product launch. You’re the product. The company you’re applying to is the only customer. Jojo builds the marketing campaign.

How it works

The workflow starts with two inputs: your resume data (a structured YAML file) and a job description (a file or URL). From there, Jojo runs a pipeline of AI-powered generation steps.

# Create a new application workspace
jojo new --slug acme-senior-dev --job posting.txt

# Generate everything
jojo generate --slug acme-senior-dev

The generate command kicks off a sequence:

1. Research — AI analyzes the job description and (optionally) searches the web to build a research document about the company, the role, and how to position yourself.
2. Resume — Your structured resume data is curated and rendered into a tailored resume, emphasizing the most relevant experience.
3. Branding — AI writes a personal branding statement specific to the company and role.
4. Cover letter — Generated from the research and tailored resume, so it references specific things about the company rather than generic platitudes.
5. Annotations — The job description is analyzed requirement by requirement, with each one mapped to your matching experience.
6. FAQ — AI generates role-specific questions and answers based on your background and the job requirements.
7. Website — Everything comes together in a self-contained landing page.
8. PDF — Resume and cover letter are converted to PDF via Pandoc.

Each step feeds into the next. The research informs the resume tailoring. The resume informs the cover letter. The annotations and FAQ feed into the website. It’s a pipeline, not a collection of independent scripts.

Every application gets its own workspace directory organized by slug:

  applications/acme-senior-dev/
  ├── job_description.md
  ├── job_details.yml
  ├── research.md
  ├── resume.md
  ├── cover_letter.md
  ├── branding.md
  ├── faq.json
  ├── job_description_annotations.json
  ├── status.log
  └── website/
      └── index.html

For daily use, there’s also an interactive TUI mode. Running jojo with no arguments launches a dashboard that shows all your applications, tracks which steps are complete, detects when artifacts are stale (because you regenerated a dependency), and lets you generate or regenerate individual steps with a keypress. The staleness detection uses file modification times. If you regenerate your research, the dashboard knows your resume is now stale because it was built from the old research.

  ┌─ Jojo ────────────────────────────────────────────┐
  │  Active: acme-senior-dev                          │
  │  Company: Acme Corp  •  Role: Senior Developer    │
  ├───────────────────────────────────────────────────┤
  │  Workflow                           Status        │
  │  1. Job Description            $   ✓ Generated    │
  │  2. Research                   $   ✓ Generated    │
  │  3. Resume                     $   * Stale        │
  │  4. Cover Letter               $   ○ Ready        │
  │  ...                                              │
  ├───────────────────────────────────────────────────┤
  │  [1-9] Generate item    [a] All ready    [q] Quit │
  └───────────────────────────────────────────────────┘

The $ indicator shows which steps call paid APIs, so you know if an action will cost something before you press the key. Steps that just combine existing artifacts (like website generation) are free.

Architecture: the command pipeline

Jojo is over 5K lines of Ruby source across ~50 source files. Most CLI commands follow the same three-file pattern:

  lib/jojo/commands/{command_name}/
  ├── command.rb    — Orchestration: validates inputs, manages file I/O
  ├── generator.rb  — Content generation: builds context, calls AI
  └── prompt.rb     — AI prompts: system and user prompt templates

So when I need to add a new command, I can create these three files, follow the pattern from the existing commands, and it (hopefully/usually) works. I don’t have to modify a central router or understand the internals of unrelated commands. The pattern helps make the codebase predictable. If you’ve read one command, you understand the shape of all of them. That helps the human and the AI assistant.

This wasn’t the original architecture. The CLI started as a monolith in cli.rb. Thor command definitions were mixed with validation logic, file handling, and generation orchestration. It worked fine for the first few commands, but soon things got messy. Adding a new feature meant navigating a growing code heap and hoping your changes didn’t break something unrelated.

The refactor extracted each command into its own module with a shared base class that provides common behavior (slug resolution, config loading, AI client setup). The CLI file shrank to a thin router with about 150 lines of small methods that delegate to command classes. Interactive mode, which breifly had a circular dependency calling back into the CLI class (eww) now calls command classes directly through a simple adapter.

Dual AI models

Jojo configures two AI models. There’s a reasoning model for complex tasks and a text generation model for simpler ones.

Company research and resume tailoring need the strongest reasoning capabilities as they’re analyzing job requirements, cross-referencing your experience, and making judgment calls about relevance. But extracting metadata from a job description (company name, location, job title) is easier. Using a powerful model for that is like hiring a senior architect to hang shelves.

The reasoning model handles research, resume curation, and cover letter writing. The text generation model handles job description processing, annotations, FAQ generation, and branding statements. Both models are configurable per provider, so you can use a frontier model for reasoning and a faster model for text generation, or whatever suits your budget and quality needs.

Even with the right model architecture, the AI still has a fundamental trustworthiness problem when it comes to factual content (welcome to AI).

Solving the hallucination problem

This was a technical decision that came from a hard fail.

The original resume generation would take the user’s resume data (stored as structured YAML), combine it with the job description and research, and ask the AI to generate a tailored resume in markdown. The prompt included extensive instructions about not fabricating information. It said things like “only include skills the candidate actually has” and “do not add technologies not present in the source data.”

The AI ignored these instructions way too often. I’d review a generated resume and find “Kubernetes” listed in my skills because the AI noticed I mentioned Docker and helpfully inferred I must know Kubernetes too. Or it would embellish a job description with responsibilities I never had. For a resume, this is not good.

The first instinct was to add more guardrails to the prompt. More emphatic instructions. More examples of what not to do. This helped a little, but it didn’t solve the problem. The AI still had the ability to modify anything, and language-level instructions are suggestions, not constraints.

The insight: different fields have different risk profiles

A professional summary should be rewritten for each role. That’s the whole point, but a list of programming languages must not be modified. The years you worked at a company are facts. Your name is your name.

The problem was that “AI shouldn’t have the same permissions everywhere.” Some fields need smart tailoring. Others need strict preservation. And still others might be removed or reordered. The idea was to define a permission system that specifies what the AI can for different kinds of content.

Permission-based curation

The solution was a permission system embedded directly in the resume data:

name: "Bob Denver"               # default: read-only
email: "bob@example.com"         # default: read-only

summary: |                       # permission: rewrite
  Polyglot developer who enjoys solving problems
  with software...

skills:                          # permission: remove, reorder
  - software engineering
  - full stack development
  - AI assisted development

languages:                       # permission: reorder
  - Ruby
  - Java
  - Python
  - Go

experience:                      # permission: reorder
  - company: "Island Adventures Inc."
    role: "Senior Software Engineer"
    start_date: "2020-07"        # read-only (nested)
    description: |               # permission: rewrite
      Full-stack developer delivering a SaaS platform...
    technologies:                # permission: remove, reorder
      - Ruby on Rails
      - Vue
      - Python
      - Docker

Four permission levels:

• read-only (default) — AI cannot modify, delete, add, or reorder. Contact info, dates, company names.
• remove — AI can exclude irrelevant items but can’t modify the ones it keeps. A database skill list can drop SQLite if the role is all PostgreSQL.
• reorder — AI can prioritize by relevance but can’t remove or modify. Your programming languages list stays complete but puts the most relevant ones first.
• rewrite — AI can generate new content using the original as a factual baseline. Professional summary, job descriptions.

In particular, the AI should never add items that aren’t in the source data. Though there is still a risk of hallucination in rewrite fields, the presence of original content in smaller chunks provides a grounding that makes it less likely.

Two-pass pipeline

The curation happens in two passes:

Pass 1: Filter and reorder. The AI receives the full resume data and the job description. It returns a filtered, reordered version that respects the permissions on each field. Skills marked remove, reorder get filtered to ~70% of the most relevant items and sorted by relevance. Lists marked reorder get sorted but all items are preserved.

Pass 2: Rewrite fields. The AI receives the filtered data and generates new content for fields marked rewrite. For example, the professional summary and experience descriptions. It uses the original content as a factual baseline.

Then an ERB template renders the final markdown. The template handles structure and formatting. The AI never touches the output templating.

What makes this work as an engineering solution is that the Ruby code enforces the permissions where possible. If the AI returns a reordered list that’s shorter than the original for a field that only has reorder permission, the Transformer class raises a PermissionViolation error:

unless can_remove
  if indices.length != original_count
    raise PermissionViolation,
      "LLM removed items from reorder-only field: #{field_path}"
  end
end

The permissions are no longer buried in prompt instructions that the AI might ignore. They’re enforced in code. The AI provides suggestions for how to curate the data, and the Ruby code validates those suggestions against the permission rules before applying them. If the AI tries to exceed its permissions, the operation fails rather than silently producing a resume with fabricated content.

The result is a skills section always contains skills I actually have. My job dates are always accurate. But my professional summary is tailored for each role, emphasizing the experience most relevant to that specific position.

What structured data enables

In order to make the permission system work, we had to switch from an unstructured markdown resume to a structured YAML format. This was a significant architectural change and it required reworking the entire resume generation pipeline. However, it was necessary to address the hallucination problem.

The permission system is the most visible benefit of using structured data, but there are other advantages:

• Narrower AI focus — With structured data, the AI can focus on curating specific fields rather than trying to parse and understand a free-form markdown document. This leads to better quality and more consistent results.
• Better output control — The ERB template handles formatting and structure, so the AI only generates smaller pieces of content. This reduces the chances of formatting errors or hallucinated sections and increases the human control over the final output.
• Easier testing — Structured data is easier to work with in tests. You can create synthetic resume data with specific permissions and verify that the output respects those permissions. With unstructured markdown, it’s harder to assert that the AI didn’t add or modify content it shouldn’t have.

Testing as a development discipline

A permission system that enforces constraints in code is only trustworthy if you actually test the enforcement. Jojo has 530 tests across two tiers, with 84% code coverage. Getting there was an intentional investment.

AI coding assistants are enthusiastic about writing features. They’re less enthusiastic about writing tests. This mirrors human tendencies. Tests aren’t as exciting as shipping the next feature, but with AI-assisted development the gap is amplified.

When the first large refactor was needed I noticed that test coverage was sitting at 31%. The code worked, but I had no safety net for refactoring. The push to 84% was a conscious decision to invest in change enablement.

Three kinds of tests

Jojo has three kinds of tests:

All tests run on every ./bin/test (or rake test:all) invocation and in CI.

Testing API-dependent code

The trickiest part of testing Jojo is that a lot of interesting work involves AI and Search API calls. You can’t run those in CI without spending money on every test run, but you also want tests that exercise real response parsing.

The solution was the VCR gem. VCR records real HTTP interactions the first time a test runs and saves them as “cassettes.” On subsequent runs, it replays the recorded responses instead of making real API calls. You get fast, deterministic tests that still exercise the full response-parsing pipeline.

Fixture discipline

One rule that has saved me more than once is that tests (and AI) never touch the inputs/ directory — no matter how much AI would like to. That directory contains real resume data from the user. Tests use test/fixtures/ exclusively, with synthetic data designed for testability.

This is codified in the project’s AI guidelines, which was previously prone to such mistakes. The instructions are explicit, emphatic, and took a few iterations to be effective. This testing discipline was part of the broader experience of building with AI.

Building with AI

There’s a meta quality to this project: it’s a tool that uses AI to generate content, and it was built with AI assistance. Both Claude and Z helped with development.

AI is pretty good at generating boilerplate, brainstorming design alternatives, and automating the tedious parts of refactoring (like updating 50 files when you rename a class).

But the decisions this post is about — the curation system, the architecture, the decision to refactor and when, the test organization — those were human decisions (as was the choice to use em-dashes just then). AI helped implement them a bit faster, but it didn’t tell me they were needed.

One nice thing about AI-assisted development was the ability to explore approaches quickly. When I was designing the permission system, I could describe different architectures, brainstorm, and get working prototypes, all in fairly short order. That kind of rapid experimentation is really helpful. The design thinking, however, still has to be yours.

One not-so-nice thing was needing to prod the AI to write tests for the features it’s helping to build. Also, let’s be honest, there’s a temptation to let the AI go a little too long before reviewing its output. Left to its own devices, an AI assistant will happily build feature after feature, with no test coverage and growing technical debt. Just like a human developer on a deadline, it needs someone to say “we’re not adding anything else until we address the technical debt, and that includes tests.”

What I learned and what’s next

A few things I’d do differently if I started over:

Start with structured data sooner. The original design used a free-form markdown resume as input. This was a frightful battle of prompt engineering from the beginning. The switch to structured YAML data (resume_data.yml) was the right call, but it required reworking the entire resume generation pipeline. If I’d started with structured data, the permission system would have been a natural extension rather than a redesign.

Build the interactive mode earlier. The TUI dashboard made the tool dramatically more usable, but it came in Phase 6 out of 7. Earlier access to the dependency graph and staleness detection would have improved my own workflow during development.

Force TDD from the start, or very near it. I had a test suite from the beginning, of course, but it wasn’t until I hit a major refactor that I made a conscious decision to invest in better test coverage. If I had enforced TDD from the start.

Basically, I would have spent a lot more time up front on planning the architecture and testing strategy, which would have made the development process smoother and more maintainable. AI assistance can be great, but it’s also really good at seducing you into bad habits.

What’s next

A few potential things for the roadmap:

• Interview prep generation — STAR-method examples drawn from your resume data, tailored to the specific role
• More and better theming options — The landing page is Jojo’s UVP, but the current design is pretty basic. More themes and customization options would let users create a landing page that better reflects their personal brand.
• Application tracking — Status tracking across all applications with dates, notes, and follow-up reminders
• Full SaaS product — A Rails app version of Jojo with a user-friendly interface and full job search management features — this would be a much bigger project but could help a wider audience.

Try it out

Jojo is open source and available on GitHub with a documentation site. It’s a Ruby CLI that requires AI and Search provider API keys. Setup takes just a few minutes.

If you’re interested in the code, the architecture, or just want to talk about AI-assisted development, I’d enjoy hearing from you. You can find me on LinkedIn or Mastodon.

Building a CLI for turning job applications into marketing campaigns. Jojo helps you create personalized, engaging applications that stand out from the crowd. It generates tailored cover letters, optimizes your resume for ATS, and even creates a custom portfolio website for each application. With Jojo, you can turn your job search into a marketing campaign that gets noticed.

Building a Ruby CLI gem for Hyrum’s Law

When you build a public API, users will depend on behaviors you never intended to guarantee. It’s called Hyrum’s Law, and it’s particularly tricky when it comes to error messages. Change “User not found” to “No such user exists” and someone’s regex breaks in production at 2am.

I built Hyrum to solve for this. It’s a Ruby CLI gem that uses AI to generate variations of status messages, ensuring users never become dependent on exact wording. It evolved from a single-provider tool into a multi-provider platform that cut costs by 10x, reduced code complexity from 12 to 1, and added quality validation for AI-generated content.

The problem with predictable messages

Hyrum’s Law states that all observable behaviors of your system will be depended on by somebody. This creates a dilemma for API designers: you want clear, consistent error messages, but you don’t want users parsing them as if they were structured data.

Error codes help with this. Return 404 or E_NOT_FOUND and the message text can evolve independently. But this only works if you’re disciplined about using codes for everything that matters. In practice, some context lives in the message text, and someone will parse it.

The traditional solution is thorough documentation warning against this. The pragmatic solution is accepting that some percentage of users will do it anyway. The solution presented here is to make the messages unpredictable by design.

Building the initial solution

The first version was straightforward. The gem takes a message like “The server refuses the attempt to brew coffee with a teapot” and generates code in your language of choice (Ruby, JavaScript, Python, Java, or JSON) that returns variations at random:

module Messages
  MESSAGES = {
    e418: [
      "Invalid Brewing Method",
      "Teapot not designed for coffee brewing",
      "Please use a suitable brewing device"
    ]
  }.freeze

  def self.message(key)
    MESSAGES[key].sample
  end
end

I started with OpenAI’s API via the ruby-openai gem. It worked well enough for the core use case. But eventually, two problems emerged.

First, cost. Running gpt-4 for simple message generation was using a chainsaw to cut butter. Second, vendor lock-in. Some projects had Anthropic credits, others used local Ollama models. I needed to support multiple providers without maintaining provider-specific code paths.

The migration decision

There were three options:

1. Build custom adapters for each provider
2. Find an abstraction layer that handled the differences
3. Accept the limitation and move on

Building custom adapters would provide complete control, but at the cost of maintaining provider-specific logic as APIs evolved. Option three was tempting but unsatisfying.

I chose option two, migrating to ruby_llm. More than swapping dependencies, this was a fundamental architecture change that would affect testing strategy, error handling, configuration, and the public API.

Key technical decisions

Cost optimization through model selection

The most impactful decision was switching from premium to budget models. For Anthropic, this meant claude-sonnet-4 to claude-haiku-20250514, a roughly 10x cost reduction.

This wasn’t about being cheap. (Okay, maybe it was a little bit about that.) It was (mostly) about matching model capability to task complexity. Generating three variations of “Resource not found” doesn’t require deep reasoning. Budget models handle it perfectly well. The quality remained identical while costs dropped by an order of magnitude.

Testing strategy: mock at the right level

The original implementation used VCR to record HTTP interactions. This is a common pattern, but it had problems:

• Maintaining cassettes for 10+ providers would be tedious
• Tests would break when ruby_llm changed request formats
• We would be testing ruby_llm’s HTTP implementation, not our code

The better approach: mock at the ruby_llm interface level. Instead of recording HTTP traffic, we mock RubyLLM.chat() directly. One mock setup works for all providers. Tests are faster, more maintainable, and focused on our actual logic.

This eliminated the need for VCR and WebMock entirely, removing two dependencies. 🎊

As an additional safeguard, I set up a GitHub Actions workflow that runs the full test suite on every push. This catches regressions before they reach main and provides confidence when accepting contributions. It’s a small addition that pays dividends in long-term maintainability.

Code simplification through extraction

FakeGenerator started at 298 lines with embedded message data. Extracting the messages to an external JSON file and refactoring the logic brought it down to 36 lines. That’s an 88% reduction.

Another big win was in AiGenerator. By letting ruby_llm handle provider differences, the cyclomatic complexity dropped from 12 to 1. Twelve decision points (checking provider types, handling edge cases) collapsed into a single code path.

This is the value of a good abstraction layer. It reduces the lines of code, sure, but it also reduces the number of things you have to think about.

To be honest, this is also currently a pain point. We swapped ruby-openai’s abastraction for a better one, but in an evolving ecosystem, we may need to swap again. The key is that the architecture is now flexible enough to accommodate future changes more easily.

Breaking changes as a design tool

The migration required environment variable changes:

• OPENAI_ACCESS_TOKEN → OPENAI_API_KEY
• OLLAMA_URL → OLLAMA_API_BASE

I considered adding migration helpers to detect old variables and warn users. But the gem was pre-1.0 with minimal adoption. Adding complexity for hypothetical users would hurt future maintainability more than it helped current users.

The cleaner approach: document the breaking changes clearly, provide a migration guide, and move forward with consistent naming. Sometimes the right trade-off is accepting short-term pain for long-term simplicity.

Implementation approach

I followed a disciplined TDD approach for the migration:

1. Write failing tests for AiGenerator
2. Implement minimal code to pass
3. Add error handling tests
4. Implement error handling
5. Verify across multiple providers

Each commit represented a logical unit of work with a clear purpose. The git history tells a story: dependency updates, test infrastructure, new generator implementation, factory updates, cleanup, documentation.

This matters for maintainability. Six months from now, when I need to add a new provider or debug an edge case, the git history explains not only what changed but why.

Validating non-deterministic output

Getting AI to generate message variations is great, but how do you know if the variations are any good?

Building a system that validates the quality of non-deterministic output requires deeper thinking about what “quality” even means in this context.

Defining useful variation

A good variation needs two properties:

1. Semantic similarity - It preserves the original message’s meaning
2. Lexical diversity - It uses different wording than other variations

These goals exist in tension. Perfect similarity means identical text. Perfect diversity means unrelated messages. The sweet spot is variations that mean the same thing but say it differently.

I created a validation system that measures both metrics and combines them into an overall quality score. This lets you validate generated variations automatically.

The initial design mistake

My first implementation compared variations to each other. Generate five variations, measure how similar they are as a group, done. This seemed logical until I tested it.

The problem: variations could be highly similar to each other but completely different from the original message. A set of variations about network timeouts would score well even if the original message was about authentication failures. They were similar to each other, but wrong.

The fix was obvious in hindsight: compare each variation to the original message, not to other variations. Semantic similarity measures how well each variation preserves the user’s intent. Lexical diversity measures how much the variations differ from each other. Two separate concerns, two separate comparisons.

Semantic similarity with embeddings

Measuring semantic similarity requires understanding meaning, not just matching words. “Server error” and “Internal server failure” share minimal text but convey the same concept. Simple string comparison would fail.

The solution: embedding models. These convert text into high-dimensional vectors where semantically similar content clusters together. Calculate the cosine similarity between the original message’s embedding and each variation’s embedding, and you have a numeric score for how well meaning is preserved.

I designed this to be provider-agnostic from the start, learning from the earlier migration experience. The validator uses RubyLLM.embed() which works with any provider that supports embeddings (OpenAI, Google, etc.). When embeddings aren’t available, it falls back to a simpler word overlap heuristic.

This graceful degradation was important. Users without embedding access still get validation, just with reduced accuracy. The feature doesn’t silently fail or block users.

API design for optional features

Quality validation needed to be opt-in. The core workflow is “generate variations and use them.” Adding validation steps would slow things down and require configuration. It needed to enhance the workflow without disrupting it.

The CLI design reflects this:

# Basic usage - no validation
hyrum -s openai -m "Server error" -f ruby

# Opt into validation
hyrum -s openai -m "Server error" -f ruby --validate

# Use in CI/CD with strict mode
hyrum -s openai -m "Server error" -f ruby --validate --strict --min-quality 75

Validation is off by default. Enable it when you want quality metrics. Use --strict to fail builds when quality is too low. Use --show-scores to include metrics in generated output.

Each flag serves a specific use case without cluttering the happy path. This is backward compatible and makes the feature discoverable through --help without overwhelming new users.

What I learned

Shipping a Ruby gem is more accessible than I expected

I hadn’t published a gem before this project. The Ruby ecosystem makes it surprisingly straightforward: follow conventions for directory structure, add a gemspec, and gem build handles the rest. RuboCop enforces community standards, and RSpec provides solid testing patterns.

Most of the learning curve wasn’t in the tooling. It was in the design decisions around versioning, breaking changes, and API stability. Understanding when to bump major vs minor versions, when breaking changes are acceptable, and how much backward compatibility to maintain. These are judgment calls that come with experience, not documentation per se.

Abstractions have a cost and a benefit

Ruby_llm’s abstraction eliminated provider-specific code paths. But it also added a dependency and gave up some provider-specific features (like Anthropic’s prompt caching). The trade-off made sense because the gem’s core use case doesn’t need advanced features. Your mileage will vary.

Model selection is a design decision

Defaulting to budget models wasn’t about minimizing costs. It was about right-sizing capability to task complexity. When your task genuinely needs advanced reasoning, use advanced models. When it doesn’t, you’re paying for capability you’re not using.

Testing at the right abstraction level matters

Mocking at the HTTP level tests the wrong thing. Mocking at the library interface level tests your code. The latter is almost always better unless you’re specifically testing HTTP behavior.

Breaking changes are acceptable in context

Pre-1.0 software with limited adoption is the right time to make breaking changes. Adding backward compatibility for a handful of users creates technical debt that affects every future user. Sometimes the generous thing is to break things cleanly.

User feedback catches design flaws early

My initial quality validation design seemed sound in theory. It measured variation quality as a group property. But the first test revealed the flaw: variations could be similar to each other while being completely unrelated to the original message.

This is why you test with real examples before building the whole system. The fix (comparing to the original message) was trivial to implement early. It would have been painful to retrofit later after building an entire validation pipeline on the wrong assumption.

The lesson: design mistakes are inevitable. What matters is catching them before they become weight-bearing walls in your architecture.

Graceful degradation beats hard dependencies

Embedding models provide superior semantic similarity measurement. But requiring them would block users whose AI providers don’t support embeddings. The word overlap fallback isn’t as accurate, but it’s better than nothing.

This pattern appears throughout the gem. Can’t access embeddings? Use heuristics. Provider doesn’t support structured output? Parse text. Each graceful degradation expands the set of valid configurations.

The alternative is failing fast with clear errors. Both approaches are valid, but for a tool that works across many providers, degradation creates a better experience than strict requirements.

Validation changes what “done” means

Before quality validation, “done” meant “generates variations.” After, it meant “generates variations that preserve meaning while varying wording.” This shift changed the entire value proposition.

The interesting part is that validation makes the AI output more trustworthy without requiring a better AI model. Same model, same cost, but now you have quantitative confidence in the results. That’s the leverage of good metrics.

The result

Hyrum now supports 11 AI providers (OpenAI, Anthropic, Gemini, Ollama, Mistral, DeepSeek, Perplexity, OpenRouter, Vertex AI, AWS Bedrock, GPUStack) through a unified interface. The codebase is simpler, tests are faster, and costs are 10x lower.

Quality validation adds confidence without complexity. Generate variations, validate they preserve meaning while varying wording, and integrate the results into your codebase with quantitative quality metrics. The validation system works across all providers and degrades gracefully when embeddings aren’t available.

Best of all, the architecture can accommodate new providers and capabilities without increasing complexity. When ruby_llm adds support for a new provider, Hyrum gets it for free. When embedding models improve, quality validation automatically benefits. That’s the payoff of choosing the right abstractions.

The project is open source on GitHub. If you’re dealing with Hyrum’s Law in your own APIs, or you’re just curious about the implementation details, check it out.

Ruby Array Methods

I wondered if this information would be more digestible using fruit. The taste test is ongoing.

🚫 = nil