Why I built Speechwave in Elixir (as a Rails developer)

I’ve been developing with Ruby on Rails for a long time. If you had asked me to bet on what I’d build my next web app with, I would have bet on Rails.

Then I started building Speechwave, a live emoji reaction app for conference talks, and I wrote it in Elixir instead

This isn’t a “Rails is dead” post. It’s still a great choice for lots of things. This is about what this particular app needed, why Elixir turned out to a great choice, and why the hosting bill is about as small as a bill can get. If you’re an Elixir developer, some of this may be familiar.

What Speechwave does

Attendees scan a QR code at a talk and are sent to a page of emoji buttons. They tap one. A Chrome extension on the speaker’s laptop receives that reaction and floats the emoji over the speaker’s Google Slides in real time. After the talk, the speaker can see which slides got which reactions.

Look at that from a server’s point of view. It’s a lot of small, short-lived, simultaneous things:

• Dozens (or hundreds) of phones holding open WebSocket connections
• Each tap needs to be rate limited, optionally saved, and fanned out to everyone who cares
• The speaker’s extension holds one more long-lived connection, and it isn’t even a web page
• The whole thing is quiet for days, then very busy for 45 minutes

That last point will matter quite a bit when we get to the hosting section.

Why it works: the three things I would have had to do in Rails

I don’t want to oversell this. Rails can do real-time. Action Cable exists, and Rails keeps improving. But in Rails I would have been assembling pieces. In Phoenix, the pieces come in the box and they were designed together.

1. A message bus that doesn’t care who’s listening

Phoenix PubSub lets any process broadcast to a topic and any other process subscribe to it. The attendee’s LiveView doesn’t know the Chrome extension exists. After the rate limit check (and saving the reaction, if a session is recording), it just broadcasts. Here’s the handler from talk_live.ex, with the persistence code trimmed out:

def handle_event("react", %{"emoji" => emoji}, socket) do
  if RateLimiter.allow?(socket.assigns.session_id) do
    # ...save the reaction if a session is recording...

    SpeechwaveWeb.Endpoint.broadcast!(
      "reactions:#{socket.assigns.talk.slug}",
      "new_reaction",
      %{emoji: emoji}
    )
  end

  {:noreply, socket}
end

The extension’s channel is subscribed to the same topic, so it gets the message. If it add a third kind of listener tomorrow, say a live dashboard, nothing in that function changes.

2. LiveView for the attendee page

The attendee page is a LiveView, so we got WebSocket-backed interactivity without building a single-page app and without maintaining a separate JavaScript frontend. The UI is a function of server state, and Phoenix handles keeping the browser in sync. For an app where nearly every interesting event is “something happened on the server, tell the clients,” that model is a very good match.

3. Cheap, isolated processes for everything else

The rate limiter is a GenServer that owns an ETS table (an in-memory key/value store built into the runtime). Reads don’t queue behind the GenServer, so a room full of people tapping at once doesn’t turn into a traffic jam. It’s a few dozen lines and there is no Redis anywhere.

Presence, which tracks how many extensions are connected to a talk so we can enforce plan limits, is also built in. So is the supervision tree that restarts any piece that crashes without taking down the rest. Here’s the top level of the tree from application.ex, trimmed a bit for space:

children =
  [
    Speechwave.Repo,
    {Phoenix.PubSub, name: Speechwave.PubSub},
    Speechwave.RateLimiter,
    Speechwave.AuthThrottle,
    # ...a few more throttles and reapers...
    SpeechwaveWeb.Endpoint,
    SpeechwaveWeb.Presence
  ] ++ backup_children()

opts = [strategy: :one_for_one, name: Speechwave.Supervisor]
Supervisor.start_link(children, opts)

I wrote down what happens when each child crashes (the rate limiter loses its cooldown state, Presence resets its count, and so on) and for this app every answer is “acceptable, by design.” Thinking about failure that explicitly is something the platform nudges you toward. In a Rails apps, I mostly find out what a crash does when it happens.

The hosting bill

This is the part I was the most excited about, and why I’m writing this post in the first place.

Speechwave runs on Fly.io, and here are the essential bits configuration:

[[vm]]
  memory = '512mb'
  cpu_kind = 'shared'
  cpus = 1

[http_service]
  auto_stop_machines = 'suspend'
  auto_start_machines = true
  min_machines_running = 0

That’s one shared CPU and 512 MB of memory. When nobody is using the app, the machine suspends. When someone scans a QR code, it wakes up.

Three things make that work, and Elixir is part of each of them.

The runtime is frugal. Because the BEAM runs lightweight processes instead of OS threads, one small machine can hold a lot of open connections. I’ve configured the app to allow up to 1,000 concurrent connections on that single 512 MB box. For context, my realistic target is a few conference rooms, not a stadium.

One process serves everything. The reaction fan-out, the LiveView pages, the Channel for the extension, the rate limiter, the presence tracking, and the scheduled backups all run inside a single Elixir application. I don’t need a separate Redis for pub/sub, a separate worker process for background jobs, or a separate WebSocket server. Fewer moving parts means fewer things to pay for and fewer things to keep alive.

SQLite is enough. The first version of this app used Fly’s managed Postgres. I’ve since moved to SQLite on a Fly volume, with Ecto’s SQLite adapter and a pool size of 2. Reactions are small writes, the data volume is tiny, and a few talk-shaped workloads doesn’t need a database server.

That does mean I own backups. A GenServer takes a snapshot every hour using SQLite’s VACUUM INTO (which gives a consistent copy of a live database, unlike copying the file) and uploads it to an S3-compatible object storage. It’s about a hundred lines of Elixir, it’s supervised like everything else, and a failed backup logs an error instead of crashing anything. If you want the details, the codebase explainer walks through it.

Put those together and the cost is tiny. Fly waives invoices under $5, and only last month did my bill creep above that to $5.49, just a little over the line. Usage of the S3 object store at Tigris has stayed within its free allowances.

I want to be careful about what I’m claiming. That number reflects low traffic on a young app. I haven’t load tested a 5,000-person keynote, and I’m not claiming this configuration handles one. What I can say is that the floor is very low, and the architecture doesn’t force me to pay for a fleet of services before I’ve earned a paying user.

Could I have done this in Rails?

Probably, yes. Rails 8 made SQLite a first-class citizen, and Solid Cable removes the Redis requirement for Action Cable. I’m not going to pretend that Rails can’t get to a similar place. A fair comparison, though, is about how much I had to assemble and tune to get there, and about how comfortable the result is at idle and at peak.

For this specific shape of app, many long-lived connections, constant broadcast, tiny writes, spiky traffic, Phoenix gave me the right design without much negotiation. In Rails I would have been making deliberate choices to approximate it.

I’ll also say what I did not get for free. Rails has an enormous ecosystem of gems that solve the boring parts of an app. In Elixir I wrote more of the glue myself. Authentication is a good example: magic links and OAuth took more work (Phoenix’s generator got me started, and a library called Assent handles the OAuth side). If this had been a plain CRUD app with admin screens, Rails may have been faster to build, and I may have shipped it sooner.

What learning Elixir was like for a Ruby developer

Easier than I expected, honestly. The syntax is friendly to Rubyists, the tooling (mix, iex, ExUnit) feels familiar, and the community writes really good documentation. The adjustments that took some effort were more conceptual:

• Data is immutable, so you transform values instead of mutating objects.
• Pattern matching shows up everywhere, and it replaces a surprising amount of if and case logic.
• You think in processes, which is a new mental model, but a pleasant one once it clicks.

If you’ve used Ruby 3’s case/in, pattern matching won’t be a brand new idea. You can destructure hashes and arrays, bind variables, and pin values with ^:

case response
in {status: 200, body:}
  render(body)
in {status: 404}
  not_found
end

The difference is how deep it goes. In Ruby, pattern matching is a feature you opt into with case/in, and most Ruby code still uses if, case/when, and guard clauses. In Elixir it’s foundational. Even = is a match operator, not just assignment. Most importantly, you can match right in a function’s arguments, so each clause of a function handles one shape of input. Speechwave’s Channel has two join functions, for example (from reaction_channel.ex, with the first one’s body trimmed):

def join("reactions:" <> slug, %{"api_key" => api_key}, socket) do
  # ...check the talk, the user, and the plan limits...
end

def join("reactions:" <> _slug, _params, _socket) do
  {:error, %{reason: "unauthorized"}}
end

The first clause only matches when the topic starts with "reactions:" and the params include an api_key. Anything else falls through to the second clause, which rejects it. No if required.

Ruby methods have one body, so you branch inside it. Elixir functions can have many clauses, and the branching happens before your code even runs. That’s the part that changed how code was structured, and it’s why pattern matching feels more like a way of working than a convenient syntax.

Here’s an example. The Channel join for the Chrome extension is a chain of four checks, and each failure has its own reason. This is the body of that first join clause, minus a couple of setup lines inside the do block:

with {:talk, %Talks.Talk{} = talk} <- {:talk, Talks.get_talk_by_slug(slug)},
     {:user, %Accounts.User{} = user} <- {:user, Accounts.get_user_by_api_key(api_key)},
     {:owner, true} <- {:owner, talk.user_id == user.id},
     {:capacity, :ok} <-
       {:capacity,
        Plans.check(
          :max_participants,
          user.plan,
          Presence.list("reactions:#{slug}") |> map_size()
        )} do
  # ...subscribe to disconnect events, schedule presence tracking...
  {:ok, join_payload(user), assign(socket, talk: talk, user: user)}
else
  {:talk, nil} -> {:error, %{reason: "not_found"}}
  {:user, nil} -> {:error, %{reason: "unauthorized"}}
  {:owner, false} -> {:error, %{reason: "unauthorized"}}
  {:capacity, {:error, :limit_reached}} -> {:error, %{reason: "capacity_reached"}}
end

I might have written that as a pile of guard clauses in Ruby. This version reads top to bottom as a list of requirements, and the else block is a table of what goes wrong.

As an aid to learning I had AI help me write a short course for Ruby developers, drawing the examples from the Speechwave codebase. If you’re coming from the Rails side, that’s an interesting place to start.

What I’d tell another Rails developer

If your app is mostly forms, tables, and background jobs, stay with Rails. You’ll move faster and you’ll have more gems to lean on.

If your app is mostly “lots of people connected at once, and something happening to all of them in real time,” look hard at Phoenix. The runtime and the framework were built for exactly that, and you may find, as I did, that the infrastructure shrinks down to a single small machine.

And if you’re already an Elixir developer: I’d love your feedback on the codebase. It’s open, it’s about 6,800 lines of application code, and I’m sure there are idioms I’m still getting wrong.

• Try Speechwave: speechwave.live
• Read the code: github.com/speechwave-live/speechwave
• Tour the internals: codebase explainer
• Peek at the AI-assisted short course for Ruby developers

Documentation search: When your Rails app needs which approach

Imagine spending an afternoon watching a developer tear out a perfectly functional search feature. They replace their solid Postgres full-text search with a vector database and RAG pipeline because, well, that’s what you’re supposed to do now, right? The new system is slower, cost them $200 a month in OpenAI API calls, and returned worse results for their specific use case.

This keeps happening. The AI hype cycle has convinced developers that every search problem needs embeddings, vector databases, and agentic loops. Sometimes that’s true. Often it’s not.

Let’s build the same feature four different ways and see what each approach actually costs you.

The use case: searching Ruby gem documentation

We’re building a search feature for a documentation site that indexes about 5,000 Ruby gems. Each gem has README content, API documentation, and code examples. Users ask questions like “How do I upload files to S3?” or “What’s the best gem for handling webhooks?”

This is a realistic scale for most Rails apps. Not Google-sized, not trivial. Just normal business software that needs to help users find information.

I’ll show you four implementations, each adding a layer of complexity. We’ll look at the code, measure the actual costs, and figure out when the added complexity pays for itself.

Approach 1: Traditional search with AI summarization

Start with what works. Postgres full-text search has been solving search problems since before your junior devs were born.

class Documentation < ApplicationRecord
  include PgSearch::Model
  
  pg_search_scope :search_content,
    against: {
      title: 'A',
      content: 'B',
      code_examples: 'C'
    },
    using: {
      tsearch: { prefix: true },
      trigram: { threshold: 0.3 }
    }
end

class DocumentationSearcher
  def initialize(query)
    @query = query
  end
  
  def search
    results = Documentation.search_content(@query).limit(10)
    
    {
      results: results,
      summary: summarize_results(results)
    }
  end
  
  private
  
  def summarize_results(results)
    return nil if results.empty?
    
    prompt = <<~PROMPT
      User question: #{@query}
      
      Here are the top search results:
      #{format_results(results)}
      
      Provide a concise answer to the user's question based on these results.
      If the results don't contain relevant information, say so.
    PROMPT
    
    client = OpenAI::Client.new
    client = OpenAI::Client.new
    response = client.chat(
      parameters: {
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: prompt }],
        temperature: 0.3
      }
    )

    response.dig(:choices, 0, :message, :content)
  end
  
  def format_results(results)
    results.map.with_index do |doc, i|
      "#{i + 1}. #{doc.title}\n#{doc.content.truncate(500)}"
    end.join("\n\n")
  end
end

This approach does one database query and one API call. The search uses proven Postgres features: full-text search with ranking, trigram matching for typos, and weighted fields. Then we send the top results to GPT-4 to generate a summary.

Cost per query:

• Database: ~5ms
• OpenAI API: ~$0.002 (about 1,000 input tokens, 200 output tokens)
• Total latency: ~800ms

When this fails:

• User queries are conceptually different from how docs are written (“async jobs” versus “background processing”)
• Important information is buried in the middle of long documents
• You need to combine information from multiple sources

The failure mode is subtle. Traditional search ranks by keyword matching and field weights. When users phrase questions differently than your documentation uses terminology, they get poor results. You can’t fix this with better prompt engineering because the LLM never sees the relevant documents.

Approach 2: Basic RAG with vector embeddings

RAG (Retrieval Augmented Generation) means embedding your documents as vectors, embedding the user’s query as a vector, and finding documents with similar embeddings. This solves the terminology mismatch problem.

# We need to store embeddings
class AddEmbeddingsToDocumentation < ActiveRecord::Migration[7.1]
  def change
    add_column :documentations, :embedding, :vector, limit: 1536
    add_index :documentations, :embedding, using: :hnsw, opclass: :vector_cosine_ops
  end
end

class Documentation < ApplicationRecord
  has_neighbors :embedding
  
  after_save :generate_embedding, if: :content_changed?
  
  private
  
  def generate_embedding
    text = "#{title}\n\n#{content}"
    
    client = OpenAI::Client.new
    response = client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    self.update_column(:embedding, response.dig(:data, 0, :embedding))
  end
end

class RagSearcher
  def initialize(query, client: OpenAI::Client.new)
    @query = query
    @client = client
  end

  def search
    query_embedding = generate_embedding(@query)
    results = Documentation.nearest_neighbors(:embedding, query_embedding, distance: "cosine").limit(5)

    {
      results: results,
      answer: generate_answer(results)
    }
  end

  private

  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end

  def generate_answer(results)
    context = results.map { |doc| "#{doc.title}\n#{doc.content}" }.join("\n\n---\n\n")

    prompt = <<~PROMPT
      Answer the user's question based only on the following documentation:

      #{context}

      Question: #{@query}

      If you cannot answer based on the provided documentation, say so clearly.
    PROMPT

    response = @client.chat(
      parameters: {
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: prompt }],
        temperature: 0.3
      }
    )

    response.dig(:choices, 0, :message, :content)
  end
end

Now we’re making two API calls per search: one to embed the query, one to generate the answer. We’re also using pgvector with HNSW indexing for fast similarity search.

Cost per query:

• Database: ~15ms (vector similarity search)
• OpenAI embeddings API: ~$0.00001 (negligible)
• OpenAI chat API: ~$0.003
• Total latency: ~1,200ms

When this works better: The semantic matching is noticeably better. A query about “background jobs” will match documents about “async processing” and “delayed tasks” because the concepts are similar in vector space. This is real improvement over keyword search.

When this still fails:

• Complex questions requiring information from many documents
• Multi-step reasoning (“compare these two approaches”)
• Questions where the first retrieval doesn’t get the right context

Here’s a concrete failure case I hit: A user asks “What’s the difference between Sidekiq and Good Job?” The vector search retrieves five documents, but three are about Sidekiq and two are about Good Job. The LLM tries to compare them but doesn’t have complete information about both systems. It hedges and gives a vague answer.

Approach 3: Agentic RAG with adaptive retrieval

This is where we let the LLM decide if it needs more information. Instead of one retrieve-then-generate pass, we give the LLM tools to search again, rephrase queries, or combine results.

class AgenticRagSearcher
  MAX_ITERATIONS = 3

  def initialize(query, client: OpenAI::Client.new)
    @query = query
    @client = client
    @conversation_history = []
    @retrieved_docs = []
  end
  
  def search
    initial_prompt = <<~PROMPT
      You are a helpful assistant that searches Ruby gem documentation.
      
      User question: #{@query}
      
      You have access to these tools:
      - search_docs(query): Search documentation with a semantic query
      - get_related(doc_id): Get documents related to a specific document
      
      Think step by step. You can search multiple times with different queries
      to gather complete information before answering.
    PROMPT
    
    @conversation_history << { role: "user", content: initial_prompt }
    
    MAX_ITERATIONS.times do
      response = call_llm_with_tools
      
      break if response[:finish_reason] == "stop"
      
      if response[:tool_calls]
        handle_tool_calls(response[:tool_calls])
      end
    end
    
    {
      results: @retrieved_docs.uniq,
      answer: @conversation_history.last[:content],
      iterations: @conversation_history.length
    }
  end
  
  private
  
  def call_llm_with_tools
    response = @client.chat(
      parameters: {
        model: "gpt-4o",
        messages: @conversation_history,
        tools: tool_definitions,
        temperature: 0.3
      }
    )

    message = response.dig(:choices, 0, :message)
    @conversation_history << message

    {
      finish_reason: response.dig(:choices, 0, :finish_reason),
      tool_calls: message[:tool_calls]
    }
  end
  
  def tool_definitions
    [
      {
        type: "function",
        function: {
          name: "search_docs",
          description: "Search documentation using semantic search",
          parameters: {
            type: "object",
            properties: {
              query: {
                type: "string",
                description: "The search query"
              }
            },
            required: ["query"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "get_related",
          description: "Get documents related to a specific document",
          parameters: {
            type: "object",
            properties: {
              doc_id: {
                type: "integer",
                description: "The ID of the document"
              }
            },
            required: ["doc_id"]
          }
        }
      }
    ]
  end
  
  def handle_tool_calls(tool_calls)
    results = tool_calls.map do |tool_call|
      function_name = tool_call.dig(:function, :name)
      arguments = JSON.parse(tool_call.dig(:function, :arguments) || "{}")

      result = case function_name
      when "search_docs"
        search_docs(arguments["query"])
      when "get_related"
        get_related_docs(arguments["doc_id"])
      end

      @retrieved_docs.concat(result)

      {
        role: "tool",
        tool_call_id: tool_call[:id],
        content: format_docs_for_llm(result)
      }
    end

    @conversation_history.concat(results)
  end
  
  def search_docs(query)
    embedding = generate_embedding(query)
    Documentation.nearest_neighbors(:embedding, embedding, distance: "cosine").limit(3)
  end
  
  def get_related_docs(doc_id)
    doc = Documentation.find(doc_id)
    Documentation
      .nearest_neighbors(:embedding, doc.embedding, distance: "cosine")
      .where.not(id: doc_id)
      .limit(3)
  end
  
  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end
  
  def format_docs_for_llm(docs)
    docs.map do |doc|
      {
        id: doc.id,
        title: doc.title,
        content: doc.content.truncate(1000)
      }
    end.to_json
  end
end

This is a real step up in complexity. We’re now orchestrating multiple LLM calls with tool use. The LLM can search multiple times, explore related documents, and build up context before answering.

Cost per query:

• Database: 30-90ms (multiple vector searches)
• OpenAI embeddings API: $0.00002-0.00006 (2-6 embedding calls)
• OpenAI chat API: $0.015-0.045 (3-5 LLM calls with larger context)
• Total latency: 3-8 seconds

Notice the variance. Some queries get answered in one iteration. Complex ones might do three searches with different phrasings, explore related documents, and make five total LLM calls.

When this works better: That comparison query from before (“What’s the difference between Sidekiq and Good Job?”) now works great. The LLM searches for “Sidekiq background jobs”, gets those docs, then searches for “Good Job background jobs”, gets those docs, then synthesizes a real comparison.

Multi-part questions work too. “How do I set up Stripe payments and handle webhooks?” triggers two separate searches that gather comprehensive information.

When this gets expensive: Every query where the LLM decides it needs more information costs you 3-5x more. If your users ask a lot of complex questions, your API bill climbs fast.

The latency is also noticeable. Eight seconds feels slow in a web UI. You need to stream responses or show progress indicators.

Approach 4: Full conversational agent with external tools

Now we’re building a real agent that can search your documentation, browse external sites, and maintain conversation context across multiple turns.

class DocumentationAgent
  def initialize(session_id, client: OpenAI::Client.new)
    @session_id = session_id
    @client = client
    @conversation_history = load_conversation_history
  end
  
  def chat(message)
    @conversation_history << { role: "user", content: message }
    
    loop do
      response = call_llm_with_tools
      
      break if response[:finish_reason] == "stop"
      
      if response[:tool_calls]
        handle_tool_calls(response[:tool_calls])
      else
        break
      end
    end
    
    save_conversation_history
    
    {
      response: @conversation_history.last[:content],
      sources: extract_sources
    }
  end
  
  private

  def call_llm_with_tools
    response = @client.chat(
      parameters: {
        model: "gpt-4o",
        messages: @conversation_history,
        tools: tool_definitions,
        temperature: 0.3
      }
    )

    message = response.dig(:choices, 0, :message)
    @conversation_history << message

    {
      finish_reason: response.dig(:choices, 0, :finish_reason),
      tool_calls: message[:tool_calls]
    }
  end

  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end

  def tool_definitions
    [
      {
        type: "function",
        function: {
          name: "search_internal_docs",
          description: "Search our Ruby gem documentation",
          parameters: {
            type: "object",
            properties: {
              query: { type: "string" }
            },
            required: ["query"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "fetch_external_url",
          description: "Fetch content from an external URL like GitHub or RubyGems.org",
          parameters: {
            type: "object",
            properties: {
              url: { type: "string" }
            },
            required: ["url"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "search_github",
          description: "Search GitHub repositories and code",
          parameters: {
            type: "object",
            properties: {
              query: { type: "string" }
            },
            required: ["query"]
          }
        }
      }
    ]
  end
  
  def handle_tool_calls(tool_calls)
    results = tool_calls.map do |tool_call|
      function_name = tool_call.dig(:function, :name)
      arguments = JSON.parse(tool_call.dig(:function, :arguments) || "{}")

      result = case function_name
      when "search_internal_docs"
        search_internal_docs(arguments["query"])
      when "fetch_external_url"
        fetch_external_url(arguments["url"])
      when "search_github"
        search_github(arguments["query"])
      end

      {
        role: "tool",
        tool_call_id: tool_call[:id],
        content: result.to_json
      }
    end

    @conversation_history.concat(results)
  end
  
  def search_internal_docs(query)
    embedding = generate_embedding(query)
    docs = Documentation.nearest_neighbors(:embedding, embedding, distance: "cosine").limit(5)

    docs.map { |d| { title: d.title, content: d.content.truncate(800), source: "internal", id: d.id } }
  end
  
  def fetch_external_url(url)
    # In production, use a proper HTTP client with timeouts and error handling
    response = HTTP.timeout(5).get(url)
    
    {
      url: url,
      content: extract_main_content(response.body.to_s).truncate(2000),
      source: "external"
    }
  rescue HTTP::Error => e
    { error: "Failed to fetch URL: #{e.message}" }
  end
  
  def search_github(query)
    # Use Octokit or similar
    client = Octokit::Client.new(access_token: ENV['GITHUB_TOKEN'])
    results = client.search_code(query, language: "ruby")
    
    results.items.first(3).map do |item|
      {
        name: item.name,
        repo: item.repository.full_name,
        url: item.html_url,
        source: "github"
      }
    end
  rescue Octokit::Error => e
    { error: "GitHub search failed: #{e.message}" }
  end
  
  def load_conversation_history
    cache_key = "agent_conversation:#{@session_id}"
    JSON.parse(Rails.cache.read(cache_key) || "[]")
  end
  
  def save_conversation_history
    cache_key = "agent_conversation:#{@session_id}"
    # Keep last 10 messages to control context size
    trimmed_history = @conversation_history.last(10)
    Rails.cache.write(cache_key, trimmed_history.to_json, expires_in: 1.hour)
  end
  
  def extract_sources
    @conversation_history
      .select { |msg| msg[:role] == "tool" }
      .flat_map { |msg| JSON.parse(msg[:content] || "[]") }
      .select { |item| item.is_a?(Hash) && item["source"] }
      .uniq { |item| item["id"] || item["url"] || item["title"] }
  end
end

This is a different beast. We’re maintaining conversation state, hitting external APIs, and letting the LLM orchestrate complex research tasks.

Cost per conversation turn:

• Database: 15-50ms
• External API calls: 200-2000ms (GitHub, external sites)
• OpenAI embeddings: $0.00001-0.00005
• OpenAI chat: $0.02-0.15 (larger context windows, multiple turns)
• Total latency: 5-15 seconds

When this is worth it: You’re building a research assistant or technical support bot where users have complex, multi-turn conversations. They ask follow-up questions, need you to check external sources, and expect the system to remember context.

A user might ask “What’s the best gem for image processing?”, then follow up with “Show me examples from the ImageMagick wrapper”, then “Is there a more modern alternative?” The agent maintains context and can search different sources for each question.

When this is overkill: Most search features. If users are doing one-off queries and moving on, you’re paying for conversational capabilities they don’t need.

Choosing your approach

I’ve built all four of these systems in production. Here’s how I decide which to use.

Start with enhanced traditional search if:

• You have well-written documentation with consistent terminology
• Queries are mostly straightforward lookup tasks
• You need predictable costs and latency
• Your document corpus is under 10,000 items

The cost difference matters. At 1,000 queries per day, enhanced traditional search costs you $2/day. Basic RAG costs $3/day. Agentic RAG costs $15-45/day. A full agent costs $100-300/day.

Move to basic RAG when:

• Users phrase questions differently than your docs
• Keyword search returns poor matches for valid queries
• You have good quality source documents
• Your corpus is large enough that keyword search becomes unwieldy (50,000+ documents)

You’ll know you need this when users complain that search doesn’t work, and you look at their queries and think “we have docs about that, but they’re using different words.”

Move to agentic RAG when:

• Users ask complex questions requiring multiple sources
• You see patterns of users doing multiple searches in sequence
• Simple RAG returns incomplete answers
• You have budget for 3-5x higher API costs

Watch your analytics. If users do three searches in a row and then give up, they’re manually doing what an agentic system would do automatically.

Build a full agent when:

• You’re building a product feature, not just search
• Users need multi-turn conversations with context
• You need to integrate external data sources
• You have engineering resources for proper tool integration and error handling

The engineering complexity here is significant. You need proper timeout handling, retry logic, conversation state management, and graceful degradation when external APIs fail. This is a feature, not a quick enhancement.

The implementation details that matter

Some practical considerations that aren’t obvious from the code samples.

Chunking strategy for vector search: Don’t just embed entire documents. Break them into logical chunks. For documentation, consider chunking by section with overlap:

class DocumentationChunker
  CHUNK_SIZE = 1000 # characters
  OVERLAP = 200
  
  def chunk(document)
    sections = document.content.split(/^##\s+/)
    
    sections.flat_map do |section|
      break_into_overlapping_chunks(section)
    end
  end
  
  private
  
  def break_into_overlapping_chunks(text)
    chunks = []
    start = 0
    
    while start < text.length
      chunk_end = start + CHUNK_SIZE
      chunks << text[start...chunk_end]
      start += CHUNK_SIZE - OVERLAP
    end
    
    chunks
  end
end

This means each document generates multiple rows in your database with different embeddings. Your vector search returns chunks, not whole documents.

Hybrid search combines the best of both:

def hybrid_search(query)
  # Keyword search results
  keyword_results = Documentation.search_content(query).limit(20)
  
  # Vector search results
  query_embedding = generate_embedding(query)
  vector_results = Documentation.nearest_neighbors(:embedding, query_embedding, distance: "cosine").limit(20)
  
  # Combine with reciprocal rank fusion
  combine_results(keyword_results, vector_results)
end

def combine_results(keyword_results, vector_results)
  scores = Hash.new(0)
  
  keyword_results.each_with_index do |doc, i|
    scores[doc.id] += 1.0 / (i + 60)
  end
  
  vector_results.each_with_index do |doc, i|
    scores[doc.id] += 1.0 / (i + 60)
  end
  
  Documentation.where(id: scores.keys).sort_by { |doc| -scores[doc.id] }
end

This gives you semantic understanding from vectors plus precise keyword matching. The reciprocal rank fusion formula is from research on combining search results. It works better than naive score addition.

Caching saves you money:

class CachedRagSearcher
  def search(query)
    cache_key = "rag_search:#{Digest::MD5.hexdigest(query)}"
    
    Rails.cache.fetch(cache_key, expires_in: 1.hour) do
      perform_search(query)
    end
  end
end

Popular queries get asked repeatedly. Cache the embeddings and the LLM responses. This cuts your API costs dramatically for common questions.

The above caching method uses the raw query as the cache key. If users ask the same question with slightly different wording, they won’t hit the cache. You might want to normalize queries before hashing them for the cache key. For example, you could lowercase the query, remove stop words, etc.

Monitor your failure modes:

class SearchMetrics
  def self.track(query, approach, results)
    SearchLog.create!(
      query: query,
      approach: approach,
      result_count: results.length,
      latency_ms: results[:latency],
      cost_cents: calculate_cost(results),
      user_clicked: false # updated when user clicks a result
    )
  end
end

Track which results users actually click. If they click the first result, your search works. If they reformulate their query three times, it doesn’t. This data tells you whether to upgrade your approach.

What I actually recommend

Build the simplest thing first. Most Rails apps should start with Postgres full-text search plus GPT summarization. It costs almost nothing, has predictable latency, and works fine for straightforward queries.

Add instrumentation immediately. Track user behavior, measure latency, and watch your API costs. You need this data to know if upgrading is worth it.

When you see concrete evidence that simple search fails for your use case, add vector embeddings. This is a real improvement for semantic search. The pgvector extension makes this straightforward in Postgres. You don’t need a separate vector database until you have millions of documents.

Only add agentic features when you can point to specific query patterns that need them. “Users ask comparison questions and we don’t have comparison docs” is a good reason. “Agents are cool and I want to try them” is not.

Save full conversational agents for when you’re building a product feature that needs conversations. This is engineering work, not just adding a library. Budget for it appropriately.

The hype cycle pushes developers toward complexity. Resist it. Your users don’t care about your architecture. They care about getting answers quickly and cheaply. Often the simplest approach gives them exactly that.

But does the code work? See this repo for an implementation of all four approaches.

Disclaimer: The code samples in this post are simplified for clarity. In production, you need proper error handling, timeouts, retries, and security considerations (e.g., sanitizing user input before embedding). The costs and latencies mentioned are estimates based on recent OpenAI pricing and typical response times; your actual costs may vary based on usage patterns and model choices. Always monitor your API usage and costs when deploying AI features. Void where prohibited.

Introduction

While studying for job interviews I wanted to create several new Rails applications to test my knowledge. I found myself repeating the same steps over and over again: creating a new Rails project, installing gems, configuring the database, setting up authentication, and customizing the project structure. Wait. Why is it failing? Oh, I forgot to add the --database=postgresql flag. It was frustrating and time-consuming. That’s when I discovered Rails application templates. Here’s an example:

# $XDG_CONFIG_HOME/rails_templates/default.rb

def add_gems
  gem_group :development, :test do
    gem 'rspec-rails'
    gem 'factory_bot_rails'
    gem 'faker'
  end
  
  gem 'devise' if @using_devise
end

def get_user_input
  @using_devise = yes?("Do you want to add Devise for user authentication?")
end

def add_rspec_config
  initializer 'generators.rb', <<-CODE
    Rails.application.config.generators do |g|
      g.test_framework :rspec,
        fixtures: false,
        routing_specs: false,
        model_specs: false,
        helper_specs: false
      g.fixture_replacement :factory_bot, dir: 'spec/factories'
    end
  CODE
end

def add_devise = generate 'devise:install'
def add_rspec  = generate 'rspec:install'

# main setup
get_user_input
add_gems
after_bundle do
  add_devise if @using_devise
  add_rspec
  add_rspec_config

  # prepare the database
  rails_command 'db:drop db:create db:migrate db:test:prepare'

  git :init
  git add: '.'
  git commit: %Q{ -m 'Initialize repository' }
  
  say
  say 'Your Rails app is ready to go!', :green
  say
  say 'Next steps:'
  say "  $ cd #{app_name} && ./bin/dev", :yellow
  say
end

The template file is then provided as an argument when creating a new Rails app:

rails new myapp -m $XDG_CONFIG_HOME/rails_templates/default.rb

However, since I always wanted to use the same template with postgres and bootstrap options, it was easier to add the following shell function to my .zshrc file:

rails_new () {
  if [[ $# -eq 0 ]]
  then
    echo "Usage: rails_new <app_name>"
    echo "rails new \"\$@\" --css=bootstrap -T -d postgresql -m \$XDG_CONFIG_HOME/rails_templates/default.rb"
    return 1
  fi
  rails new "$@" --css=bootstrap -T -d postgresql -m $XDG_CONFIG_HOME/rails_templates/default.rb
}

Now I can create a new Rails app with the desired configuration by simply running:

rails_new myapp # plus any other options

That’s it! No more manual setup, no more forgetting to add the right flags.

A few additional features

1. The location of the template can also be a URL. This is useful for sharing templates across teams or projects.

2. You can define custom generators, such as for controllers, models, or scaffolds, to further customize the project setup.

3. Application templates can also be applied to existing projects to add new configurations or dependencies. This might be useful if you need to update mulitple projects. In that case, the command would look like this:

   rails app:template LOCATION=~/project_zebra_updates.rb

Check out the Rails Guide for Application Templates for more information.