If I described the systems I built at SoundCloud and Honeycomb as AI agents, the description would be technically convenient and mostly wrong.
There were conversational interfaces. There were LLMs. There were automated workflows. There were systems that could retrieve information, reason over it, generate outputs, and eventually trigger additional work.
But the model was the least interesting part.
The difficult work was building the pipeline around it.
I needed to get fragmented company data into a form that could be reasoned over consistently, preserve source evidence, distinguish deterministic work from interpretive work, encode repeatable analytical methods, connect the system to the places people already worked, and evaluate whether its conclusions were actually trustworthy.
The architecture was much closer to this:
source systems, extraction, normalization, persistent evidence, retrieval, task-specific reasoning, audience-specific output, action, new evidence.
At SoundCloud, the sources were heavily weighted toward support interactions, product behavior, knowledge, and business intelligence.
At Honeycomb, the same architecture expanded across Gong, Salesforce, Snowflake, Product, Research, Support, Marketing, Slack, competitor information, and other GTM systems.
The technology changed somewhat between implementations. The design principles did not.
I started by deciding what the LLM should not do
One of the easiest ways to make an AI system unreliable is to ask the model to do work that normal software already does better.
I did not need an LLM to count competitor mentions. I did not need it to calculate a rate. I did not need it to determine whether a timestamp occurred in the last 30 days. I did not need it to join account identifiers, retrieve a known field, or perform straightforward aggregation.
SQL, Python, APIs, application logic, and workflow automation are better for that work because they are deterministic. The same input should produce the same result.
The LLM became valuable when the work stopped being deterministic.
A support customer might say: "I'm paying for Go+, why are you still asking me to upgrade?" The system may need to infer that the underlying issue concerns entitlement, billing state, or subscription recognition even though the customer never uses those words.
A prospect might say: "We're trying to get away from needing a bunch of people every time something goes wrong." That may imply an observability workflow problem without naming a competitor, product category, or feature.
Those are semantic interpretation problems. LLMs are useful there.
The division of labor became simple in principle. Use software when the answer can be calculated. Use the model when the answer has to be interpreted. Then evaluate the interpretation.
That boundary was one of the most important architecture decisions I made.
The first layer was always the raw evidence
At SoundCloud, the raw qualitative layer included Zendesk tickets, Otto conversations, Help Center interactions, survey feedback, related support artifacts, Product and Engineering documentation, and Marketing content. The quantitative and contextual layer included product usage, business metrics, and the existing business intelligence environment.
At Honeycomb, the evidence surface expanded considerably. It included Gong, Salesforce, Snowflake, product usage, Support, User Research, Product and Engineering documentation, Marketing content, Slack, Apollo, Pylon, and external competitive information.
The common mistake would have been to dump all of that into one enormous retrieval system and let the model figure it out. I did not want that.
Raw company data is full of ambiguity. The same concept can have different names in different systems. A support system may use one taxonomy while Product uses another. Salesforce may identify an account differently from Gong. A buyer may mention a competitor in shorthand. Product documentation may describe intended behavior while support evidence describes actual behavior.
Retrieval does not fix those differences. The system needed a normalization layer.
I converted conversations into structured evidence
This was the first place the LLM became genuinely useful.
Support tickets and sales calls contain a tremendous amount of information, but most of it is embedded in natural language with a ton of pii.
First, I redacted identity before I extracted evidence
Support tickets and sales calls contained useful business evidence, but they also contained personally identifiable information that the reasoning layer didn’t need.
Before a conversation entered the extraction pipeline, I ran it through Google Cloud Sensitive Data Protection, which was previously called the Cloud DLP API. Its inspection method returned structured JSON describing what it found, including the type of sensitive information, the likelihood of the match, and its exact location in the text.
I used those findings in the agent code to redact the detected spans before sending the conversation to the model. That kept redaction deterministic and inspectable. I wasn’t asking an LLM to decide what looked private every time it processed a ticket or call.
The harder design question was what to preserve. A Support ticket still needed to tell us what the customer was trying to do, where they got stuck, which product area was involved, what the automated system attempted, and whether the issue was eventually resolved. A sales call still needed to preserve the buyer’s problem, objections, evaluation criteria, technical concerns, alternatives, and language.
The system needed the evidence. It didn’t need the person’s identity.
Doing this before extraction also meant the redaction carried through the rest of the pipeline. Structured signals, retrieval results, analytical prompts, and downstream outputs could reuse the conversation’s meaning without repeatedly moving its PII through every layer of the system.
Then, I turned the evidence into something meaningful.
At SoundCloud, I wanted to extract concepts such as customer intent, product area, issue type, suspected root cause, resolution path, channel, bot outcome, human outcome, content used, customer segment, and journey stage.
The broader 5W1H model helped impose structure: who is experiencing the problem, what are they trying to accomplish, when and where does the problem occur, why does it appear to happen, and how are they attempting to resolve it?
At Honeycomb, the extraction problem changed but the pattern stayed the same. Gong conversations could be transformed into signals around competitors, alternatives, customer pain, buying criteria, objections, technical concerns, desired capabilities, perceived weaknesses, win themes, loss themes, and buyer language.
The key design choice was that the structured extraction did not become the conclusion. It became evidence the next layer could reason over.
That distinction matters. If an LLM analyzes 1,000 calls from scratch every time someone asks a question, the system is expensive, slow, difficult to evaluate, and inconsistent. If those calls are first converted into reusable structured signals while preserving the connection to the original conversation, many downstream questions become much easier.
The model interprets once. The system reuses the evidence many times.
Normalization made cross-source reasoning possible
Extraction by itself was not enough. The signals had to mean the same thing across sources.
At SoundCloud, one of the central problems was inconsistent categorization between Otto and human support. A customer issue could effectively become a different issue depending on which path the customer took through the support experience. That made aggregate analysis unreliable.
The answer was not simply better embeddings. The answer was a shared conceptual model.
If several phrases all describe duplicate billing, the system needs to know they belong to the same underlying issue while still preserving the original customer language. If a support conversation concerns uploads and a Product document calls the same area track ingestion, the intelligence layer needs a relationship between those concepts.
At Honeycomb, similar normalization was necessary for competitor names, product areas, account identity, opportunity relationships, objections, themes, and buying criteria.
The taxonomy did not need to eliminate nuance. It needed to create enough common structure that evidence from different systems could be compared.
This became one of the least glamorous and most important parts of the architecture. AI can reason over messy information surprisingly well. That does not mean you should make it.
I separated evidence from interpretation
Once information was normalized, I wanted to preserve another important boundary. Evidence and interpretation should not occupy the same layer.
Imagine the system finds 27 opportunities where buyers mention a particular competitor. Twelve mention pricing. Nine raise a particular technical concern. The company wins eight and loses nineteen. Those are observations.
The system may then infer that the competitor has become more dangerous in price-sensitive deals. That is interpretation.
A strategist may then recommend changing packaging, positioning, or Product investment. That is a decision.
If those layers collapse together, the AI can easily turn a plausible inference into an apparent fact.
So I increasingly thought about the system as a chain: source, extracted observation, aggregated pattern, inference, recommendation. Each step moves farther from the original evidence. Each step deserves more scrutiny.
This principle also applied at SoundCloud. A spike in tickets is measurable. A cluster of customer comments about confusion is evidence. The conclusion that a recent design change caused the confusion is an inference. The decision to redesign the workflow is a recommendation.
A strong system should help humans move through those layers without pretending they are interchangeable.
The retrieval layer needed to be selective
Another architecture mistake I wanted to avoid was treating more context as automatically better. It is not.
Sending every support ticket, every Gong call, every Product document, every Salesforce field, and every Slack thread into the same prompt would make the system worse.
The relevant context depends on the question. If a PM asks why adoption changed for a particular feature, the system may need product telemetry, recent support themes, relevant Product documentation, a recent launch history, and perhaps customer research around the workflow. It probably does not need every competitor battle card.
If an AE asks how to handle a competitor in a specific account, the useful context changes. Now the account, opportunity history, Gong conversations, competitive patterns, relevant product capabilities, recent releases, and previous win/loss evidence matter.
The job of the context layer was therefore not to maximize retrieval. It was to assemble the smallest useful evidence set for the decision at hand.
This is one reason I resist describing these systems as "chat with all your company data." That sounds appealing. It is also a poor architectural objective. The better goal is to retrieve the right evidence, with enough source context to evaluate it.
Claude Projects gave me a controlled reasoning environment
At Honeycomb, Claude Projects became one of the working environments for the intelligence layer. Projects made it possible to create a more controlled context around the reasoning than a blank general-purpose chat.
But Projects alone were not enough. A model with access to the right information can still reason inconsistently if every analysis begins with a different set of instructions.
That led to one of the most important choices in the system: reusable skills.
I treated skills as code
I did not want high-value analytical workflows to depend on whoever happened to write the best prompt that day. So repeatable reasoning patterns became Claude skills.
A Win/Loss Mining skill had a defined purpose and analytical method. It could look for recurring loss patterns, competitive signals, objections, and repeatable win themes.
A Discovery and Multithreading skill could evaluate whether the right stakeholders had been engaged, what had been learned in discovery, and where important context was missing.
Whitespace Analysis looked for opportunity expansion. Deal Hygiene and Enablement focused on the quality and completeness of commercial execution. Board, GTM, and QBR narrative skills converted evidence into executive-level synthesis.
The important architectural point was not the specific list. It was that I was moving reasoning out of ephemeral prompts and into shared operational infrastructure.
Those skills lived in a version-controlled GitHub repository. That meant they could be reviewed, changed intentionally, compared over time, shared, reused, improved, and eventually governed.
If a particular analytical method is going to influence a real decision, I want to know which version of that method produced the answer. That is difficult when the logic lives in somebody's chat history. It becomes possible when the logic is treated more like code.
I used deterministic tooling around the reasoning layer
The model did not need to orchestrate everything itself. Python, JavaScript, SQL, APIs, VS Code, workflow tooling, and systems like n8n could handle the mechanical parts of the pipeline.
A typical workflow might pull records from a source system, normalize identifiers, remove irrelevant fields, perform known transformations, calculate aggregations, and then hand a bounded evidence set to the model for interpretation. After the model returned a structured result, normal software could validate required fields, write output to another system, trigger a notification, or schedule the next run.
This approach made the architecture much more reliable than asking one giant autonomous agent to decide every step itself.
The agent did not need to decide whether 17 was greater than 12. The agent needed to determine whether 17 customer conversations represented the same underlying concern. Those are different problems. I tried to assign each one to the tool best suited to solve it.
MCPs expanded the system beyond retrieval
At Honeycomb, MCP became an important part of the way I thought about system connectivity. I eventually reduced integration capability to four verbs: read, write, trigger, and monitor.
Read means the model can retrieve evidence from a system. Write means it can persist an output or update a record. Trigger means it can initiate another action. Monitor means it can watch for a condition and surface something when that condition changes.
That progression is more important than the protocol itself.
A read-only system can answer what we know about a competitor. A system that can write might save a newly identified competitive insight into the relevant operating record. A system that can trigger might launch an analysis workflow when a significant opportunity reaches a certain stage. A system that can monitor might identify that competitor mentions have increased sharply over the last two weeks and alert the appropriate team.
The same progression applied to SoundCloud. Read support conversations. Write structured classifications. Trigger deeper analysis when a pattern crosses a threshold. Monitor for unusual increases in a support theme.
That is the difference between a chatbot and an operational intelligence layer.
Persistent state mattered
One of the weaknesses of conversational AI is that valuable work can disappear into the conversation.
At Honeycomb, Notion played an important role as persistent operational state through the Opportunity Hub. The system could capture opportunities, experiments, workflows, findings, implementation status, ownership, and future work outside of the conversational interface.
That separation mattered. Claude could reason. Notion could persist. Slack could distribute. GitHub could version the logic. Snowflake and source systems could hold structured company data.
No one tool needed to become the entire operating system. I preferred composing the system out of tools that already had clear strengths.
Slack became the distribution layer
There were two basic ways people could interact with the intelligence system.
The first was pull. Someone had a question and asked the system.
The second was push. The system recognized that something had changed and brought the information to the appropriate person.
Slack was useful for the second pattern because that was already where people worked.
The design problem was restraint. An AI system can generate infinite alerts. That does not make it useful.
A useful notification needs a threshold for significance. A competitor mention increasing from two to three occurrences may not deserve anyone's attention. A new objection appearing across several large active deals might. A support theme increasing 300% after a release might.
The intelligence layer therefore needed not only detection but judgment about whether the detected change was meaningful enough to interrupt someone. That is where deterministic thresholds and LLM interpretation can work together.
I designed for provenance from the beginning
The more consequential the conclusion, the more important it became to preserve the source.
If the system said customers increasingly cared about a particular capability, I wanted the user to be able to inspect the evidence. Which calls? Which customers? Which tickets? Which accounts? Which Product documentation? Which time period? Were the conversations closed-won, closed-lost, or early-stage? Was the observation concentrated in one cohort? Did telemetry agree?
The answer should not require trusting the model because it sounds confident.
This was especially important when the system connected qualitative and quantitative evidence. Suppose product usage drops while support complaints rise. Those facts may be related. They may also be coincidental. The system should make the relationship visible without pretending correlation proves causation.
Source traceability gave the human reviewer a way to inspect that distinction.
I deliberately preserved contradictions
Most summarization systems are biased toward coherence. They take several sources and produce a clean story. That is sometimes exactly the wrong behavior.
At SoundCloud, Support could show customers repeatedly trying to use a capability while product analytics showed low successful adoption. That contradiction might reveal friction.
At Honeycomb, Sales might say customers were demanding a feature while Research suggested that the feature request was actually shorthand for another problem. Marketing might believe a differentiator was strategically important while win/loss analysis showed that it rarely affected buying decisions.
Those disagreements should survive synthesis. I would rather have a system tell me that Sales and product usage disagree than produce a neat paragraph explaining why everyone was basically saying the same thing.
Organizational intelligence is valuable partly because it can expose where different parts of the company are operating from different versions of reality.
Evaluation had to happen at several layers
Another lesson from the SoundCloud work was that there is no single AI accuracy metric for a system like this.
The extraction layer can be wrong. The normalization layer can be wrong. The retrieval layer can select poor evidence. The reasoning layer can misinterpret the evidence. The output can be technically correct but unhelpful. The final decision can still be bad even if every earlier step worked.
So evaluation needs to match the stage.
For support classification, I cared about whether intent was correct, whether routing worked, whether the right content was retrieved, whether the user resolved, whether they escalated, whether the conversation looped, and what happened downstream. At SoundCloud, combining bot and downstream ticket evidence improved decision accuracy by roughly 20%.
For competitive intelligence, I would evaluate whether competitor mentions were identified correctly, whether themes were supported by actual conversations, whether loss reasons matched human review, whether buyer statements were distinguished from seller interpretation, and whether aggregated conclusions remained true when humans inspected representative evidence.
I would also look at false positives and false negatives. Missing an emerging competitor can be costly. So can declaring a competitor strategically important because of several noisy mentions. The evaluation system needs to understand both errors.
This is where my principle of evaluating the evaluator came from. The reasoning system itself becomes another system that needs measurement.
Autonomy came after reliability
I used an autonomy ladder that moved from Human Only, to Human in the Loop, to Human on the Loop, and eventually to Fully Autonomous.
The objective was never to move everything toward full autonomy. It was to decide what level made sense for each part of the workflow.
A model can probably classify competitor mentions with limited human involvement once its accuracy is established. It can cluster repeated support issues. It can retrieve related evidence. It can draft a preliminary synthesis.
That does not mean it should autonomously decide to change a roadmap. The further the system moves from observation toward strategic recommendation, the more consequential the judgment becomes.
The question I used was: what evidence would I need before I trusted this system to make the decision without me? If I could not answer that question, the workflow was not ready for greater autonomy.
Trust before scale.
The next layer is temporal intelligence
If I were continuing these systems today, one of the biggest areas I would deepen is time.
Most internal AI systems answer questions about the current body of evidence. I care at least as much about what changed.
Which customer problem is appearing faster than before? Which objection emerged this month? Which competitor suddenly entered a segment where it rarely appeared historically? Did support contacts change after a Product launch? Did a new Marketing message alter how buyers described the problem? Did Product fix something but customer perception lag?
Those are temporal questions. They require storing evidence in a way that preserves time, not simply indexing the latest state.
A mature intelligence system should be able to answer both what is true and what became true recently. Often the second question is more strategically valuable.
I would also weight evidence more explicitly
Not every signal deserves equal influence.
A single early-stage prospect mentioning a competitor is different from the same concern appearing repeatedly in closed-lost opportunities. A support ticket from one highly unusual user is different from a theme appearing across thousands of users. A stated preference in an interview is different from observed product behavior.
A Product document describes intention. Telemetry describes behavior. Support describes experience. Research describes interpreted human context.
None of those are inherently superior. They are different kinds of evidence.
A more mature version of the architecture would explicitly retain source type, recency, sample size, outcome context, confidence, and other factors that help the system reason about how much weight to assign each signal. That would make synthesis more rigorous without pretending every conclusion can be reduced to a mathematical score.
The final architecture is a learning loop
The architecture I ended up converging on across both companies can be described fairly simply.
The company generates raw evidence through customer interactions, sales conversations, product behavior, business metrics, research, documentation, and market activity.
The system extracts reusable signals from the messy sources. Those signals are normalized enough to be compared across systems while keeping the original evidence attached.
Deterministic software handles calculation, filtering, joins, schedules, and known rules. The LLM handles ambiguity, semantics, synthesis, and audience-specific interpretation. Task-specific skills constrain repeatable reasoning. Persistent systems hold state. Slack and conversational interfaces distribute the result. Humans make or supervise consequential decisions.
What happens next creates new evidence. Then the loop runs again.
That is why I do not think of the systems I built as agents. An agent is one component. The thing I was really building was an intelligence pipeline that allowed an organization to continuously learn from information it was already producing.
And once I saw the pattern at SoundCloud, I recognized it immediately when the same problem appeared at Honeycomb.
The data was different. The business questions were different. The architecture was not.