Why Most AI Fails on Benefits Data

Every carrier leadership team is circling the same question right now: how do we point AI at the data that actually drives distribution and strategy?

The answer most vendors offer is seductive in its simplicity. Take Form 5500 filings, attach a large language model, let people ask in plain English. Summarize this filing. Compare these employers. Tell me about this broker.

The demo lands. The production results disappoint. And the reason is not that Form 5500 is a weak source; it is one of the most comprehensive regulatory datasets in employee benefits. The reason is that a model pointed at filings can only ever give you filing summaries. A model pointed at structured intelligence can give you market answers. The distance between those two outcomes is the engineering described in the first two articles of this series: normalization, validation, resolution, attribution, estimation, mapping.

Most AI products skip every bit of it. They ingest filings or filing text, drop it in a vector store, and ask a model to narrate what it finds. In benefits, that approach breaks in three specific, repeatable places.

Ask a model wired to raw filings: "Who brokers the dental coverage for this employer?"

It reads Schedule A on the latest filing and returns a name, usually a large brokerage brand listed as broker of record. Confident, fast, and frequently the wrong office.

What it cannot do is tell whether that name is the office that produced the business or a central lockbox that merely filed the paperwork. National brokerages routinely file out of one hub while the relationships live in branches across the country. Schedule A names the filer. The market runs on the producer. They are often different offices inside the same firm, and the model has no concept that the distinction exists.

A distribution manager who acts on the lockbox answer builds a territory around an accident of paperwork. A carrier measuring broker penetration counts hub volume instead of real production. The model is not lying; it is reading the filing exactly as written, and the filing was never built to answer the question.

Make it concrete. A carrier's Southeast director asks which broker office to prioritize for a dental cross-sell push in Atlanta. The raw-filing model returns one national brand as broker of record on twelve target employers. Structured intelligence returns three distinct producing offices: one in Atlanta, one in Charlotte filing centrally, one in Nashville running a satellite relationship. Each comes with attributed premium, compensation benchmarks, and book overlap computed at the office level. The campaign built on the first answer calls the wrong offices. The one built on the second is a plan.

Benefits relationships move. Carriers get swapped at renewal. Brokers carry accounts with them. Employers merge, rebrand, change EINs. Plans amend mid-year.

A filing database stores each submission as a row, and a model pointed at those rows treats them as independent documents. It summarizes the 2024 filing and the 2022 filing as separate things, with no thread connecting them. Three failures follow:

• It misses the switch. It can name the current carrier and tell you nothing about when the change happened, what came before, or whether the move tracked a broker change or a compensation shift.
• It invents churn. An employer changes EIN after an acquisition, and the model sees two different employers. A broker looks like it lost an account it still services under a new entity.
• It cannot draw a trend. "How has this employer's premium moved over three years" needs aligned plan histories, not three disconnected summaries. Without temporal alignment, the model recites three numbers and calls it analysis.

Plan hierarchy (employer, plan, product, carrier, office, agent) combined with a real timeline is what makes multi-year analysis possible. A model on flat filing text has no hierarchy to walk and no clock to follow. It can describe any single snapshot. It cannot model change.

The third failure is the quietest and the most expensive for a strategy team.

A model on filings narrates what a filing contains. It does not reason across how entities relate.

Ask: "Which brokers gained share in my territory over the last three years?" A summarization approach simply cannot answer. It can summarize a filing. It can compare two employers if you hold its hand. It cannot walk a broker-employer-carrier graph across 791,300+ employer records, compute share shift at the office level, and rank brokers by change in attributed premium, because that requires relationship mapping on a resolved, attributed, time-aligned graph.

Ask: "Show me employers where dental premium is flagged high against peers and the broker's dental compensation is above benchmark." That needs per-product estimation, compensation classification across 14 fee types, and benchmark flags, none of which exist until modeling has run.

Ask: "Which employers in my territory carry high 401(k) loan rates and thin voluntary coverage?" That needs retirement and group benefits fused on one profile, 838,500+ retirement plans connected to group data for cross-sell signal.

These are relationship questions, and they are the questions distribution and strategy teams actually ask. A model that summarizes filings is answering a different and lesser question, one that demos beautifully and falls apart in a quarterly review.

The distinction runs through the whole series:

Filings are the input. Intelligence is the engineered output. Point AI at filings, get summaries. Point it at intelligence, get answers.

One question a carrier analyst runs constantly:

"Who is the producing broker for this employer's dental line, and how does their dental compensation compare to peers?"

A model on raw filings answers:

"Per the 2024 Form 5500, the broker of record on Schedule A is [Large Brokerage]. Total plan premium is $4.2M. Reported broker compensation is $142,000. The filing does not break out dental-specific compensation or identify the producing office."

Accurate. Useless for a dental cross-sell conversation.

A model on the intelligence layer answers:

"Dental for [Employer] is attributed to [Producing Office] in [City, State], not the firm's central filing hub. Modeled dental premium is $380,000, flagged high against peer plans of this size and industry. Dental compensation breaks down as commission ($28,400) and service fee ($12,100), both flagged above benchmark for dental in this segment. The producing agent is [Agent] with verified contact information. Two similar employers within 15 miles run the same office with dental premium flagged normal, a clear positioning angle."

Same employer. Same filing source. Different engineering beneath the question, and a different category of answer.

The AI conversation in benefits is moving fast, and most of what launches is a model bolted onto filings calling itself intelligence.

The products that will matter are the ones wired to structured intelligence, the full pipeline from ingestion through semantic intelligence cubes. They inherit resolution, attribution, estimation, and mapping. They answer relationship questions instead of summary questions. They fail honestly when data is missing instead of confidently filling the gap.

That is the architectural bet we made. The intelligence layer is the product. AI is how you reach it, but only after the foundation exists.

The next article makes the architectural case directly: why the data layer beneath the model, not the model itself, is the actual product, and why schema, validation, and relationship modeling decide outcomes far more than model choice.