What the Data Says About AI Agents in Marketing Today

We were running blind for longer than we'd like to admit. Vendors were telling us their agents would cut content production time by 60%, and we had no real way to verify that number against our own workflows. So we built the measurement infrastructure ourselves - and what we found was messier, more useful, and significantly different from what anyone had claimed.

The Setup: What We Were Actually Dealing With

Marketing decisions were being made on vendor slide decks and single-campaign tests. Someone would run one agent on one brief, see a time saving, and extrapolate that into a company-wide rollout plan. That's not data. That's a story built around a data point.

The obvious fix - run a proper A/B test - had a real constraint. One test on one campaign type, with one content team, in one month, doesn't tell you much. It tells you what happened in that specific context. Agents perform differently across task types, user skill levels, and brief quality. A test that doesn't account for those variables is just a more expensive hunch.

We needed something broader. Specifically: real usage data across dozens of teams, actual time saved versus claimed savings, and enough signal to separate the cases where agents genuinely helped from the cases where they just shifted work around.

How We Collected and Structured the Data

We pulled from four sources. Agent usage logs, which every run generates automatically. Time-tracking integrations, which we set up to capture task duration before and after agents were introduced. Content output metrics - draft count, revision count, approval rate, publish rate. And user surveys, run bi-weekly, to capture things the logs couldn't: whether people actually used what the agent produced or rewrote it from scratch.

The schema we built was straightforward. An events table with columns for agent_id, task_type, user_role, time_before_agent, time_after_agent, output_used (boolean), revision_count, and approval_status. That last column - output_used - turned out to be the most important field in the table, and it was almost the one we didn't build.

The early version of our tracking didn't capture whether users actually incorporated the agent's output. We were logging task completion time and assuming that if the task was done, the agent had contributed. It hadn't, in a lot of cases. Users were running the agent, reading the output, and then writing their own version anyway. Silent rejections. That skewed our time-saving numbers significantly upward. We caught it during a manual audit of ten tasks where the logged time saving looked implausibly high. Fixed it by adding the output-used flag and back-filling a sample through user interviews.

What the Data Actually Showed

The headline finding was counterintuitive: agents cut content production time by roughly 40% on average, but that average masked a split that mattered a lot. Teams with clear, structured briefs got close to the claimed savings. Teams with vague inputs - where the brief itself was underspecified - saw agents generate output that required so much editing it negated most of the time gain. The agent didn't make a bad brief better. It made it faster to produce something that still needed rework.

Breaking it down by role showed the same pattern in a different form. Copywriters saved about 35 minutes per week on average. Editors saved 12. Product marketers saved under 5. The reason wasn't that agents were better for copywriters - it was that copywriters had more routine, repeatable tasks where agents could do meaningful first-draft work. Editors were spending most of their time on judgment calls that agents couldn't make. Product marketers were dealing with briefs that changed frequently and required context the agent didn't have.

The vendor pitch that contradicted most cleanly: we were told agents would reduce revision cycles. They didn't. They reduced first-draft time. Revision cycles stayed roughly constant and in some cases went up slightly, because the agent's first draft was good enough to get reviewed but not good enough to get approved without changes. More things entered the review queue. The queue got longer. This dynamic is worth understanding in depth - the question of why AI agents still need editorial judgment at the review stage gets at exactly why faster drafts don't automatically mean faster approvals.

The Gotcha: Where the Numbers Broke Down

We measured time saved. We didn't initially measure quality consistency. And that turned out to matter a lot more than we expected.

Teams that used agents heavily had faster output but higher rejection rates on first review. One content team using agents for brief generation was saving 8 hours a week in production time. Looked great in the usage data. But their approval rate had dropped from 92% to 78% over the same period. That 14-point drop translated into real rework - briefs going back to writers, extra review cycles, delayed publish dates. When we calculated the cost of that rework in hours, it erased more than half of the time saving they were logging.

The fix was a quality-adjusted metric: time saved minus time spent on rework, measured at the task level rather than the team level. That meant we needed to connect the events table to downstream approval data, which required a join across two systems that hadn't been built to talk to each other. It took a few days to wire up. But once we had it, the picture changed substantially. The agents that looked like clear wins on time alone looked more modest on quality-adjusted time. And a few agents that looked marginal on raw time actually came out ahead once you factored in their lower rework rate.

Why This Matters: The Broader Lesson

Vendor claims about AI agents are almost always incomplete. They measure what's easy to measure - speed, draft count, words generated per session - and skip what's hard to measure: quality consistency, downstream friction, and the cost of rework that happens three steps later in the workflow. That's not always bad faith. It's just that vendors measure what makes their product look good, and first-draft speed makes any AI product look good.

If you're evaluating agents for your team, the headline number is the wrong place to start. Instrument your own workflow. Track what actually matters to your specific output: approval rates, revision cycles, downstream rejection rates, time to publish - not just time to draft. Most B2B teams also make a related mistake at the measurement layer itself - defaulting to content ROI metrics that obscure downstream friction rather than exposing it. Measure across a real distribution of task types and brief quality, not just the best-case scenario. And build the quality-adjusted metric before you scale anything, because the teams that skipped that step are the ones who rolled out agents broadly, saw slower timelines six months later, and couldn't explain why.

The data exists in your own systems. The question is whether you've structured it in a way that lets you read it honestly. Part of that is also a governance question: once you have the measurement infrastructure in place, you still need to decide how much editorial autonomy to extend to your agents as their performance data improves.