AI Memory Platform Turns Model Selection Guesswork Into a Built-In Benchmarking Engine with Claude on AWS Bedrock
At a glance
A conversational AI memory platform for developers had no way to independently prove which foundation models performed best on accuracy, cost, or latency for a given workload, leaving customers to trust marketing claims or run costly manual comparisons themselves.
NMD built a model evaluation engine on AWS Bedrock that runs any developer-defined scenario, translation, summarization, Q&A, multi-turn reasoning, across multiple models side by side, scoring each on lexical accuracy (F1, ROUGE), semantic similarity, and LLM-as-judge benchmarks, with a “virtual mode” that estimates cost and latency instantly using cached pricing data instead of live API calls.
The platform can now offer developers a self-service way to test and compare models before committing production traffic to any of them, turning an internal QA exercise into a customer-facing product capability that differentiates the company from single-model AI memory competitors.
Industry
Use Case
Model Evaluation & Benchmarking, RAG / Conversational AI, MLOps & Platform Modernization
Solution implemented
- Built dual-mode evaluation engine on AWS Bedrock: virtual (estimate-only) and live (real API execution) scenario runners
- Implemented automated scoring: F1, ROUGE, cosine similarity, and LLM-as-judge comparison against golden outputs
- Integrated published memory benchmarks (LongMemEval, Locomo) as standardized accuracy references
- Stored per-model pricing, latency, and capability metadata in DynamoDB for instant virtual-mode estimates
- Ran turn-by-turn and session-by-session memory write/retrieval testing to isolate failure points
- Designed scenario framework (input, golden output, model list, embedding config) for repeatable, front-end-ready testing
The value equation
- Replaces manual, one-off model comparisons with a repeatable, self-service evaluation workflow
- Gives developers cost and latency estimates before running a single live API call
- Positions the platform as an evaluation layer, not just a memory API, opening a new product surface
- Provides defensible, standardized accuracy scoring instead of self-reported benchmark claims
Company Snapshot
Location
North America
Customer Situation
The company’s core product lets developers give AI assistants persistent memory, but proving that memory implementation was more accurate than a competitor’s required a defensible, repeatable measurement approach rather than internal spot-checks.
Developers building on top of the platform faced the same underlying problem from the other direction: with dozens of foundation models available, each with different pricing, latency, and accuracy profiles, there was no fast way to know which model fit a given workload without burning real production budget to find out.
This is an industry-wide gap: as the number of viable foundation models multiplies, most AI-native platforms lack any built-in way to let their own customers evaluate model tradeoffs before committing to one.
NMD Solution
NMD reviewed the company’s memory-write and retrieval pipeline and found that inconsistent, non-deterministic memory-write completion (calls returning without a memory operation ID) was suppressing benchmark accuracy independent of model quality. The fix required a turn-by-turn ingestion and polling architecture on Bedrock-hosted models, paired with a scoring layer combining lexical (F1/ROUGE), semantic similarity, and LLM-as-judge methods against LongMemEval and Locomo benchmark standards.
What We Delivered
The platform runs a full model evaluation engine on Bedrock: developers define a scenario, select models and embedding configurations, and run it live or virtually, receiving cost, latency, and accuracy scores per model in a single session. The evaluation logic (rate-limited memory writes with completion polling) also improved the reliability of the company’s core memory benchmark scores.
Ready to Transform Your Software Business?
See how New Math Data can transform your software business with AWS-powered innovation.