Production Agent Eval Gates That Catch Silent Drift
Accuracy in a notebook is not a release check. Hold a fixed eval set and stop promotions when the fail rate crosses a written threshold.
Chainweb Insights
We use this space to share how we think about product engineering, AI systems, blockchain delivery, automation, and IoT platforms in real client environments.
Accuracy in a notebook is not a release check. Hold a fixed eval set and stop promotions when the fail rate crosses a written threshold.
Live MQTT is for decisions that expire in minutes. The historian is for shift reviews and audits. Mixing the two in one operator screen creates stale alarms.
A Server Action is a public endpoint. Put auth and tenancy checks in the action, not in the calling component, and fail closed on missing session.
The risky LLM migration is identity, not model brand. Split the key, tag the workflow, then change models.
Split the trace into retrieval, model, and post-process. If retrieval is the long span, cache and join first. Quantize only after the eval still passes.
Treat the plant tag list as the API. Keep the names, keep the raw values, and only add sensors for points the PLC does not already have.
Stream the story. Block the price, the entitlement, and the button that spends money. Measure time-to-correct-price, not only LCP.
A shared model key hides the only number that matters. Tag every call with a workflow, cap that workflow, and charge the team that can change the prompt.
Write the request budget first. If the agent cannot finish inside it with one tool and no retries, take it off the click.
Use the factory password once, to join. After that each device has its own certificate, a 30-day lifetime, and a path to rotate across an air gap.
A server fetch is not free because it runs on the server. Set an RSC payload budget, select only what the component renders, and fail the build when the query grows.
Team-name tags die when the org chart moves. Charge the durable service, keep people in a mapping table, and do not move money until that mapping survives a reorg.