--- title: "27 findings from a consultancy review, worked through" description: "A Nordic consultancy tested an AI planner and returned a long list of faults. How that became 27 root-caused tickets, each fix verified against a running stack." url: https://torn.studio/en/work/consultancy-review-27-findings locale: en published: 2026-09-01 updated: 2026-06-25 --- # 27 findings from a consultancy review, worked through > **Short answer:** A Nordic consultancy tested an AI-driven planning canvas and reported it as untrustworthy. Torn Studio turned the report into 27 traceable tickets, root-caused them before writing any code, fixed them and verified against a running stack. At the verification on 25 June 2026, 22 were implemented, 3 partial and 2 deferred. **Client:** A Nordic digital transformation consultancy · **Delivered:** 2026-06-25 **The studio’s relationship to the client:** The consultancy is an independent party that evaluated the planning canvas in Changemkr, a platform Torn Studio owns part of and held the product role in. The studio received their material and did the work that followed. ## The brief A consultant at the firm had tested the planning canvas and its AI assistant in earnest and summarised the result in a presentation. The job was to turn that narrative into something that could be worked off, and to be able to show what actually got fixed. ## What made it hard - The material was a narrative of one session in the order things went wrong, with 27 complaints that might be fewer faults or more — which it was could not be read off the list. - The most serious finding was that the assistant wrote it had made changes that never happened, which makes the rest of the tool unusable however many layout faults get fixed. - Several tickets turned on model output that varies between runs, so they could not be verified with an ordinary test that expects the same output every time. ## Figures a reader can count | Figures a reader can count | | | | --- | --- | --- | | 27 | tickets from one testing document | TP-01 through TP-27 in the fix plan in the repository, each carrying the tester’s own wording and a mapped root cause. | | 22 / 3 / 2 | implemented, partial, deferred | Countable in the status table in the fix plan, dated 25 June 2026. Two rows cover several tickets each, which is why 24 rows carry 27 tickets. | | 11 / 1 | tests passed and skipped | The verification report in the repository names the seven specification files that ran against the stack and the command that re-runs them. | | 4 | tickets that were product decisions | Lifted out of the ticket list and settled as decisions in the plan, with the choice and its consequence written out. They have no code fix at all. | | 1 | bug found beyond the report | Ticket TP-15: an invalid reconnection deleted the existing connection. Found during the fix work and described in the status table. | ## How it was done ### The narrative became traceable tickets The presentation’s complaints were translated into 27 numbered tickets, TP-01 through TP-27. Each ticket carries the wording the tester used, so anyone can go back and read what the person actually saw. That part decides whether a testing round leads anywhere. ### Analysis before the first line of code A full pass ran with the code left untouched. Every ticket was mapped to a confirmed or hypothesised root cause, tickets sharing a cause were grouped, and four that turned out to be product decisions were lifted out and settled as decisions. Grouping first is what lets one fix close several symptoms. ### The most dangerous ticket first The assistant claiming changes it had not made was taken before anything else, because a user misled once stops trusting the next answer. The prompt now forbids the phrasing, and the interface shows a proposal as unapplied until a person presses the button. ### Verification against a running stack The fixes ran against the whole containerised stack, with Playwright driving the web app, the business API and the AI service at once. Eleven tests passed and one was skipped, the last because of a gesture no headless browser can drive. The report says which one and why. ### Residual gaps written out per ticket Every row in the report carries a column for what remains unresolved. A ticket where the tester’s case is fixed but an edge survives is marked partial, with the edge described. A testing round reported as entirely green cannot be trusted the next time round. **Stack:** Next.js, React Flow, FastAPI, LangGraph, Playwright, Podman, PostgreSQL ## The limits of this proof This shows that a sprawling body of test feedback can be turned into a plan that can be followed up row by row. It shows nothing about the product succeeding: two tickets were deferred with reasons, three are partial with the edge still there, and the three large creation journeys were never run end to end because they depend on model output that varies. ## Common questions ### Why is the consultancy unnamed? The company has not agreed to be named, so the name stays off the page. The description — a Nordic digital transformation consultancy — is true and enough for a reader to understand where the material came from. The day a company agrees to be named, the name goes on the page. ### Why run an analysis pass with the code untouched? Because 27 symptoms are rarely 27 faults. Grouping by root cause shows where one cause produces several symptoms, so a single fix closes several rows. The pass also lifted out four tickets that turned out to be product decisions, which have no code fix at all. ### What was the most serious finding? That the assistant wrote it had made changes that never happened. A layout fault is visible and can be worked around, while a false receipt makes a user stop trusting the whole tool. That ticket was taken first, and the fix is both a prompt rule and a visible marking in the interface. ### What does partial mean in the report? That the case the tester reported works, while an adjacent edge survives. The report describes that edge on the row. The marking exists because a round reported as entirely green makes the next round worthless — whoever reads the report has to see where the boundary sits. ### How long did the round take? Intake, analysis, fixes and verification all sat inside June 2026, with the verification report dated the twenty-fifth. That is the pace one person with AI tooling holds on a bounded body of material where the tickets are already described by someone who used the product. ### Can a client buy a round like this? Yes. It sits under Product Management when the input is user reports that need to become a prioritised plan, and under AI and automation when the faults sit in an AI layer. The studio sets a fixed price for the round after reading the material.