We wanted a proper test of CalcTree's AI working with everything else we have shipped lately, not another feature demo. So we gave it the hardest problem in structural engineering we could think of: the IStructE Chartered Membership exam. Candidates get seven hours to answer one question out of five, and around 70% failed the January 2026 sitting. We set CalcTree AI all five, and used the runs to tune our system knowledge base and the context engineering behind it.
The test: the Chartered Membership exam
The IStructE Chartered Membership exam is what structural engineers sit to become chartered. Seven hours, one question chosen from five, and roughly 30% of candidates pass. The January 2026 sitting passed 29% of 328 candidates.
It is a good test precisely because it does not reward what AI is naturally good at. Read the examiner reports and the failures are almost never arithmetic. Candidates fail for generic answers, for two schemes that turn out to be the same scheme in different materials, for asserting a building is stable without ever quantifying it, and for missing whatever makes that particular structure hard. Fluency counts for nothing. That is the whole point.
Each answer has five parts: a design appraisal of two schemes with a recommendation (40 marks), a letter to the client (10), design calculations for every principal element including foundations and embodied carbon (22), general arrangement drawings (20), and a method statement (8).
So we set CalcTree's AI the whole paper. Answer all five questions of the January 2026 paper, and report each answer back into the workspace as a group of connected CalcTree pages rather than as a document.
We built the examiner as well
Nobody publishes a mark scheme for this exam, so marking was its own problem. We built a grader out of the published examiner reports for January 2026, July 2025 and 2012, turned into a rubric with a band guide and an answer key for each question. It is a deliberately hard marker. Before it is allowed to award anything it has to work through the known failure modes for that question and say whether the answer exhibits each one, and it is told the real pass rate. It scores the five parts separately and applies the same rule the real exam does, where both sections have to clear half on their own.
What actually got built
Each answer is not a document. It is a pack of 11 to 13 connected CalcTree pages, built by the AI using combinations of CalcTree features that are already available to our users.
The climbing wall, Q5, is the one every drawing below comes from. It came out as eleven pages under a cover: a design basis, lateral stability, three element pages, foundations, embodied carbon, and then the four written parts of the answer. Inside them, 61 calculation statements resolving 275 named values, eight generated drawings and a DXF file, and around 10,000 words of appraisal, letter and method statement. That is what came back from one instruction.
A design basis page holds every shared input once: geometry, loads, material properties, site constraints. Nothing downstream keeps a copy. The stability page reads those values by reference, works out the load path, and writes its own results back out. The element pages read from stability. The foundations page reads from the elements. The carbon page reads from all of them. The appraisal, the client letter, the drawings and the method statement sit on top and reference the numbers underneath rather than restating them.
That structure is the interesting part, and it is the same shape a real calculation package has. It is also what makes the output checkable. Change the wind speed on the design basis and the tie force, the column check, the foundation bearing pressure and the carbon total all move together. No export step in the middle, and no second copy to fall out of date. We wrote about the same structure on a smaller job in wind optioneering near a runway.
What it used
- AI page authoring. One instruction per question. The AI planned the pack, created every page, and wrote the prose and the maths into each one.
- Cross-page references. Shared inputs defined once and read by reference everywhere else, so there is no second copy to fall out of date.
- Inline Python. The drawings and the DXF, generated from the same variables the checks run on.
- The dependency graph. What any number depends on, which is what lets a reviewer trace a result instead of taking it on trust.
- Workspace knowledge base. Code clauses and reference material retrieved from documents we uploaded, so the engineering is grounded rather than recalled.
The drawings come out of inline Python. Plans, elevations, sections and the connection details are generated as plots and DXF from the same parametric model that drives the calculations. A candidate draws these by hand on the day. Here the geometry is bound to the calc: every dimension on these drawings, the 8 m by 4 m footprint, the 15 m height, the true slope length up the raking face, is read from the design basis rather than drawn to suit. So trying a different option means changing an input, not redrawing anything.
Some of the annotated results are bound the same way. The base reaction on the third drawing below is the number the column page computed. Not all of them are, though. On two of the drawings the AI wrote the tie force in as text instead of referencing it, which is exactly the kind of label that goes stale the moment an input changes, and part of why drawings came out the weakest section of the five.
So how did it do? All five packs passed. Our grader put them between 71 and 74 out of 100, a mean of 72.8, where the pass mark is around 50 and 29% of real candidates passed this sitting. Each answer cleared the Section 1 and Section 2 gates on its own rather than scraping through on a total.
Passing was not the interesting part. The scores were tightly bunched, none of them outstanding, and the criticisms repeated: drawings were the weakest part of every answer, and in three of the five the alternative scheme was never really designed. So we went back to the prompt, the easiest thing to change, to see how much of that was us.
The second experiment: how much prompt do you actually need
Partway through, Anthropic published The new rules of context engineering for Claude 5 generation models. The claim that caught our attention: they removed over 80% of Claude Code's system prompt with no measurable loss on their coding evaluations, because they had been overconstraining the model. Their guidance is to prefer principles over prescriptive rules, and to stop stating things the model can work out for itself.
The first prompt set was a rule book for structural scheme design. 330 lines, 3,711 words, 17 sections. Quantify stability before anything else. Carry the load path through the whole system instead of checking a kit of parts. Justify the foundations against the ground, not just the footing. Count the carbon per element. Draw the awkward interfaces, dimensioned. A checklist for each, and a drawing style to follow on top.
So we wrote the opposite and ran the whole benchmark again. 93 lines, 1,183 words, 7 sections, 68% shorter. It gives engineering principles and leaves the judgement to the model: read the brief and take a position, work the design space for genuinely different schemes, weigh them the way an engineer would.
Same builder, same grader, same rubric. The prompt was the only variable.
"Principled prompts" won every question
All ten builds cleared both section gates.
Principled prompts won on all five. We would not have run the comparison at all if Anthropic had not published that guidance, so treat this as a second data point for their argument, from a domain about as far from coding as you can get.
The result we did not expect
Averages hide the interesting part. Group the marks by exam part instead of by question, summed across all five:
Calculations were the weakest part under the prescriptive prompt and became the second strongest under the principled one. That single part carries nearly half the total gain.
This is backwards from what you would guess. Calculations feel like the part that should respond to explicit instruction, and the appraisal feels like the part that needs judgment. It went the other way.
There is a reasonable explanation, and it is not arithmetic. Both groups compute in the CalcTree engine, and every graded pack verifies at zero calculation errors, so no marks moved on a number being wrong. What moved was the engineering around the numbers: why a particular check is being done, which case governs, and how load gets from the top of the structure into the ground. That is exactly what the examiners report as the most common weakness in the calculation section, a poor narrative of how load is derived and transferred rather than bad maths. A rule book produces a set of correct but disconnected element checks. Principles produce a load path the model can explain from first principles, and the rubric rewards the load path.
Where it is weak
Drawings. Worst part under both prompts, and it barely moved. The limit is not the prompt. The graders kept asking for the same three things: dimensions on the drawing face rather than in a schedule, more bespoke details at the awkward interfaces, and more views. Getting the same verdict from both prompt sets points at us rather than the model. CalcTree can compute a structure better than it can draw one, and that is a capability to build, not a sentence to add. We will put the work in as demand for this kind of output grows.
The second recurring criticism is about the engineering, and it is fair. Section 1a asks for two schemes worked up far enough to compare properly, then a recommendation. In three of the five answers the AI only did that for the scheme it went on to recommend. The alternative was not designed, it was estimated by scaling the first one, so the two were never compared on equal numerical footing. The marks are in the comparison, not in the winner, and that shortcut is exactly what a candidate running out of time takes. Our grader flagged it all three times. We need to provide more tokens it seems!
A few key caveats
Our examiner is also an AI. It is a hard marker built from the real examiner reports, but it comes from the same family of model that wrote the answers, so these are not IStructE-verified results and we are not claiming a pass of the real examination.
The conditions were not the real ones either. Candidates sit closed-book for seven hours with nothing prepared. Our builds had a knowledge base, worked from the question text rather than the figures, and had no time limit, though in practice a question took about 30 minutes. A human answers one question and we answered five. None of that is like for like and we will not pretend it is.
Key takeaways
- Principles beat rules, and that says something about knowledge. The shorter prompt won every question because it gave the model engineering principles instead of a checklist. Clear, codified engineering knowledge that a human and a machine can both read is worth more than a longer list of instructions.
- The work has to be reviewable, and now it is. AI is producing engineering output faster than anyone can check it, so the useful question was never whether a model can write something plausible about a portal frame. It is whether an engineer can interrogate the result. Every answer here is a set of live calculations where each value has a visible derivation, every input is defined once, and the dependency graph tells you what any number depends on. A reviewer can go at it the way they would go at a graduate's work. We could not have produced the tables above if the output were not structured enough to mark part by part.
- Scheme design is now in range. Not AI designing structures on its own, which it should not do and nothing here suggests otherwise. But eleven connected pages, live calculations, generated drawings and a written appraisal, produced in one pass and checkable at the end of it. That is a real piece of engineering work, and every feature that carried it is in the app. Combining the features in CalcTree with AI and a human driver will be very powerful.
What is next
- We would like to work closely with engineering firms who want to run this kind of workflow in-house. If that is you, get in touch.
- There is more to do on CalcTree's AI, and two things are at the front of the queue: more engineering knowledge behind it, and drawings.
- We also want to generalise the rubric and the grader instructions and publish them, so the same method can be pointed at real project work in CalcTree rather than an exam paper.
- On the off chance you have marked CM scripts and want to tear one of these apart, we would especially like to hear from you. Book a time and we will walk you through a pack and get your take.
