Build in Public on Alfero Chingono

Designing Multi-Agent Systems: Lessons from Building an 8-Agent Engineering Orchestra

Thu, 28 Aug 2025 09:00:00 +0000

A lot of “multi-agent” demos are really one agent wearing different hats.

The names change. The prompts change. Sometimes the avatars change. But the authority model, the memory model, and the execution model are all still basically the same. That is fine for a demo. It is much less convincing when you are trying to build a system that can do real engineering work.

Building CueMarshal made that distinction impossible for me to ignore.

What I wanted was not eight personalities for marketing. I wanted a working system where planning, coding, review, testing, DevOps, documentation, and quality control could be separated cleanly enough to be trustworthy.

That is how the “engineering orchestra” idea emerged.

The roles mattered because the boundaries mattered

CueMarshal’s cast eventually became:

Marshal for orchestration
Ava for architecture
Dave for implementation
Reese for review
Tess for testing
Devin for DevOps
Dot for documentation
Linton for linting

What made that useful was not the naming. It was the fact that the roles had different responsibilities, different tool access, and different default model tiers.

That is the first lesson I would pass on to anyone designing a multi-agent system:

1. Different roles need different authority

If your reviewer can rewrite production code, your reviewer is not really a reviewer.

In CueMarshal, least privilege is deliberate. The reviewer is configured without write/edit permissions. The docs agent is restricted from shell access. The linter acts like a gate, not a developer with nicer manners.

That kind of restriction sounds limiting until you realize it is what gives each role meaning. Boundaries are not friction here. They are the mechanism that creates trust.

A good multi-agent system is not just a cluster of competencies. It is a set of constrained responsibilities.

2. Coordination needs durable state outside the model

One of the reasons I anchored CueMarshal in Git is that I did not want coordination to depend on hidden model memory.

Tasks become issues. Work becomes branches. Proposals become pull requests. Reviews become durable comments and approvals.

The Conductor receives webhooks, uses Redis and BullMQ to manage asynchronous flow, and dispatches work through Gitea Actions. The runners themselves stay stateless; they rebuild context from the repository, the issue, and the tool layer every time.

That has been a much better trade than magical continuity.

Models forget. Git does not.

3. Identity is part of the architecture

Another thing I underestimated early on was how important identity separation would be.

Each CueMarshal agent has its own account, token, and audit trail. That means the Git history shows who planned, who implemented, who reviewed, and who approved. Even when the “who” is an AI agent, the distinction still matters.

This has two benefits.

First, it improves explainability. The system becomes easier to inspect when actions are attributable.

Second, it changes how you think about safety. Once every agent has a clear identity and permission scope, you stop designing from a vague “assistant” mindset and start designing from explicit operational roles.

That shift is subtle, but it is foundational.

4. The tool layer is what makes the orchestra playable

This is where MCP became important for CueMarshal.

All of the agents connect to a structured tool layer instead of improvising raw integrations on the fly. The same Gitea, Conductor, and System capabilities can be used by the runner agents over stdio and by the orchestration layer over HTTP/SSE.

That matters because multi-agent systems are not only about reasoning. They are about coordination through reliable interfaces.

If the tools are vague, agents collide. If the permissions are sloppy, trust collapses. If the transports are inconsistent, reuse gets expensive.

The protocol is not the whole story, but it is the difference between a collection of prompts and a real system surface.

I wrote more about that in MCP in Practice, because it deserves its own treatment.

5. Model routing is architecture, not optimization

Another lesson I came away with: not every role deserves the same model.

Architecture work is more expensive and more consequential than documentation cleanup. Review often needs stronger reasoning than linting. Mechanical work should not burn premium tokens if a cheaper tier can do it reliably.

CueMarshal’s tiered routing reflects that reality:

heavy reasoning for architecture
balanced capability for implementation, review, testing, and DevOps
lighter-weight models for docs and linting

That is not just a cost decision. It is part of how the system stays sustainable.

Too many agent systems treat model choice as an afterthought. I think it belongs in the design doc.

6. Closed loops beat hero agents

The more I build these systems, the less I believe in the “super-agent” story.

What works better is a closed loop:

detect work
route it clearly
execute with constrained roles
review it
merge it with human control
feed the next signal back into the system

CueMarshal’s self-improvement workflow made this even clearer to me. Once SonarQube findings, scanners, issues, PRs, and agent roles all started participating in the same loop, the system became more useful than any single agent inside it.

That is why I think orchestration matters more than agent count.

My current takeaway

If you are building a multi-agent system, start with these questions:

What roles genuinely need to be different?
What permissions should each role have?
Where does coordination state live?
How are actions attributed?
What is the closed loop that turns outputs into the next inputs?

If you cannot answer those, adding more agents will mostly add more noise.

If you can answer them, the number of agents becomes much less important than the quality of the structure around them.

That has been the real lesson for me. The point of the orchestra is not to have more instruments. The point is to make the handoffs musical instead of chaotic.

If you want the adjacent pieces, Why I Started Building My Own DevOps Platform covers the motivation, and How I Run SonarQube in My Own CI Pipeline (And Let AI Fix What It Finds) shows what this architecture looks like when the feedback loop closes on itself.

References:

Teaching Kids to Code With Bayesian Knowledge Tracing: Why I Built FireFly

Thu, 08 May 2025 09:00:00 +0000

Most kids’ coding products are better at rewarding momentum than understanding.

They make it easy to complete the next exercise, collect the badge, and keep moving. What they do not always make easy is answering the question that actually matters: does the learner understand what the code is doing?

That question is the reason I started building FireFly.

I wanted something that combined three ideas:

kids should write real code
they should be able to see it execute step by step
progression should depend on demonstrated understanding, not just content completion

That third requirement is what led me to Bayesian Knowledge Tracing.

Why I did not want a purely linear curriculum

One thing I dislike about a lot of beginner coding experiences is that they quietly assume exposure equals mastery.

Finish the lesson? Great, on to the next topic.

But anyone who has taught programming knows that this is not how understanding works. A learner can complete an exercise and still have a shaky mental model. They may get the right answer for the wrong reason. They may be copying a pattern without being able to transfer it.

I wanted FireFly to behave more like a patient tutor than a content playlist.

That meant the platform needed a way to estimate whether a concept was becoming solid over time, not just whether one attempt happened to be correct.

Why Bayesian Knowledge Tracing felt like the right fit

Bayesian Knowledge Tracing (BKT) gave me something I liked immediately: it is simple enough to reason about, but still more honest than a naive pass/fail progression model.

In FireFly, each concept gets a mastery probability. After every attempt, the platform updates that probability and decides whether the learner has actually crossed the mastery threshold.

The model in the current implementation uses fixed, interpretable parameters:

initial knowledge (pL0) = 0.10
learning transition (pT) = 0.20
guess probability (pG) = 0.25
slip probability (pS) = 0.10
mastery threshold = 0.80

I like this because it mirrors how real learning feels. One correct answer can move a student forward meaningfully, but not crown them an expert. A wrong answer does not erase everything, but it does lower confidence. Consistent performance matters more than isolated wins.

That is much closer to how a thoughtful teacher actually thinks.

Why BKT alone is not enough

A mastery model is only as good as the learning experience feeding it.

That is why FireFly is not just a scoring engine. The heart of the product is still the visual code stepper.

Kids write Python in a Monaco editor, run it through a sandboxed execution engine, and watch the program unfold line by line with stack, heap, variables, and output. Under the hood, the backend wraps Python code with a sys.settrace() tracer before submitting it to Judge0. The result is a structured trace that the UI can render frame by frame.

That matters because novices do not just need answers. They need a way to build mental models.

If a child can see:

which line ran
what changed in local variables
what got printed
where a loop is getting stuck

then the feedback stops being mysterious.

Once you combine that with mastery tracking, you get something more interesting than “did the code run?” You get a system that can ask, “is this learner genuinely building understanding over time?”

Why I made the experience age-adaptive

Another thing I did not want was one aesthetic and one teaching voice for everybody.

An eight-year-old and a fifteen-year-old can both be beginners, but they do not need the same interface. FireFly uses three age modes:

Fun for younger learners
Balanced for middle learners
Pro for older learners

The execution engine stays the same, but the tone, visuals, density, and celebration style change. I wanted the underlying system to stay serious while the presentation adapts to the learner.

That principle carries into the AI tutor as well. The goal is not to hand out solutions. The goal is to provide age-appropriate hints, explanations, and nudges that keep the learner thinking.

What I learned while building it

The more I worked on FireFly, the more I became convinced that the real bottleneck in beginner programming is not lack of content. It is lack of visible causality.

Kids get stuck because code feels magical until it does not work. Then it feels hostile.

The combination of execution tracing and mastery tracking is my attempt to close that gap:

the trace makes program behavior visible
the tutor makes the explanation conversational
the BKT layer prevents false confidence from becoming progression

That combination feels much closer to how I would want a patient, thoughtful mentor to teach.

Why I still think this matters

I did not build FireFly because kids need more gamification. They already have plenty of that.

I built it because I think programming is one of the few subjects where you can watch thought become behavior almost immediately, and that is incredibly powerful when it is made visible.

If a learner can write code, see it run, get unstuck without being spoon-fed, and unlock the next idea only after they really understand the current one, then we are not just teaching syntax. We are teaching how to think.

That is the kind of learning experience I wanted to build.

References:

Why I Started Building My Own DevOps Platform (And What I Learned)

Sat, 15 Feb 2025 09:00:00 +0000

For a while, I had the same reaction to most AI-for-software-delivery demos: impressive in a narrow way, but not something I would trust with real work. One tool could write code. Another could summarize a diff. Another could review a pull request. But the hard part of software delivery is rarely one isolated step. It is the handoff between steps.

That was the itch that eventually pushed me to start building CueMarshal.

I did not start with the ambition to build “an AI company” or some abstract autonomous future. I started because I wanted a more coherent delivery system: one place where a task could move from idea to issue to branch to pull request to review without losing context every time responsibility changed hands.

The problem I actually wanted to solve

CI/CD was never the whole problem. In many teams, the pipeline is the most deterministic part of the process. The mess usually lives around it:

the design decision that only exists in a chat thread
the issue that says too little
the reviewer who has to reconstruct intent from commit history
the documentation that is always “we’ll do it after”
the growing pile of tools that all know a little, but none of them own the workflow

What I wanted was not another dashboard. I wanted a delivery surface that respected how engineering work already happens.

That led me to a simple conviction: Git should be the source of truth, not just the storage layer.

If work already becomes legible through issues, branches, pull requests, labels, and reviews, then the orchestration layer should live there too. Not beside it. Not behind it. Inside it.

Why I built it myself

There were three constraints that mattered to me from day one.

First, I wanted the system to be self-hosted. A lot of AI tooling assumes you are comfortable sending your code, your process, and your delivery metadata into someone else’s black box. Many teams are not. I wanted an approach that made data sovereignty a feature, not an apology.

Second, I wanted the system to be role-aware. Real software delivery is not “one super-agent with a clever prompt.” Design, implementation, review, testing, DevOps, and documentation are different jobs. Sometimes one person does multiple jobs, but the jobs are still different. That distinction matters.

Third, I wanted human control to remain the final gate. I am interested in automation, not surrender. If an AI system cannot work inside a reviewable pull-request workflow, I do not think it is mature enough for serious engineering work.

Those constraints eventually turned into the shape CueMarshal has now: a conductor service in TypeScript, specialized agents for architecture, development, review, testing, DevOps, docs, and linting, a Git-native workflow in Gitea, and a tool layer built around MCP so the same system can reason over structured interfaces instead of raw shell scripts and ad-hoc API calls.

The architecture came later. The principles came first.

Long before the implementation solidified, the design principles were already obvious to me.

1. Git is a better coordination layer than most agent UIs

An issue is a task. A branch is a workstream. A pull request is a proposal. A review is a decision record. A merge is a controlled state change.

That sounds almost too obvious to say out loud, but it changed how I thought about the whole problem. Once I stopped treating Git as the place where code merely ends up, and started treating it as the place where engineering decisions become inspectable, the rest of the architecture got much simpler.

2. Specialization beats a “do everything” agent

In CueMarshal, the system is intentionally split into named roles: Marshal for orchestration, Ava for architecture, Dave for implementation, Reese for review, Tess for testing, Devin for DevOps, Dot for docs, and Linton for linting.

That is not branding for its own sake. It is an operational choice.

The moment one agent tries to be planner, coder, reviewer, tester, and documentarian all at once, you lose clarity. You also lose accountability. Specialization makes prompts sharper, tool permissions narrower, and outputs easier to judge.

3. Tool contracts matter more than prompt cleverness

One of the biggest lessons from building CueMarshal is that the quality of an agentic system is heavily constrained by the quality of its interfaces.

If an agent is forced to improvise around loosely structured APIs, fragile shell commands, or browser automation for tasks that should be typed and validated, the system becomes harder to trust. This is one reason MCP clicked for me so quickly later on: it gave a clean shape to something I already knew was essential.

Good tool contracts do not just help the model. They help the human operator understand what the system is even allowed to do.

4. Stateless workers are a feature, not a bug

CueMarshal’s runners are intentionally stateless. They reconstruct context from the repository, the issue, the pull request, and the tool layer every time.

That may sound less magical than the “persistent AI teammate” narrative, but it is much easier to reason about. It scales better. It fails more cleanly. And it produces a better audit trail.

In practice, that has made me more skeptical of systems that depend on hidden memory to feel smart.

5. Human control is product design

The more I worked on this, the more convinced I became that “human in the loop” is not enough as a slogan. It has to be built into the workflow itself.

That is why I prefer issue-driven execution, reviewable pull requests, typed tools, explicit handoffs, and merge control. Those are not bureaucratic constraints. They are the difference between a system that can support real engineering and a system that is only good for demos.

What I learned from building in public

The most useful part of this project has not been proving that agents can write code. We already knew that. The useful part has been learning where coordination breaks, where trust gets earned, and what kinds of structure make AI assistance actually usable.

It also made one thing clearer for me: the next layer of software delivery is not “more CI/CD.” It is better orchestration around the work humans and machines are already doing together.

That is the reason I started building CueMarshal, and it is still the reason I keep working on it.

If you want the more technical follow-up, I wrote about what MCP actually changed for developers and the coordination lessons from building an eight-agent engineering orchestra.

References: