I initially started down this path with Home Assistant, adding all of our IoT devices into Home Assistant running on an RPi, which worked fine, but when I took the next step of buying a couple Home Assistant Voice Preview Edition devices I found the performance of the RPi was painfully slow.

Moving the entire stack to my desktop off of the RPi made a huge difference, and this is where the paid Claude account paid for itself, with Claude planning and then scripting the entire migration process that went super smoothly. After that migration the voice path ran much faster, but digging into the HA implementation I saw that it was the equivalent of yelling at a bag of regexes. This was the perfect place to use an LLM. Now, first world benefits, I happened to have a computer laying around the house with an RTX 5000 (Turing) GPU with 16GB of RAM. I wondered how far I could push that hardware to be the House Brain I imagined.

Now full credit to the Home Assistant folks for creating a great basis for all of this. For the device management and control Home Assistant is open source and just works. The UI is fairly complex, verging on the byzantine, but HA also supports a REST API that allows enumerating and controlling all of your devices. With that as the foundation I embarked on building what I call “My Jarvis”, the Home Brain. Now the naming isn’t incidental, the Home Assistant folks have done training on various wake words, and then boiled those down into tiny models you can run on as EPS32-S3 device, and one of those wake works is “Hey, Jarvis”.

Getting Claude to spin up a Go program that coordinated the ESP32 Voice devices and TTS, STT, and mapping that in a crude way to Home Assistant calls will be a blog post for another day, but after that worked I pivoted to running LLMs on the RTX 5000 machine to first handle the mapping from voice commands to Home Assistant API calls. Once that worked I expanded the actions to answering queries across two distinct datasets. The datasets are Wikipedia and an Obsidian vault with personal information in it. The key here was to index both datasets using Qdrant.

And this is where it got interesting, because there’s a whole slew of models and indexing strategies to try, particularly trying to run all of this on a five year old GPU with only 16GB of RAM. I did look at a bunch of benchmarks, but in the end nothing is a substitute for just testing the actual thing yourself.

The setup

MyJarvis is a Go voice-assistant pipeline for Home Assistant: an ESP32 hears a wake word, audio goes to VAD → STT → an LLM, and the LLM either calls a Home-Assistant tool or answers a question from RAG (Obsidian notes or a 43-million-document Wikipedia index in Qdrant).

Every user interaction is dominated by two LLM calls, not the code:

1. Routing — pick the right tool (turn on a light? check a list? look something up in Wikipedia? in my notes?).
2. Synthesis — for a lookup, turn retrieved chunks into one spoken sentence.

Latency is the whole game for voice. A two-second answer feels like an assistant; a fifteen-second one feels broken. So we want not only to be correct, but fast, so the first test harness tested

• 33 prompts spanning home automation, personal-notes lookups, and general-knowledge lookups.
• Each asserts the model’s first tool call equals the expected tool.
• Synthetic Home-Assistant entities so it’s reproducible.
• A machine-readable result line so a script can sweep it across models.

So I’ve got the time, and the tokens are free, so let’s test this against a large number of models; five families across sizes, context windows, and quantizations, each run with and without chain-of-thought where the family supports the switch:

• Qwen: 3:4b, 3:8b, 3:8b-128k, 3:8b-q8_0, 3:14b-64k, 3:14b-q4_K_M, 3.5:9b, 3.5:9b-64k, 3.6:latest
• Gemma: 4:latest, 4:16k, 3:4b, 3:12b, 4-26B (Q4 GGUF)
• Nemotron: 3-nano:4b, mini:latest
• Granite: 4:latest, 3.3:8b
• Mistral / Phi: mistral:7b, phi4-mini, phi4, phi4-reasoning, phi3

The results were interesting:

Group 1 — 100% routing accuracy (33 prompts, warm, chain-of-thought on, ranked by mean latency):

I absolutely did not expect IBM’s Granite to top out the list here.

Group 2 — ran but mis-routed (unusable as routers):

Group 3 — no tool-calling support: gemma3:4b, gemma3:12b, phi4:latest, phi4-reasoning:latest, phi3:latest, and the 26B Gemma GGUF all return Ollama’s “does not support tools” — no function-calling template.

Chain-of-thought on vs off was tested for every Qwen and Nemotron config — the extra runs that take the sweep to 36. The deltas were small and inconsistent — granite4 0.54→0.55 s, qwen3:8b 3.02→3.06 s, but qwen3:8b-q80 actually _improved 4.98→4.28 s while qwen3:4b worsened 4.81→5.99 s, and nemotron-3-nano slipped 100→97%. Disabling “thinking” was not a reliable latency reducer.

Now the above only tests if the LLM chooses the correct tool to call, we still need to measure how well the LLM does as querying Wikipedia and the Obsidian vault via Qdrant and organizing that into a coherent answer.

A separate suite ran real Wikipedia questions through the full retrieve-then-answer path, scoring factual correctness, TTS-cleanliness, source attribution, and latency. Synthesis turned out to be easy: 16 of 17 models were 100% factually correct (the answer is in the retrieved text; the model just has to phrase it). The differentiators were latency and cleanliness:

All 17 chat-capable models, 6 Wikipedia questions each, ranked by mean latency (facts = answer contains the correct fact; clean = TTS-safe, no markdown; attrib = cites the source article):

Again, Granite being the winner here, fastest speed with 100% on facts and cleanliness, was a total surprise.

So I thought I had a win here, but then a simple query blew the whole thing apart.

“How big is the moon?”

That query produced a 600-character ramble stitched across several unrelated articles including Phoebe — a moon of Saturn; a basin on Mars; an orders-of-magnitude list, etc.

Probing the 43 M-doc index directly was the eye-opener: no short query surfaces canonical articles. “Moon”, “how big is the moon”, and the LLM’s keyword expansion all return bands, people, and disambiguation pages — never the Moon article. It is indexed (a long, content-rich query finds it at rank 4–5), so this is a ranking problem, not a data problem. The token “moon” is swamped across 43 million documents; sparse hybrid search (SPLADE + BM25, DBSF fusion) can’t promote the canonical article from a two-word query.

Because retrieval is fast (~70 ms), I could afford to be clever:

Use the question, not just the keywords.

The router distills the user’s question into keywords; sometimes that drops the very term that finds the article. I probed 12 factual questions, retrieving three ways and scoring whether the expected canonical article landed in the top 5:

So simply adding the original query in along with the keyword query made a huge difference.

Adversarial re-rank.

Search results kept hitting Wikipedia disambiguation pages that aren’t really useful. A simple fix was to drop them outright; demote “List/Index/Outline of …” pages below real articles.

One conditional re-query.

If the top hit was still junk (disambiguation/index), ask the model for a better article title and retrieve once more. Bounded, no loop, and it only fires on the junk signal so clean queries pay nothing. This rescued “What is the capital of France?” — which had missed the Paris article in every retrieval mode — by re-querying into “Paris”.

After these, the synthesis suite (now including adversarial cases) was 8/8: the Eiffel Tower question cites the Eiffel Tower article, not “Eiffel Tower (disambiguation)”.

This has been a fun project, and I’m continuing to work on it and improve functionality and correctness, but the biggest lesson I learned along the way so far has been to just simply measure things and not rely on rankings/ratings from other parties. While they might give you a general idea of a model’s capabilities, you won’t really know until you apply them to your specific problem. I had originally just gone with qwen3.5:9b based off it’s rankings against the other models, but in actual measurments the Granite model is not only more accurate on tool calling and synthesis, but also over 8x faster!

Now the moon query is faster, but it still ends up referencing the Wikipedia article on Habash al-Hasib, so clearly there’s still work to do.

Now that we have our smaller list of tasks, one of the first thing you will want to do is look at the critical path, that is, the longest set of tasks in your plan that all depend on each other and define the longest path from start to finish in your project.

Let’s consider the following project, where we have tasks A, B1, B2, and C. Note that B1 and B2 are both “successors” of A, i.e. A has to finish before they can begin. Also note the B1 and B2 are “predecessors” of C, that is both of them much complete before task C can begin. The final thing to note is that B1 takes twice as long to complete as B2.

So if B1 takes four weeks to complete, B2 only takes two weeks to complete. In this case the critical path of the project is A -> B1 -> C, which you can see as highlighted in blue in the above chart. Any delay in A, B1, or C will delay the completion of the project. On the other hand, if B2 takes a few days longer than planned, actually anywhere up to taking twice as long, and the project will remain on time.

The critical path is an important tool in project planning because it tells you the tasks you really need to monitor closely because they are the ones that determine the overall project lengh. Also, these are the tasks you need to focus on when trying to shorten a project. And who among us hasn’t been on a project where you’ve planned to do the work in X days and you’re asked, what would it take to get it done in X/2 days?

In the above example how much effort should you put into shortening task B2? Well, none, because even if you got B2 down to just a single day, that will not have any affect on when the project gets finished:

What you really want to focus on in this particular example is reducing the length of task A. It clearly makes up a large portion of the project timeline and reducing that task will have the largest impact on finishing the project sooner.

That’s the general idea of critical path analysis, find the critical path, then find the “long poles” on that critical path, that is, the longest duration tasks that appear on the critical path, and focus on shortening them to bring down the total project duration.

While simply calculating the critical path will certainly help you run your project, you must be aware of, and always on the lookout for, hidden critical paths. Let’s look again as our first example:

But now let’s assign a level of Uncertainty to each task. In this case we will use Jacob Kaplan-Moss’s multipliers for measuring uncertainty:

So what does a moderate level of uncertainty mean? If we presume a task has a duration of 6 days, then that task could be completed from anywhere from the low side of $$(6 / 1.5) = 4$$ days, or on the high side of $$(6 * 1.5) = 9$$ days.

If all the tasks in the project below have a low uncertainty except for ‘B2’ which has an extreme level of uncertainty then (totally depending on what the distribution of the uncertainty of B2 looks like), B2 may actually end up on the critical path just as often as B1.

That is, B2 will complete somewhere between $$[(1w / 5), (1w * 5)]$$ or somewhere in 3 to 35 days, and given that uncertainty in B2 there’s roughly a 50% chance it’s actually on the critical path.

While this might seem like a pretty academic exercise, looking for hidden on critical paths was instrumental on getting one very large profile project to finish on time: careful attention found a long pole task on a hidden critical path that could be accelerated, which we did accelerate, which was lucky because other parts of the project finished early and the hidden long pole did end up being on the critical path and our acceleration of that task turned into a huge win in getting the project done in time.

The Axioms of Project Management:

1. Starting is definite, finishing less so.
2. Divide and conquer to reduce uncertainty.

So now let’s apply that to an ambiguous task and see how we can break it down into more manageable chunks. Let’s start with the classic example of building a house:

Now that’s pretty ambiguous, we have really no idea how long that will take, maybe anything from 3 months to a couple of years. As a first step let’s split this task in two on the time axis.

Here we can split the task, first getting the permits needed to build the house, and then beginning construction.

Note the dependency between those two tasks and each one’s uncertainty.

We can’t start building the house until all the permits are secured. (Starting is definite, finished less so.)

The uncertainty around getting the permits is a subset of the uncertainty around the whole project and wil be much less than the original single task. (Divide and conquer to reduce uncertainty.)

And we can further subdivide the “Get Permits” task, because before you do that you need to know all about the lot you are building on. Again, we’ve take a task and broken it down into two tasks in the time dimension, one task coming before the other:

Let’s assume we’re building in an area without city water and sewer, so we’ll also need to plan and build out a septic system and know the dimensions of the lot we’re building on. Both of those can happen at the same time, in this case think of splitting the original “Survey” task into two parallel tasks with each task being done by separate people. The first “Survey” means getting a surveyor out to survey the land, and the second task is getting a soils person out to test the soils.

Finally, we can put together the final information we need for the permits once we have both the survey and the soils report, at which time we can layout the house envelope and the septic field. This is like a game of tetris as you try to fit these things on the same lot, but still being aware of the setbacks, i.e. the septic field should be at least 25 feet from the house, but also needs to be 100 feet from the water well, etc. But I digress, lets’ get back to our chart. All the work has to happen before the permit application, and after the survey and soils report:

Note that at each step we are sub-dividing a task either in time or in resources, specifying things that come before and after, or tasks that can take place in parallel. And as each task gets smaller, the more the uncertainty will shrink.

Now the diagrams you see above are called Gantt charts, and if you work in the software field you’ll know that a subset of people in the field will be begin shuddering, averting their eyes and muttering “waterfall”, “agile”, and “scrum” under their breath.

You see, somewhere in the distant misty past of software development people stopped looking at Gantt charts not as the output of a process to reduce ambiguity, but as a mandate from on high on how to exactly run a software project. Or maybe some weak and ineffective managers decided to use the Gantt chart as a command and control mechanism to manage software development. Either way the original use of such a chart got labelled as “waterfall” development and the word “waterfall” became vilified.

But that’s completely wrong, and not how these charts came about. They were invented to tackle large ambiguous projects, doing so from the bottom up, and I’ve got the receipts to prove it.

Let’s jump back to 1956, almost 70 years ago!, to the development of the Polaris Missle System.

The Polaris missile program’s complexity led to the development of new project management techniques, including the Program Evaluation and Review Technique (PERT) to replace the simpler Gantt chart methodology.

Here’s where we hit a little bit of complexity because language isn’t fixed and the meanings of words change over time. Back in 1956 a Gantt chart was just a horizontal bar chart that did not include dependency relationships between tasks. PERT came along and showed the importance and power of including the relationships betwen tasks, so then Gantt charts started including inter-task dependencies, and yet still retained the name “Gantt” chart.

A two part report was publishing on how the PERT process was computerized and applied to the project:

Program Evaluation Research Task (PERT) Summary Report - Phase 1

Let’s look at some key quotes from this document, stating on page one:

Three factors, however, set research and development programming apart. First, we are attempting to schedule intellectual activity as well as the more easily measurable physical activity. Second, by definition, research and development projects are of a pioneering nature. Therefore previous, parallel experience upon which to base schedules of a new project is relatively unavailable. Third, the unpredictability of specific research results inevitably requires frequent change in program detail. These points are acknowledged by all experienced research people. Yet, even though it be ridiculous to conceive of scheduling research and development with the split- second precision of an auto assembly line, it is clear that the farther reaching and more complex our projects become, the greater is the need for procedural tools to aid top managers to comprehend and control the project.

Would it be bad form to point out that the entire edifice of “Agile” software development is built on a bed of lies? Anyway, we can clearly see that the entire point of the enterprise is to reduce ambiguity around research and development work, the kind of work with the highest levels of uncertainty.

And this process is emphatically not a top-down process. Still on page one:

This last point introduces a most important matter in research administration. The people most qualified to speak on what they have done, are doing, can do, and might do in a development project are the development people themselves. To interpose a substantial layer of evaluation organization between top management and the development people stretches the time of progress reporting, risks distortion of reports through successive interpretation on the way to the top, and generally adds to the remoteness of top management to the tasks it is managing. 1 A system should be a close coupling between the laboratory and top management and should serve both the planning and evaluation interests of both, each at the proper level.

And flexibility was baked in from the beginning, on page 3:

Actual day-to-day happenings never follow the stated or “nominal” schedule exactly. They should bear a reasonable identity, but the “actual” schedule will continuously change and flex within the general limits of the nominal schedule.

Does it scale? Yes, yes it does, from page 4:

The development of the FBM incorporates a tremendously complex system of event achievement. It is estimated that there may be upwards of 5,000 events which should be portrayed in the evaluation process. The computations that must be undertaken for each event as well as the interactions between events require something more than unabetted human contemplation . For this reason, the PERT procedure has been laid out so as to be compatible with processing on modern electronic computers.

The amusing part is that this project took place so long ago and the computers were so slow that they spent a bunch of time and math speeding up the analysis by avoiding taking cube roots. The bonkers thing is that folks have just been copy and pasting the formulas on page 7 of the report to this very day as the right way to estimate task duration even though today we have computers powerful enough to, checks notes, take cube roots.

Anyway, you should absolutely go read Program Evaluation Research Task (PERT) Summary Report - Phase 1, it’s eye opening how forward looking the project was, and how much we’ve lost, and then poorly reinvented, since then.

If you were to search across the web today you’d find a slew of randomly enumerated fundamentals of project management, including, but certainly not limited to:

• 5 C’s of project management
• 5 P’s of project management
• The 5 principles of project management
• The 12 principles of project management
• 50+ Axioms on the Art and Science of Managing Projects

Being a trained mathematician I can tell you for certain that if you’ve got 50+ axioms, you clearly don’t know the meaning of “axiom”.

Anyway, for you dear reader, I am going to break down project management into just two axioms.

The first axiom of project planning:

Starting is definite, finishing less so.

So what does that mean? Well maybe let’s start with a more colloquial saying, which is:

A journey of a thousand miles begins with a single step.

See how we know when the journey begins, when we take that first step, and we can definitely say when we are going to start. But when, exactly, will we finish that journey of 1,000 miles? Presuming we’re going to walk, that’s going to take quite a while and the exact finish date of our journey is going to be highly variable. Consider hiking the Appalachian Trail:

Completing the entire 2,190+ miles of the Appalachian Trail (A.T.) in one trip is a mammoth undertaking. Each year, thousands of hikers attempt a thru-hike; only about one in four makes it all the way.

A typical thru-hiker takes 5 to 7 months to hike the entire A.T.

–The Appalachian Trail Conservancy

If only 1/4 of the hikers actually finish hiking the trail in any year, the remaining 3/4 don’t finish it, the average time to finish the project is infinity. Infinity! I don’t think I’m going out on a limb when I say that [5 months, ∞) is a huge amount of uncertainty.

Just the list of failed and overbudget custom software projects is worth an entire Wikipedia entry.

And there are projects that have gone on so long that they’ve become legendary, like Duke Nukem Forever which took 14 years to ship.

The longer a task takes, the higher the uncertainty of when it will finish.

Think of a simple task, like vacuuming the house. This has a short time, and you can probably predict with pretty good accuracy how long it will take you to complete the task. And yes, leave it to your kids and it might never get done, but let’s not go there.

Compare that to taking on a larger project, like building a skyscraper, a sub-division, or something never accomplished before, like a fusion reactor, and the uncertainty rises dramatically.

But what to we do in the face of such uncertainty?

The second axiom of project planning:

Divide and conquer to reduce uncertainty.

Yeah, really, it’s that simple. If you have a large, complex, or ambiguous task, then break it down into smaller more manageable tasks. For example we could divide up our Appalachian Trail hiking into first figuring out how many days of food and water we can carry, along with preliminary test hikes to see how many miles a day we can cover on similar terrain.

And this loops back around to our first axoim, the shorter the sub-task the less the ambiguity.

So now we have our two axioms:

The Axioms of Project Management:

1. Starting is definite, finishing less so.
2. Divide and conquer to reduce uncertainty.

In my next installment I’ll talk about how to apply these axioms to a project to reduce uncertainty, and how everything you’ve learned about project management is probably wrong.