Giving It a Memory
Building My Personal AI, Step 3
At the end of Step 2, I had a model at home that answered in about a second.
And forgot everything the moment it finished.
Ask it about my own life and it knew nothing. Not my bills, not my plans, not what I'd decided last month. A brilliant stranger.
Step 3 gives it a memory. One that lives on my machine, in files I can read, and that I can delete.
I changed my mind four times
I started with trading notes. Then life admin: the morning news, the calendar, email. Then I mapped my own day hour by hour, and the days of five different families. Then I talked to a friend about his.
Each time, the use case moved.
I feel this is the most important thing .. Use Case(s) but i feel once we get "hooked" on one like Visicalc the flood gates will open for other "uses"
What didn't move was the plumbing. Whatever I ended up asking, the AI would need the same things: a place to keep my files, a way to find the right page, and an answer that shows where it came from.
So I built that first, and left the use case open.
The idea: the files are the memory
My memory lives in a folder called the vault, on the Mac Studio. Plain files: PDFs, notes, emails, Word documents. Things I can open without any special software.
The AI builds an index from those files so it can search them quickly. But the index isn't the memory. I can delete it at any time and rebuild it from the files. Same answers.
The rules of the vault are simple:
- It lives on the Studio only. Never in GitHub, never in the cloud.
- Only my user account can open it, and the disk is encrypted.
- To forget something, delete the file. To forget everything, delete the folder.
A checklist confirms all of it every time I run it. This week: 42 checks, all passing.
Two ways to search
When I ask a question, the AI searches my files in two ways at once.
By meaning. Each passage of each file is turned into a list of numbers that captures what it's about. A question gets the same treatment, and the closest passages win. This is how "old course" found my golf notes, even though those words weren't in the file. And how a question with a typo still found my year-end card statement.
By exact words. Names, amounts, dates, account types. Meaning search can blur these. Word search doesn't.
Each catches what the other misses. The results are merged, and the best eight passages go to the model.
Then it answers
The main model reads those eight passages and answers my question. With rules:
- Only from the passages. Nothing from its general knowledge about my life.
- Every fact cited, like [2], so I can check it.
- Numbers copied exactly, and any math shown.
- If the answer isn't there, say so.
- Never decide for me.
That last rule is what I've started calling BuWe: the machine carries the understanding, I keep the discernment. More on the word here.
The numbers
| What | Result |
|---|---|
| Reading 2 real PDFs | 72 passages in 3.4 seconds |
| Searching my files | 0.05 seconds |
| First word of an answer | 1.2 to 1.7 seconds |
| Full answer, with sources | 1.3 to 2.8 seconds |
| Search model | qwen3-embedding, 639 MB |
| Answer model | Qwen 3.6, the main model from Step 2 |
| Privacy checklist | 42 pass, 0 fail |
An answer from my own files, written on a machine in my house, in under three seconds.
The best answer was "I don't know"
The test I cared about most was a question it couldn't answer.
My vault has a page for golf, with a blank where my handicap goes. I haven't filled it in.
So I asked: "What is my golf handicap?"
It said: "Not in my memory."
No guess. No plausible number. Just the truth about what it knows.
An assistant that admits a gap is one I can start to trust with the things it does know.
What went wrong
| What happened | Why | Fix |
|---|---|---|
| A blank template ranked #1 for a question about spending | Its placeholder text mentioned spending | Leave unfilled lines out of the index |
| The page with the total didn't make the top 5 | Every page started with the same header, so they all looked alike | Strip headers and footers that repeat on most pages |
| After the fix, nothing changed | My own upgrade check missed indexes built before it existed | Rebuild, and a fix so older indexes upgrade by themselves |
| I pasted search results with private details into a cloud chat | Search shows what's in the file | Share only file names and pages. Answers are written at home |
The first two came from real files on the first real run. The stand-in model I used for testing would never have shown them.
The last one was mine to learn. Search results show what's inside the file. Pasting them into any cloud chat, even to get help, sends that content out of the house. Now I share only file names and page numbers, and the answers are written at home.
What's still missing
- Scans. One of my files was a scanned tax form with no text in it. The AI flagged it and skipped it. Reading images comes later.
- A proper test. Ten questions about files I know, scored: was the right page in the top three? Was the answer right?
- Remember, forget, why. Saving a note in one line. Removing a file from memory. Seeing exactly which passages an answer used.
- Access from the other Macs and my phone. Today I ask on the Studio itself.
The how
The vault, the scripts, every number, and what went wrong:
Next: the recall test, then teaching it to remember and forget.