Scientia - Custom Physics courses generated by AI
06 July 2026 (03:00 AM):
I never thought about building simple apps with AI until I began reading Patrick Collison's posts. However, today that has changed.
Physics is fascinating. And personally, I prefer learning by my own account. However, as anybody who's been a self learner probably knows (or maybe I'm just be too unorganised), getting a consistent, linear and organised workflow can be quite hard.
So, I went ahead and built myself an AI app that will create personalised courses, with graded exercises included, on any physics topic that you prompt it.
The app searches official sources only (MIT, arXiv, CERN, etc), and for anything where it does not have >95% certainty (arbitrary figure), it will send out agents to double-check it.
You give it a prompt regarding what you want to learn (e.g: I want to learn about vacuum energy),and it then gathers a broad landscape on what concepts are required to fully grasp that topic.
It then shows a pop-up that tells you what information is required to grasp its concepts, and then you pick where you want to start from. Right below that, you pick what depth of the course you want.
It goes up to graduate level, which is 12 modules long; the shortest is survey level at 4 modules long, which gives you a deep enough overview of the topic to really appreciate it's significance.
It generates the entire course as a PDF that you download.
Here's a preview of what it looks like. It eats up tokens as if there were an infinite supply of them... I gave it $5 worth of tokens to create a survey-level course, and it barely got through a quarter of the course generation before it ran out. So you're probably looking at a cost of $15 - $50 per course in tokens.
You can get very specific with this too, so you can go really in depth on any topic you choose. Ideally, it would be a very close equivalent to asking a Nobel Prize-winning physicist to lecture you on a topic.
06 July 2026 (9:52h PM):
As it turns out, every module generation failed because of some arbitrary time limit in the module research process.
As Opus would research the topics for each module, instead of receiving the information in real time and saving it, it would just wait until the entire chunk was gathered.
This took multiple minutes, without a response, so it would time out and throw an error and turn all your precious tokens to dust.
So I've updated the module generation process so that it gets all the information in real time, and builds everything up in real time, and then saves it to memory in case the module generation process fails at any point, saving your precious tokens.
So now I managed to get module 1 generated for this course. It's pretty good. A very academic read and doesn't really explain the concept, but rather tells it. That'll be something to fix later. Hm..
However, now that the module had the chance to complete, I realised how many damn tokens the thing uses. It consumes tokens like a black hole consumes matter. $10 of credit got me the first module, and a little of the second module.
So for this I went ahead and split up the module generation into processes with different AI models. The PDF framework and research (intake, syllabus, error finding, etc) is performed with Opus 4.8. The actual execution part (so, generating the PDF, fact checking, exercises, etc) is performed by Sonnet 5.
Hopefully this will decrease token usage noticeably. I'm hoping for around 25% decrease. Yet to test out, though, as I've thrown $25 in tokens, and I am currently on the ramen diet. That's about 13 lunches right there thrown out the window.
To save (not really, but sort of) on tokens I've also added a feature to connect your Claude Code account, so it can generate a course with your Claude AI monthly subscription instead. Going to test this out later and see how far it gets me
Update - 08 July 2026 (2:45 AM):
Well, I've been able to significantly reduce token usage. Or better said, Mr.Fable and I have. Team effort FTW!
Fable added token caching into the model partitioning mix, which not only has reduced token costs by absolutely astronomical scales, but also sped up the process somewhat (seemingly negligible on complex topics such as Quantum Chromodynamics, but very noticeable on matters that build up on each other, such as cosmic inflation)
Overall, I've seen up to a 10x token cost reduction. The cosmic inflation survey course (4 modules) cost me right around $1.20 to generate, and used some 2.2M tokens, compared to the first course (mentioned on __, which cost around $25 to generate and used some 1.5M tokens; one of the complex courses cost around $10 per module, which has now been reduced, in a QCD grad-level course generation, to around $2.20 per module, which is still nearly 5x cost reduction).
However my biggest gripe is how ridiculously long this takes. And I'm not talking long relative to modern standards, which could be like 20 - 30 minutes. A survey course will easily take an hour or more to generate. A grad-level course? Well, it took about 2 hours to generate 5 modules (longer time since it's more complex, and thus requires more research and proof-checking), so an estimated 4.8 hours for the entire grad course. Jeez. Thankfully it runs in the background when my Mac is turned off.
So I added a speed selector. You can use Anthropic's "fast mode", which costs twice as much and can operate up to 2.5x faster, however, real time savings will probably look more like 50% reduction, which is still considerable. That'd make the grad-level course a 2.4 hour generation and the survey a 30 minute generation - not too bad considering Claude's research tool can easily take up to 30 minutes.
Now, on to accuracy, which is the most important part, and the entire reason LLM's aren't widely adopted for STEM, especially hard physics. Apparently, LLM's are some 35% accurate on hard physics, which is just... useless.
So far - and unfortunately the edge cases are exactly the things that aren't within my scope of knowledge, so I can't really proof-read them beyond a basic overview of 'it looks right' - it seems to be >90% accurate. The main issues are actually on the LaTeX generation, where some code slips into the complex equations.
It can also repeat itself in some modules, or as in my latest cosmic inflation course generation, flat out write in the course that "... are questions this module has not been able to fully document from verified sources here"
At least I know it tries its hardest to avoid hallucinating! Though it may be worth finding new sources. Hm..
As for new features, I'm going to make a whole mathematics section for this too. There's some mathematics in some of these modules that I'm just guessing on, or haven't seen in a while. That should be about as easy as this to make. It's going to be a carbon-copy of the physics course, except it'll spit out mathematics instead of physics.
Update - 08 July 2026 (3:22 PM):
So, I added the math course generation. It's exactly the same as Scientia, except for math. There isn't much to it.
Update - 08 July 2026 (01:56 AM):
Tested it with Grok 4.5. It's supposed to use less tokens than Opus 4.8 and be significantly faster - and therefore ultimately cheaper -, so I thought I'd give it a try. For this, I tested it using the cosmic inflation course, to see how it would fair with the Claude setup in both cost and speed.
Lord. That was expensive.
It would've been a good shot if I hadn't implemented all the cost saving measures with Claude. But it failed on 3/4 modules, and blew through $14 worth of credits. Ouch. There goes a week worth of lunch.
For the one module it did generate, I honestly preferred its output. It felt more complete. Sure, maybe a little heavier on the math, but that's what makes everything come together. But there again, it isn't a fair test. Frontier Grok model vs Sonnet. I mean, that's like sticking Usain Bolt next to a 14 year old and having them race. Sorry kid, you're not going to win that one... I really want to stick Fable 5 into there and see what it throws out.
For fun, I hid the model names and asked Grok which document it thought was better. It picked its own output. Then I told it one model had used Nobel Prize sources while the other relied heavily on arXiv papers, and asked it to match the documents to the source lists. It got it exactly right and reasoned why the arXiv-heavy version ended up more mathematical with the same thoughts I had. That was unreasonably amusing.
Unfortunately, I've blown through more money in tokens than I'd spend in 3 weeks for food, and I've maxed out this week's Claude pro usage plan, so I can't run the same test with Claude. Fortunately, in less than 48 hours the limit resets. At least I got every penny's worth out of this subscription.
With that being said, the experiment was genuinely useful (and fascinating). The issues encountered with Grok (except for the domain whitelist) were completely reasonable for it being its first ever run. Claude has been optimised for cost savings (roughly 80% of the way there), whereas this was just tokens thrown at Grok. On Claude's first run using only Opus 4.8, it also failed on most, if not all, modules, and was actually more expensive. Grok 4.5 may be a great model to use for a 'high quality + speed' tier in the course generation process, where you're willing to spend more for higher quality and speed.
On another note - Fable 5 for coding has been genuinely impressive. It would be nice to have the budget to just throw tokens at it 24/7.
Update - 10 July 2026 (12:50 PM)
As it turns out, the cheap, $1.20 survey was just an outlier. I ran a test earlier in the morning, and it ate up $6 worth of tokens right before it got to writing the PDF modules. So, it doesn't really make sense to use Claude anymore. It cost the same as Grok 4.5, is slower, and I prefer Grok's output.
So today I'll be adding Grok API back to the app with an even crazier cost savings measure.
Update - 3:15 PM
Applied the first set of optimisations. Holy shit. That was fast. And cheap.
Grok 4.1 fast w/ reasoning + Grok 4.5
Entire research phase took 1 minute and 28 seconds, vs the 14 minutes and 42 seconds from Claude, and the 40+ minutes from the first optimisation layer. Cost? $0.56. No kidding.
The module generation took about 4 minutes and a half more or less, and cost $3.50 per module. That's expensive. Why? Well, since I upgraded fact-checking to Grok 4.5 (vs Sonnet 5 with claude), it actually became the slowest and most expensive part of the entire process. It accounted for >94% of the entire cost. If it would've been performed with Grok 4.1 fast w/ reasoning, it would've been about 10x cheaper. Damn. There goes my money. Shit.
So that's the next step.
15 & 16 July:
I've settled with Grok.
I've gotten the generation times for a survey, undergrad and grad course (respectively) down to an est. 14 minutes @ $1 to $3 per course, est. 27 minutes @ $3 - $5 per course and est. 35 minutes @ $5 - $9 per course. I can further reduce the times by about 2x with no cost savings, but that's for the future.
Then, as I was on Reddit looking at a post about Richard Feynman, I saw somebody had just posted a question regarding self-study physics courses, and I realised that hey, this could be useful to other people. So I set it up on my personal domain.
Now of course for people to use it, courses need to be linked to each specific user, so I quickly (literally, it took about 15 minutes of work) implemented account creations, passkeys, and user-assigned courses.
From here, I redesigned the entire UI to go from the raw, unfinished design to a nice, polished UI.
← Projects