- Oct 9, 2013
- 3,471
- 14,474
I don't know how popular this is going to be here, but let's go ahead anyway.
I'm going to start a youtube channel soon for, "vibe coding", really "Agentic Software Engineering", but the term vibe coding is popular, so we'll go with that.
Reality: What we're building is
"An Engineer in the loop agentic development system with pattern based guardrail learning and a meta-learning layer built on top of that"
It's not an easy one to put into 1 line. Let me use AI to break it down into something more readable.
The elevator pitch structure:
1. AI writes code (that's table stakes now)
2. Code gets reviewed and issues get found
3. System traces issues back to what caused them
4. Those lessons inject into future sessions automatically
5. The loop closes - most AI tools don't do this
(Everything I write if bolded/italic is AI. The rest is me.)
The framework is called Falcon AI.
The philosophy is that agents can not and should not be making decisions within your entire building system that are outside the scope of that specific task and its guidance.
That's a bit of a mouthful. This is what it means..
An implementation agent shouldn't be making architectural decisions during implementation.
A human senior dev doesn't need every single decision made for him ahead of time when it comes to the architecture. Docs are usually 70-80% complete. A lot of things are taken for granted.
With agents we can't assume this. 95% of the time agents like opus 4.5 will make the right decision, but 5% of the time they'll make a strange decision that will be missed and go entirely undocumented.
No other framework that I've come across has this philosophy and it's this philosophy and structure that makes Falcon a truly enterprise-grade production development system.
Not just production-grade. ENTERPRISE-grade. :
And the best part is, you don't need to read a single line of code. You should't be reading code in fact. It's not scalable.
You should be acting as an architect. You should understand the architecture and the flow of the system.
Ie, you should know how to build a car if you're given an engine, brake pads, clutch and so on.
You don't need to know how the engine works down to the smallest detail, or the materials science behind the brake pads.
You do need to know what happens when you turn the key, how every system is connected, its purpose.
And it's with this shift in mindset that Falcon AI was born.
This system is aimed at software engineers, but it's usable by anyone if they're willing to slow down and learn about software design and architecture.
It's completely open source and I'm licensing it under the MIT license, so you can use it, build with it, and if you're a complete ass-hat you can build a commercial vibe coding app with it. ;-)
How It Works
Note: You need at least a $100 max claude code plan. I have 3x$200/mo subs right now. This system is about quality > quantity. It's for building software on a par or better than humans can build, but in a fraction of the time, and it is also at a fraction of the cost. 1 or 2 max plans will allow you to achieve a lot, however. If you've only got a $20/mo plan, this really won't work. You'll run out of credits in 2 or 3 tasks.
It's a little complicated just now, but it will get easier over time. I have a vision for turning this into a revolutionary agentic control dashboard style system. I won't go into that now, but it'll make building anything with 20+ agents a breeze.
It's broken up into 3 layers.
Layer 1
The docs/
The docs/ are composed of
Tier 1: docs/design
Tier 2: docs/systems
Tier3: docs/support
Design is where you work with the AI to design your software.
Systems is the formal, leave-nothing-out documentation that layer 2 will use to build your software.
Support is all the runtime/dev-ops where agents will know how to manage your servers, testing, CI pipelines and more.
Layer 2
There's a bridge agent that turns your design into milestones and tasks and right now, will add everything on linear.app. It'll plan out what needs doing first and what can be done in parallel and break everything down into tasks that aren't too big and aren't too small for an llm like opus 4.5.
This layer does the following
(It's all through linear. Each of these are statuses on a linear workflow. You only need to type /checkout LIN-XXX and the agent knows what to do.
Create context pack->check context pack->improve context pack(if fails check)->create spec->harden spec and add tests->review spec(the review spec does fixes in place)->implement, test, then create PR->Code review with scouts and judges->full test agent->merge->doc-review->update_docs
That's PER TASK.
With the code review being
1 opus 4.5 orchestrator
Runs 6 scouts(for now. I will add more scout types) - These are sonnet 4.5, but you can choose opus
Then the PR REVIEW opus 4.5 orchestrator makes the final decisions and presents to the engineer.
This is where you read and manually decide what to do next.
If you agree with everything, you just ask it to comment on the PR with all the issues that need to be addressed. If some it thought aren't issues, but you disagree, you ask it to include those.
Then you open up a fresh agent, and instead of /checkout LIN-XXX, you do /checkout LIN-XXX --fix - The only time you specify something different. Without the --fix, it'll just review again.
Once everything is fixed, you manually go into linear and mark it as passed, then you can do the full test phase(but this isn't always needed and is sometimes overkill)
Final stage is to merge and do a final doc-review.
There's multiple layers of identifying undocumented decisions in the codebase. For a human team this is fine. For agents, this can lead to problems down the road. EVERYTHING must be documented in docs/ for the agents to keep being able to work effectively on the project unlike humans who just "know" their codebase.
Layer 3
This is the most advanced layer.
It contains multiple parts and is essentially a full self-learning layer with a meta-learning layer on top used by the judges. (Meta learning is learning how to learn.)
1) Principles
Principles are injected into different phases depending on the type of task. It automatically determines which core principles should be injected. Ie, some principles are related to authentication, and if the task is related to authentication it will be injected. Ie, it's smart-injection.
2) Patterns
Patterns are derived as the system works.
For example, if one of the scouts finds a security problem, it uses advanced attribution to determine what decision from what part of the system lead to that, then it logs an incident. If this happens 3 times, a pattern is created that can be injected into that part of the system to guide it away from the negative outcome.
It also has a special mode that if something is critical, it will create a temporary injection for up to 14 days. Ie, this prevents critical issues re-occurring before they've had a chance in the learning-system to turn into re-usable injectable patterns.
A pattern is basically something that lead to a 'finding'. A finding is an issue in the scout/judge system. Ie, a problem.
By making the agents aware of certain patterns, they can avoid those patterns.
Principles = things to follow. Follow principle and good things happen.
Patterns = things to avoid. You're showing the agent a pattern that lead to a finding. It can extrapolate that pattern leading to other issues and come up with better solutions..
And multiple patterns, can instead be distilled into a new principle when enough of them are similar. They are opposites. Once is do this for good, other is avoid this for good.
3) Derived principles
Once multiple patterns are created, the system automatically determines if they can be turned into a principle and auto-injected as a principle, not just a pattern.
4) The most complicated part which I just designed last night is the true meta-learning.
Every decision that every part of the system makes is logged along side every finding the judges confirm as a finding. They're grouped by tasks.
Once we have enough of them, a correlation layer is run to identify decisions that correlate with findings.
From these, we create a new structure which we inject into relevant parts like context pack creator or spec creator depending on where the decisions originated.
They look like this :
decisionGuidance {
Basically guidance on decisions. Chosen based on touches overlap. Ie, what parts of the system both the decision and the task touch. auth, api etc.
We show potential consequences and remind that these are co-occurrences so as not to confuse the model.
There's a severity and a correlationFactor. Both 1 to 5
If severity is 5, we include decisions(based on an intelligent inclusion algorithm) that have correlation factors of 2-5.
If severity is 4, then correlation is 3-5
Severity 3, correlation 4-5
and severity 1-2, only correlation factors of 5.
We also give it a confidence number which is the correlation strength. Separate from our correlation factor.
This helps it make a decision. You're showing it patterns here. Not giving it exact instructions. These patterns help advanced models like opus 4.5 avoid potentially negative consequences.
THEN
And this part is the true meta-learning. Everything until now is just self-learning.
self-learning = Learning how to produce good results.
meta-learning = Learning about what parts of our learning are effective and what aren't.
Ie, which decisionGuidance helps and which doesn't?
So when decisionGuidance blocks are injected, the agents' decisions are influenced by it.
We then create decisionGroups which contain all the guidance blocks used. I'm still designing this and thinking about where to put it.
Probably it's going to go in a separate special judge layer. I still have a lot of work to do on this layer, and to be honest it's one of the last things. This becomes powerful if we get 1000's of people using Falcon AI and we have a central repository of decisionGuidance blocks. (I'll add an option to opt-in to the central. If you share decisionGuidance blocks, you get access to the meta-learning from it)
The github repo is here - https://github.com/tbelfort/falcon-ai
As I said, I don't know how much interest this'll get..
It's mainly just somewhere to write and log stuff. X isn't really the place to post about this. Neither is linkedin. Something like this only works with a youtube channel and following, and to be honest, 99% of my time is spent building. I'm an engineer, not an influencer, but I will try to slot in some time to build a youtube channel where I can at least make some practical videos about real world vibe coding.
But meanwhile, I'll post updates in here and anyone who wants to come along in the journey is welcome to use it, ask questions, share problems, make suggestions etc.
You can think of this as a journey and AMA rolled into 1.
I'm going to start a youtube channel soon for, "vibe coding", really "Agentic Software Engineering", but the term vibe coding is popular, so we'll go with that.
Reality: What we're building is
"An Engineer in the loop agentic development system with pattern based guardrail learning and a meta-learning layer built on top of that"
It's not an easy one to put into 1 line. Let me use AI to break it down into something more readable.
The elevator pitch structure:
1. AI writes code (that's table stakes now)
2. Code gets reviewed and issues get found
3. System traces issues back to what caused them
4. Those lessons inject into future sessions automatically
5. The loop closes - most AI tools don't do this
(Everything I write if bolded/italic is AI. The rest is me.)
The framework is called Falcon AI.
The philosophy is that agents can not and should not be making decisions within your entire building system that are outside the scope of that specific task and its guidance.
That's a bit of a mouthful. This is what it means..
An implementation agent shouldn't be making architectural decisions during implementation.
A human senior dev doesn't need every single decision made for him ahead of time when it comes to the architecture. Docs are usually 70-80% complete. A lot of things are taken for granted.
With agents we can't assume this. 95% of the time agents like opus 4.5 will make the right decision, but 5% of the time they'll make a strange decision that will be missed and go entirely undocumented.
No other framework that I've come across has this philosophy and it's this philosophy and structure that makes Falcon a truly enterprise-grade production development system.
Not just production-grade. ENTERPRISE-grade. :
And the best part is, you don't need to read a single line of code. You should't be reading code in fact. It's not scalable.
You should be acting as an architect. You should understand the architecture and the flow of the system.
Ie, you should know how to build a car if you're given an engine, brake pads, clutch and so on.
You don't need to know how the engine works down to the smallest detail, or the materials science behind the brake pads.
You do need to know what happens when you turn the key, how every system is connected, its purpose.
And it's with this shift in mindset that Falcon AI was born.
This system is aimed at software engineers, but it's usable by anyone if they're willing to slow down and learn about software design and architecture.
It's completely open source and I'm licensing it under the MIT license, so you can use it, build with it, and if you're a complete ass-hat you can build a commercial vibe coding app with it. ;-)
How It Works
Note: You need at least a $100 max claude code plan. I have 3x$200/mo subs right now. This system is about quality > quantity. It's for building software on a par or better than humans can build, but in a fraction of the time, and it is also at a fraction of the cost. 1 or 2 max plans will allow you to achieve a lot, however. If you've only got a $20/mo plan, this really won't work. You'll run out of credits in 2 or 3 tasks.
It's a little complicated just now, but it will get easier over time. I have a vision for turning this into a revolutionary agentic control dashboard style system. I won't go into that now, but it'll make building anything with 20+ agents a breeze.
It's broken up into 3 layers.
Layer 1
The docs/
The docs/ are composed of
Tier 1: docs/design
Tier 2: docs/systems
Tier3: docs/support
Design is where you work with the AI to design your software.
Systems is the formal, leave-nothing-out documentation that layer 2 will use to build your software.
Support is all the runtime/dev-ops where agents will know how to manage your servers, testing, CI pipelines and more.
Layer 2
There's a bridge agent that turns your design into milestones and tasks and right now, will add everything on linear.app. It'll plan out what needs doing first and what can be done in parallel and break everything down into tasks that aren't too big and aren't too small for an llm like opus 4.5.
This layer does the following
(It's all through linear. Each of these are statuses on a linear workflow. You only need to type /checkout LIN-XXX and the agent knows what to do.
Create context pack->check context pack->improve context pack(if fails check)->create spec->harden spec and add tests->review spec(the review spec does fixes in place)->implement, test, then create PR->Code review with scouts and judges->full test agent->merge->doc-review->update_docs
That's PER TASK.
With the code review being
1 opus 4.5 orchestrator
Runs 6 scouts(for now. I will add more scout types) - These are sonnet 4.5, but you can choose opus
- Docs scout - Checks for compliance with docs/
- Tests scout - Checks quality of the tests and checks for reward hacking in the tests
- Spec scout - Checks for adherence to the spec
- Decisions Scout: Checks for undocumented decisions made by the implementation agent.
- Bugs scout: Checks for bugs, logic errors and common bug patterns
- Adversarial scout: Checks for security issues, edge cases and failure modes.
Then the PR REVIEW opus 4.5 orchestrator makes the final decisions and presents to the engineer.
This is where you read and manually decide what to do next.
If you agree with everything, you just ask it to comment on the PR with all the issues that need to be addressed. If some it thought aren't issues, but you disagree, you ask it to include those.
Then you open up a fresh agent, and instead of /checkout LIN-XXX, you do /checkout LIN-XXX --fix - The only time you specify something different. Without the --fix, it'll just review again.
Once everything is fixed, you manually go into linear and mark it as passed, then you can do the full test phase(but this isn't always needed and is sometimes overkill)
Final stage is to merge and do a final doc-review.
There's multiple layers of identifying undocumented decisions in the codebase. For a human team this is fine. For agents, this can lead to problems down the road. EVERYTHING must be documented in docs/ for the agents to keep being able to work effectively on the project unlike humans who just "know" their codebase.
Layer 3
This is the most advanced layer.
It contains multiple parts and is essentially a full self-learning layer with a meta-learning layer on top used by the judges. (Meta learning is learning how to learn.)
1) Principles
Principles are injected into different phases depending on the type of task. It automatically determines which core principles should be injected. Ie, some principles are related to authentication, and if the task is related to authentication it will be injected. Ie, it's smart-injection.
2) Patterns
Patterns are derived as the system works.
For example, if one of the scouts finds a security problem, it uses advanced attribution to determine what decision from what part of the system lead to that, then it logs an incident. If this happens 3 times, a pattern is created that can be injected into that part of the system to guide it away from the negative outcome.
It also has a special mode that if something is critical, it will create a temporary injection for up to 14 days. Ie, this prevents critical issues re-occurring before they've had a chance in the learning-system to turn into re-usable injectable patterns.
A pattern is basically something that lead to a 'finding'. A finding is an issue in the scout/judge system. Ie, a problem.
By making the agents aware of certain patterns, they can avoid those patterns.
Principles = things to follow. Follow principle and good things happen.
Patterns = things to avoid. You're showing the agent a pattern that lead to a finding. It can extrapolate that pattern leading to other issues and come up with better solutions..
And multiple patterns, can instead be distilled into a new principle when enough of them are similar. They are opposites. Once is do this for good, other is avoid this for good.
3) Derived principles
Once multiple patterns are created, the system automatically determines if they can be turned into a principle and auto-injected as a principle, not just a pattern.
4) The most complicated part which I just designed last night is the true meta-learning.
Every decision that every part of the system makes is logged along side every finding the judges confirm as a finding. They're grouped by tasks.
Once we have enough of them, a correlation layer is run to identify decisions that correlate with findings.
From these, we create a new structure which we inject into relevant parts like context pack creator or spec creator depending on where the decisions originated.
They look like this :
decisionGuidance {
Code:
interface DecisionGuidance {
id: string;
// Scope: Start PROJECT-scoped to avoid cross-project leakage
scope: Scope; // { level: 'project', workspaceId, projectId }
decisionClass: DecisionClass; // Reuse existing enum
title: string; // "Timeout selection for outbound calls"
exampleDecision: string; // Short example (value + rationale + constraints)
// TWO SEPARATE SIGNALS (not conflated)
severity: 1 | 2 | 3 | 4 | 5; // 5 = severe consequences if wrong
correlationFactor: 1 | 2 | 3 | 4 | 5; // 5 = strong decision→violation correlation
confidence: number; // 0-1 evidence strength (separate from severity!)
// "What might go wrong" as check targets
// NOTE: Co-occurrence, NOT causation
potentialConsequences: Array<{
principleId: string; // DerivedPrinciple ID (baseline or derived)
note: string; // "Missing timeouts often leads to unbounded waits..."
association: number; // 0-1 correlation strength
}>;
// Injection filtering (same as patterns)
touches: Touch[];
technologies?: string[];
injectInto: 'context-pack' | 'spec' | 'both';
// Lifecycle (decay/expire by default)
status: 'active' | 'archived' | 'expired';
expiresAt?: string; // Default 90 days
createdAt: string;
updatedAt: string;
}Basically guidance on decisions. Chosen based on touches overlap. Ie, what parts of the system both the decision and the task touch. auth, api etc.
We show potential consequences and remind that these are co-occurrences so as not to confuse the model.
There's a severity and a correlationFactor. Both 1 to 5
If severity is 5, we include decisions(based on an intelligent inclusion algorithm) that have correlation factors of 2-5.
If severity is 4, then correlation is 3-5
Severity 3, correlation 4-5
and severity 1-2, only correlation factors of 5.
We also give it a confidence number which is the correlation strength. Separate from our correlation factor.
This helps it make a decision. You're showing it patterns here. Not giving it exact instructions. These patterns help advanced models like opus 4.5 avoid potentially negative consequences.
THEN
And this part is the true meta-learning. Everything until now is just self-learning.
self-learning = Learning how to produce good results.
meta-learning = Learning about what parts of our learning are effective and what aren't.
Ie, which decisionGuidance helps and which doesn't?
So when decisionGuidance blocks are injected, the agents' decisions are influenced by it.
We then create decisionGroups which contain all the guidance blocks used. I'm still designing this and thinking about where to put it.
Probably it's going to go in a separate special judge layer. I still have a lot of work to do on this layer, and to be honest it's one of the last things. This becomes powerful if we get 1000's of people using Falcon AI and we have a central repository of decisionGuidance blocks. (I'll add an option to opt-in to the central. If you share decisionGuidance blocks, you get access to the meta-learning from it)
The github repo is here - https://github.com/tbelfort/falcon-ai
As I said, I don't know how much interest this'll get..
It's mainly just somewhere to write and log stuff. X isn't really the place to post about this. Neither is linkedin. Something like this only works with a youtube channel and following, and to be honest, 99% of my time is spent building. I'm an engineer, not an influencer, but I will try to slot in some time to build a youtube channel where I can at least make some practical videos about real world vibe coding.
But meanwhile, I'll post updates in here and anyone who wants to come along in the journey is welcome to use it, ask questions, share problems, make suggestions etc.
You can think of this as a journey and AMA rolled into 1.
