Simple Scalable Architecture Heuristic
Prefer simple, efficient architectures that improve as data and compute scale
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 91%
This heuristic treats simplicity, efficiency, and scalability as evidence that an architecture may capture the essential structure of a problem. First define the required behavior, then reduce the design to the fewest mechanisms that can produce it. Test whether the resulting architecture uses resources efficiently and improves predictably as more data or compute is supplied. The transformer exemplified the pattern: its attention mechanism was surprisingly simple, parallelizable, and able to scale beyond recurrent alternatives. The heuristic also reflects the “bitter lesson” that general methods powered by increasing computation often outperform systems packed with handcrafted domain assumptions. Simplicity does not prove correctness, but it identifies designs worthy of aggressive empirical testing and scaling.
Origin
Google Brain co-founder Greg Corrado described the transformer's elegance and linked it to a broader observation that efficient solutions tend to survive in nature. The hosts connected this with Rich Sutton's “bitter lesson” about scalable computation.
Core principles
- 01Simple mechanisms are easier to scale and optimize.
- 02Resource-efficient solutions have a structural survival advantage.
- 03A scalable architecture can matter more than a handcrafted algorithm.
- 04More data and compute compound value only when the architecture can absorb them.
- 05Elegance is a promising signal, not a substitute for empirical testing.
How to run it
- 1
Reduce the problem to essentials
Describe the required inputs, transformation, and outputs without committing to existing implementation details.
Pro tip Ask which mechanism directly creates the desired result and which mechanisms merely support historical choices.
Watch out Do not remove safety, validation, or observability requirements simply because they add complexity.
- 2
Design the simplest viable mechanism
Build an architecture with as few interacting components and special cases as practical.
Pro tip Prefer general operations that can be executed repeatedly and in parallel.
Watch out Superficial simplicity can conceal complexity shifted into data preparation or operations.
- 3
Measure resource efficiency
Evaluate computation, memory, latency, and coordination costs under representative workloads.
Pro tip Compare total system cost rather than one isolated benchmark.
Watch out An elegant algorithm can still fail if its surrounding system is inefficient.
- 4
Run scaling tests
Increase data, compute, and workload size to determine whether performance continues improving predictably.
Pro tip Plot both quality gains and resource costs to expose diminishing returns.
Watch out Do not extrapolate scaling behavior from a single small experiment.
- 5
Challenge it with complex alternatives
Compare the simple architecture against richer designs and retain added mechanisms only where they deliver repeatable gains.
Pro tip Use controlled ablations to isolate the value of each additional component.
Watch out Elegance can become an aesthetic bias if evidence favors a more complex design.
- 6
Standardize and compound
Once validated, consolidate work around the architecture so improvements in tooling, data, and infrastructure benefit every use case.
Pro tip Build shared evaluation and deployment paths around the core architecture.
Watch out Standardization too early can suppress exploration before the architecture is proven.
In the wild
Google researchers sought an alternative to recurrent language models and LSTMs, which retained context but were computationally intensive and difficult to parallelize. The transformer used attention across the input, enabling parallel execution and improving as models grew.
→ The architecture surpassed Google's LSTM-based translation system and became the foundation of modern large language models.
An engineering team compares a bespoke workflow containing many domain rules with a smaller general pipeline. It stress-tests both with larger datasets, removes rules that do not improve measured quality, and keeps only the simple design if its performance scales more predictably.
→ The team gains a system that is easier to operate and improves as additional data and compute become available.
Common mistakes
Treating elegance as proof
A simple idea can still be wrong. Require experiments demonstrating quality, efficiency, and scaling behavior.
Ignoring hidden complexity
A clean model may depend on fragile preprocessing, expensive infrastructure, or manual operations. Evaluate the complete system.
Adding clever rules too early
Handcrafted exceptions can improve a benchmark temporarily while reducing generality and scalability. Add them only after measuring durable benefits.
Is it for you?
Best for
Research and engineering teams comparing alternative technical architectures for a rapidly scaling problem.
Not ideal for
Safety-critical systems where necessary safeguards and redundancy might look like removable complexity.
From the transcript
“you're probably on to the right solution when it's really simple and really efficient versus a complex idea.”
“you just figure out a scalable architecture and then the more data wins.”
“more data, more compute, better results.”
From the episode
Google Part III: The AI Company