All Articles
Foundations // LEARNING THE STACK•22 MAY 2025•6 min read

First Principles: From Traditional Software to Transformers & Graph ASTs

We came from traditional engineering backgrounds. To build an AI co-engineer, we went back to first principles: linear algebra, self-attention, token probabilities, and graph structures for codebases.

To build something that acts like an engineer, we needed to go beyond standard API integrations. We dedicated ourselves to studying machine learning from the mathematical foundations upward. We studied: * Linear algebra & calculus for ML optimization * Deep learning topologies & neural network weights * Transformer architectures and the mechanics of Self-Attention * Graph structures and Graph Neural Networks for codebase mapping * Foundational Generative AI architectures and prompt orchestration This foundational study demystified how LLMs ingest text, predict code structures, and represent code context, helping us understand the absolute limits and probabilistic nature of artificial intelligence.

Key Lessons & Engineering Principles

  • 1
    Foundational AI knowledge prevents you from treating model behaviors as unpredictable magic.
  • 2
    Graph structures (ASTs & dependency matrices) are crucial for mapping imports, function links, and cross-file relationships.
  • 3
    Orchestrating codebases requires managing token probabilities and semantic relations deterministically.