Can an Artificial Neural Network Grow Its Own Shape?

1. Background: Today’s AI Models Are Trained, but Their Bodies Are Designed in Advance

Large language models have become remarkably capable. They can write software, explain scientific ideas, translate languages and answer questions across hundreds of subjects. Yet almost all these models share a fundamental limitation: although their knowledge is learned from data, the shape of the network in which that knowledge develops is largely decided beforehand.

Researchers determine how many layers a model will have, how wide each layer will be, how many attention heads it will contain and how information will move between its components. Training then adjusts billions of numerical weights inside this fixed structure. It is rather like constructing an enormous office building before knowing which organisations will occupy it. The rooms, floors and corridors are fixed; only their contents can change.

Mixture-of-Experts models introduce some flexibility. Instead of using the entire network for every input, they contain multiple computational “experts,” and a router selects a small number of them for each token. This can make a model much larger without proportionally increasing the computation required for every response.

However, these experts are also normally designed in advance. Researchers decide that the model will contain, for example, 64 or 128 equally sized experts. Training may cause different experts to develop somewhat different functions, but the model cannot ordinarily decide that it needs 17 large experts, 36 small ones, three shared pathways and a new specialist module for an emerging category of problems. It learns within an anatomy that humans have already prescribed.

This raises a more fundamental question:

Could an artificial neural network develop its own internal anatomy as it learns?

The question matters because present models may be carrying considerable computational redundancy. A model capable of medicine, coding, poetry and mathematics does not necessarily need to activate the same internal machinery for every request. If useful computational circuits could separate into reusable modules, the model might become faster, cheaper and easier to adapt.

But efficiency is only one motivation. A dynamically developing architecture might also reduce interference between skills, support continual learning and produce modules that can be updated without retraining the entire system.

2. The Core Problem: Nature Does Not Begin With a Fully Formed Adult Brain

Biological brains offer a useful—but imperfect—parallel.

A human brain is not assembled as a finished collection of fixed-purpose departments. Genetics provides a broad developmental plan, but experience helps shape how different regions specialise and connect. Neural pathways strengthen through use, unused connections are pruned, and circuits differentiate as they repeatedly process different kinds of signals.

The brain also does not activate every neuron for every task. Reading, movement, face recognition and spatial navigation recruit different but overlapping networks. Some pathways are specialised, while others provide shared capabilities such as attention, memory and coordination.

Nature therefore appears to combine several principles:

• a general starting structure;
• specialisation through experience;
• sparse and conditional activation;
• pruning of unnecessary connections;
• reuse of common circuits;
• adaptation at several timescales.

Current neural networks reproduce some of these principles, but not their full combination. A conventional dense model learns weights but not its architecture. A Mixture-of-Experts model learns routing, but its expert boundaries and sizes remain mostly fixed. Neural architecture search can identify useful structures, but it is usually treated as a separate optimisation process rather than as part of the model’s continuing development.

A second parallel comes from teacher–student learning.

In knowledge distillation, a large “teacher” model trains a smaller “student.” The student learns to reproduce the teacher’s outputs, probabilities or internal representations. But the student’s architecture is normally selected before distillation begins.

This may be a missed opportunity.

The teacher’s behaviour could be used not only to train the student’s weights, but also to reveal what shape the student should take. Whenever the student repeatedly struggles with a family of inputs, the system could ask whether it needs a new computational pathway. Whenever two pathways behave almost identically, it could consider merging them. When one module receives incompatible learning signals from different tasks, it could split into specialists.

The core problem can therefore be stated as follows:

Can a student model discover a modular and efficient architecture while learning to imitate a larger teacher, rather than receiving its final architecture in advance?

Such a system would not merely be compressed. It would develop.

3. The Core Idea and Research Questions

The proposed system begins with a small, relatively undifferentiated student network and a capable pretrained teacher. The student is given a limited number of shared computational modules rather than a complete fixed set of experts.

During distillation, the system observes not only whether the student matches the teacher, but also how learning pressure is distributed inside the student.

Suppose one module receives frequent updates from programming examples and creative-writing examples. If those updates consistently push its weights in opposing directions, the module may be trying to represent two incompatible functions. The system could duplicate it, route the two families of inputs separately and allow the copies to specialise.

Conversely, if two modules receive nearly identical inputs, produce similar outputs and can replace one another without hurting performance, the system could merge them.

A module might grow when it is overloaded, divide when its incoming examples contain distinct clusters, become shared when it is useful across most tasks, or disappear when it makes little causal contribution.

This produces a developmental cycle:

learn → observe → split or merge → distil again → evaluate → repeat

Architecture is therefore treated as part of learning rather than as a fixed container around it.

Several research questions follow.

First, what signal should cause a module to split? Candidate signals include conflicting gradients, high routing uncertainty, persistent teacher–student disagreement, capacity saturation and multimodal activation patterns.

Second, what makes a module genuinely independent? Two groups of neurons may activate together without forming a reusable capability. A meaningful module should survive intervention: it should perform a coherent function, contribute causally to the result and interact with the rest of the model through relatively limited interfaces.

Third, will semantic specialisation emerge naturally? The resulting modules may correspond to recognisable categories such as coding, translation or mathematical reasoning. They may instead specialise in less obvious features such as punctuation, numerical formatting or particular hidden-state patterns. Either outcome would be scientifically informative.

Fourth, does structural evolution outperform a carefully designed fixed architecture? The comparison must control for total parameters, active parameters, training data and computational budget.

Fifth, can the resulting structure improve continual learning? A modular student may be able to learn a new domain by adding or adapting a small component while preserving older capabilities.

The research hypothesis is:

A student network trained through distillation can improve its accuracy–efficiency trade-off by dynamically splitting, merging, sharing and pruning computational modules in response to teacher disagreement, gradient interference and real usage patterns.

4. A Small Experiment to Test the Idea

The first experiment should be deliberately modest. Its purpose is not to build a biologically realistic AI system, but to determine whether architecture-changing distillation produces measurable value.

A suitable teacher would be an openly available language model in the range of one to seven billion parameters. The student could contain roughly 100–300 million parameters and begin with four interchangeable feed-forward modules plus one shared module.

The training dataset should contain several clearly different categories:

• general text completion;
• question answering;
• basic mathematics;
• Python programming;
• translation;
• summarisation;
• structured information extraction;
• creative writing.

The student would initially use a router to choose among its four modules. During distillation, the system would record four kinds of evidence:

1. the student’s disagreement with the teacher;
2. the gradients produced by different examples;
3. which inputs are routed to each module;
4. the similarity between module outputs.

At regular intervals, the system would evaluate possible structural changes.

A module would be eligible to split when it receives high traffic and contains two stable groups of inputs associated with conflicting gradients or different teacher errors. The two child modules would inherit the parent’s weights with small perturbations and then specialise through further training.

Two modules would be eligible to merge when they receive similar traffic, produce highly correlated outputs and can be combined without significantly increasing validation loss.

A module would be pruned when it is rarely used and its removal has little effect. A feature would be moved into the shared module when it is repeatedly required across most task categories.

Three students should be compared:

• a conventional dense student;
• a fixed Mixture-of-Experts student with the same maximum parameter budget;
• the dynamically restructuring student.

The models would be evaluated on teacher imitation, downstream task accuracy, active computation, latency, module utilisation and performance on previously unseen prompt types. Researchers should also measure whether the evolved architecture remains stable across repeated training runs.

A promising result would not require the evolved model to discover perfectly labelled “math” or “coding” regions. It would be enough to show that:

• its module structure changes consistently rather than randomly;
• splitting occurs where learning interference is high;
• merging removes genuine redundancy;
• it matches or exceeds a fixed model at the same compute budget;
• and the resulting modularity helps it learn new tasks with less damage to old ones.

Even a negative result would be valuable. It might show that modern neural representations are too entangled to separate through simple splitting and merging, or that computational efficiency requires stronger architectural constraints than natural emergence provides.

The larger vision is an AI system whose knowledge, routing and anatomy all adapt together. Instead of training a static machine, we would create a developmental process: one that begins simply, encounters a varied environment and gradually discovers the internal structure required to reproduce a capable teacher.

That would move neural-network design from constructing finished architectures toward cultivating systems capable of organising themselves.