Since consciousness had first dawned, in that laboratory so many million for miles sunward, all Hal’s powers and skills had been directed towards one end. The fulfillment of his assigned programme was more than an obsession; it was the only reason for his existence. Undistracted by the lusts and passions of organic life, he had pursued that goal with absolute single-mindedness of purpose.

- 2001: A Space Odyssey, Arthur C. Clarke

Computers are fascinating things. It is said, we conjure the spirit of computers with our spells!

How do these spirits think? Is it like a human brain?

One of my first working software was a neural network, $3$ days to write on paper and code in numpy. It was trained on the Titanic dataset. Going from regression (n$00$b) to NN gave $~30\%$ improvement and when I added some eval data, it became even better (n$00$b). But it was very cool! A .npz file with numbers that could do a better “if-else” than I could code and I could put it in a pendrive and carry it around. I thought these weights would be like gold and NN would take over the world (not n$00$b).

$10$ years later that prediction was true. LLMs are a type of NN and they are already insanely good. AGI is finally here (sarcasm) and it has changed culture forever. But they haven’t replaced the if-else code we write, rather NN provides a very rich perception and augmentation to that decision making.

I recently wanted to do research in AI.

Just like everyone-fucking-else.

But I knew it was not LLMs, Agentic, AGI but teaching NNs to behave like computers.

So I packed up my bag, decided to leave Hype city. Meet strange computers and find the Grand Old Automata. Learn how to build a new computer. I will travel away, finding the roots of how we got here. Study things in jungly fringes of the field. Understand the neural alchemy from shiny beings. I have left these essays as a map for future travellers walking on the path of automata.

• #$1$ - Automata: This covers the Chomsky’s Hierarchy and machines required to solve them.
• #$2$ - IRL Automata: The four types of Register Machines.
• #$3$ - Diverse Automata: Computing is not just limited to CPUs.
• #$4$ - ???
• #$4A$ - ???
• #$5$ - ???
• #$6$ - ???

All are independent essays can be read in any order.

I have also created a lot of supplementary material to understand existing work done in this direction.

• #$1$ - Task Visualizer: There are about 33 algorithms that if a model generalises to, we can claim to have built a general purpose neural computer i.e. a neural automata. These include simple things like Dyck-n languages, binary arithmetic to complex tasks like Travelling Salesman Problem.
• #$2$ - Stack Memory Models: There are some RNN based models that connect to an external stack memory. These are useful to solve many tasks and can be programmed to be simple computers.
• #$3$ - Matrix (Random Access) Memory Models: Moving a step ahead from Stack, we get to models that can access any part of the memory thus being able to solve complex graph problems.
• #$4$ - Implicit Memory Models: These models do not have a dedicated memory like LSTMs or Transformers. These are generally not very powerful but there are some up-coming ideas to look out for.

It turns out I briefly touched this subject $6$ years ago in the early part of $2020$. Aah the covid days.

• Teaching Machines Rules and Algorithms
• Reading list on Neural Graph Execution

What do we do when we meet the Grand Old Automata? Will we discover a new type of comptuer? Or will we return empty handed? Always remember what Ilya, the Monk once said:

Let's start by reading #$1$.