When Knowledge Stops Being Scarce

In 1993, I was working in a research lab at Dublin City University, building a user interface for a research project where we needed to communicate with an Ericsson MD110 exchange. The application ran on a Sun workstation using XView, the toolkit for the OpenWindows desktop, and I needed to work out how to handle data arriving asynchronously from the exchange without blocking the UI.

I knew O’Reilly published what were effectively the definitive books on the X Window System, and that the answer was probably somewhere in the XView volume. I went into Hodges Figgis in Dublin city centre, searched out a copy of the XView Programming Manual, found a quiet corner, and worked my way through the book until I found the relevant section. I transcribed the code example and took it back to the lab to try it out (I was a poor student, the book was expensive, and ultimately I only needed a few lines of code).

That feels absurd now, but at the time it was quite normal. The information I needed existed; the difficulty was getting hold of it. Books and manuals were important, as were colleagues who happened to have solved the same problem before. A lot of the skill was simply knowing where to look.

More than thirty years later, I’m back at university, doing an MSc in Artificial Intelligence, and this time round the way I learn is very different. I can move quickly between a paper, alternative explanations, implementations, source code, and my own experiments. If something still doesn’t make sense, I can ask an AI system to explain it differently, challenge the explanation, or relate it to something I already understand.

The gap between “I don’t know how this works” and “I can experiment with this myself” has narrowed dramatically. That’s obviously a good thing, but I think something more interesting is happening than simply making learning easier.

From information scarcity to knowledge abundance #

For much of my early career, technical information was relatively hard to get at. Search engines, open source, Stack Overflow, technical blogs, and online courses made information much easier to find.

But abundant information is not the same thing as abundant knowledge.

Even in the mature Internet era, you still had to find the right sources, reconcile conflicting explanations, work out which parts applied to your situation, and turn all of that into something you could actually use. The bottleneck had moved from getting access to making sense of what you found.

AI is taking that a step further. It is not just helping us find information. It can pull different sources together, explain things at the right level, relate them to the problem in front of us, and help us put them to use. That makes useful knowledge much easier to get at, not just information.

There is an important difference between asking “how does this API work?” and asking an agent to inspect a codebase, work out how the API is currently used, propose a change, implement it, write tests, run them, diagnose failures, and iterate. The first gives you knowledge. The second starts to look much more like capability.

The interesting question is no longer just whether these tools can help us work faster. It’s what happens when they stop being individual productivity aids and start changing what we delegate and how the work is divided up.

What does that mean for learning? #

One consequence is that it is easier than ever to get an answer without necessarily understanding it. There was a lot of wasted effort involved in learning technology thirty years ago, and spending an afternoon trying to find a manual is not, in itself, character building. But some of the friction was useful. Having to work through a problem, read around it, test assumptions, and reconcile different pieces of information often helped build understanding.

AI can remove a lot of that friction. That makes it possible to learn much faster, but it also means we need to choose where to slow down.

I’m seeing this directly in my MSc. A lot of my current work involves reading research papers on reinforcement learning. Mnih et al.’s Human-level control through deep reinforcement learning, for example, can take some effort to unpack: why experience replay matters, what the target network is doing, how the agent learns from reward, and what the Atari experiments actually demonstrate. AI can help me work through that much faster. I can ask it to explain a component, trace a claim back to an experiment, or challenge my interpretation of what the paper is saying.

That’s incredibly useful, but I still need to make sure I understand the idea myself, rather than just accepting an explanation because it sounds convincing. There have been times when I realised that an AI explanation made perfect sense while I was reading it, but afterwards I couldn’t reconstruct the argument. That is a useful warning sign: an explanation making sense in the moment is not the same as understanding it.

So the scarce resource moves again. Information became abundant, but attention and making sense of what we found became the harder parts. AI is now making that easier too. Judgement therefore matters more: knowing what to trust, what needs to be understood deeply, what can safely be delegated, and when the apparently easy answer deserves another hour of work.

The same thing is happening in software engineering #

AI-assisted software development is still often discussed in terms of productivity: whether developers can work faster or get more done. But agents are changing something more fundamental. Autocomplete and code generation improve the existing development process; with agentic development, instead of asking AI to generate a function, we can give an agent a goal and allow it to perform the sequence of activities needed to achieve it. It can investigate, plan, modify code, execute tests, inspect results, and revise its approach.

That changes the unit of work we can delegate.

Not very long ago, using AI while developing software mostly meant having a conversation about some code or accepting a suggested completion in an editor. I now routinely find myself describing an outcome, giving an agent access to a codebase, and allowing it to investigate how the system works before deciding what needs to change. I am still responsible for the result, but my interaction with the implementation is different. More of my effort goes into specifying intent, providing context, reviewing decisions, weighing trade-offs, and deciding whether the result actually solves the problem. Typing code does not disappear, but it occupies a different place in the work.

More of the value moves to the why and the what, rather than the how.

Communication becomes part of the craft #

If implementation becomes cheaper, deciding what to implement matters more. Engineers need to understand the problem, not just the specification. They need to understand why a feature matters, who is going to use it, which trade-offs are acceptable, and how we will know whether it is working.

That does not make product management redundant. Quite the opposite. Good product judgement and a real understanding of customers become more valuable. But some of the old boundaries between roles may become less rigid as it gets easier to move from an idea to a working implementation.

Communication becomes more important here too, not less. If engineers are expected to understand problems rather than simply implement specifications, they need to be able to listen carefully, ask good questions, explain trade-offs, and write clearly. The same applies when working with AI. An agent can do a great deal with good context and a well-framed objective, so the quality of the communication matters just as much as it does when working with people. Communication is not something separate from the technical work; it is part of the craft itself. Listening well enough to understand the real problem, writing clearly enough to express intent and constraints, and talking through uncertainty with colleagues or customers all become more valuable as implementation gets easier.

What happens to teams? #

An engineer working with agents can move more easily across implementation, testing, infrastructure, documentation, and analysis. Tasks that previously required several hand-offs can increasingly be handled by a smaller group of people. Specialisation will not disappear - security, architecture, domain expertise, product knowledge, and operational experience still matter - but it becomes much easier for an individual engineer to work across a broader range of areas.

That could have significant consequences for how we organise engineering teams. Many software organisations have grown around the assumption that producing software involves a lot of specialised activities. Requirements are defined, designs produced, code implemented, tests written, environments configured, documentation created, and software deployed. Each transition can introduce another role, team, process, or hand-off.

Some of those divisions exist for very good reasons, but others exist because there was simply too much work for one person, or one small team, to do effectively. AI changes that constraint.

The obvious conclusion is that teams get smaller, but I suspect the reality will be more complicated than that. If individuals can work effectively across more of the development lifecycle, smaller teams can own larger problem spaces. Teams can become more autonomous, and some organisational layers whose main purpose is coordination or translation may become less important. That does not necessarily mean organisations simply shrink. They could become flatter in places, while the same number of people tackle more ambitious problems. Some of the role boundaries we have built around the cost of producing software may move as well.

This is where things get more interesting. The capabilities available to an individual engineer can change very quickly with AI, while career paths, management structures, team design, ownership models, and ways of sharing knowledge tend to change much more slowly.

One of the risks of agentic development is that an individual can produce a surprisingly large amount of software very quickly. If the understanding of that software remains primarily with the person and their AI tools, the result may be more output without more shared understanding.

Team learning matters more as a result. Code still needs to be understandable, architectural decisions need to be shared, and reviews need to test understanding rather than merely confirm that the tests pass. Teams need ways of turning individual learning into shared understanding, otherwise we simply create much faster-moving silos.

Communication is central to that too. A team does not learn simply because information exists somewhere. It learns when people can explain what they have discovered, when others can challenge it, when decisions are written down clearly, and when context survives beyond the person who did the original work. As individuals produce more, the harder problem may be making sure everyone else can understand and build on it.

Are we copying knowledge, or offloading it? #

To make AI useful on real problems, we spend a lot of effort giving it context. We describe our architecture, business rules, terminology, conventions, constraints, previous decisions, and examples of how things should work. We improve the documentation, structure repositories so agents can navigate them, provide instructions and tools, and make previously implicit knowledge explicit.

That is useful in itself. Organisations have always struggled with knowledge trapped in people’s heads, and making it explicit is generally a good thing. But as our AI systems become more capable, we are also writing down more of the domain knowledge that used to sit with experienced people. The agent becomes useful because that knowledge is now available to it.

As more of that expertise gets written down for the agents, I wonder how much of it people will still internalise, and how they develop good judgement if they don’t. I don’t think the answer is that expertise becomes unnecessary. Someone still needs to create, challenge, maintain, and evolve all that context. But the distinction between what we know ourselves and what we can make available to people and machines is becoming increasingly important.

Organising the agents #

Another sign of where this is heading is the growth of frameworks for organising agents - assigning responsibilities, coordinating work, managing context, sequencing tasks, and deciding where humans fit into the process.

Microsoft Agent Framework, CrewAI, and Paperclip are examples. They approach the problem differently, but I’m less interested in which one, if any, wins out than in the fact that people are trying to make this kind of coordination systematic.

At that point, the questions become familiar organisational ones: who owns the outcome, who provides context, how is work reviewed, which decisions can be delegated, how do we maintain standards, and where do humans add the most value? To me, those questions seem much more consequential than whether an AI can generate a good piece of Rust, TypeScript, or Python.

What remains valuable? #

With every technological shift, there is a temptation to conclude that everything that came before is obsolete.

A lot of the underlying craft of engineering remains. Understanding systems matters. Being able to decompose a difficult problem matters. Domain knowledge matters. Knowing which trade-offs are important matters. So does the ability to listen, write, and explain ideas clearly. Curiosity matters, as does being able to recognise when something does not look right.

If anything, those things become more valuable when the cost of producing plausible-looking output falls.

In 1993, the obstacle between me and a few lines of useful XView code included a trip into Dublin city centre and an afternoon in a bookshop. I certainly don’t miss that friction. But removing the friction from learning and engineering does not remove the need for expertise.

It changes where the expertise needs to be.