An organization may already have a data governance policy in place. It may already have a document for data inventory, its classification, its source systems, and its respective data owners. Yet the organization may still suffer from poor data governance, causing low trust in data. Why? Because those things may not be operationally connected.

The policy document may say sensitive data must be protected, but classification is not linked to access control to be seamlessly enforced. The governance committee may approve standards, but data assets get created faster than they are documented. The spreadsheet says a dataset exists, but analysts cannot find it, verify its lineage, or know whether it is safe to use. And data owners may be named in the spreadsheet, but incidents or any downstream concerns may not be routed back to the owners. Worst, there are no data owners.

The book Data Governance: The Definitive Guide argues that trust in data increases when (1) someone is accountable for the data, (2) the data are easily observable, and (3) the data are secure from unauthorized access, misuse, or corruption.

That’s Accountability, Observability, and Security. They encapsulates the capabilities the many tools and artifacts that is commonplace in data management and governance practices. Examples includes data contracts, metadata management, data catalog, data lineage, data quality tests, access controls, encryption, etc. Some contributes to more than one. Data lineage and data catalog, for example, not only improves observability by improving data discoverability; they also improve accountability by making data asset ownership more explicit and granular, closer to the logical data structure.

One way to model the interaction between these three capabilities is a self-reinforcing governance loop that continually builds trust in a systemic way:

That is to say, the relationship between these capabilities is cyclical: assigning ownership, with the proper incentives, establishes stronger accountability. It motivates data owners — through checks and balances such as data contracts and SLAs — to be accountable to the curation of metadata and improve documentation, making data easier to discover and observe, increasing observability. It also exposes a weak ownership structure if data issues are rampant. Better observability, in turn, enables stronger security controls and faster detection of misuse or quality issues. Those incidents can then be traced back to owners who should be held accountable — such as by institutional policy — reinforcing responsibilities and driving further improvements to observability and security. Thus, each cycle increases trust in the data.

In the world of digital data abundance, software plays a more important role in supporting this system, seamlessly connecting the three important capabilities in data governance, as I noted in DFCE 2025: Regulation, Innovation, and Friction. Yet paperwork may never be extinct. It’s still required for audits, legal accountability, or C-suite sign-offs, but it won’t be enough to operationalize data governance at scale. Beyond paperwork, however, software helps in making it easier to do the right thing with data, and difficult to do the wrong thing.

The Governance Loop on Software

Generally, data catalogs, used for metadata management, focuses on the observability capabilities (by ingesting technical metadata and curating them) and accountability (by assigning data owners to the ingested metadata). Security, meanwhile, is naturally integrated with the AI and data analytics platform, where data analytics or data engineering works — together with policy enforcements — happens.

To appreciate software’s role in data governance beyond bureaucratic paperwork, in this post I’ll use solutions such as OpenMetadata (as the data catalog) and Snowflake (as the AI and data analytics platform) as examples. Note, though, this isn’t a recommendation of the two solutions. Another powerful competitor to Snowflake is Databricks; and a good curated list of data catalog solutions can be found in the awesome-data-catalogs repository on GitHub.

Accountability

Documenting asset ownership usually means the following: Either an owner or an asset was identified first. An agreement then has to be made on who is accountable for the asset — an asset which should be valuable (otherwise why bother?) — to some stakeholders. Ownership usually lies where it is in his or her interest to keep the asset running, reliable, and sustainable (lest stakeholders can’t extract value and blame the owner).

In the corporate world, this is usually structured by some KPI to incentivize the asset owner to do the “right” thing with the asset. So far, this kind of incentive structure is straightforward and common. Examples abound, such as KPIs on incident resolution timeliness, data privacy compliance rating, external audit rating score, etc.

But generic KPIs are not enough. The data assets must also be measured against their intended use, so that dataset used for operational monitoring should have different definition of “good” data quality from dataset used in sales forecasts, for example. The owners are therefore not only accountable for IT-centric issues; equally important is accounting for data consumer needs — to ensure the data is fit-for-purpose and fit-for-use — which corresponds more strongly to the trust placed in data. Indeed, trust in data is not purely absolute (e.g., no duplicated records); it is also relative to the domain context and intended use (e.g., the records satisfies use case requirements).

In the world of data abundance, however, this effort of establishing the link between data owners (and their responsibilities) with their data assets has become cumbersome. The reason can be explained briefly in terms of the three V’s of big data: Volume, Velocity, and Variety¹.

Increasing use of digital workflows increases the rate of growth of organization’s data footprint. The sheer amount of data volume being generated per day — or even minutes — may overwhelm unprepared organizations answerable to regulatory policies. At the same time, the rate — or velocity — at which data are being created, captured, and processed also increases, each of which proliferates new data assets that stakeholders rely on. Problems and confusion arise when this rate of data creation and usage is faster than the organization’s capability to document, manage, and control its data assets. Finally, different use cases necessitate the adoption of different variety of data storage formats and data processing technologies, from simple fixed-schedule batch-processing of tabular formats to more complicated event-driven unstructured data processing. Combined, the large volume, high velocity, and wide variety of data assets can compound accountability problems.

This brings us to another related problem: How do you define the data ownership boundaries? Where does the line begin that delineates one owner’s assets from the others, so that there’s no ownership overlap? And how do you even begin to draw that line²? The complexity of these questions is further compounded by the three V’s above. High velocity means data ownership boundaries change frequently. Ownership then needs to be frequently reviewed and updated. That means the inventory to keep track of the increasingly growing data assets with wide varieties needs to be kept up-to-date as frequently, too.

You can’t beat this data governance problem in a world of data abundance with more bureaucracy and paperwork. Since you can’t manage what you can’t observe — and what you want to manage scales out quickly in every dimension — we need to increase our capability to observe at scale, both organization-wise (as seen with data ownership concerns) and technology-wise (along the volume, velocity, and variety dimensions).

Observability

The oldest observability mechanism in the book involves writing paper documentation of your data assets and manually performing inventory bookkeeping. That usually translates to cumbersome yearly or bi-annual (because any faster will be exhausting) cross-team efforts of surveying and interviewing different teams to understand the data landscape: What are the databases that are currently running? Which database contains sales data? Which ones contains PII data? Which database gets its data from which other databases? If a specific table is corrupted, which downstream stakeholders are affected? As you can imagine, this approach to observability means such information gets stale quickly, and doesn’t scale well when data footprints grow larger and quicker over the years. Answering each question above may take days or weeks, depending on your three V’s.

Despite the increase in data footprints, it’s not uncommon to see digital things being inventoried manually in an Excel sheets or Word documents. That should change. With regulatory requirements, for example, data catalog plays an important role for data observability.

To remain compliant with PDPO in Brunei, for example, an artifact called Data Inventory Map was introduced. The fundamental reason for its introduction is, of course, to ensure organizations and regulatory bodies have the capacity to observe data assets and their use. The following is an illustration provided by AITI — the supervisory authority behind PDPO:

It captures important metadata about data assets containing personal data: the purpose of collection, the retention period, where they’re stored, the legal basis behind the collection, etc. The problem with the Data Inventory Map isn’t the artifact itself; it’s about keeping a live, granular, and accurate metadata of the data assets.

This is where data governance platforms like OpenMetadata or DataHub comes in handy. They can automatically discover, extract, and ingest the technical metadata — logical and structural information about your data assets, like table schema, view definitions, and lineage — in your data landscape ³.

They generally work as follows. First, once they have read-only access to your data systems (e.g., Oracle and SQL Server RDBMS, or Snowflake data warehouse), they can extract the metadata information from the source systems. For example, on SQL Server, you can find this in the information schema views; on an Oracle database, the data catalog views; on Snowflake, the information schema (aka “data dictionary”) views. The good news is that instead of querying these views yourself — which usually isn’t meant for human consumption — platforms like OpenMetadata already have their own built-in technical metadata ingestion tools (called connectors) that reads from the catalog views and parses them further to enrich the metadata in the governance platforms. The caveat is that if there’s no built-in connector for a source system you want to ingest, you’d have to create a custom one.

Once the data governance platform ingests the technical metadata, the accountability-observability-security activities in the loop above become more streamlined and agile. Data assets inventory can be updated quickly in a matter of seconds or minutes, instead of weeks or months, greatly reducing the data inventory staleness problem. The bottleneck to governance is no longer on extracting information about data in systems, but extracting tacit knowledge in people.

The technical inventory of data assets is also more reliable and accurate because it’s inferred directly from source. Storing the inventory in the platform allows it to be indexed for search via fuzzy words matching or tags, which improves discoverability.

The technical metadata can further be curated with information useful for operationalizing data governance, such as data owners for each data assets, the business glossaries, classifications and tagging, and data quality status. While these tasks are traditionally done by human data stewards, future context-aware AI agents might help stewards speeds up tedious data curation exercises.

In short, the link between the governance contexts of the data — that is, the purpose of collection, legal basis, and intended recipients, etc. — and the technical data assets should be unambiguous and clear. Without data catalog, the link gets blurry, giving rise to ambiguity about data ownership and the actual scope of policy implementation.

With a more precise technical metadata structure, metadata management such as data labeling or classification can become less ambiguous, giving way to transparent traceability of data use and enabling precise policy enforcement, and thus — as the loop suggests — enabling better data security.

Security

If you can’t manage what you can’t observe, it goes without saying that you can’t secure what you can’t manage. In the context of data governance, securing data goes beyond the prevention of data exfiltration and data misuse. It also extends to include data corruption: the degradation of data integrity, such as inaccurate data updates, or simply technical data consistency failures (remember ACID?). Data corruption hence deserves a similar treatment as the former two, because violating any of them can cause considerable harm to the organization.

All these can be done by internal or external actors, whether it was done with malicious intent or otherwise. This is why observability is the key enabler for securing data. With the metadata ingested and properly curated, we know what data exists across our data landscape and how it is classified or labeled (e.g., PII, Confidential, Public, etc.). This observability capability allows organizations to better plan and be more proactive to establish the appropriate level of monitoring and controls to right data with the right security controls.

Let’s expand on how observability through metadata management enables security across the three security controls introduced above. The first is the most obvious one, which is to prevent data exfiltration by malicious actors. This means you want to safeguard access to your data systems, such as your enterprise data warehouse, to the right people with the right level of privilege. To ensure the control is in place, not only you should be able to know who can access your data systems, it is also important to monitor their usage to alert suspicious activities (a sudden overnight spike in database queries by a single user outside office hour should raise some suspicion); usage are, after all, metadata that can be ingested by data governance platforms. Naturally, the access control safeguard happens at the data systems level (e.g., SQL Server, Oracle, and Snowflake) — not the data governance platform such as OpenMetadata or DataHub. How data governance platform complements access control, however, is by making it easier for organizations to design controls commensurate with the classification of data attached to the ingested metadata.

The second control is relatively harder to manage, which is to control against data misuse. For example, you may have submitted an online survey form containing your personal email — with your consent in a checkbox — to allow the researchers to contact you for further inquiries about the research survey. However, privacy laws may prohibit the researchers from using your email for commercial purposes — as that would have been a misuse of your personal data (and a betrayal to your trust!)

Purpose-based access control is harder to manage because of low traceability of data usage from data access provision. Mostly, you’ll find the clues about the purpose the data access — and hence its usage — in some email trails, such as emails about data engineers requesting read-only access to a specific table. Indeed, in most cases, purposes are not a first-class citizen in data platforms. Often, they are only weakly inferred from the combination of usernames and roles, and the grants applied to them.

To tackle this issue, OpenMetadata allows us to represent purpose as metadata with the use of tags or classification attached to catalogued data assets. This can be made even tighter on the data systems level: On Snowflake, for example, access to data is restricted by roles defined not by job description (e.g., SALES_ANALYST), but also by purpose-specific roles (e.g., PURPOSE_REVENUE_REPORTING, PURPOSE_OPERATIONS_MONITORING). These purpose roles can then be combined with row-level access policies, masking policies, and other constraints so that the same data is exposed differently depending on the approved purpose of use.

The third control on data corruption, is one that’s often overlooked in the context of data security. Corrupted data may impair decision-making too, and, in the case of AI/ML models, makes them unreliable. This can be damaging to the organization. Again, observability helps. To tackle data corruption issues, organizations should catch data errors as early as possible, ideally within a defined SLA commensurate with the criticality of the data asset. This is usually done by running queries or scripts — often on schedule — that tests data assets for the expected data profile or quality. This can be as simple as rigid rule-based checks, such as looking for blank values, or as complex as regression analysis using machine learning models.

OpenMetadata, for example, has features that allow data stewards to define and run data quality assertions from its web UI. Test failures notify the stewards — or whoever is interested to subscribe to the notification — when quality assertion fail. If the data asset catalogued in OpenMetadata has an owner (as it should!) stakeholders would know who is accountable to move their team to investigate and fix the reported data issue. As discussed above, this is only possible because of accountability that drives the organization to this level of increased observability.

Remediation that follows the violation of the controls above feeds into accountability, as discussed earlier, in the form of a governance loop. We’ve seen how observability enables security across the three controls, thus increasing security capability. These controls force organizations to answer data ownership scrutiny. Who defines the quality metrics? How do you guarantee SLAs? Are your data consumer needs met? These questions, at least in theory, reinforce better accountability, and in turn, their incentive to do better on observability and security.

Conclusion

Data governance is ultimately a systems problem whose objective is to increase trust in data: Trust that the policy intents are applied where and when they should be applied; trust that data is discoverable, understandable, and fit for intended use; trust that sensitive data is protected from misuse, exfiltration, or corruption; and trust that someone is accountable when something breaks.

By modeling this problem using three underlying reinforcing capabilities — Accountability, Observability, and Security — we can reason and explain why their connectedness is important for operationalizing data governance to increase trust. Without closing the connection gap between them, ambiguity rises, accountability becomes blurry, and conformance checks become more costly to conduct. As we’ve seen, data catalog, by automatically ingesting technical metadata, reduces ambiguity that arises when the data asset inventory does not reliably capture (if at all) granular technical metadata information.

Indeed, data governance succeeds when the system — an amalgamation of people, process, and technology — makes doing the right thing with data easier than doing the wrong thing.

Recently I stumbled upon a tweet by an open-source maintainer, ranting about AI and how it affects his ability to monetize his OSS project (emphasis mine):

All my new code will be closed-source from now on. I’ve contributed millions of lines of carefully written OSS code over the past decade, spent thousands of hours helping other people. If you want to use my libraries (1M+ downloads/month) in the future, you have to pay.

I made good money funneling people through my OSS and being recognized as expert in several fields. This was entirely based on HUMANS knowing and seeing me by USING and INTERACTING with my code. No humans will ever read my docs again when coding agents do it in seconds. Nobody will even know it’s me who built it.

Look at Tailwind: 75 million downloads/month, more popular than ever, revenue down 80%, docs traffic down 40%, 75% of engineering team laid off. Someone submitted a PR to add LLM-optimized docs and Wathan had to decline - optimizing for agents accelerates his business’s death. He’s being asked to build the infrastructure for his own obsolescence.

Two of the most common OSS business models:

• Open Core: Give away the library, sell premium once you reach critical mass (Tailwind UI, Prisma Accelerate, Supabase Cloud…)
• Expertise Moat: Be THE expert in your library - consulting gigs, speaking, higher salary

Tailwind just proved the first one is dying. Agents bypass the documentation funnel. They don’t see your premium tier. Every project relying on docs-to-premium conversion will face the same pressure: Prisma, Drizzle, MikroORM, Strapi, and many more.

The core insight: OSS monetization was always about attention. Human eyeballs on your docs, brand, expertise. That attention has literally moved into attention layers. Your docs trained the models that now make visiting you unnecessary. Human attention paid. Artificial attention doesn’t.

Some OSS will keep going - wealthy devs doing it for fun or education. That’s not a system, that’s charity. Most popular OSS runs on economic incentives. Destroy them, they stop playing. Why go closed-source? When the monetization funnel is broken, you move payment to the only point that still exists: access. OSS gave away access hoping to monetize attention downstream. Agents broke downstream. Closed-source gates access directly. The final irony: OSS trained the models now killing it. We built our own replacement.

My prediction: a new marketplace emerges, built for agents. Want your agent to use Tailwind? Prisma? Pay per access. Libraries become APIs with meters. The old model: free code -> human attention -> monetization. The new model: pay at the gate or your agent doesn’t get in.

I don’t agree with his implication that OSS is now left to “wealthy devs doing it for fun or education.” I do agree, though, that eventually there will have to be some kind of pay per access feature to libraries/content for this to be economically sustainable (see Agentic Web). Despite that, I think open-source won’t die out, even without money in the picture, because:

• Programmers will still want to code to contribute to open-source software (with or without the help of AI)
• We still benefit immensely from the compounding value of using and looking at the same code. This is because of something called Inverse Tragedy of the Commons. Open-source won’t die out; at least not the ones that the commons care about.

How the Open-Source Machine Works

To understand why open-source won’t die out, and will still be sustainable despite AI, let’s understand how open-source software not survive and self-perpetuates across many industries.

I find Eric S. Raymond’s (aka ESR) essays very helpful in explaining why free and open-source works in great detail—which I think can be summarized as a virtuous cycle.

First, for individual programmers, there’s that itch. A programmer will work on the things that they care about the most; if they’re doing it for free, they’re doing it for themselves. Such was how Linux was conceived, and many others that preceded it or followed it. This is especially true when there’s no other (free) software that exists—or (free) software that exists but is deemed not good enough—that solves the programmer’s problem. To quote ESR:

Every good work of software starts by scratching a developer’s personal itch.

Now that a programmer has scratched the itch, either by forking an existing (dead) software, making incremental contributions, or by creating a new one from scratch (pun not intended), what keeps them going? Here lies the question of incentives. If they plan to sell the support or maintenance service (e.g., as with enterprise Red Hat Linux), then the incentive is obvious: money. But what about the software that exists without a business model? If we eliminate money from the equation, what’s the driving force for programmers to maintain and write code, besides the joy of it?

To answer this question, ESR first described two different ways humans commonly organize themselves to allocate scarce resources: command hierarchy and market economy. As the name suggests, command hierarchy follows a top-down approach of central planning that allocates resources for everyone in the organization. Perhaps the most popular example of command hierarchy in history is also a textbook example of why it doesn’t scale well and is bound to crumble: the communism in 20th century Soviet Union failed to deliver its promise—instead of a rich Utopia for the masses, it delivered chronic shortages, breadlines, and eventual economic collapse. The clear winning model for allocating scarce resources is that there should be little to no planning at all; instead, in a market economy, individual transactions are driven by supply and demand. This scales better, as prices become the signal for resource allocation, operated over the exchange of money for goods and services.

Yet neither of these economic models explains the behavior of programmers in the open-source ecosystem: While command hierarchy and market economy are economic models that can explain the allocation of scarce resources, it doesn’t fit well in a world of abundance! Unlike the resources managed in a command hierarchy model or market economy model, software is a resource that can be duplicated with virtually zero cost, at a very large scale and speed, given the abundance of compute, storage, and network bandwidths (thanks to the 90s dot-com bubbles). Abundance, ESR writes, makes “command relationships difficult to sustain and exchange relationships an almost pointless game.”

There’s a third model, however, that can help explain the allocation of resources in the open-source ecosystem characterized by abundance: the gift economy. In contrast to a market economy, where goods and services are exchanged for money, a gift economy operates differently. ESR explained the gift economy through an anthropological lens, using Lockean theory of property ownership as an analogy to explain the motivation behind the ownership and “homesteading” of open-source projects. Here’s how he explained it:

On a frontier, where land exists that has never had an owner, one can acquire ownership by homesteading, mixing one’s labor with the unowned land, fencing it, and defending one’s title. … A piece of land that has become derelict in this way may be claimed by adverse possession—one moves in, improves it, and defends title as if homesteading. This theory, like hacker customs, evolved organically in a context where central authority was weak or nonexistent.

If we think of the open-source ecosystem as a frontier—ungoverned, with no central authority assigning ownership—it follows that hackers observe the same customs that Lockean theory describes: claim it through labor, maintain it, defend your title. Let a project go unmaintained, and someone else will fork it:

The Lockean logic of custom suggests strongly that open-source hackers observe the customs they do in order to defend some kind of expected return from their effort. The return must be more significant than the effort of homesteading projects, the cost of maintaining version histories that document “chain of title”, and the time cost of making public notifications and waiting before taking adverse possession of an orphaned project.

In other words, open-source software maintainers expect something in return, or a net benefit, for their efforts, even if it isn’t directly about money.

When the exchange of goods and services becomes pointless (everyone has access to all goods and services in the world), monetary exchange currency becomes worthless. ESR argues that the principal currency in a rich society blessed with abundance is reputation:

… prestige is a good way (and in a pure gift economy, the only way) to attract attention and cooperation from others. If one is well known for generosity, intelligence, fair dealing, leadership ability, or other good qualities, it becomes much easier to persuade other people that they will gain by association with you.

Obviously, if your gift is crap, the reciprocal is a crappy reputation. So there has to be some value associated with the open-source software that’s being maintained or built that gives the programmers their expected returns. The higher the value of the software, the better your prestige or status; the better you can make it when you enter the market economy or command hierarchy.

This is another property of open-source software that makes it different from traditional resources: Not only is it abundant (i.e., not scarce), but it also increases in value the more you use or consume it. The other thing I know that has this property is knowledge, which is befitting, since source code can be thought of as an encoding of someone’s or an organization’s knowledge.

ESR called this phenomena Inverse Tragedy of the Commons, after the popular concept called Tragedy of the Commons:

Over every attempt to explain cooperative behavior there looms the shadow of Garrett Hardin’s “Tragedy of the Commons”. Hardin famously asks us to imagine a green held in common by a village of peasants, who graze their cattle there. But grazing degrades the commons, tearing up grass and leaving muddy patches, which re-grow their cover only slowly. If there is no agreed-upon (and enforced!) policy to allocate grazing rights that prevents overgrazing, all parties’ incentives push them to run as many cattle as quickly as possible, trying to extract maximum value before the commons degrades into a sea of mud.

…

[W]idespread use of open-source software tends to increase its value, as users fold in their own fixes and features (code patches). In this inverse commons, the grass grows taller when it’s grazed upon.

The Inverse Tragedy of the Commons is what completes the open-source virtuous cycle loop. Somewhere down the line there’ll be some disgruntled programmer who uses the commons, creates a patch or adds new features because he has an itch. While doing that, he’s also doing it for himself, to earn reputation, and for self-satisfaction. And the open-source software (the commons) only grows taller.

AI in the Open-Source Machine

If intelligence is one of the ingredients that fuels the open-source machinery and produces free software, then the programmers are vehicles of the intelligence that transforms ideas to runnable software. Yet while intelligence is scarce, intelligent programmers that are willing to write free software are even scarcer!

And then AI comes along, which sort of commoditize intelligence, and suddenly the scarce intelligence becomes abundant. I’ve tried Claude’s Opus 4.5, and so have many others including many veterans in the industry, and there’s no denying that AI will effectively be a huge part of the intelligence that creates software.

What this means to our open-source machinery cycle described above is the following: The cost of scratching an itch is cheaper, either because it takes less time to substantiate code, or because of lower skill ceiling. Whether this is a good thing or a bad thing for the overall quality of open-source source codes, the barrier to entry to scratch an itch is lower. Using the Lockean theory analogy above, one might think that the cost of reaping the benefits of homesteading an open-source project is now cheaper, and thus the return is higher (i.e., it is easier to gain reputation). Unfortunately, that’s not what happens.

A consequence of the lower barrier to entry, however, is that it devalues reputation. Or put in terms of currency, the value of reputation is being inflated. But not all reputation inflates equally: What AI commoditize is the execution—substantiation of code or working the CLI at breakneck speed. What remains scarce is the judgement: identifying which itch is worth scratching, making sound architectural decisions, and stewarding open-source projects over time (i.e., collaborating with some randoms on the Internet, establishing a hacking subculture, etc.)

This is like the barrel vs. the ammunition analogy. From my post Of Kind Chess and Wicked Programming:

This doesn’t mean I completely disregard the notion of AI replacing human programmers in the future. The key difference between the AI we’re using right now and the one that Jensen Huang portends is in their capacity (or lack thereof) to be a manager instead of the individual contributor; to be an architect instead of the builder; or, to borrow Ben Thompson’s analogy, to be an Artificial Super Intelligence (ASI)—the rifle barrel—instead of Artificial General Intelligence (AGI)—the ammunitions:

What o3 and inference-time scaling point to is something different: AI’s that can actually be given tasks and trusted to complete them. This, by extension, looks a lot more like an independent worker than an assistant — ammunition, rather than a rifle sight. That may seem an odd analogy, but it comes from a talk Keith Rabois gave at Stanford…My definition of AGI is that it can be ammunition, i.e. it can be given a task and trusted to complete it at a good-enough rate (my definition of Artificial Super Intelligence (ASI) is the ability to come up with the tasks in the first place).

In this perspective the future of programming is certainly bright: We may not be too far from a future where we can “hire” cheap individual contributors—the ammunitions—to help generate 100% of the code. But ammunitions are only useful when we—the rifle barrel—point them to the desired targets. It gets trickier in a wicked world, where the targets would often dance unpredictability. Often times, it is even unclear which targets we should point to. In other words, creativity is about knowing what and where to aim; simply being the projectile isn’t!

Back to Marc, the disgruntled open-source maintainer from the tweet above: Marc’s moat—and the foundation of his business model—was knowledge, or specifically, the kind of execution-layer expertise that AI now replicates at scale. His deep contextual knowledge about his own codebase was valuable when humans couldn’t easily acquire it. But now, with AI commoditizing intelligence that can not only easily acquire that knowledge, but also execute on it, his OSS income funnel threatens to collapse.

This is a classic example of how new technology decimates information asymmetry. To quote Freakonomics:

It is common for one party to a transaction to have better information than another party. In the parlance of economists, such a case is known as an information asymmetry. We accept as a verity of capitalism that someone (usually an expert) knows more than someone else (usually a consumer). But information asymmetries everywhere have in fact been gravely wounded by the Internet.

Then it was the Internet; now, AI collapses that information asymmetry even more by not only giving away the information, but also acting on it.

Of course, this is only saying that he’s losing his moat to monetize on his open-source project, but this does not mean that his open-source project is in any way less valuable. In fact, to the community as a whole, his open-source project may get more valuable, thanks to AI agents making it easier (and cheaper) to capture value.

AI doesn’t break the commons, but it does break the existing OSS monetization model. Yet programmers will still want to contribute to open-source projects: The broken monetization model is just one way of converting the currency of reputation to fiat currency in the market economy. AI may break that specific monetization model, but the mechanism of converting reputation to monetary currency hasn’t broken yet.

Pushing the Money Downstream

While AI doesn’t kill OSS, it doesn’t make it any easier for people to make money off it either. If not in the contribution to—or in the possession of knowledge moat of—OSS projects, where can monetary rewards be captured?

I believe OSS will still be alive due to the virtuous cycle I described earlier: Programmers will still get the itch, they will still benefit from the reputation reaped from their OSS contribution in the gift economy, and this in turn will make the open-source ecosystem much better. But while there’s no money to be made in this cycle, there’s money to be captured downstream from the products of this cycle, as illustrated below:

The key difference between the commons and the uncommons lies in the nature of the problems they solve. The commons layer addresses problems shared across a domain—problems whose boundaries often map to industry lines (oil and gas, finance, pharmacy, and so on). Contributors and users benefit from network effects: the more people use and improve the same codebase, the more valuable it becomes for everyone. This isn’t charity either. Every contribution reflects a cost-benefit calculation, whether explicit or intuitive: is it cheaper to privately fork and bear the full maintenance burden, or to contribute upstream and let the community carry the weight?

The uncommons, by contrast, solves problems specific to individuals or organizations—the differentiated logic, the proprietary integrations, the things you don’t want your competitors to have. But because the uncommons depends on the commons, programmers working in the uncommons layer inevitably encounter friction in the foundations they build on. When they do, they face a choice: fork and maintain privately, or fix it upstream. For non-differentiating fixes, contributing back is almost always cheaper. The uncommons doesn’t just extract from the commons—it feeds the commons whenever doing so costs less than going it alone.

It is also interesting to note the dynamic between agentic AI and the commons and the uncommons. Due to AI slops, thanks to unguided code generation, the maintainers of the commons may react negatively to AI generated contributions. More slops means it can take more time, energy, and resources for the maintainers to do code review, which may lead to stricter governance on how code contributions are made, slowing down progress. Downstream in the uncommons, it’s a different story: AI helps individuals or organizations with specific problems in the uncommons, problems that otherwise wouldn’t have been addressed in the commons, and thus are more welcomed. In fact, using AI to aid software development is often encouraged in the name of productivity, as we’ve seen in many enterprises in the past few years. This is because the closed-source projects in the uncommons enjoy the luxury that open-source projects in the commons don’t have: a handful of familiar contributors.

This isn’t really a happy answer for OSS maintainers like Marc, who depends on the moat of deep source knowledge and expertise to fund his projects. With AI easily collapsing that moat, his choice to close-source his future projects behind a paywall is entirely rational, and perhaps necessary for surviving this paradigm shift. Perhaps the only consolation is that his prestige and personal brand—the judgement, taste, and stewardship that AI cannot replicate—remains valuable. The monetization path might have changed, but the currency of reputation hasn’t disappeared.

Last week, I had the opportunity to attend DFCE 2025 organized by AITI. At the conference, I wore two hats from work: that of a data governance enabler and a data architect for our effort in migrating (and re-architecting) a data platform to the cloud. There are a lot of interesting topics covered, generally around data governance (specifically, the Personal Data Protection Order, PDPO) and, of course, AI, with optimism mixed with cautionary reminders.

At the heart of the conference, though, is an important message: That to facilitate economic growth through digital innovation, trust is the foundation that business organizations and the governing bodies need to build to attract more consumers and extend their reach to broader regional markets. And it needs to be built at the outset—while we’re still early in our digital transformation phase—using policies as the tools, though admittedly at the cost of friction.

In this post, I explore the concept of regulatory friction—which I agree is a crucial foundation for a digital economy—and its impact on innovation, and how we can utilize metadata management tools to streamline the implementation of policy enforcement.

Regulatory Friction vs. the Speed of Innovation

On the drive back home after the event, I listened to a recent a16z podcast where Marc Andreessen lamented the EU’s heavy-handed regulatory approach—among other things:

Marc Andreessen: And so the startup process is a way to especially get smart young people to do ambitious things. So that’s great. It’s all fantastic. Against that is just this incredible drag by bad governments and bad policies. As you mentioned, in Europe, you’re persistent shooting itself, not just in the foot but in the other foot and in the ankle and in the knee and in the gut. And they’re just on this absolute frenzy to regulate and kill tech in Europe [and] the UK.

Eric Torenberg: And they’re proud of it.

Marc Andreessen: And they’re proud of it. Yes. The actual European line now is we quote—this is in the [Financial Times]—this is an actual quote from a European senior politician: “We know we cannot be the global leader in tech innovation, so therefore we will be the global leader in tech regulation.” And you can imagine being like a German or French tech founder and reading that just being like, “Oh God, … get to the U.S. embassy and apply for a visa as fast as possible.”

And going back just a year ago in July 2024, Stratechery wrote an article aptly titled “The E.U. Goes Too Far”, where the author lamented about “Europe’s data obsession”, which leads to the annoying consent pop-ups, personal data collections, and unnecessary frictions:

… for the first time in a while, I was traveling as a tourist with my family, and thus visiting things like museums, making restaurant reservations, etc.; what stood out to me was just how much information all of these entities wanted: seemingly every entity required me to make an account, share my mailing address, often my passport information, etc., just to buy a ticket or secure a table. It felt bizarrely old-fashioned, as if services like OpenTable or Resy didn’t exist, or even niceties like “Sign In With Google”; what exactly is a museum or individual restaurant going to do with so much of my personal information - I just want to see a famous painting or grab a meal!

The friction faced by consumers, thanks to heavy-handed regulation, is the least of the problems. The more critical secondary effect of regulations is how they either cultivate or stifle innovation and influence economic growth. Spending more time to ensure regulatory compliance means less time spent getting products to market faster; unless, of course, you spend more resources to account for that time. This also means additional risks that otherwise would not have existed without stringent regulations, scaring off startups and investors. A recent report published by the European Commission highlights the lack of appetite by venture capital firms to invest in AI in the EU compared to the US and China:

The EU’s efforts in advanced technologies, such as artificial intelligence and cloud computing, are far from matching those of the US. The main instrument available to the EU, the European Innovation Council, had a budget of 256 million euros in 2024, while the US allocated more than 6 billion dollars for this purpose, including 4.1 billion from the Defense Advanced Research Projects Agency and 2 billion dollars from other related agencies.

The situation is repeated when looking at venture capital investment. In 2023, they invested about \$8 billion in venture capital in artificial intelligence in the EU, compared to \$68 billion in the U.S. and \$15 billion in China. The few companies that are creating generative AI models in Europe, such as Aleph Alpha and Mistral, need large investments to avoid losing the race to U.S. firms. However, European markets do not meet this need, pushing European firms to look outside for funding.

To get some idea about the cost of this friction from GDPR enforcement alone, you can head over to the GDPR Enforcement Tracker website¹. It’s no surprise, then, as the senior EU politician is well aware, that the EU lags behind the US and China in the AI race, in many respects, due to the continent being the “global leader in tech regulation”, instead of lack of talent—which, if this trend continues, may eventually be true due to brain drain. From the same report:

Excessive regulation and administrative barriers in the EU are obstacles to technology companies deciding to settle or simply stay in Europe. In fact, if between 2008 and 2021 147 unicorns were founded in Europe, i.e. companies whose valuation exceeds 1 billion dollars, 40 moved their headquarters abroad, the bulk of them to the United States.

Now, I don’t mean to undermine AITI’s—and the Brunei Government’s—good intentions behind PDPO. To the conference’s credit, this was discussed at some length in a panel discussion titled Striking the Balance: Innovation, Regulation, and Building Public Trust. Here’s the description of the panel discussion in the programming document:

As technology reshapes economies and societies, policy must ensure innovation delivers public value while safeguarding trust.

How do we ensure that regulatory frameworks remain agile enough to respond to fast-evolving technologies like AI and IoT? Can Brunei position itself as a model for digital trust in the region by embedding governance into innovation from the outset?

Clearly, AITI understands the need to strike a balance, hopefully to be careful so as to avoid the missteps of the EU, where the economy is stagnating. And given Brunei’s current economic climate, the last thing we want is to repeat the same mistake.

So if there’s one key observation I can take away from the conference, it is that AITI—and by extension, the Minister of Transport and Info-communications and the Government of Brunei Darussalam—is aware of the general perplexity from the private sectors about what PDPO asks from them, and the potentially enormous complexity and efforts it might entail. Hence, the many presentations around AI and data governance throughout the conferences, which I thought were well delivered and pertinent to the discussion around balancing innovations and regulations, are welcome.

Among the many presentations that stood out to me (and still too many to discuss in one article) are the presentations by DST on how it transformed itself into a more digital company using AWS, embracing economies of speed, and by Drew & Napier—a Singaporean law firm that contributed to the PDPO—on principles of data ethics, IoT securities, and a little overview on concepts (or “Accountability Tools”, as the DPO guide calls it) such as Data Protection by Design (DPbD), Data Inventory Mapping, and Data Protection Impact Assessment (DPIA). The former boasts about innovations that leverage public cloud services, while the latter emphasizes data ethics, security, and the importance of regulations as key enablers. Again, the key question is, can we strike a good balance between innovation and regulation?

I think DST’s presentation is pertinent to the question of balance between innovation and regulation, at least to the local Bruneian audience: First and foremost, DST is familiar to everyone in Brunei—everyone who owns a mobile phone pre-UNN most likely was a customer to DST—and those who stick with DST may observe how much they’ve embraced digital transformation throughout the last five years, perhaps much more than their competitors do, at least amongst the three salescos. After all, they have an “army of developers”, as one panel speaker puts it. Secondly and more critically, they’re able to make this transformation to the cloud in spite of the regulatory scrutiny by the regulators from AITI, while PDPO was still being drafted as early as 2021. In fact, DST’s response to AITI’s public consultation on PDPO highlighted that the PDPO draft “should not hinder any innovation development.” I don’t see this sense of concern about PDPO potentially slowing down innovation in either Progressif’s response or Imagine’s response. DST may own the bragging rights at the conference (and deservedly so), but the main beneficiary of the presentation here is AITI, who want to see increasing confidence in the private sector to comply with PDPO. I’d go further to suggest that it is no coincidence that DST is making such a timely presentation about its transformation into a digital organization using public cloud technology—especially for a company that holds a huge portion of Bruneians’ personal data—at the same conference where AITI announced that it is publishing its guide for appointing Data Protection Officers (DPOs). (PDPO requires an organization to assign at least one DPO to be responsible for ensuring that the organization complies with the law.)

From this perspective, at least, the conference fosters awareness, trust, and confidence in the practical feasibility of complying with PDPO for local companies—especially for those that want to innovate more by leveraging the cloud. Of course, all this is in the hope that this buildup of trust and confidence doesn’t stop there; when done right, the consumers and investors should feel it too.

Reducing Friction with More Software

To be sure, friction isn’t inherently bad—we need friction as a society to be stable. During COVID, we had to take an ART test every few days before we could enter the office; it’s pretty tedious and uncomfortable, but at least everyone feels a bit safer. But what if the ART test takes seconds instead of minutes, and is less intrusive to carry out, and yet still offers the same accuracy (or better) for its test results? Everyone would still feel safer, but with less friction!

To that point, what’s not explicitly discussed in the conference, though, is how the costs of friction introduced by regulations like PDPO can be reduced significantly with scalable technological implementations to meet PDPO requirements. Within the last decade, the rising costs and complexities incurred by regulations like GDPR in the EU and CCPA and HIPAA in the US have increased the demand for technology that streamlines tasks and processes for compliance with these regulations .

Take, for example, the Data Inventory Mapping exercise, which is an essential exercise as highlighted in the presentations by AITI and Drew & Napier. “Producing a Data Inventory Map” is listed as an “Accountability Tool” under the DPO guideline. From the guideline, in section 4.1.2, the exercise is described below:

(a) A Data Inventory Map is a type of data mapping which identifies the data assets and the flows of data relating to a system or process.

(b) It maps out the types of personal data, the purposes of its collection, the access controls, the methods of transfer, the types of storage and the disposal method.

(c) It captures the processes by mapping the flow of data including the assigned staff involved in the management of personal data ranging from internal departments or external third-party organisations.

(d) Organisations should review and update its inventory map periodically along with any developments relating to the system or processes.

(e) Generally, the data inventory map can be implemented by project managers involved in the development of the system or process, with the assistance of the DPO

If this exercise sounds too resource-intensive as organizations and their data grow exponentially—it is because it is indeed a taxing exercise. Data catalogs or metadata management tools, such as DataHub, Amazon DataZone on AWS, Microsoft Purview on Azure, and Unity Catalog on Databricks—among many others—exist to make this exercise less taxing through automation and streamlining metadata curation workflows. All share similar features to make data observable, thereby fostering the accountability and trustworthiness of the data.

For example, DataHub supports the ingestion of metadata from Oracle and SQL Server databases (among many others), so that it can (1) parse the SQL scripts in the databases to infer the schema of the tables and views in the databases, (2) infer the lineage between tables and views, even across data systems², (3) use AI to classify whether a table or view contains PII automatically (and have it vetted manually), and, if so, (4) automatically test for policy compliance such as “Does this dataset have an owner?” or “Has this dataset been stored beyond its retention period?” Since parts (1) and (2) of this flow are done automatically, the DPO’s responsibility for carrying out task (d) in Data Inventory Mapping becomes operationally trivial.

This development in data governance software isn’t surprising. What DPOs are responsible for doing is inherently dealing with software problems; that of ensuring data in their organization’s software systems is secure (with DPbD), observable (with Data Mapping), and measurable (with DPIA). That data in discussion just happens to be personally identifiable.

If we observe the general trend of how software is eating the world, the pattern is that most of the time, the answer to software problems is to add more software on top! This is true for many aspects of IT: from upfront IT infrastructure procurement to using X as a service on demand; from manual testing and deployment to CI/CD; and from centralized and intermediary-dependent payment rails to decentralized value-settlement networks without intermediaries. So, it is not surprising that the data governance aspect follows a similar trajectory too.

Organizations with more sophisticated data engineering teams will find the effort to do the exercise above a much less tedious and time-consuming activity, as they are capable of leveraging the emerging capabilities of data systems and their integration with data catalogs or metadata management tools. Those without such a team would be doomed to resort to Excel (or worse, Microsoft Word!) as their data inventory map repository. Of course, this is a moot point when discussing organizations with data that grows very slowly or are not large enough to necessitate such a scalable approach.

More Than Just Good Intentions

Good intentions don’t work. Mechanisms do.

— Jeff Bezos

The conference did well to underscore the importance of building trust as a strategy to pave the way for economic growth through digital transformation. Pertinent to that strategy is the challenge of balancing regulation and innovation, and being cautious of the potential second-order complexities, such as those faced in the EU. As usual, the market finds its ways to even things out, and data catalogs and metadata management tools emerge from the necessity for reducing regulatory friction. Looking ahead, the next conference in 2026 should aim to close this gap with more examples showcasing how conformance to regulatory compliance is done in the real world at scale, not just with good intentions in mind, but also supplemented with effective mechanisms that make it generally feasible.

Magnus Carlsen announced in a podcast that he would no longer defend the title he’d won in 2022 after consecutively remaining undefeated in the World Chess Championship since 2013. Many critics speculated he had lost his passion or was too afraid to lose to the next generation of players. About two years later, in March 2024, he announced the inaugural tournament of the Freestyle Chess Grand Slam Tour—a series of five major Chess960 tournaments held across different continents throughout 2025.

Chess960, also known as Freestyle Chess or Fischer Random, radically increases the possible combinations of the starting positions in a game by randomizing the positions of the back-rank pieces, leaving the front-rank pawns untouched. As the name suggests, the number of possible starting positions is increased to 960 possible combinations. The Freestyle Chess Grand Slam Tour, though, bans the traditional Chess starting position, including swapping the King and Queen.

Clearly, the motivation behind the Freestyle Chess Grand Slam Tour is to make chess more interesting and entertaining in the hope of drawing in a larger audience. This is reflected by the fact the tournament offers a bigger prize pool than traditional chess tournaments and features an innovative player-focused gameplay approach, such as the “confession booth.” Players are also equipped with heart rate monitors to provide “a novel layer of drama,” as Carlsen puts it.

I’m no chess fan, but this seems like a good revitalization of the ancient game, sparking enthusiasm and debate amongst the sport’s fans and grandmasters alike (though the first tournament of Chess960 dates back to 1996). It is definitely pertinent now with how AI has crept into the sport over the last three decades, starting with a brute-force approach of Deep Blue by IBM in 1997, to the use of artificial neural network and reinforcement learning pioneered by Google DeepMind with AlphaZero in 2017. It is no surprise then of AI’s increasing influence on how the game is played. Magnus Carlsen wrote in The Economists for the By Invitation section:

Why freestyle? Following my fifth consecutive victory in the World Chess Championship in 2022, I announced that I would no longer defend the title. Many speculated that I was exhausted, or that I was scared of the next generation of players. On the contrary, my passion for chess remains as strong as ever, and I am as ambitious as I’ve always been. What changed was my perspective on the format of the classical world championship itself.

⁠⁠The challenge wasn’t the games, which often stretch for hours. I enjoy the length of time allowed under the rules. My title defence in 2018 against Fabiano Caruana, for instance, pushed us, over 12 drawn games, to our mental and physical limits, before I emerged victorious in the tiebreak. The issue lay elsewhere, in the months of grinding preparation leading up to the event. Modern World Chess Championships demand endless memorisation of computer-generated opening lines, reducing the sport’s artistry to rote learning. As someone who treasures the creativity of chess, I wanted to focus more on this aspect of the game. Also, life beyond chess deserved my attention too.

Saving Creativity

In a way, I share Carlsen’s sentiment. Chess960 was originally introduced to reduce the emphasis on opening preparation and encourage creativity. By moving to freestyle, player decision-making in chess may be free from the need to internalize patterns over years of grinding to gain a competitive edge. Going freestyle is how the grandmasters strike back against AI with a renewed hope for creativity in the sport.

Should the same thought be entertained about programming? Programming, after all, is an art. It is a skill that’s improved with creativity rather than mere rote memorization of patterns of codes (though having some repertoire of knowledge on software design patterns does help). Many prominent programmers have expressed similar sentiments on the importance of taste, elegance, and beauty with regard to programming. Here’s how Linus Torvalds, the original hacker behind the Linux kernel and Git, explained good taste in 2007:

To me… the sign of people I really want to work with is that they have good taste… Good taste is about really seeing the big patterns and kind of instinctively knowing what’s the right way to do things.

And what are instincts but the use of implicit patterns so deeply familiar that we no longer remember learning them after years of repetition and experience—embedded deep in the back of our mind? If this is true, then AI is just a monstrous instinct machine that has learned to recognize the big patterns of our collective knowledge that we’ve put out on the Internet. It doesn’t truly understand the right way to do things, but it can infer them from the big patterns—and it will only get better.

To that point, AI has already internalized a wealth of creative output of perhaps some of the best programmers in the world and makes them readily available as generative code assistants. They help us scaffold code from a blank page, suggest code improvements for refactoring opportunities, and even fix our code plagued with cryptic error messages. If this effectively takes away the creative aspect of programming from the programmers, then will the programmers too, strike back?

The problem with this line of thinking is that chess is an adversarial zero-sum activity (one must lose for another to win), while programming—like many productive economic activities—is a positive-sum activity (everyone wins something). So while AI takes away the creative value of chess from the players (by eating away their creative decision-making opportunities), AI adds creative value to the programmers (by adding more options for creative decision-making opportunities). It makes no sense to strike back. We’re living the dream!

Exploiting the Kind and Capitalizing on the Wicked

There’s also another interesting angle of AI’s influence on chess, and it is related to learning and decision-making. Carlsen’s issue with the overreliance on rote learning computer-generated patterns and his desire to revive the creative aspect of chess reminds me of two thought-provoking ideas introduced by Robin Hogarth about how we learn and make decisions: the kind learning environment and its antithesis, the wicked learning environment.

To frame the two antithetical learning environments into questions: Does our decision-making improve with more experience? Or does having more experience make us myopic to the broader worldview, making us more susceptible to being biased in our judgment?

If your belief leans more to the former, then you’re in the camp of expert intuition who believes that good decision-making comes with years of repeated practice and experience. Otherwise, if you’re skeptical that having more experience is a good predictor for excellent decision-making, then you’re in the camp of heuristics and bias, who believe that experience leads to overconfidence and results in more decision errors.

A seminal paper titled Conditions for Intuitive Expertise: A Failure to Disagree reconciles the opposing views of these two camps. The two authors, Daniel Kahneman (camp heuristic and bias) and Gary Klein (camp expert intuition) touched on their differing perspectives and their reconciliation:

In this article we report on an effort to compare our views on the issues of intuition and expertise and to discuss the evidence for our respective positions. When we launched this project, we expected to disagree on many issues, and with good reason: One of us (GK) has spent much of his career thinking about ways to promote reliance on expert intuition in executive decision making and identifies himself as a member of the intellectual community of scholars and practitioners who study naturalistic decision making (NDM). The other (DK) has spent much of his career running experiments in which intuitive judgment was commonly found to be flawed; he is identified with the “heuristics and biases” (HB) approach to the field.

A surprise awaited us when we got together to consider our joint field of interest. We found ourselves agreeing most of the time. Where we initially disagreed, we were usually able to converge upon a common position. Our shared beliefs are much more specific than the commonplace that expert intuition is sometimes remarkably accurate and sometimes off the mark. We accept the common-place, of course, but we also have similar opinions about more specific questions: What are the activities in which skilled intuitive judgment develops with experience? What are the activities in which experience is more likely to produce overconfidence than genuine skill?

And so the answer to the above questions is that it depends. Ask a chess grandmaster, and you may get a nod of agreement. Ask a senior programmer, and he may disagree vehemently (simply because programmers are an angry bunch). What sets them apart is the domain in which their decision-making is applied. One lives in a world where the rules seldom change, and the other has to adjust and adapt frequently to meet the demands of the rapidly shifting world (and gets angry every time a new feature is requested “at the last minute,” or a change introduced an unexpected behavior).

Chess is considered the gold standard of a kind learning environment. It has simple and predictable rules and provides immediate feedback, and yet it is quite complex thanks to the astronomical size of the possible positional configuration of the pieces on a 8x8 board. (For traditional chess, a recent estimate is in the order of $~10^{44}$ reachable legal positions.) On the contrary, a wicked learning environment is the opposite: the rules are unclear and complex and may change overnight. Feedback in the environment is often delayed. And what you learn today may not be applicable tomorrow. These are the defining features of problems faced by any white-collar workers—and sure is true for programmers!

The table below summarily compares the two:

In a kind, zero-sum environment like chess, the incentive is to use AI’s super computational capability to exploit the solution space—commonly through “prepping”—to devise optimal opening strategies and gain competitive leverage over competitors. This effectively shrinks the space for human creativity, turning expertise into a matter of rote memorization rather than tactical ingenuity. In contrast, in a wicked, positive-sum environment like programming, the incentive is to capitalize on existing knowledge and tools using AI to accelerate discovery and expanding our horizons. Here, AI doesn’t limit our creativity; rather, it enables us to explore and traverse more complex solution spaces that were previously beyond our reach. In other words, AI’s interaction with the kind environment reduces the creativity space for humans; but it expands the creativity space for humans in a wicked environment.

Though in this post I limit myself to only use chess vs. programming to drive my musing, I’m sure this observation rings true to many other domains as well. As I’m writing this post, another example that came to mind is that of F1 (a kind, zero-sum environment) vs. forecasting long-term stock prices (a wicked, positive-sum environment).

In the case of F1, teams uses AI to exploit the useful patterns from real-time data to help in making fast pit strategy decisions, a job which traditionally relies on a team of engineers to make the call. This narrows down human choices to those already exploited by AI. As for traders interested in the long-term future price of stocks, the logical thing to do is capitalize on the wealth of financial and economical knowledge—a job better suited for AI than human—to help them make better decisions.

Well, does this mean AI’s involvement in chess—or any other kind environment—is bad? Not really. While it is generally true that in a kind environment AI reduces the number of available choices to go for optimizations, it also shift the creative frontier downstream into regions of small pockets of wicked environment within the domain. This is apparently Magnus Carlsen’s play style: by going for offbeat opening choices, he embraces ambiguity and face it with ingenuity. By steering away from exploited openings, he’d purposely land himself and his opponents into small pockets of wicked environments in chess—in the middlegames and the endgames—where creativity triumphs over rote memorization.

Creativity is About Knowing Where to Aim

With the advent of AI assistant tools like GitHub CoPilot and Cursor, I’ve come across many claims about the impending irrelevancy of programmers in the future where AI reigns supreme. In the mean time, the programming culture spawn yet another subculture, called vibe coding—where the main language for software development is English, and codes generated by AI are not meant to be fully understood (if you do review, understand, and test the code, you’re not vibing).

Perhaps one of most vocal leader who shared this thought is Jensen Huang, the CEO of NVIDIA. In 2024 last year at World Government Summit, he mentioned something about the future of programming to the chagrin of many programmers (myself included):

I’m going to say something, and it’s going to sound completely opposite of what people feel.

You probably recall, over the course of the last 10 to 15 years, almost everybody who sits on a stage like this would tell you it is vital that your children learn computer science. Everybody should learn how to program. In fact, it’s almost exactly the opposite: it is our job to create computing technology such that nobody has to program, and that the programming language is human.

Everybody in the world is now a programmer.

This is the miracle; this is the miracle of artificial intelligence. For the very first time, we have closed the gap. The technology divide has been completely closed, and this is the reason why so many people can engage with artificial intelligence. It is the reason why every single government, every single industry conference, and every single company is talking about artificial intelligence today.

His speech would ring true, if only programming is only constrained to the problem of writing code to make the interpreter or compiler happy. But programming is more than just about generating code. Software needs to be useful in order for it to continue to exist, and much of the input to the activities that generate real values in the software product requires humans. While writing code is an important part of the business, it is not complete in itself to deliver real software value. To give some examples, think software architecture and domain-driven design—both of which require very hard-to-master skills (especially as programmers), like developing business acumen, making architectural decisions, and making complex topics stick.

We should also be aware of the fallacy of thinking that the programming domain is a zero-sum game: That for one party to win, another must lose; that if AI dominates programming, then the programmers are pushed out of job. This worldview disregards the fact that programmers and software developers operate in a collaborative positive-sum world. In fact, as AI joins us in this positive-sum world, it is just so that AI needs to capitalize on human programmers’ inputs for its outputs to be useful, and in turn, human programmers need to capitalize on AI to remain relevant.

This doesn’t mean I completely disregard the notion of AI replacing human programmers in the future. The key difference between the AI we’re using right now and the one that Jensen Huang portends is in their capacity (or lack thereof) to be a manager instead of the individual contributor; to be an architect instead of the builder; or, to borrow Ben Thompson’s analogy, to be an Artificial Super Intelligence (ASI)—the rifle barrel—instead of Artificial General Intelligence (AGI)—the ammunitions:

What o3 and inference-time scaling point to is something different: AI’s that can actually be given tasks and trusted to complete them. This, by extension, looks a lot more like an independent worker than an assistant — ammunition, rather than a rifle sight. That may seem an odd analogy, but it comes from a talk Keith Rabois gave at Stanford…My definition of AGI is that it can be ammunition, i.e. it can be given a task and trusted to complete it at a good-enough rate (my definition of Artificial Super Intelligence (ASI) is the ability to come up with the tasks in the first place).

In this perspective the future of programming is certainly bright: We may not be too far from a future where we can “hire” cheap individual contributors—the ammunitions—to help generate 100% of the code. But ammunitions are only useful when we—the rifle barrel—point them to the desired targets. It gets trickier in a wicked world, where the targets would often dance unpredictability. Often times, it is even unclear which targets we should point to. In other words, creativity is about knowing what and where to aim; simply being the projectile isn’t!

In an era where AI reigns supreme, the game we must learn to play will be that of unpredictability, chaos, and ambiguity. It is no longer about perfecting the established art, but mastering the art of exploration and experimentation. And if history is any guide, humanity has always been remarkably good at just that.

Digitalization, or digital transformation, has long since become a key factor responsible for the progress and development of the world’s economies. “Every company is a technology company,” or variations of the phrase, is an oft-spoken mantra for highlighting the importance of digitalization to every organization, no matter the industry. Recently, the push for digital transformation is usually induced by the furor surrounding emerging technologies, such as artificial intelligence (AI), cloud computing, big data, and blockchain.

With the hype around the emerging tech, many organizations, perhaps due to fear of missing out, are racing to transform their business to be “digital”. But what does that mean? If it’s about adopting digital technology, we’ve had a track record of that going back to a few decades ago. (The first digital alarm clock was first patented in 1956.)

From experience, when people talk about going digital, they usually mean adopting emerging technologies to realize its lofty promises: that big data and data analytics enable data-driven business decisions, and in some way or another, AI is a silver bullet that can solve an organization’s myriad problems.

Of course, the promises of big data and AI are real. However, they have led some organizations to develop a lackadaisical strategy for digital transformation. Decisions for digital transformation are mostly pivoted around the procurement of emerging technologies, rather than by reimagining the operating system of the business and changing the ways of working; that is, transforming the business operating model.

This is unfortunate. Digitalization should be an enabler and offer opportunities for lifestyle change instead of a mere leap in the technology being used. To do otherwise is to only assume the appearance of a smart organization. A masquerade.

Masquerading Digitalization with Digitization

Simply carrying out a mere transition from paper to its digital forms (e.g., PDF documents, image files, etc.) does not imply digitalization. Here’s an example.

There are a few local restaurants that I’ve personally experienced where their effort to digitalize backfires miserably, to the point where it would have been much easier to order the menu without the digital technology.

There’s no longer any physical menu placed on the counter or the dining table. QR code stickers, however, are conspicuously glued onto each table. When scanned, it would lead to a shared Google Photos link containing the images of the restaurant menu. With a bad cellular connection, it would take a while for the thumbnails of the images to load at their full quality. There was quite a number of images; one image for each page. After a few swipes left and right or scrolling up and down the list of images, it can take a while to read most of the menus to find what I’m looking for before I can finalize my order.

In the end, replacing physical menus with digital copies does not make discovering (and re-discovering) menus easier. The overall experience turned out to be frustrating. To make things worse, everyone can see who has visited the shared Google Photos link.

Clearly, whoever decided to replace the physical menu with digital pictures muddled digitization for digitalization. Lifting and shifting your physical copies of data into a digital landscape does not make your organization digital. In fact, the meaning of digital transformation is evolving, with more than 72% of industry leaders viewing digitalization as much more than lift-and-shift.

To give them the benefit of the doubt, for all I know, the restaurants were just trying to save some paper, albeit at the cost of a somewhat disappointing customer experience. But some of the worst offenders yet are those who lauded and coveted digitalization the most: Enterprise IT.

Masquerading Digitalization with Emerging Technology

Migrating to the cloud is a no-brainer when organizations decide to “go digital”. The cloud had risen in favor over on-premise infrastructure for more than a decade since Amazon first revealed a unique business model at the time, stemming from an idea—a multi-billion dollar idea¹—that their existing compute and storage infrastructure can be offered as a service, with great agility and scalability.

According to a 2023 cloud adoption statistics by Zippia, 61% of businesses migrated their workloads to the cloud in 2020, although the share of workflows on the cloud is quite low, at only 25%, according to The Economist.

In Brunei, a 2019 report by the Authority for Info-communications Technology (AITI), a statutory body that focuses on the development of IT in Brunei, showed that most IT services infrastructure (e.g., data analytic tools, business process automation, HR management system, etc.) are on-premise², with less than 5% cloud adoption for each sector. That was five years ago. (Unfortunately, I couldn’t find recent reports on cloud adoption in Brunei.)

Nevertheless, over the past few years, signs for cloud adoptions by local businesses are increasingly popping up, such as the adoption of AWS Managed Services by Datastream Digital Sdn Bhd (DST) and Progresif’s partnership with Alibaba Cloud. Brunei’s Digital Economy Masterplan 2025 by the Digital Economy Council (DEC) even mentioned “Public Cloud Adoption” as part of its key projects in 2021, among other digital transformation initiatives.

It is all the more important, then, to understand what it really takes for an organization, especially those already operating for decades with the support of Enterprise IT to undergo digital transformation.

Many Enterprise IT are prone to hold on to their long-held traditions. Most are unable to forgo IT processes and operational culture that are already well-established and deeply entrenched in their organization. The processes are followed almost down to a dogmatic level, in the name of minimizing risks and ensuring stability and reliability.

The typical tradition includes loathsome bureaucratic IT processes. They are the manual and tedious checks and balances organically injected at almost every step of the value streams throughout the organization’s lifetime, thanks to the learnings from past incidents and failures. In other words, they’re the organization’s counter-responses to past disappointments, which, unfortunately, end up introducing waste in the organization’s value streams. Fittingly, in the DevOps community, they are referred to as “scar tissues”.

Traditional change management processes are one example of scar tissue, where each “normal” change requires a lengthy process of approvals, careful planning and coordination, and manual testing and deployment methods. To make things worse, these are usually exacerbated by the lack of cooperation between teams in organizations that suffers the symptoms of organizational silos.

Organizational silos are the defining feature of traditional IT, where teams are locally optimized, but end-to-end cross-team value delivery are not. Thus, works that require cross-team collaborations are impeded due to friction. The existence of the friction is easy to explain from economic standpoint: the builders (e.g., software engineers, data engineers, data scientists) are incentivized by making more changes or adding more features in response to the changing business needs, whereas the infrastructure team and the support team are disincentivized because they want to keep the software reliable and easy to support (by the virtue of less frequent changes and fewer features to support). As economists are wont to say, our actions are driven by incentives. When the incentives are not aligned, you get friction.

Without fixing the organization’s processes and organizational structure, whatever fancy technology the organization is trying to adopt will go to waste. Suppose you have set up Continuous Integration and Continuous Delivery pipelines (CI/CD) which automatically run unit tests and integration tests against your application each time a change is made to your code in your version control repository. Suppose also that you’ve painstakingly written some custom scripts to allow an application custodian to pick a snapshot version of the application to be automatically deployed to the production environment. If your organization still fanatically follows a traditional change management process—the likes that require approvals across multiple stages and require you to fill in numerous forms—the value proposition of the continuous property made available by the CI/CD pipelines will be diminished. In short, the inherent culture of dogmatism and fanaticism for traditional ways of working inhibits digital transformation from squeezing out its maximal value outcome.

Understanding the implication of this masquerade, as described above, is paramount for successful digital transformation. In the effort to transform an organization to be digital, it would be neglectful to be obsessed with the procurement and implementation of emerging technologies without expending more energy and resources on transforming their people and the operating model they’re clinging to.

Seeing Through the Masquerade

Mixing emerging technology with traditional IT culture is a recipe for disaster. As Gregor Hohpe, an Enterprise Strategist at Amazon Web Services wrote in his book Cloud Strategy:

If you don’t know how to drive, buying a faster car is the worst thing you can do.

Viral videos online showing amateur drivers driving recklessly and crashing their cars are all too common. Its dour consequences are obvious; and, grim as they may be, they serve as effective reminders and warnings to other drivers.

Organizations driving the digitalization vehicle do not normally have this luxury. Instead, digitalization mishaps are difficult to notice: We’re using the cloud now, we’re digital!, or We’re using Docker to do the deployment! are just fanciful masquerades if symptoms of organizational silos are still rampant across the organization, or deliverables take months before they can be deployed and realized.

For digitalization strategy and execution to be successful, it is paramount that everyone across all levels in the organization can see through the masquerades and understand the desired outcome of digitalization for the organization that maximizes its business value.