Between evidence and experiments-based regulation: hybrids and nudges

This is a transcript of my lecture at the “Best in Class Regulation: Meeting the Challenge” workshop, organised by RegNet: School of Regulation and Global Governance, Australian National University, 8 August 2016.

Best in class workshop slide


Good afternoon everyone,


Experimental regulation and governance

Experimental regulation and governance has made quick inroads in the development and implementation of regulation. And over the last years, scholars have begun to actively study this approach to rulemaking and implementation.

But why experiment at all?

The short answer to that question is that traditional forms of regulation rely on stability, predictability and uniform applicability. Road safety is a typical area where you want stable, predictable and uniformly applicable regulations.

Road safety is, however, a fairly non-complex area for regulation. Many areas of regulation are much more complex. They are ever evolving, see ongoing technological innovation and social change, and involve a wide variety of actors with different needs and capacities.

Typical areas of such complexity are healthcare, food, and the built environment.

Actors in these areas (may they be providers or consumers) often critique traditional regulation for causing excessive burdens on society. And they often critique traditional regulation for limiting the innovativeness of the sector and its competitiveness.

Of course, politicians and regulators are receptive to such critique. They are open to introducing new or amended regulation to aid these critical actors. Experimentation provides them a means for doing so.


How does experiments based regulation work?

There is no single blueprint for how to carry out a regulatory experiment. But, there is some consensus about what makes for meaningful experiments.

It is argued they need to be time fixed, because otherwise there is a risk that an experiment silently becomes a policy program before lessons have been drawn. Experiments need to focus on a limited group of participants or limited jurisdiction. This to aid the collection and analysis of data, and communication with participants.

It is often argued that experiments need to be developed and carried out as a collaboration of experimenters and participants. Meaningful experimentation is not just turning some dials and see what happens, but it is a collaborative and iterative process where experimenters and participants work together to find a regulatory solution to a specific complex problem.

Finally, it is strongly advised to set upfront goals, as well measures to assess whether or not they have been achieved. And it is recommended to carry out ongoing evaluation of the experiment. This all to maximise the lessons that can be drawn from the experiment.

But that’s experimentation on paper. How does it work in practice, you may ask.


Experimentation in practice

Between 2012 and 2016, I have studied over 30 experiments in 16 cities in 8 countries. All these experiments sought to develop, test and implement novel regulatory instruments in the area of sustainable city development.

Realizing the transition from our currently high resource intensive, energy inefficient cities to cities that are more sustainable is a highly complex regulatory problem. This transition comes with rapid technological innovations, many of which that are available but not yet regulated. It comes with huge economical costs and vested interests. It comes with a vast array of actors, all with their own capacities and needs. This holds for the supply side as well as for the demand site.


Four experiments

Let me briefly talk you through a handful of experiments that I have studied.

Green Door was an experiment run by the Queensland Government in collaboration with a number of construction industry stakeholders. It sought to fast track development approvals for projects that proposed high levels of urban sustainability. Often these projects make use of solutions that are not yet accepted under current regulation, for instance the reuse of rainwater or being fully off the grid in terms of energy provision. For these projects special case managers were assigned that had considerable regulatory freedom in assessing the “deemed to satisfy” status of proposed solutions. The initial idea was that these ambitious projects could inform future regulatory change, but the experiment was terminated in 2012 when a new government came to power in Queensland.

CitySwitch Green Office challenges office tenants in Australia to reduce their energy consumption. Tenants interested in participating sign up and commit to achieving energy efficiency improvements well beyond what is required by mandatory regulation. In return they are supported by local governments. Sometimes support is financial, but more often it is the provision of information about how energy reductions can be achieved. Office tenants often find it difficult to find such information themselves, and CitySwitch addresses that exact problem. This is an intriguing experiment as it has a strong focus on influencing office tenant behaviour. This is an area that is normally not addressed through building regulation.

The Better Buildings Challenge in the United States challenges property owners to improve the energy performance of their existing buildings with 20 percent compared to mandatory requirements. It helps building owners to find funds for retrofits. And, like CitySwitch, it supports building owners with information regarding how to achieve energy efficiency improvements. The experiment seeks to understand whether a significant increase of building energy efficiency is possible with current technology and knowledge, within the current regulatory framework, and at no long term cost for the regulator or property owners and users involved.

A final noteworthy experiment is Eco-Mileage in Seoul. It works a little like frequent flyer miles you get from airlines. Under the experiment, the City of Seoul awards households with points, termed “Eco Miles”, when they reduce their energy consumption by at least 10 per cent over a two-year period. Higher levels of reductions result in exponentially higher amounts of points awarded. Points can be used to buy energy efficient appliances and services. This experiment is particularly interested to understand whether considerable energy reductions are possible at household level without changing existing building codes. It is further of interest as it is run by a city, Seoul, that itself cannot change the national building codes of Korea.

But what has been learned from these experiments?


The main lessons learnt

Unfortunately, my research does not result in overly positive conclusions.

Overall, I do not see much systematic experimentation. Often experiments do not have a limited time-frame. Often they do not focus on a limited jurisdiction or a limited group of participants. But most problematically, often they do not begin with setting a baseline or benchmark against which to evaluate performance. Many experiments had not in place processes to collect and assess data. And in many of the experiments it became clear, over time, that it is very difficult to obtain trustworthy data from participants.

This implies that the experimenters often do not know much about the transferability of lessons from their experiments, or whether or not it makes sense to scale them up to policy programs and formal regulatory regimes.

Still, it was possible to get some insight in the overall performance and improvements achieved by these experiments. For most of the experiments I was able to set a benchmark against which to assess their performance, but in hindsight. For example, for CitySwitch Green Office I took the energy consumption of the full Australian office market in the year the experiment was scaled up to the national level. This was 2011. I then took the reported performance in the year 2015 and contrasted this with the benchmark. For CitySwitch it became clear that it had only achieved a reduction of half a percent of the benchmark. In other words, after four years of much time and effort, the experiment has reduced office energy consumption in Australia with half a per cent annually. Such marginal performance was the rule, rather than the exemption in my study.

Despite this overall marginal performance, often these experiments had over time been silently transformed and formalised to permanent policy programs. A partial explanation of this trend is that the experiments were not bounded in time or space at the start. Without a formal end-date and without a formal boundary it was easy to keep pushing them on in time, and broadening them up in space.

More problematically, and another part of the explanation of these silent transformations is a very normal human tendency to ignore failure. With much time and effort invested in these experiments it is difficult for the experimenters and participants to terminate them. On top of that, I indicated what I term a “leadership delusion”. Many of the more than 200 experimenters and participants that I have interviewed talked up the handful of good practices achieved. They argued that once the leaders are on board of these experiment the rest will follow automatically. My research indicates, however, that this does not happen.

So how to move on from here? If traditional regulation is too restrictive for certain areas, but experimental regulation does not achieve its anticipated results, then the answer is perhaps somewhere in between.

And that is where diffusion of innovation thinking as well as insights from the behavioural economics come in.


Insights from diffusion of innovations thinking

This is the ideal type representation of the uptake of any innovation thinkable. The model is based on over 50 years of research on the uptake of innovations. It indicates that any innovation begins with acceptance from leaders and early adopters. They inspire the majority to use it, and ultimately the innovation become so persistent and normalised that laggards will have to come on board also.

There is however a very critical moment in the spread of an innovation throughout the market. The transition from leaders and early adopters to the majority.

And another general insight from the diffusion of innovations literature is that sometimes innovations do not achieve this transition. If an innovation is not of interest to the majority it gets stuck, and that is exactly what has happened with many of the experiments I studied. They got stuck at the level of leaders and early adopters.

Now, why is that? Well, over the years it has become clear that the diffusion of innovations model is indeed a good predictor for homogenous markets, but not for very non-homogenous markets. And the market for sustainable buildings and infrastructure is all but a homogenous market. In such markets innovations likely face a chasm that is hard to overcome.

I argue that one of the core reasons here is that the experiments I studied have too strong a focus on getting the leaders on board. They build on leadership narratives and leadership rewards. They talk about how great it is to be seen in as a leader in the industry and society. Such leadership rewards were often considered more important than for instance the actual cost savings that could be achieved.

Yet, these specific leadership rewards do not resonate well with the majority and the laggards.

The behavioural economics provide some answers why this is. So let’s have a look at them.


Insights from the behavioural economics

Most of you will have heard of this famous book and the theory underlying it; nudging. It argues that people make irrational choices. Choices that are not in their own best interest. The book further argues that by changing defaults regulators can help them making better choices.

A well known example of nudging is to present healthy food first in a school café and the less healthy alternative further down the line. This will likely result in students choosing the healthy alternative over the less healthy one. Another example is to make people first sign forms to promise that they will fill it out honestly, rather than letting them sign a form after filling it out.

Whilst nudging may work very well for these relatively simple regulatory problems it comes with some complications for more complex ones. And more and more critique is being expressed to nudging as a regulatory mechanism.

To begin, so ask some people, is not all regulation ultimately a form of nudging? Isn’t it old wine in new skins? So why then would nudging provide a better answer to regulatory problems than other approaches? Another key problem with nudging is the paradoxical situation of providing a rational answer to irrational behaviour. Are regulators not equally irrational as those they seek to regulate?

Most problematically, I feel, when I look at the current rise of nudging as a regulatory strategy is that it is often forgotten that “irrational behaviour” comes in many forms. And those on the right side of the screen are but a few that the behavioural economics have uncovered over the last years. And the list of irrational behaviours keeps growing. The more complex a regulated area, and the more actors involved, the more forms of irrational behaviour may exist. Is there a one-size nudge to address them all? Certainly not.

Still, when taking a step back and look at the various forms of irrational behaviour that nudging and the behavioural economics talk about, it becomes clear that they can be roughly clustered in two categories. Loss aversion and status quo bias.

Loss aversion relates to the tendency of people to value loss larger than gain. The tendency to want to have a small gain in the now rather than a large gain in the future. And the unwillingness to pay a little in the now for a large gain in the future. Some forms of nudging, I believe, are suitable here.

Status quo bias relates to the tendency of people to consider the here and now the best there is. Any deviation from the here and now will, in their perception, result in a situation where they are worse off. This, I feel, is a very difficult bias to overcome with nudges.

Now, let me super-impose these insights on the diffusion of innovations model.


Combining diffusions of innovations thinking with insights from the behavioural economics

In doing so, it looks to me that the notions of loss aversion best explain the “irrational behaviour” of the majority. In short, they do not appreciate the long term benefits of the experiments over the short term costs.

The notions of status quo bias resonate with the behaviour of the laggards. Whatever novel regulatory instrument presented, they will express preference for no change.

The real puzzle now is, what types of regulatory instruments are suitable for these two groups? I argue that this is where hybrid forms of regulation and nudge type approaches come in. Let me give a number of brief examples before I round up this presentation.


Towards hybrids and nudges

A first hybrid situation would be to move away from the current distinction between voluntary experiments, and mandatory regulation. Many of the experiments I have studied ask for voluntary participation, and often set voluntary performance levels for participants. This allows some participants to go the extra mile and achieve even larger leadership rewards.

The problem of this model is that you have two distinct groups. Those in the experiments and those not in it. Those in the experiment may influence each other, but it is unlikely that they will influence those outside it.

So why not move to a situation where everyone is part of a new experimental regulatory regime, but the level of performance is voluntary. Suddenly those who were earlier not participating are now the poorest performers in the group.

This model has for many years been successfully applied by NABERS, the National Australian Built Environment Rating System. But it is not hard to imagine health care classification or food labelling regulation along similar lines.

A second approach I can envisage here builds on insights from the behavioural economics. Voluntary opting in, as was the case with many of the experiments I studied may be a cause of their limited impact. In areas such as organ donation and top-up pensions it has been shown that people are likely to stick with the default option of not opting in when given the choice to voluntarily participate.

So why not move to opting out systems? Imagine you buy an existing house or office building and the bank or local council offers you a top-up mortgage to retrofit the place. You are given information about the advantages of retrofitting, the personal losses of not doing it, and so on. Of course, if you don’t want the top-up mortgage you are free to opt out of it.

Again, insights from the behavioural economics indicate that for such regulatory problems there is a good chance that people stay with the default option. In this case, having a top-up mortgage. The two systems are almost the same, but by changing the default considerably different outcomes may be achieved.

A third way would be to work towards evidence based regulation. What we see now is that experiments and regulatory change are often not aligned. This may result in a situation where regulatory change in no way resembles what has been evidenced with experiments. It could even result in a situation where practices from an experiment are still not included in regulation after many years.

An example here is the Melbourne City Hall that was opened in 2005. Back then it was a leading example in one of the experiments studied. In Australia it is still considered to be among the best examples of sustainable building practice. But the building is now 10 years old and the Australian mandatory building codes have still not caught up with what was possible 10 years ago.

So, why not move toward rolling rule regimes. In such regimes, experiments are used to inform future regulatory changes. Say, the best practice in an experiment today will be the bottom line mandatory requirement in five years from now. The best practice in five years from now will be the bottom line in ten years from now. And so on. This will force the laggards in the market to follow the leaders.


Bringing it all together

Now let me, by means of conclusion, bring this all together. Experiments, as currently applied, seem only of interest to the absolute leaders in industry and society. Still, the lessons learnt from them might help developing hybrid forms of regulation and nudge type interventions. These may help addressing loss aversion by building on strong narratives of personal gains and by providing clear personal rewards.

At the same time experiments, as well as the hybrids and nudge type interventions, might help develop evidence based regulation for the laggards. They can be introduced following a narrative of “new social norms” to address the status quo bias of this group, and more than working with positive personal rewards they can be introduced and enforced using the traditional force of law.

Of course, this is an ideal type dynamic regulatory regime. I have not seen it applied in practice. But it provides food for thought and sums up the take home lessons of my presentation.

The take home lessons are that in complex regulatory areas we need to think about the differences in the actors we seek to regulate—the leaders, the majority, the laggards. We need to think about the different biases they have towards new forms of regulation—wanting to be seen as a leader, loss aversion, and status quo bias. And we need to think about the different narratives that are required to present new regulatory instruments to these groups—leadership narratives, gain narratives, and new social norm narratives.


With that I like to round up. Thank you for your attention. Are there any questions?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at

Up ↑

%d bloggers like this: