Over six months, Nesta worked with the Public Innovation Team (EiP) in Colombia’s National Planning Department (DNP) on a series of live pilot projects. One year on, we reflect on what we learned through the experience and present a few lessons learned for how to design, manage and use pilot projects more effectively to drive innovation in government.

In spite of several years of experience and lessons learned about innovation projects in government, there is still a tendency to set them up to fail. This is because a large amount of innovation efforts come in the form of “innovation pilots”, which are run on the promise of being “demonstration projects” capable of illustrating the value of a new approach, method or way of working. Pilots are often expected to deliver positive, demonstrable change in response to a challenge - a result that is usually described as a “solution” to the challenge.

The consequence of these expectations is that most of the actual lessons and valuable insights gained during the project get lost, and its success gets judged essentially on a binary scale: did it work or did it fail? In one typical scenario, the road ahead is rarely organisationally secured, and the pilot then evolves into the task of creating conditions for the successful approach to spread and be sustained.

In a second typical scenario, this setup either sends the innovation pilot into eternal oblivion, or it forces the people responsible for the project to set up a kind of “innovation theatre” where they present a narrative about the pilot’s effective value beyond the evident results. Neither are very productive for driving innovation efforts in government.

In this blog, we share some insights from our own recent experience and some overall considerations for enabling, driving and supporting innovation through pilots in the future.

As part of States of Change, Nesta’s Innovation Skills Team collaborated with Colombia’s Equipo de Innovación Pública (EiP) on a six-month programme supported by the Newton-Caldas Fund to help build innovation capacities and skills in the Colombian government.

This involved providing better strategic support for innovation across the Colombian government by advising the country’s new innovation strategy, strengthening the innovation team’s role and mandate (read more about this here) and working with two innovation pilot projects.

The first pilot aimed to address the huge Colombian (and global) issue of environmental sustainability, focusing on how to reduce black carbon emissions for a range of potential benefits in health, environment and infrastructure. The second pilot focused more on internal systems transformation: how to develop better ways of managing and transmitting knowledge and data in and across public organisations, to strengthen collective intelligence over time and enable better decision-making and service delivery. You can read more about both of the pilots at the end of the article.

Our aim was to define the status of these pilot projects, scope out useful next steps for their progress, and from this identify learnings about processes for innovating from within the Colombian government. However, as is often the case with innovation programmes and projects, the expected needs of the pilots were not those that emerged during the programme.

We’ve found that innovation practitioners leading pilots often end up having to “sell” the project as the answer to a complex problem. The implications of this are that, instead of exploring new possibilities through an explorative and often unpredictable process, innovation pilots have to promise a valuable result.

So while innovation pilots should be a way to deal with lack of knowledge about what might work, the bureaucratic logic translates them into processes of proving a specific theory of change or a “best practice” specification. Paradoxically, when innovation pilots are considered to be failures, they are often more likely to have been failed by their authorising environment and wider networks of support.

These are patterns we see happening in governments around the world, which reiterates the importance of the overall framing of innovation projects and what expectations are set. The Colombian government is currently on the path to embracing and embedding a more experimental culture - as are a number of our partners within the States of Change learning collective.

This means pivoting from running innovation pilots as demonstration projects to doing experiments as a systematic way of accelerating learning about possibilities and testing out assumptions in real-life settings.

This transition is - among other things - based on the following lessons learned in our collaboration.

Lesson #1: Demonstration project vs. experiment to accelerate learning

It quickly became clear that the two innovation pilots were expected to be demonstration projects - rather than exploring and learning about new possible ways of addressing the challenge. The project teams struggled to navigate these expectations and worked hard to try to test new hypotheses built on new data generated about what might work. But if the expectation is a demonstration of an “innovation recipe” for success, this is an uphill battle.

Lesson #2: From one-off authorisation to an ongoing process of building the authorising environment

It also became apparent that the organisational commitment was unclear - largely due to a lack of involvement from the executive level throughout the process. The projects would have benefitted from more than a one-off endorsement, but rather an active participation throughout the journey.

It is therefore useful to think about authorisation not as a formalised approval, but an ongoing creative process of building the authorising environment. In practice, this means constantly signalling an organisational commitment to carry on with the project, creating clear paths to find and secure resources needed to implement and evaluate prototypes, and connecting and unlocking support from potential internal and external partners.

Lesson #3: Prioritising and embedding learning outputs as part of a strategic project portfolio

Building on the above lesson, the innovation pilots also illustrated the importance of a dedicated space for exploration and experimentation where learning is valued over traditional outputs. This not only requires an openness to challenging initial assumptions of a project, but also a commitment to effectively use learning.

The innovation pilots would have benefitted from developing a strategy for how stakeholders should be involved, what was expected from them and how they could apply the insights and concepts discovered in the pilots. There is no point in doing pilots if what is learned is left to gather dust.

Lesson #4: Creating a better link between small scale experimentation and large-scale ambitions (systems change)

We found that there wasn’t a clear plan or process in place for how these small-scale pilots were part of addressing a large-scale system challenge (urban pollution/climate change and knowledge management in government). The project teams found themselves caught between the excitement of working on an important issue and the demotivation of limited support and resources allocated to address a challenge of this magnitude.

Pilots could open up new perspectives and possibilities if it is clear how they fit into a larger strategic effort of addressing the challenge. Otherwise, the more likely outcomes for people are demotivation and paralysis.

Lesson #5: Create a mandate to evolve the project brief into a portfolio of experimental interventions

Building on the above two lessons, the question is whether it’s worth thinking about innovation pilots as ways of evolving (simpler) project briefs into a portfolio of experimental interventions.

Instead of fixating on a single problem description and designing a single intervention in response, a portfolio approach implies looking at a problem from different angles simultaneously and deploying an array of prototypes. This may then more efficiently reveal promising paths to achieve systemic change.

Recognising that most public organisations - to fit with bureaucratic procedures - still work from fairly static problem definitions and project briefs, can pilots be used to open up new possibility spaces to develop and evolve portfolios of experimental efforts to address complex challenges?

Lesson #6: Maintain pace and momentum of the project to mobilise and sustain change coalitions

One key overall lesson was the importance of maintaining the momentum of the pilot. Too often innovation pilots in government are delayed due to bureaucratic constraints and complex stakeholder management.

It was no different in this context, where one of the pilots was drawn out over more than two years. By then, the initial excitement for adopting a different approach is gone – and the project slowly drifts into business as usual project management routines. Planning for more intense experimental pilot sprints with heavy resource allocation in the front-end is needed in order to commit people strategically and practically.

Lesson #7: Setting up the right team to ensure the right mix of skills, mindset and leadership capability

The Colombian pilot projects demonstrated that embedding innovation in government isn’t easy, but requires an openness to enter into an explorative and experimental collective learning process. Each individual team member should bring skills and attitudes that align with the ambitions of the projects and include those outlined in the Competency Framework for Experimental Problem Solving. Innovators must have a balance of the competencies to:

1. Work together with citizens and stakeholders in a creative problem-solving process (as opposed to serving particular interests)
2. Lead change by mobilising resources and creating an permissive environment to make it happen
3. Accelerate learning by applying new methods to explore, develop and iterate new idea concepts to open up new possibilities and develop innovative solutions

Lesson #8: Develop pilot selection criteria to ensure the right conditions and expectations

Working with the innovation pilots, it also became clear that using more explicit selection criteria would be useful to ensure the right conditions and expectations were set from the beginning.

For experiments to succeed within the public sector, they need to have a balance of the right project, people and permission. The design and selection of the project area should be informed by realistic timescales and should consider the route to scale. People should be capable of and willing to reflect on current practice, with a view to adopting a greater level of experimentation and exploration in their work. Teams need to be able to leverage senior level buy-in as well as influence the right people and resources for change.

The capacity to achieve these enabling conditions should be evaluated from the outset and used as the main input for deciding whether to establish a full-blown innovation initiative.

Lesson #9: Secure skills and resources for strategic communication for different audiences and objectives

Effective communication is a cross-cutting need for applying these lessons in practice. Communication needs to be used strategically to create trust in stakeholders, secure permanent executive buy-in, reduce uncertainty, share and synthesise learnings, and engage public officials and citizens in the development of an innovation project.

Different audiences and objectives require different communication approaches, but many innovation teams frequently lack the sort of professional skills and resources needed to define them and bring them into action.

Although time has gone by, there is still potential with both pilots to reframe and revitalise them. In the case of black carbon emissions, increased awareness about the issue and growing evidence about the negative effects of black carbon indicate that the challenge is as relevant and urgent as it ever was.

Equally, better-adapted knowledge management approaches are still a pressing need in both the DNP and broader Colombian public sector. Building on the lessons learned, the EiP will continue working to enable innovation approaches in these and other policy challenges in Colombia.

This means becoming better at setting up pilots as more than just demonstration projects. Instead, the focus should be using them to learn about new opportunities for innovation, to accelerate the systematic testing of assumptions about what might work, and to test the organisational capacity to challenge business as usual and deliver on innovation projects.

Black carbon emissions pilot

The was a collaboration between the Ministry for the Environment and Sustainable Development and the EiP that aimed to find new approaches for reducing the quantity of black carbon emissions, specifically those from informal urban freight transportation.

When the collaboration begun, the pilot had already been running for 20 months and had involved an extensive exploration phase, in which a systemic understanding of the issue was established, the target population of informal freight carriers was identified, and efforts were carried out to strengthen innovation capabilities in the individuals and teams involved.

In partnership with stakeholders (such as ViveLab, Ruta N and AMVA), options for a product-service system enabled by technology were then developed. The main hypothesis was that freight carriers could be incentivised to behave in a more positive environmental way by developing a new market that used digital platforms to mediate interactions between carriers, their clients and other agents in the ecosystem.

By design, access to the platforms would be granted only to those carriers capable of demonstrating environmentally positive behaviors, thereby incentivising those behaviours system-wide.

Actually testing this hypothesis meant not only securing political and organisational buy-in, but also the financial and human resources required to translate the idea into action. However, the immediate context was an upcoming presidential election, which meant a variety of restrictions and uncertainties.

The difficulties in securing buy-in and resources led to a stall in the project’s development. Furthermore, the challenge of translating a systemic vision into cheap but workable prototypes that could be implemented locally was identified as a consistent barrier.

Knowledge management pilot

On the other hand, the knowledge management pilot was an early-stage project. It was a collaboration between the EiP and the General System of Royalties (SGR), a large unit in charge of managing funds derived from the extraction of non-renewable natural resources, which can be used to finance local development strategies. The SGR provides support to territorial governments to develop investment projects that can be potentially funded with these royalty funds.

This task requires specialised knowledge and a nuanced understanding of the institutional dynamics of each territory, which has been found to be key factor for project success. However, such knowledge remains tacit, only to be found “in the heads of people”. The aim of the pilot was to extract this knowledge from the most experienced people in order to transfer it to incoming officials, reducing the impact of high turnover, raising the quality of service provision and ultimately increasing user satisfaction.

As part of the MA programme in Design Management at Universidad Jorge Tadeo Lozano (UTadeo), over six weeks students, professors, and public officials from DNP went through a co-creation sprint to conceptualise and prototype ideas using a design thinking methodology. The resulting prototypes took a systems approach to articulate different ways of learning and diverse channels to produce and access knowledge, in order to allow for contextualised and personalised learning paths.

However, when the prototypes were presented to the SGR, there was no clarity on how to secure the resources required to iterate and implement them at scale. Furthermore, the unit was undergoing a change in its leadership positions, which also contributed to a stall in the project when previously built momentum was lost.