NAB is set to ‘innersource’ some of its key business platforms, after successfully applying the model to the development and maintenance of more internally-focused code libraries and tools.
Innersource is an increasingly popular set of software engineering practices that are used to create an open source-like culture inside of an organisation.
NAB has been innersourcing code for about two-and-a-half years, with the model forming part of a broader set of practices known as the NAB engineering foundation or NEF, which is designed to help development teams get code into the cloud and into production faster.
Another ‘big four’ proponent of innersource is CBA, which revealed its own work to iTnews earlier this year.
NAB engineering manager Matt Cobby told the Innersource Summit 2021 last month that the bank adopted innersource initially to remove duplication of coding effort and costs as different teams worked to make their products cloud-ready.
“We migrated Australia’s first highly confidential banking workload into public cloud in 2016, and we enabled teams to move rapidly and take control of their own outcomes, but this led to duplication of tooling and there was a need to reduce the cost per workload to scale faster,” Cobby said.
“It was in this situation that I found myself looking for a tool to automate AWS Credentials setup, and I found 20 different versions of the same tool across Github. Some were supported and some weren’t, and some were fully functioning and others less than perfect.
“I felt that this was definitely one of these places where we were not being as efficient as we could be, and I felt that the techniques of open source development could help us improve.”
Cobby said that the bank also wanted to reduce that cost of experimentation “in order to help teams develop new ideas and test out new business solutions quickly and efficiently.”
That led NAB to adopt innersource, which it defines as “the sharing of knowledge, skills and code across teams in NAB using open-source collaboration techniques”.
By creating formal ways to share the work of different teams and to collaborate on further development, the bank hoped to remove “undifferentiated heavy lifting” of multiple teams reinventing code libraries and tools, and in doing so, refocus the efforts of teams to reach a business outcome much faster.
Innersource setup
With “hundreds” of development teams across the bank that each had their own way of working, the bank focused its innersourcing efforts “on the interfaces between teams”.
“This allowed us to create a safe environment for engineers to talk to other engineers across the bank: to reach out, to understand their codebases, and to share their work,” Cobby said.
NAB said its adoption of innersource had to balance “the needs of the bank in terms of architectural endorsement, security endorsement, ownership, accountability and auditability, with the needs of an open source community to be creative.”
To do so, it appointed “community champions” that act as on-the-ground “evangelists”.
“They make sure that their peers know about innersource,” Cobby said.
“They’re running community showcases for new products … where they do peer review on the products, they check if the product meets certain criteria such as do we know what problem it solves, that it isn’t the duplication of an existing product, that it meets our minimum standards and that it has a strong ownership. It’s at this point as well that security and architecture both have a voice and can endorse or query any individual product.
“Where we have multiple solutions to the same problem, we’ll build a small community around that problem and we’ll work with all the interested parties to reduce that duplication and come up with a better solution for everyone.”
On the other side of the model, Cobby said a strong culture of product ownership has been established.
“This is where we make sure that each product within innersource has a distinct product owner,” he said.
“The owner’s responsibilities are around making sure that the product meets the minimum standards, that it has a workflow, that there’s somebody there to read and evaluate the pull requests, and to make sure that these pull requests meet certain SLAs.
“They’re there to provide technical support for the products and to take questions from people when they’re asking about contributions to the products.
“We also provide these product owners with a playbook in order to help them innersource their own platforms and their own products.”
Products that are to be innersourced are classified as either “curated” or “community”.
“The purpose of this is to show that when ‘consuming’ teams are looking at what they can use [from elsewhere in the bank], they have the confidence that the code they are using is endorsed and has production-level support - but we don’t set the bar so high that the community projects can’t get started,” Cobby said.
“Typically, a curated product is proven in production. We know that it’s gone through all our normal existing operational processes, that it’s running in production with customer workloads, that it’s been security tested, that it encapsulates many years of learning and experience across the organisation, and that there’s often significant investment behind it.
“This means that there are very few curated products, but they are very high quality.
“On the community side we embrace our open source origins and this is more of an incubator for new ideas. We make sure there’s a very low barrier to entry.
“We tend to use a more open-source style support model where it’s often by best endeavours, and the typical products we see in this space are around tooling or individual pipeline components which are used in the delivery of applications.”
While maintaining a “light touch”, Cobby said there are some minimum standards that all code repositories have to meet to “create a safe space for teams to reach out and work on other teams’ repositories in a clear and consistent way.”
“We make sure that every innersource repository and every innersource product has a README [file] that makes it very clear what the product is doing and what problem it solves,” Cobby said.
“We make sure the CODEOWNERS [file] is maintained and up-to-date so that external developers know who to talk to when they have a question.
“There’s a contributing guide so that when you want to make a change there’s a very clear path for you to do so, and a code of conduct to make sure that you know the acceptable behaviours for the team [that created the product or tool].”
Benefits so far
NAB said that code quality, collaboration and learning opportunities had all increased under innersource.
“When we write code in the open, we tend to write better code,” Cobby said.
“We’re improving discoverability and the ease of finding the source of truth for a piece of information, and we’re reusing intellectual property across the different domains.”
Cobby said that the openness made it easier to understand why certain architectural decisions were made.
“We peer review each other’s work and our discussions are in the open, so that we can always find out why a certain architectural decision was made or why this decision was made not to use a particular technology,” he said.
Teams can also move faster by making changes to existing code libraries directly, where required.
“When we have the ability to read another team’s repository, we have the ability to remove bottlenecks,” Cobby said.
“If you’re dependent upon a team and they can’t implement your change, you have the ability to make the change yourself and get it accepted into the core product.
“We’re then also breaking down the silos of the organisation and helping learnings from one area be applied into different areas.”
The bank saw some unanticipated benefits around “mentorship, cross-skilling and learning.”
“We found through some of the innersource hackathons that we ran that we had senior engineers mentoring junior engineers, we found frontend developers learning how to be API developers, we’ve had backend business service operators learning how to be frontend React developers,” Cobby said.
“This is one of the real unexpected benefits from innersource and is something which is giving us probably far more return on investment than we ever expected.
“So one of the main benefits for us is this cross-skilling of people across the organisation.”
Expansion opportunities
Still, Cobby indicated there are opportunities for the bank to strengthen its innersource adoption as well as to broaden its use.
“We’ve been looking at how we can innersource our business platforms,” he said.
“With some of the benefits that we’ve seen before about decoupling teams and removing the blockers from coupled backlogs, there’s a real business potential here for understanding how we can ease delivery through the organisation and across multiple platforms.”
He also saw further opportunities to automate some of the metrics NAB used around innersource; the bank is working with Github on this particular area of improvement.
“We’re looking at the number of people collaborating across teams, we’re looking at things such as product reuse,” he said.
“We have automation that scans Github for dependency management and tells us how many reuses of an individual library that we’re seeing. We can then quantify that library reuse into financial terms in terms of how much it cost to develop, and how many times it’s been reused.
“We’re also using the metrics for operational health of the innersource products, because it’s very important to check that products don’t end up in some wasteland. We’re using some of these metrics in reporting to find products which need some help, or that need an owner, and then we step in and get them some help.”
He continued: “We’ve just scratched the surface in terms of what we can do [here].
“I believe that the source code of an organisation is often an untapped source of intelligence, and there’s a lot of information there we could look at to help us understand what are the flows of information across the organisation.”
