In a new series of blogs we give you insight about software engineering, as seen through the eyes of our technology team. Here, Josh Samuel, explores the benefit of sharing prior experience.
For many years, software engineers have strived to avoid reinventing the wheel, including designing and documenting architectural patterns, writing libraries and open-sourcing code. What does this mean practically for an organisation that strives for creativity and innovation?
How many times do you look across teams and individuals, to see that they have built solutions for the same problems, in different ways?
Have you completed a major portion of a large project, only to be asked why you didn’t just use the neighbouring teams’ existing solution?
Team creativity
If you ask a room full of creative people to draw the instructions on how to make toast, you will get a lot of very different drawings. Many people will have a step that involves putting bread into a toaster — unless someone gets creative with a toasting knife.
Who had the most creative solution? Did they draw a standard toaster or some other method of making toast? How many of the options had mistakes or gaps in the instructions? When you followed through with testing of the possibilities, how many further problems were picked up? Did anyone’s toast burn?
More importantly, did any groups of people, or individuals compare notes/diagrams, or team up to offer help to their colleagues?
While most people know how to make toast, collective wisdom can often produce a better solution. Different people will bring their focus to various aspects of the process, filling in omissions or fixing errors that others skipped over.
Making toast consistently
“Can I have some Vegemite on toast?” calls one of the kids. “I want cheese on my toast!” calls the other.
Just like when making toast for the kids, in almost all cases, you should not need to think about starting from scratch when engineering software. Service templates or application templates have been used for decades as a quick-start for engineers.
A search of github shows that templates and quickstarts are popular.
If you build it, they will come — or perhaps not.
On Github, a quick search says that there are approximately 366k repositories, 1.6M issues and 214k wiki pages that relate to templates or quickstart as of January 2020.
If you’re a Microsoft-based programmer, you’ll know that Visual Studio has for decades had a set of screens that allows you to quickly create a new application shell. The boilerplate is there, and all you have to do is add your business logic. There is now also the ability for Visual Studio to set up your continuous delivery to a cloud service, if your CD tooling and cloud service are both also owned by Microsoft.
Visual Studio — Create a new project dialog
Nowadays, the actual deployed application code is often less as there are frameworks that provide much of the boilerplate of a service or application. This, though, is offset by the expectation (and often requirement) that there is a build & deploy pipeline, test runner, linter, dependency security analysis and more — many of which are not provided by most frameworks.
When building many services, every small difference in this boilerplate costs time, in both upfront and maintenance, as requirements change and as the supporting tooling is uplifted. So having a shareable template for the full lifecycle of the code becomes more important.
One template is unlikely to fit all of your requirements, and as the number of templates grows, discoverability becomes hard. You are likely to get duplicates with small differences, and teams that run-away in their own direction because the other template didn’t do exactly what they needed.
Focusing the creativity
For some organisations, having a recipe book, such as a technical catalogue, that can enable the exploration of recipes may suffice. This enables creativity without the need to start from zero. If no recipes match, it is also an excellent trigger for the engineer to create a new method or approach and then provide this back to the community.
For this to work, the recipes need to be frictionless or as close to frictionless as possible. The formula must match the environment and align to the technology stack the engineer is working with, while enabling non-functional requirements, such as security, performance and resilience. With a few button-clicks or via the command-line with scripts, an engineer should have something that can be deployed safely to production.
As a reward to teams that contribute back to the recipe book, they will get a quick-start way to create new services, as well as having their non-functional requirements reviewed just once for the template, rather than individually for each service.
As the number of teams grows, you will find that you have a team/organisational problem, rather than a tooling problem — “Not Invented Here” syndrome may pop up between office locations or between larger teams. At this point, you may want to consider a guild-like structure where like-minded engineers across the teams and locations can form to keep things converging and improving.
Attributes of a great technical catalogue
How you present the catalogue can make a big difference to its adoption.
Consider:
- Presenting it via your well-established intranet or collaboration platform (the place your engineers go for information).
- Ensuring search-ability of solutions by use case, type, language/framework or business domain.
- Having a standard format for presenting all the relevant information on a solution. Make sure the introductory section has all the most relevant information and is easy to read.
- Listing any limitations and costs for the solution.
- Linking to your security and design reviews.
- Adding relevant diagrams to make it look pleasing to the eye without compromising on readability and accessibility.
- The ease of getting started (through to your first production deployment of “nothing”) from your instructions. Make sure the steps are clear and concise, assuming no prerequisite knowledge.
Once built, execute some internal marketing to let people know that it exists, or boot camps to seed your initial users. Most importantly get feedback from your teams. It’s likely that your first cut won’t be right. It will need improvement, so measure and improve.
Perhaps you do need to reinvent
Sometimes, the barriers are just too high. A team is prototyping a new tech stack that differs vastly from before, or you have just acquired a business with a very different way of operating their environment, or perhaps your patterns are getting old and bloated and you have made a conscious effort to rebuild rather than refactor.
In these situations, don’t prevent the reinvention, but embrace it. Bring the inventors into the fold and encourage their learnings to be shared. If a guild is formed, encourage them to join. Technology changes faster than business requirements change, so a little reinvention is still good for your technical health.
Where can this work?
I have seen this pattern used successfully over the years, including:
- Migrating database abstraction layers away from a deprecated library — with a catalogue of new patterns for people to look up.
- Templates for new services — a catalogue of service types. Upon choosing a service type, and hitting create, it creates everything including the build and deploy pipelines.
- Class templates — with a catalogue of templates documented and extending the developers IDE to allow the fast creation of classes of differing purposes.
As you can see, it can be used for anything from the micro to the macro.
What can prevent success?
If your organisation is siloed and your teams don’t want to talk to each other, they will likely have their own information repositories on the collaboration platform they use. As a result, each team may end up with their own catalogue.
If the platform you use to collaborate on is a mess, and discovery of the catalogue is poor, it will make it harder for you to succeed. People will likely give up on that mechanism for searching altogether — making shared collaboration difficult.
Finally, if your recipes are too complex — if you need a large number of steps to get started using a template or require implied knowledge, people may decide that it is easier for them to just start from scratch. Automation and documentation can go a long way towards reusability, though clearly this is not a good recipe in any case, due to its complexity.
