Frequently when I’m on site I am not directly asked but I am expected to provide answers to my customers how to get the best use of a technology. In this post I’m examining a recent scenario around providing structure around deploying OpenShift in order to provide a collaboration environment that would aide the use of this technology. We were also deploying OpenShift but writing about OpenShift deployment is a well covered subject across the board.
I’ve recently visited a customer that wished to containerise the world and provide to their developer community a Container As A Service (CaaS) – a single enterprise kubernentes cluster that would allow groups of developers to develop and deploy, as well as an Enterprise Kubernetes cluster As A Service (KaaS) offering – a series of clusters that would be ordered on demand by different management chains and in different security groups. Although I think the first one is easy to do and would fit many use cases, the second one is definitely more complex; big vendors and service companies still struggle in updating and maintaining multiple clusters of Kubernetes distributions especially when those distributions have massively different configurations.
When I first went on site, I realised that I was in London and my primary contacts were working remotely. This is quite uncommon for consulting engagements but it was a common theme for the organisation I was working with: distributed teams with minimal travel budgets. I need to pick my fights as to what I can change, so I set course to meet my primary contacts in a central European city that would suit them to organise a series of workshops that would help us agree on ways of working, tools, technologies, architecture etc. Even if I was working on this project remotely for a few weeks this was a major breakthrough to the pace of work. This was a highly effective method of getting to know and trusting each other. Other than time and experience in the field, a few techniques that I used played major role in that too.
Using Open Practice Library practices in a distributed team
At the time, I had recently finished a precursor of the current DevOps Culture and Practice Enablement – DO500 course and I was eager to put what I had learned in practice. I thought that these methods are always effective and able to bring people together talking about the right things.
When I arrived in the mutually agreed location, I was given a list of objectives to help the organisation deploy OpenShift Container Platform (OCP) as a service. We started first discussing why we were trying to achieve what we were trying to achieve and what success would look like using a Start At The End method. This was very useful to give us context and direction as we wanted to make sure that the business would get the most of this. It made us focus on what the end goal is: user (developer) satisfaction by creating seamless integration with current customer systems, ease of testability and and engagement with the product.
We then followed on agreeing a list of things we would continue doing to make sure that collaboration and focus doesn’t wane; we built our foundation:
- We decided to use pair programming techniques whereby having two people delivering a feature and many when learning something new in the platform. When using this in delivering features to the platform we ensured that knowledge is distributed across the team. This also enabled a constant channel of communication being open between distributed team members. Old fashioned video conferencing and screen sharing was sufficient at the time but we later explored
tmuxconfiguration for shared command line access to machines. Anything beyond that was a struggle regarding pair programming tooling as the environment was quite locked down to allow the live share functionaly of VSCode or something similar.
- It was important for us to ensure that everything we did was repeatable so all the changes we wanted to do whether it was a configuration change or a build change or deploying new servers we codified first. We mainly used Ansible playbooks or Jenkins pipelines and followed the everything as code practice. We used git which made our code versionable and when we released a new stable version of the platform we tagged that to indicate that point. We could always revert to a working version. This helped us a few times especially at the beginning when we needed to spin a new cluster very quickly to test new functionality.
- We agreed on a set of rules we’d all abide by: including core hours of work, remote-scrum times, and potentially a sense of humour. We wrote our social contract and signed it and then transcribed it to our wiki. This gave us an understanding of when and how it was best to collaborate even with our different cultural backgrounds and timezones.
I’ve seen a few of these deployments in the past and one of the main success or failure criteria that I have seen is development and business engagement. Therefore, it was important to ensure that developers were engaged as much as possible to use this platform to test and develop.
From the initial set of practices we used to collaborate we found that they worked quite well but needed a few modifications and additions. Below are things that we changed or would have liked to have changed:
- OpenShift and Kubernetes in general is a fast moving platform, while learning about all the new components, integrations and modifications, it was important to educate our users too. We set up time during our days to absorb new material in the community by reading blog posts, following tutorials and adapting some of it for consumption of our users. This is something we then added to our social contract.
- Empathy mapping and user interviews for increasing user engagement was something we were all interested in and it was a key factor in getting the platform moving forward. We wanted to ensure that new users of any container technology would first try and potentially succeed with OpenShift. We came up with a list of teams that were aiming to create cloud-native workloads or could benefit from a modernisation and came up with a list of questions to understand their current frustrations when developing and constraints. This gave us a direct line with our main users after we started enabling features for the platform.
- Using principles such as everything as code is great 80% of the time for everything that is well understood. However, there was a good 20% that the value of automating something needed to be proven by testing a change worked manually first. This 20% gap was later minimised by introducing automated tests as part of building a new cluster that would give us an answer as to if our changes were sane and worked.
- Not all scrum events worked well in this distributed team. Our daily standup ended up in a debugging session more often than not. Although this was useful I feel like we were missing the point in focusing time a bit better. I understand why too; the setting was exactly the same we were on a video call all day to each other. It improved a little bit after one suggestion: to stand up during our phone call. However, I feel like it would have been much easier to have a scrum master to enable us.
- Visualising our workload was something that we had to do using digital tools like wikis and digital kanban boards. However, having physical copies of our social contract and actual boards to write on would have helped massively in actually re-focusing every time we looked around or went for a coffee. Space was something that wouldn’t allow us to do that but I believe that it would bring even better results.
These are the things I would do differently next time.
I would love to push that initial collaboration meeting a few weeks forward. This was the catalyst in actually working better together. It created connections that are so difficult to forge over the phone or video conferencing and a lot of trust.
Product owner involvement was not as high as I expected and delegated to the team. Although this was good as it gave us more power, creating the initial connections to the developers was slow and frustrating. If I were to do this again I would stress even more how important time with the team and the developers would be.
So far, with the practices used above, I’ve seen not only successful deployment and use of OpenShift but a clear shift as to how people talk to each other and want to contribute to this project. Whether it’s a small company, or a global supertanker of an organisation, everyone wants to improve their ways of working. This was keenly felt here. These practices are easy to try but they take discipline and good humour to continue especially in the context of widely distributed teams. I would thoroughly recommend trying them and reporting back on which ones you picked and why.
If you want to learn more about practices used in this blog post visit https://openpracticelibrary.com.
If you are interested in OpenShift and learning more about the technology visit https://docs.openshift.com.
If you are interested in automation around self-created IaaS and OpenShift follow the CASL project. This was used as an upstream OpenShift deployment tool with pre-and post installation hooks and upstream changes were made to ensure the the community would be able to work around the customer’s required changes.
2 thoughts on “OpenShift – From Design and Deploy to Deliver and Transform: Optimising Distributed Teams with Agile Practices”
Attempt6: Public DNS is necessary for OCP 4 IPI. When trying with a private DNS zone (on a bootstrap node that can see the private DNS) the installer fails.