A Productive Pipeline: A Focus on Developer Productivity Engineering

With software projects getting more and more complex, delivering new features into production quickly and safely has become a major challenge for software organizations today. For this reason, making sure developers are able to develop and deploy new features in rapid cycles is a key success factor, but his task has long been neglected. The most successful software organizations are prioritizing this key success factor through a new practice Developer Productivity Engineering. This practice is based on a culture where the entire organization collaborates to maximize developer productivity.

This session answers the questions:

• What are the challenges organizations are facing regarding developer productivity?
• Who is responsible for maximizing developer productivity?
• Which skills are required in developer productivity engineering?
• What tools do teams use to measure and improve their productivity?

Transcript:

Cameron: So our next speaker has taught technical concepts since 2005, and has embraced the open source spirit with Gradle. She has a doctorate in computer science teaches Java and Python using open source platforms, and takes great pride in teaching developers to deliver clean, scalable, readable and maintainable code. Please welcome Dr. Amanda Martin. And Amanda the stage is yours.

Amanda: Thanks. Thanks, Cameron, for the great introduction. So I know a lot of people here haven't heard of Gradle. So my goal with this talk is to inspire you to think a little bit about the pipeline that you're using to build data, and how you can change that pipeline to be the most productive. The way I like to picture Gradle for all users is software engineers, data scientists, they make all of the parts. And then Gradle is like the conveyor belt that moves the parts along so that someone else can like assemble them. So that's the picture to have in mind of like what we do, and where we're starting to optimize things. So my name is Amanda Martin. And like Cameron said, I have a doctorate in computer science.

And I'm the developer advocate for Gradle build tool. And we're completely open source and that's the way I like it. If you want to get social with me, feel free to take a picture of the screen there or my Twitter account is on almost every single slide as we're moving forward. So first, to personalize this, why did I join computer science? Now, this is probably a very familiar feeling to you, when you first were sitting down on a computer, when you were a kid, you did a Hello World, and bam, something changed on the computer. And you had this experience that whatever you did to the computer, you saw that feedback come and you realize that computers are really powerful, whether it was just drawing a circle on the screen, and you saw it move or maybe a ball bouncing. Development on a computer is a very creative process.

And people come to computers, partly for that creativity, and knowing that it can do the next best thing. Now, I like to think of computer science is kind of like a test, you send a test to the computer, and you execute that test and see what the results of the test. It's very similar to kind of have scientists have a hypothesis and they enter a dialogue with that via an experiment. And whether that hypothesis is correct or not, you will get that feedback. Well, hypothesis is our code. And the dialogue is the compiler, the unit testing we have or the performance test or any other interaction we have with it. And when we're interacting with computer, we want to send our test, send our experiment and get that feedback cycle really quickly. And if we don't have a really quick feedback cycle, then we no longer will feel that excitement with computer science. And I see a lot of people suffer with this. And they start to not enjoy science as much and computer science as much. So one of the things that we've embraced is the best code is written by happy developers.

So we should strive to foster developer joy, because software innovation are augmenting global innovation and solving the biggest challenges. One of the reasons why I got into computer science, because I saw that it was you know, defeating mosquitoes that had malaria and solving placements of refugees. And it's just amazing with things computer science can do. And by making everyone more efficient, we can solve even more problems over time. But sadly, software development especially is creating a very unique set of frustrations. So this is one of my typical days, you'll see that the date on this is the 25th. And for this cycle, I generally will have some coding time, I'm liking my coding time, I'm happy with my coding time. And then I send it into that experiment to see what's happening. So I could get a feedback cycle, no breaks. So I have to spend some time fixing that code. And well, that gets me until about lunch when I've given up where I send my test and then I can code some more. This cycle repeats. And I get blocks a lot by how the code works.

And I know I'm not alone in this because, you know, developers are lacking that joy. And whether you're a developer or a data scientist, you're still doing a lot of development. And you're feeling kind of that time crunch, where you're spending a lot of time fixing things rather than doing that creative process. Now, one of the other things I really like is this is a statistic that I pulled and it said that we had almost a million software engineers shortage in 2019. So that's a lot of empty jobs. And if you've been on a hiring team for one of the tech companies, you've probably felt this that, you know, your job position will stay open for like 90 days until you find someone if you find someone. And the other problem with that is we're training a lot of software engineers, yes. However, we're not quite doing it at the rate that we need to do it. When you compare the amount of people that we're training for software engineering, with the amount of people that we need for software engineering, it doesn't quite match up.

And so rather than training more and more people, we should make our people more efficient, as efficient as we possibly can. Now in the right on this side, is the turnover rates and technology software, it's actually a really high turnover rate, because people go into their job and they see this as their calendar for a given day. And they don't like it as much they don't have that joy and that excitement that they had when they were a kid. So this is something that we need to fix. And this is just a use case about how we started addressing this problem. Okay. So some of the problems faced by many engineering organization organizations today is the collaboration between the engineers is not as efficient as it should be, especially when they're starting to solve a problem. And then like I said, before, there's these really slow feedback cycles, you send something in and you want to see if it worked.

And you don't get that instantaneous response. Sometimes it takes a day for your code to build, sometimes it takes 15 minutes for your code to build, even if it takes 15 minutes to send that result and get feedback that's too high. Because at that 15 minute mark, you know that the person is going to context switch, and, you know, go check the slack, or the discourse or whatever thing to see if there's something they have to do. There's also this lack of knowledge about how software is built in the entire organizations. A lot of the software that we're working with is these huge structures, these monolithic repos that kind of interface together or there's microservices that we need to know how to integrate. And having the knowledge of how all of that interacts is kind of too much for somebody to have the cognitive load is too high. Some other things is the high CI infrastructure cost and the unreliable local development. If you send a test, sometimes that test fails, send the same test, sometimes it pass, that's the take a flat flaky test. And we need to make sure that the tests aren't flaky. So ultimately, there is a competitive disadvantage, for shipping features slower than needed, and lower quality than expected. And this reduces that satisfaction. And it also reduces that motivation.

So one of the ideas that that people have had is this new idea called developer productivity engineering. And this is considered the next big thing after DevOps. So this is kind of the timeline for some of the big process changes that we've had in technology, ability to code, we had JIT manufacturing, a lot of you further on, and more recently, remember learning about agile practices, which are still one of the most use today. And then of course, we had DevOps. Well, the next big thing is developer productivity engineering. And it's not quite DevOps, because this is focusing on kind of how do you use technology to solve the developers frustrations. So it's a subtle difference. But I think by essentially having productivity engineers inside your cycle, you can start looking at how to fix the problem and make all the developers happy. It takes time though.

So this is an approach to productivity. So how do you make it so that when you send a test in you get the results back fast? It uses acceleration and analytic technologies to improve the developer experience. And so how can we make it so the developers' job is happier over time. Now, a lot of times when people want to hire a DP engineer on their team, feel like oh, man, it's just yet another engineer, I need to hire another engineer that I have to, you know, get on boarded and everything else. But when you have that DPE on your staff, then you have a faster TTM (Time to Market). You have reduced costs even though you have an extra salary or more and you also get an improved quality and less turnover rates for other ones. So one of the surveys that I like, is this was done by TechValidate, and DPE fosters this sense of developer joy. So 81% of surveyed IT professionals who knew DPE is agree that DPE impact on their tool chain makes their job more enjoyable.

Okay, so you know that it's a good thing, you know why you should have DPE? But now the big question is, how can you implement DPE. And that's a big sticking point for a lot of people, you know that you want your people to be more productive, you want them to be more happy. You want them to have faster feedback cycles, and spend more time doing the creative coding that you know they love. But how do you actually do it? Well, that's where you need to think about, what problem are you trying to solve. And so for example, as a software engineer, we spend a lot of times writing software. And so the problem that we're trying to solve is how do we feed back the software cycle for a developer, and there's a few other pain points I'll address later, too. But the problem we're having is somebody is going to be sitting down and doing their code, they send their test or they compile their code, and it takes too long. Since it's been a few minutes, when they get the results back, they then have to go and find out where it's broken.

If that code takes 15 minutes to run, they're not going to want to run it very much. And so they're going to do a lot of changes on their code, so that they have more features when they implement it. So if something breaks, and it comes back, they have a lot more to look at, to see where the error was. And so when you take this one developer, and you know that one developer works for 240 days a year, then the time starts to really add up, and the productivity really starts to go down. But then for some companies, especially those who have already implemented DPE, we're talking 100 developers. And so that can actually result in millions of dollars wasted, as you're having developers kind of wait for their test to build or track down bugs that are a little bit too big that they would have found before if they could, you know, test their code more frequently. So here are some of the pain points that we noticed for software engineering. One of the pain points is there's a lot of idle or wait time, when we're sending our code to compile or a test, then if it's about 10 minutes or so the developer will just actually wait, you know, it's only eight minutes, I'll just drink my coffee here and wait for my code to compile. It's, you know, a meme at this point that you're waiting for your code to compile as an excuse to just, you know, goof off and play lightsabers or something. But in addition to that, there's the inefficient troubleshooting.

When you send so much code to your compiler, and it comes back as a breaking, you don't necessarily know why it broke, if you sent in 200 lines of code, it could be any one of those 200 lines. Another problem that people complain about is having really flaky testing. And so that means you send it and it fails, you look for the bug, you look for the bug can't find the bug. So you give up, you send it again, and it passes, even though you have no real change. And then two of the other pain points is that there were no metrics or KPIs for observability. And there is inefficient use of CI resources. So let's look at these are the pain points that we addressed. And now so that we can make our software engineers more productive and more happy, we talk about how we should fix these pain points. So the idle/wait time, a good way to fix that is to have a faster feedback cycle. So how do we make it so they get the results from their compilation faster? Well, one solution is to cache your build or have distributed testing, ideally, both.

And so if a lot of the build is cached, because you haven't worked on this method, or maybe this repo you haven't touched, then you don't have to compile those, you could just send the part that you actually change because everything else is already packaged up. And then the same thing with test distribution. If you're only wanting run running one little test, you don't need to run every single test, a lot of the time testing takes most of the time. And so it's really nice if you can kind of pick and choose which tests you need to run. But also, if you need to run more than one, you can split them up and distribute them. So you could run them simultaneously.

So it's really important that we have fast feedback cycles, because then we can reduce that idle/wait time. Now you can map out the effects of kind of solving this pain point. If they have a faster feedback cycle, then they're going to have you look at the top graph, they're going to have less idle/wait time. And that means you're going to have less context switching, you know where they're working on one project. And if it takes longer than that 10,15 minute mark, then they're going to jump to a different project while the other one compiles. It takes a few minutes to actually realize what you're doing and where you are at that new project. So if they work on that for 30 minutes, and then they compile and need to go back. That's more context switching. And every time you context switch, you have wasted time. So if you have less context switching, you're going to have more focused developers. And that's going to lead to higher quality and higher productivity.

Another consequence of this faster feedback cycle, if you look at the bottom chain is if they know it's only going to take them, you know, two minutes to see the results, then they're going to build more often because they want to see that joy, that dopamine hit of who my Hello World is working. So they're going to make a change, and they're going to see what actually happened. So we're going to have a lot smaller change sets. So if there is an error in the code, or if there's a merge conflict or something, it's a lot easier to solve. And this, of course, will mean that you get a lot faster into the production quality code that you need. And again, higher quality and higher productivity. So let's look at kind of some of the numbers for this type of analysis. We have two teams. And the first team has 11 devs and their total build time, that feedback cycle is about four minutes. And so that's not enough to really go to another project. So it's just four minutes of them kind of sitting and watching their builds. Team Two has six devs but it only takes their build one minute.

And so the number of times that they've built is 1010. Now, these are actual data numbers for a software team. And so you might be asking yourself, why is it that the team that's almost half the size has a lot more builds? Well, that's because they know the build is less, and so they could actually send it more often. And see that feedback more. This is something that might not seem impactful for you know, this team one, Team Two, but as you're experiencing it, you're going to notice that and if you're starting to multiply your team's out, then you're going to really see that production increase. So the build cache delivers a fast build and test feedback cycles. So that's one of the things that we've developed. And you could actually use and implement. But the other thing is, test distribution will make tests even faster, because they're distributed outward. Now, there's another consequence to having distributed test.

Now, remember that second pain point was that inefficient troubleshooting. And we want them so that people can test faster and troubleshoot faster. And so the thing that we did for this is, well, if we have distributed tests, and we have the ability to test again, because it's faster, then we can start to address the issue of the flaky test as well. There's a quote, I don't know where this is from, but I've heard it before. And it's flaky tests, the pit of infinite sorrow. And we've all been there where those flaky test, we don't know if the test is because it's an error in our code or an error in our test. But regardless, we hate them. And so we know that there is this quote by Marc Benioff speed is the new currency of business, we want to make sure that the tests happen quickly, fail fast and fail often is another thing to say.

And so when we have these test distributions, that means we can split the test out, we can execute multiple tests in parallel. And so that means if a test generally takes four minutes, now we can split out with four. And we could do one minute total. And so you can distribute your tests across horizontally. Now, this is really useful, because if you have to scale it a lot, then you can just always expand horizontally. Now, the great thing here is if you know, a test might be flaky, or if a test fails, then just run it again. Because you don't really have to wait for that one test to run. So if a failed test, you can say, I want to run all failed tests three times and see if it actually is a failed test. So at scale, when you have this happening a lot this is the difference between an unreliable and problematic build environment, and a reliable and trustworthy one. Because now once you send all of your tests, if you get a fail, you know that it's actually a fail.

So that's kind of how we address the flaky tests as well. So this is three pain points that we really solved with just two things, essentially, by taking how we build software and only building the part we need, and also distributing our tests so that we can essentially run them simultaneously. Now, there's another issue that has come up a bit and there's a really great book out about observability. And the question they pose is, what are you observing? Yogi Berra says you can observe a lot just by watching. And so my question to you is, what are you watching? Is there someone in your team that is watching how long it takes for your data scientist or your software engineers to get that feedback cycle.

So who is watching the build times? Do you know which tests can be run in parallel? Do you know which tests are giving your people that constant problem being flaky and not knowing why they're flaky? Who isn't in charge of this? Someone looking, oftentimes, if it's one of those things, that everyone is in charge of productivity, then nobody's going to be in charge of productivity. And it's really important so that your project can scale and scale well, that we have somebody that's designated to look at these problems. Another great quote is how does a project get to be done a year late, one day at a time, it's really easy to fall behind on a project just every little day, you falling a little bit more behind. So this is kind of how we mapped our pain points to the benefit, and then what we do in practice, to kind of solve that pain point. And this is something I think every single person and company can do, think of the pain points that are making your data scientist and software engineers not as effective as they could be.

And it's important to really go through and start to address how you can actually solve them. One point of caution, though, is this isn't something that you just solve, and are done with. Because things change over time, if you think you have a completely good build cache, and then something changes like how you cache or what you need to cache, those infrastructure changes will lead to those idle wait times sneaking back up again. And so it's not something you could do just one time, but somebody has to continuously watch it, you know, perhaps if you have a new annotation processors and new versions, then things kind of trickle back into the system. And so it's really important that you watch this, and somebody watches it again, and again and again. So they say that DPE will become standard practice, because the world should foster developer joy. Now, I don't know if you've heard of DPE, it's one of the things that, you know, I've learned about and I absolutely love. But there's actually a lot of companies that are starting to use DPE.

And so these are from recent job postings, at some of the big companies, or blog posting, either or so the one on the top left is Spotify, how we improve developer productivity, and our DevOps teams. And so that's the ability of using DPE, inside a DevOps team to kind of make their developers really happy again. We have the one in the middle, is Netflix, that's a job posting, they're looking for a senior full stack Software Engineer of developer productivity. So they actually have a title to watch that. And that's the same thing on the right is Twitter, Senior Software Engineer, developer productivity, LinkedIn on the bottom left productivity in IT, and the last thing elastic, cloud tech, lead engineering productivity.

And my own company itself also wants a back end engineer- engineering productivity. So there's a big huge market for knowing how to do this well, and then taking it to the next step no matter what company you're at. And if you need the book for this, here's kind of a link for you to scan for the book. It's a really good high level read. There's no code, and it just makes you think about what are you watching? And how can you improve it so that your own data scientists and software engineers can be more productive? Okay, so that is all I have. And I will leave the time for questions. I'm right at time. So..

Cameron: Wow, super Amanda. Thanks. That was really interesting. Hey, questions are already coming in here. Firstly, let's talk let's talk timeline. Just for a minute. So company they're looking at

Amanda: You are cutting out..

Cameron: What timeline, are we looking like you know, for key stakeholders get engaged and provide developer access training? What sort of timeline are we looking at?

Amanda: Um, I would say it really depends on the company and how much resources that you can devote to this because if you are a really big company with kind of a lot of inertia, it can take years to do some changing. If you are a smaller company and really hungry to see that improvement, then you know you can start looking at different practices now and seeing what you can change in your code base to make those changes right away. Some little changes like you know, maybe you have back to back meetings all day and 15 minute breaks. I mean, little things like that you can change like right away without actually doing any technology tool, just how you design meeting times.

Cameron: Okay, cool. Are there any like programs or third party vendors that you'd recommend to supplement skill development? Well,

Amanda: of course, there's a lot of things at Gradle. If people want to go and learn more about DPE, at Gradle, the book is really good. We are a big advocate of DPE. We have a whole DPE summit November planning, just because we're seeing it be beneficial for a lot of people like LinkedIn, elastic, and whatnot. And since they've benefited from it, we're trying to, to make it well known so that a lot of smaller companies that really don't have the extra resources can also benefit from it. So check out the book, it's really worth a read. And if there's anything else, you want to know more about it, there's my Twitter, and I can send you some more resources.

Cameron: Super. I got another question that came in here. Who do you recommend should be actually in charge of tests like the flaky tests that you talked about earlier? Is that a person? Is that a taskforce? Who's incharge?

Amanda: Yeah, so testing at our company is an entire task force. And, you know, we have about 50 engineers, and about five of them are in charge of like testing. And they're the ones that kind of look at how you manage test and do test and what makes a test flaky. And how do you fix those flaky tests. And one of the best ways of doing it is just, you know, trying to run it again, and seeing if something actually changes. But again, depending on the size of the company, if there's a company of just like five, then sadly, doing testing might be just one person's thing.

There's also something I really recommend, if it is a small company is to look at test driven development. And that's something that really changed the way I like to code because you start with a test, and then you actually go and make the code after the test. And that made my test a lot more reliable. And it also made, so I had a lot better coverage in my code. And I also commented my code a lot better. And it was more readable for other people when I started that TDD development. So depending on the size of the company, start with the individual people. And then as you build more tests, you're going to expand it.

Cameron: Super. So it seems like we've talked a lot about like, you know, people and processes and everything, but what about like tools and technology and things I can go by and install on my machine?

Amanda: Well, the Gradle is open source, and it's for most of the things on the JVM. And so if you're already using a JVM based thing, then you just on a Mac, you do homebrew install Gradle. And then you set up your project that way and kind of go from there. So for JVM ecosystems, it's really convenient. Now, if you want to do something like this with data science, then I don't know of any tool that has already been built yet. It's a really new field. All the careers page is people that are normally in the JVM that you know, the Java Kotlin world. And so it's really, that's a huge ecosystem, and why it's taken off so much. But some of the other ecosystems, they're really looking for kind of like the people to start it and start being the evangelist for making sure software engineers and data scientists are productive.

Cameron: You mentioned Spotify as a use case for the improving their developer productivity, did they was there training that these companies that they completed?

Amanda: Yeah, we have trainings on our website. And I know that at least one person from Spotify has done one of those trainings. And there, I've done some of the trainings, I've taken the trainings, because I think they're really good. But you essentially go to a live training much like this one, and you ask questions, and kind of learn different techniques and whatnot. And some of them are deep dives to get into like the technology behind it. And if you go to some talks at summits and stuff, it's more high level like this one.

Cameron: Super. One last question, what's your definition of a small versus big company?

Amanda: I would say probably about 10 devs, is my definition of that breaking point between small and big, cuz after about 10, devs, you need a lot of processes to how they work together. Maybe something as small as like five devs, depending on, you know, what the devs are doing and how much other things they have to do, because sometimes devs aren't just Devs, right?

Cameron: And can you handle that maybe breaking it up into teams and things like that segmenting the product? Okay. And then where do things just get completely off the chain? Like what is what is too many developers?

Amanda: Too many developers for one project? Oh, yeah, probably more than five you definitely have to break it out. And you know, there's the management cycle of more than I think seven is what? Someone told me once so..

Cameron: Super. Well Excellent thank you so much. It was super interesting we're out of time for today but I really appreciate you coming in and sharing your insights

Amanda: All right. Thank you for your time. Have a good one.