Designing Developer Surveys: The Final Problem

Analysis, Visualisation and Communication

Sherlock Holmes, the greatest detective of all time, once said

"There is nothing more deceptive than an obvious fact."

As Product Managers, this statement is particularly relevant for how we use and interpret data, especially if we do it on our own with no support from Data Scientists or Product Analysts.

I titled this last article in the Designing Developer Survey series, The Final Problem, because this last piece is as important as designing the survey. I was also low-key excited to quote the greatest detective of all time at one point in my Substack journey, so there's that too! 

In this final article, I will aim to give a high-level overview of what to do once we have collected all the survey data.

I will caveat this and say that we should collaborate with a data scientist or a product analyst for this type of project. If that isn't an option, there are steps that you, as a Product Manager, can take:

Analysing Data and Interpreting Data:

Once you have collected a sufficient number of responses you are happy with (I mentioned previously in my posts that you should aim 60-75% response rate), it's time to review all the data and decide on your approach. It would be best to do this with the Product team or, at minimum, involve your Tech Lead and Engineering Manager. Steps to consider are:

Data Cleaning: Make sure you review all the responses and remove any duplications, errors, missing responses, or outliers. This could be a PM-only exercise, as you will bring the data in great shape, making it much easier to collaborate with the team and/or Engineering counterparts.

Quantitative analysis: For questions with rating scales or multiple-choice options, perform quantitative analysis. This is much easier when you are using a tool like Typeform, SurveyMonkey, or Google Forms, as they usually take care of this for you, and you also get decent visualisation with these tools. (Long live the pie chart!). 

Suppose your tool doesn't have this functionality out of the box. In that case, look for patterns, trends, and correlations that provide insights into the developer experience or, more generally, into their sentiments, likes, and dislikes.

If you are looking for a good read-up on how to do linear regression and correlation analysis, I found this from

Lenny Rachitsky

to be quite valuable.

Qualitative analysis: This is the most fun part for me. If you have open-text questions, this is the part where you have to read through the responses and summarise common themes and/or topics. It might make it easier for you to break down into things like CI/CD, Observability, Documentation, Infrastructure, etc.; you can even go more granular by having sub-categories into each of these main sections, whatever makes it easier for you. You can do this as a fun exercise with your team and MIRO (or Mural) whiteboard.

Segmentation: I suggested in part two of this article that it might be helpful to start the Developer Surveys by asking demographic-focused questions. If you have already collected demographic information, consider analysing how different responses vary across segments. You might have to drill down into the quantitative data to get a more comprehensive picture, but it's worthwhile.

It will allow you and the team to identify any significant differences in feedback based on job titles, experience levels, programming languages used, etc.

Capture Key Findings: If you are going through all the steps above, you should understand your key findings well. As much as we all love good feedback, do focus more on looking for pain points and areas for improvement. Then compare these against what you are doing well and the positive highlights.

Storytelling

This is the part where you get to practice what you preach. The worst thing you can do if you start on this journey is not to do anything with the information you have. Ensure you dedicate enough time to this and draft a high-level comms plan to help you tell your story. Some steps to consider are:

Share a one-pager summary: Prepare a summary highlighting the key findings, trends, and insights from the survey. Use a lot of visualisation for this, abstract technical jargon (remember your audience isn't centred around Engineering), and include recommendations and next steps in closing.

Communicate and, better yet, over-communicate: Share the one-pager with all the key stakeholders involved from the beginning and with any decision makers you might need to get alignment from.

Then do a company-wide short talk/presentation. You can use a forum like a company all hands, lunch and learn, guilds meetings, etc.

Engage in conversations to ensure the results are well received, understood, and considered in the broader product strategy.

Roadmapping areas for improvement: If you decide to take any actions based on the results, prioritise and refine them alongside the other work together with the team and other engineers, where needed. This should also be included as part of your one-pager summary.

Close the feedback loop: Share the summary with all Product and Engineering and be explicit about what actions you and the team are taking as a result of the survey. I would also include a sample of the data analysis here as some Engineers will be curious to do a deep dive into the results. And remember to keep everything anonymous if you are going down this route!

In closing the feedback loop, you demonstrate that their feedback is valued, and it informs parts of your product strategy. It also helps to build trust and engagement with your Engineering community, showing them you are a champion for an excellent internal Developer Experience.

This article concludes my three-part Designing Developer Surveys series; however, this is a constant work in progress, so some things I wrote now won't be as relevant in a few years, in which case I will gladly come back with Version 2.

See ya next Friday!