How do you measure the success of a Developer Advocate, someone responsible for helping grow the maturity and understanding of a technical community using a product? It is something I have periodically explored both at Google and as Chief Architect back at Magento. What is the best way to educate a developer community?

First, you need to be careful to measure the right metrics. For example, “there are two types of programming languages: ones people complain about and ones people don’t use.” Negative sentiment is not always the best metric to measure success. Similarly, a metric such as “number of questions in forum” might not give a good indication of how effective you are. Are people confused, or are you just popular? A better metric might be the number of unresolved/unanswered questions. Have you created a community where members help each other?

One approach I like to think about is the maturity level of a developer community. To help measure this, I find it useful to write up statements that are indicative of levels of maturity. For example:

• Developers know my product’s name
• Developers understand the products benefits and can place it relative to other products
• Developers are talking about my product (blogs, conference talks, etc)
• Developers are talking about it positively

But wait! That last one! Is a Developer Advocate’s responsible for how good the product is? Isn’t that the responsibility of the product team? Is it fair to measure someone by things outside the control of a Developer Advocate? Why should I be responsible for something outside my control? To me, one of the responsibilities of a Developer Advocate is bringing complex community feedback back to the internal product team and helping *them* understand it, and ultimately improve upon it. Your job is not just external communication, its driving internal change based on lessons learnt from the community.

Another possible way to measure the maturity of a developer community is to look at support tickets / forums / stack-exchange. Here is a possible different set of maturity levels:

• There are no questions on forums/stack-exchange
• There lots of newbie questions not being answered
• There newbie questions are being answered by other community members
• There are advanced questions
• There are advanced questions and they are getting resolved by community

No questions means nobody cares. Newbie questions means you have people having a look. Newbie questions being answered by the community means you are building an engaged community. Advanced questions means people are tackling a wider range of problems. The community resolving advanced questions means your community is able to solve the harder problems – they are engaged and knowledgable about your product.

Once you understand the level of maturity of your community, you can then use that information to inform what actions you should to focus on. Do you need to focus on product benefits and positioning? Do you need to create more introductory tutorials? Do you need to engage community experts to help others and share more? Do you need to hold expert deep dives for product experts? Evaluating the maturity of your community can help inform your strategy and goals for the year. If you have an engaged enough audience, you can start relying on them to help the newcomers more and so focus more on helping your community experts.

Measuring the success of Developer Advocacy is still hard. There are not as clean metrics as latency at the 95th percentile of an API. But it is wrong to think there are no metrics that can be measured. I find maturity levels a useful framework for thinking about what you are trying to achieve, then how to measure your progress towards those goals, and then finally to feed that into action plans to help the community grow to the next level of maturity you define.