In my last two articles, I introduced Masteries - our framework for clarifying career progression within Riot Engineering. I covered the design of the system and how we arrived at version 1.0. In this article, I’ll discuss the rollout, lessons learned, and our plans for the future of Masteries.

Rollout Goals

Before pulling the trigger on Masteries, we defined three goals for ourselves. I’ll dig into each:

• Validate the distribution of mastery levels across Engineering

• Validate the distribution of mastery levels within specific roles

• Gather sentiment from our engineers on the value of the system

Firstly, if you recall, a fundamental goal of each Mastery was an appropriate “leveling.” We wanted a normal distribution of levels across the organization. We set an extremely high bar for Brown and Black belts, so we expected most of Engineering to fall into the Orange and Red range. The real data would inform us whether or not the definition of each Mastery represented our total collective skills.

Secondly, we wanted to examine Mastery levels within each unique Engineering role. While a competency system is highly useful as a tool to identify career growth opportunities, it’s the application of those competencies in the context of a specific role that determines the value created. As a meritocracy, we care deeply about creating value for teams and, most importantly, players. Consequently, we wanted to know how Masteries mapped to what we thought was ideal for Tech Leads, Engineering Leads, and Engineers.

Lastly, we were concerned that people would misinterpret the intent of the framework. I’ve mentioned in previous articles that many of us have worked at companies that used similar systems in ways that are contradictory to Riot’s value of being “data informed” and not “data driven.” There’s significant nuance in understanding someone’s skills, performance, readiness for promotion, and ability to take on more scope and larger challenges. We will never live in a world where Masteries tells us what to do - instead, it’s a tool for engineers to understand how to increase their impact and create more player and Rioter value. And if it’s a tool for engineers, we wanted to know if they found it useful by gathering sentiment towards the system from across the discipline.

Rollout

By the time we arrived at this point, we had been discussing and aligning on the Masteries system for almost two years, starting with the original RFC and then following with subsequent conversations, pilot groups, feedback, and iteration. Our plan was for each engineer to complete a Masteries profile using the document that I referenced in my previous article. At the same time, managers would go through the same exercise on behalf of the engineers they managed. Then, managers and engineers would sit down to discuss any differences and align on the next steps for growth. This conversation was the most important part of the whole system, with Masteries providing a common vocabulary and a visible artifact. Once these conversations were complete across the organization, we would send out a survey to all participants to gather their feedback.

Because of the crucial importance of framing the system and communicating its intent, we developed a training program for all of our Engineering Leads and ran them through it in small groups. The training focused on what the system would and would not be used for, and how managers could facilitate positive discussions with the engineers on their teams.

When the training was complete, we messaged all of Engineering to kick off the rollout process. Engineers had several weeks to complete their profiles, submit their data, have discussions with their managers, and provide feedback. We had a total of 351 engineers participate, including all managers and leadership within the discipline. This last point was important - we want Masteries to be used by everyone in Engineering.

Results Analysis

The data show that the distribution matched our expectations, which we attribute to the interviews we conducted with engineers and the huge amount of feedback we received from all over Riot to help align on the content of each level before release:

As expected, the discipline’s hard engineering skills (programming and SME) are more developed than everything else. We also have a non-0 percentage of engineers in the white belt category for each attribute. While we consider Yellow “the bar” for being an engineer, we also recognize that there are some Rioters in Engineering who index highly on other skill areas – like Subject Matter Expertise – and may still be developing their other competencies.

So what about titles? We’d expect that associates will, on average, be around Yellow and Orange belts, mids somewhere between Yellow and Red belts, and seniors to be pushing Red and higher belts. Here’s the spider graph breakdown by title:

The distribution across titles flows as we expected. As engineers make their way from associate to mid-level to senior, their impact grows across all dimensions. A similar breakdown can be seen when considering roles. Engineers who move into an Engineering Lead role tend to specialize in People Leadership. Likewise, Engineers who move into a Tech Lead role tend to focus on Programming, Subject Matter Expertise, and Craft Leadership. Given that Architects and Senior Engineers typically fill the tech lead role and Engineering Managers typically fill the Engineering Lead role, it’s not too surprising that this graph looks suspiciously similar to the Architect and Management graph.

 

Engineers can also fill more than one role. The graph below depicts Mastery levels for engineers who are dual-classing - unsurprisingly, their impact increases across multiple attributes. While leading people may be the primary focus, engineering leads who also fill the tech lead role are also getting their hands dirty in terms of Programming, SME, and Craft Leadership.

 

Feedback & Lessons Learned

Overall the results of the qualitative feedback were positive (3.2-3.4 is slight agreement, 3.4-4 is agreement, >4 is strong agreement):

The system is generally well-regarded, although there’s certainly work we can do to improve. We aspire to results that are greater than four, and we’ll continue iterating until we get there.

I’d like to highlight several key takeaways we gleaned from the survey and feedback:

• The highest value engineers gained from the system was discussions with their managers on their current profile and opportunities for growth.

• We had some work to do in clarifying the connection between Masteries and player value.

• Engineers felt that the Programming mastery was less about demonstrable value and more about tacit knowledge.

• Engineers felt that the jump from Red to Brown was significantly more difficult than between other levels.

While that’s not the complete set of feedback, it underscores the most important points that we have been incorporating as changes to the system. As I write this, we're preparing for our next profile update. We plan to run this process every six months so we can track changes over time, and ensure engineers have a couple points throughout the year to check in on career growth.

The Future of Masteries

With all the feedback and data we received, plus continuing conversations about how we evolve the system over time, we’re in the process of implementing a number of key changes. The first of these is a mechanism for ownership and ongoing updates to the system. We want Masteries to be “by the engineers, for the engineers.” Rather than information coming from an ivory tower, the system will be maintained by the engineers who are closest to players. We’ve identified individual engineers who will be owners for each of the Masteries, and they will be responsible for analyzing data after each profile refresh, incorporating feedback, and then recommending changes once per year. They will keep the system fresh and relevant.

The second major evolution of the system is to further refine our definition of the Subject Matter Expertise attribute. After receiving feedback that the SME attribute was broad and difficult to apply to the various sub-disciplines within engineering, we identified the need for more specificity. While an attribute like Programming is core to every engineer’s role at Riot, Subject Matter Expertise can look drastically different in different environments. We have specific focal areas within the Engineering discipline outside of purely software engineering like infrastructure, security, and data that needed clear definitions of impact. In light of this, we’re developing “overlays” for engineers in the infrastructure, security, and data-focused roles to clarify the means of achieving expertise within those domains while still leveraging the larger system.

Lastly, we’re hitting our stride when it comes to using Masteries as a common nomenclature in all areas of talent development within Engineering at Riot. We talk about expected Masteries when considering a new hire or the promotion of a current engineer. Although Mastery progression is not the deciding factor in either of those areas, it is a data point we consider, and those conversations are important in driving alignment across our 500+ Engineering organization.

While we certainly still have work to do, we’re excited about the gains made to career progression clarity at Riot. We now introduce Rioters to the system as part of our onboarding process, and managers continue the conversation as they help engineers build development plans around these codified areas of value and impact. I hope you’ve found this series useful and that it provided some insight into how we’re approaching this particular problem for Riot Engineering. We’d love any questions and feedback.

---

For more information, check out the rest of this series:

Part I: Titles 
Part II: Levels and Attributes
Part III: The Future of Masteries (this article)