Debugging Titles: Part II

In my last article, we discussed a problem with titles: they poorly describe the many ways an engineer can develop their career. In response, we developed a system we call Masteries comprised of six attributes that, when combined, create a more well-rounded perspective on an engineer’s capabilities and growth path. In this second article, we’ll explore how we designed the levels and attributes, including the “minimum bar” we expect for engineers at Riot. We hope the results deliver compelling and aspirational paths in each mastery.

For some background in this conversation, consider two major challenges we encountered when constructing the leveling system. The first was determining what the levels measured in exact terms - our efforts to provide better career clarity would fail if a mastery is just as ambiguous as a title. The second was delivering a system that, in short, doesn’t feel bad. It was essential to avoid creating unhealthy comparisons, labeling, silos, and other unhealthy “management” behaviors. We felt strongly that this system could be effective, but we also had to combat the baggage that many of us (myself included) had from other, similar systems that were poorly used at companies we’d previously worked at. I’ll address both of these challenges by telling the story of the leveling system’s evolution, while talking about where we went wrong, and where we landed.

Like most ideas, we started by brainstorming and throwing some ideas on paper. We reached out to people across engineering who were known for being subject matter experts and gathered feedback on the attributes we thought defined our careers as engineers. As a reminder, those attributes are:

• Programming

• Subject matter expertise

• People leadership

• Craft leadership

• Product sense

• Delivery methods

Because of our affinity for games, and all of us being passionate League of Legends players, we playfully modeled the original version of the levels after ranked levels. We started at “Unranked” and included six levels, from Bronze to Diamond. It’s worth a brief sidebar here about the Unranked level. As I mentioned in the introduction, one of the goals of the levels was to define the minimum bar for being a Riot engineer. We knew that aspiring engineers might want to understand the difference between being at that bar and being below it, so we made sure to invest effort here. This has yielded significant benefits and clarity for us when we’re interviewing entry-level candidates or when transferring Rioters from other disciplines into Engineering.

Having the levels labeled, we then threw together a Google doc and shared it with everyone interested. We started adding bullet points to each level of the attributes, which created a giant 6x6 grid. We iterated until we thought we had a solid rough draft, and then shared the document with all of Engineering using our RFC process. We quickly learned through RFC feedback that we’d made some mistakes in the original design. I’ll list them here, and then touch on how we approached each one in turn:

1. The use of Bronze through Diamond as labels felt awful to most engineers

2. The intended distribution across levels wasn’t clear

3. What the levels were measuring wasn’t consistent

4. The bullets drove people to “check boxes” rather than being more subjective

5. The time horizon for measuring yourself against them wasn’t defined

6. The intended use of the whole system still wasn’t clear

Starting at the top: the most fascinating lesson-learned I personally took away from this project was in labeling the levels. While I love playing League, let’s just say I’m not turning pro anytime soon - I hang out mostly in Bronze knowing that the echelons of Platinum and Diamond are beyond my reach. As a result, I had little appreciation for the distress caused by thinking of your programming or leadership skills as “Bronze” or “Silver.” In fact, after several other attempts, including labeling them 0 - 6 and other such discrete hierarchies, we stumbled on the major issue: the comparative language. If you look at the six attributes carefully, you should arrive at the conclusion that no single person could possibly master all six. You simply can’t be the best people leader and programmer and subject matter expert, and so on. That meant you had to make choices about where to invest your personal growth time. If I decide to focus on leveling up my people leadership, to some extent I’m doing so at the expense of growing in technical leadership - and of course that’s totally fine. We needed labels that didn’t make people feel bad about those choices. We landed on a belt system that colored each level, with White being the lowest, then Yellow, Orange, Red, Brown, and Black. This resonated with most people and eliminated the concern about being “Bronze” or a “Zero.”

The second issue dealt with the expected distribution of Rioters across the masteries. Was a Black belt intended to be fairly easy to achieve, or was it highly aspirational? We wanted the levels to stand the test of time. If we had to redefine them every year, we’d lose the ability to track progression or measure anything meaningful - not to mention the amount of work that would go into such an effort. We also wanted to allow our engineers to aspire to an extremely high bar. Therefore, we decided that the Black belt should be highly aspirational, and we intentionally set each level such that the distribution among the belts would ideally be a bell curve (more on that in our next article). We essentially wanted most of our engineers to fall into the Orange/Red belt with few, if any (currently) Black belts.

The third and fourth issues were related, and they dealt with how we defined the contents of each level. As I mentioned, we originally put a list of bullets together based on the collective experience of our subject matter experts. But, some bullets talked about things you needed to know, while some talked about things you needed to be able to do. That created a feeling of inconsistency across the different attributes. What we really needed was to align on what we were measuring. At Riot, player value is paramount - everything we do is about generating value for players. Adding value as engineers on product teams needed to be the focal point of what we were measuring. Skills and knowledge are important, but much less so in a vacuum - they need to be applied to the efforts of a team. Ultimately, it’s the value an engineer adds that creates the opportunity for growth, taking on new challenges, and being entrusted with larger responsibilities. With this in mind, we had a new problem: how do we actually measure that value?

We consulted Riot’s research team and decided the best approach would be interviewing a large sample of our Engineering discipline and asking a series of questions to help determine what value looked like at a low, medium, and high benchmark. We selected 65 engineers across junior, mid, and senior ranks of each attribute and spent several weeks conducting hours and hours of interviews. We then analyzed the results, looked for themes in each attribute, and wrote a descriptive paragraph that we call the tl;dr. Here’s what they look like: