If you're a brick layer, it's very easy to determine who the best brick layer is. Count the bricks. Some smart, clever brick layer may come up with a more effective way to lay bricks. That’s wonderful and they may become a lead brick layer and be deemed a "leader".
If we want to improve things, we might decide that it's a good idea to have someone coach a team of brick layers. We might say that a coach should work with 11 brick layers to find ways to make them all more effective. Determining the most effective coach is pretty easy. Count the total number of bricks laid by each team. You'll find that some coaches are more effective than others.
Experience and brick laying skills are probably important to the coach. If the coach has previously worked as a brick layer, it's reasonable to assume they'll be more empathetic and effective. Good coaches will also have different ways of organizing their teams, different methods of laying bricks, different attitudes towards laying bricks, etc. Clever coaches will use different techniques to maximize their teams. If you’re laying bricks in the mountains you might want different types of people than if you’re laying bricks in a dessert. Still, it’s pretty easy to measure a coach’s effectiveness.
Now, let's say that we want to create a larger organization. We might have a division. A division consists to 10 teams of brick layers. In order to resolve disputes and make things more organized, we might say that we have a division head. This is necessary because everyone is trying to improve so there are competitions between our brick laying teams. [Editor: teams from the Big-10 seem particularly skilled at laying bricks…]
Now, the skills necessary to be an effective division head are not necessarily the same skills necessary to be a good brick layer or even a good coach. Also, complicating the issue, how do you measure who is an effective division head? Looking at the total number of brick laid might be effective, but they are pretty far removed from the brick laying process and the things they handle on a day-to-day basis are probably not similar to the brick layers or the coaches.
If we elevate this one more level and create a league, the problem becomes even more obvious. The league is composed of 10 divisions. Some of these divisions are in the mountains, others are in the dessert and others have to deal with great lakes and snow storms. The types of bricks that each division can get are different just as laying bricks at 2,000 feet above sea level presents different challenges from laying bricks in the dessert, which is completely different from laying bricks in the middle of a Wisconsin winter.
How do we determine if our league commissioner is effective? A brick layer or a coach may have innovative ideas and it's easy to asses their effectiveness. But determining the effectiveness of a division or league commissioner is more problematic.
How would you approach this?