(Sept 9) U18 Evaluation Process Information

If you did not get a chance to check out the SPMHA Evaluation Town Hall meeting held on Wednesday, Sept 7, please feel free to check out the posted video on-demand here. This meeting disseminated a ton of information about the SPMHA Evaluation Policies and Process, discussed hot-button topics such as Evaluators, Evaluation Games, Team Creation, and a formal Grievance Process that was introduced in the 2021-22 season.

Several questions arise each year from the U18 membership.  One of the most prevalent is how does my player move up and down the “tiers” based on their game evaluations scores.  In U18 we have a unique scenario in terms of a lower number of registrants, players returning from Elite-stream tryouts at any time throughout our process, as well as insufficient numbers to simply play multiple games with players of a broad skill level.  Therefore, we must mix players of varying skill together in some games and do our best to evaluate these players based on where their skill places them against all players in the division.

We have published the Player Movement Matrix to help visualize the evaluation process for the U18 level for the 2022-23 season.  In addition, the SPMHA Evaluation Policies and Procedures Manual is the authoritative source for the Evaluation policies for the Association.

 

Finally, several key Questions and Answers regarding evaluations can be found below:

---

Q. I noticed the evaluators are looking at their phones an awful lot.  Aren’t they supposed to be evaluating the players on the ice, not messing around with their devices?

A. SPMHA is using an App-based system called TeamGenius to assist in the process of collection and reporting of the evaluation reports, hence why they may seem glued to their phones.  TeamGenius allows an evaluator to watch the game and use a slider to assign a score of between 0.5 and 5, with half increments available.  Comments may be entered on individual players and skill criteria.  An aggregate score is then reported for each player based on the entire group that played in that game.

---

Q. Why is Little Johnny’s father in amongst the evaluators while Little Johnny is playing in that game?

A. Probably because the Division Director has concluded that Little Johnny’s father is  decent hockey mind, however that does not mean that Little Johnny is getting an advantage.

To avoid Little Johnny’s father from affecting the scores for other players competing for a spot in a tier, Little Johnny’s father would be assigned to evaluate players who are not in the same position as Little Johnny.

(i.e. if Johnny plays forward, Little Johnny’s father will be assigned to evaluate Defencemen in that game.  In fact, most directors will not allow Little Johnny’s father to evaluate any forwards at all throughout the entire evaluation process, even the games in a level above or below where Little Johnny might be currently being evaluated in.  This ensures that Little Johnny’s father does not have the opportunity to purposely sandbag the scores for any other players competing against his little angel at the same position.)

---

Q. My U15 or U18 player requested to play Checking in their registration.  Why did he end up assigned to a non-checking team?

A. Part of the job of the Divisional Director is to ensure that each player is placed in the group where that player can ultimately have the best chance for success throughout the entire season.  There are also safety concerns that we must take into account for some players who have never played body checking hockey in previous years, or who evaluate at a lower skill level than the players who end up in the Checking groups.  Using skill-based scores from the evaluations enables a director to determine which players will be able to maintain the pace of the top three groups in terms of speed and skill, an important factor when determining the viability of the player to play Body Checking hockey.  The last thing we want to do is place a player into a situation where they could sustain a potentially serious injury by participating in a Checking game that might be above their capacity to play in.

---

Q. What are the evaluation criteria used to rank my player?  Who decides on them?

A. Depending upon the level of hockey being evaluated, whether there is Body Checking or not involved, and the position (Defence, Forward or Goaltender,) players will be evaluated on a set of 6 or 7 different criteria.  Each criteria has a weight assigned to it, with some criteria (skating ability, game understand/hockey IQ for example) being assigned a heavier weighting than other criteria (such as shooting, passing and scoring).

These criteria are selected by the Evaluation Committee within the Hockey Operations group within SPMHA, based upon the recommendations of Hockey Canada.  The weighting for each criteria is determined by the Evaluation Committee with input from Divisional Directors who are intimately familiar with the idiosyncrasies of their respective divisions.

---

Q. Why did my player start with one group of kids, move throughout the process and then end up with some, but not all of the same players?

A. Throughout the course of U18 evaluations, we often do not have the numbers to play an entire game with all similarly-skilled players.  We must mix a variety of skills and evaluated scores.  For example, in evaluating which players will end up in tier 5 U18, we play games with a mix of players at approximately the tier 4 and 5 level, and then one with players at approximately the tier 5 and 6 level.  A player who scores low in a “4-5 mixed game” likely will score well in a “5-6 mixed game”.  Ultimately those players will likely end up on a tier 5 team.  The same can be said for a player who ends up in Tier 4 – they may show less well in a tier 3-4 mixed game than in a tier 4-5 game, a good indicator they need to end up in the middle – tier 4.

---

Q. My player played with a particular player (or set of players) over several season but ended up on a lower tier team than those players – what gives?

A. The best, and worst, part of the evaluation process is that players are given the opportunity to display their skillset on the ice, against a group of like-skilled players.  While your child may have played with certain individuals in past seasons but not this season can be for a wide variety of factors:

• • Some players develop strength, speed or agility at different rates as they grow physically and return year over year with different strengths.  The player that had no strength to battle in the corner suddenly dominates in puck battles and receives much higher scores than in previous years
  • Some players make the effort to improve their hockey skills or overall fitness over the off-season; some put in a tremendous amount of effort and improve their placings dramatically
  • Some players spend a lot of time playing video games over the off-season or otherwise not developing their hockey skills
  • There are times when evaluators pick up on the good, or the bad, parts of a player’s game and that helps or hinders the player’s overall evaluation.  While every evaluator does their level best to see as much of a player’s games, certain shifts can make or break that game’s evaluation score for that player.  This is one reason to encourage your player to make sure EVERY shift is their level best – if a players stays on the ice too long, evaluators may notice that and can negatively score the player on not knowing when to come off the ice, or may see that player give up on a crucial back-check because they are tired, again resulting in a negative score.

---

Q. My child scored multiple goals in an evaluation game.  Why isn’t he being moved up a tier?

A. Goals are not the only criteria evaluators are looking for.  Yes, a multiple goal game in an evaluation scenario may be due to your child having a high degree of skill.  It could also, however, indicate that they were in the wrong place at the right time, had a period of good fortune, or could also indicate that they were on the ice against a purposeful lesser-skilled line up.

---

Q. Why does SPMHA not use independent evaluators?  I would happily pay the few bucks extra for the use of independent evaluators who won’t screw my kid over year after year.

A. The Executive has debated this topic extensively every single year. The number one complaint we hear from parents surrounds the evaluation process, the use of parent-evaluators and perceived bias against certain players. In the end, there are several reasons independent evaluators are not used with the exception of goaltender evaluations – a highly technical position not easily evaluated by the majority of coaches and Association-based evaluators:

• •  First and foremost, the cost for SPMHA to hire a panel of independent evaluators for all divisions has been estimated several times over the years and it is not just a few bucks that could be absorbed easily by a small fee increase.  Either the players would end up receiving a single evaluation session with a modest increase in registration fees, or the players would receive a sufficient number of evaluation sessions with an increase of approximately $75-150 per player.  This is not a small increase that can be absorbed by the Association and would go directly into player fees.
  • Each year, SPMHA converses with a multitude of other MHAs, both within our league and across the province. There is a constant theme in all of these conversations that, regardless of who is completing the subjective evaluation piece, there are still concerns brought forth from parents who disagree with the placement of their child.
  • We have seen some quotes as little as $25 for three games. At this rate, how many evaluators are coming out and what quality of evaluators are you getting? Are you individuals who are invested in ensuring the players are actually placed appropriately, or are you getting college kids or beer league players who simply want to earn a few bucks towards their own playing fees and who don’t care where our players are assigned to in the end.
  • We have had other Associations tell us that they have confirmed proof that third-party evaluators in the past have shared results with non-sanctioned teams and that this has actually resulted in some of their higher-end talent players being recruited away from that Association. This talent drain is obviously bad for the Association and something SPMHA has consciously tried to avoid.

In the end, SPMHA is committed to completing the Evaluation process with as much integrity as possible. We believe that if we commit to a process and maintain a transparent check-and-balance system, we can put forth a reasonable effort to create fair and balanced teams. We have redirected funds that might have been used to subsidize third party evaluators to development programs that directly impact our children through in-season development programs, equipment to better train and develop the players, along with coach development and mentorship, etc.