How It Works

Each team is ranked separately according to six key variables (listed below). For the most part, the lower the ranking the better—the only exception being ease of schedule, where a lower ranking simply implies an easier schedule. For example, a team ranked #1 in recruiting means that, in theory, they were the most talented team throughout the 2005-2009 time period. The rankings correlate with a team’s overall win percentage during the 2005-2009 time period, and they basically state “this is why each team had the winning percentage that they did during this time frame.” I think it will make this concept easier to digest if I only introduce two variables at a time.

The more complicated version: Team Win % = b_o + b₁R + b₂D + b₃S + b₄H + b₅CL + b₆G + ε

• R — Recruiting Ranking
• D — Player Development Ranking
• S — Ease of Schedule Ranking
• H — Home Field Impact Ranking
• CL — Clutch or Luck Factor Ranking
• G — Gameday Factors Ranking

The b‘s are parameters to be estimated, and ε is a random error term.

Clutch or Luck Ranking

Click to enlarge.

Clutch or Luck Ranking is determined by the difference between a team’s actual performance versus its expected performance based on average net point differential. The lower the ranking, the more “lucky” or “clutch” (depending on which way you lean) a team was during that point in time. This is a topic that we’ve already covered in depth, so I’m going to keep this explanation short. In the short-term, I think that coaches and programs do get lucky. But luck tends to flatten out in the long run, and so significant deviations from the average are more likely the result of something related to the coaching staff’s ability in close games. Therefore, in a five-year study, this ranking should provide pretty good insight into which coaches have an ability to prompt their teams to perform well in close games and which ones don’t. However, some have argued that this type of metric is still more of a measure of luck, which is why we’ve kept “luck” in the title.

Clutch or Luck from ND’s Perspective

Yearly comparison (click to enlarge).

Weis (click to enlarge).

Kelly (click to enlarge).

Under the guidance of Charlie Weis, Notre Dame ranked 34th in clutch or luck from 2005-2009. This should strike ND fans as a bit odd after witnessing the 2009 team suffer through six losses from games decided by a touchdown or less—including four in a row to close out the season. But if you look at the yearly deviation from expected chart, Weis performed above or close to the expected overall win percentage up until 2009. He had what appears to be a lucky 2006 campaign, which included “miracle” victories against Michigan State and UCLA, to thank for offsetting his hard luck in 2009.

Cincinnati holds the top rank as the most clutch or lucky program from 2005-2009. It’s important to note that Mark Dantonio was the coach for two of those seasons, but Kelly deserves much of the credit for that ranking. Kelly ranks 3rd among all D1 coaches that were active throughout the 2005-2009 time frame in the clutch or luck metric and he was good for an average of one win per season above the expected based on net point differential. Or, to put it another way: Kelly’s presence on the sideline added an additional win per season because of how well his teams performed in close games. Whereas the average D1 coach evens out over the long run—meaning their presence tends not to affect the outcome of a close game one way or the other—Kelly only had a single season where his team’s actual winning percentage was below the expected level based on net point differentia

In fact, Kelly is heading into the 2010 season having won the last seven games which his teams have played that were decided by a touchdown or less. If Charlie Weis could say that, Notre Dame would have been 12-0 after the 2009 regular season and in the national championship game. A bit of light may have been shed on how Kelly prepares his teams for late game situations during his August 26th press conference when discussing having the team practice 49 different in-game scenarios he’s come across during his coaching career:

“Tomorrow we will have our bench control script which is about 49 different scenarios that will occur during the game. I was telling the team that when I first started back at Grand Valley, I had about 16 scenarios, between 16 and 18, and we have worked our way up to 49. So I don’t know if that says a lot about my experience dealing with different things in the game or the obsessiveness of the coach trying to cover every scenario that may never occur, but you want to be covered anyway….

…Obviously, I have learned some of them by being involved and one of them was against West Virginia a couple of years ago. We had a 19-point lead with about a minute and three seconds left on the clock. Every scenario, we had to make certain played out, and it did. And that was to decide do you take the safety or punt the ball. So we put in what we call turtle punt now which kills approximately six to eight seconds and you can hold. We had the whole team just hold because we don’t care if we get a penalty in that situation. But we have to get eight seconds off the clock. So imagine that scenario was a learning experience because if we had taken off a few more seconds in the West Virginia game, they wouldn’t have had a chance to tie it up and send it into overtime. So turtle punt is an experience just two years ago—I think it’s number 47 now on the list.” –Brian Kelly, August 26, 2010 Press Conference

This also serves to illuminate the value of having a man at the helm with as much head coaching experience as Kelly has to fall back upon. While critics jump on the fact that Kelly spent such a large portion of his career at Grand Valley State, he still undoubtedly gained situational head coaching skills that can only come from tangible experience. Does it really matter which level of football you pull those on-field experiences from? I say, for the most part, no. That’s why you don’t hire head coaches to learn on the job at a place like Notre Dame.

Game Day Factors Ranking

Click to enlarge.

We’ve come to the 6th and final variable in the model—and the most theoretical one. It’s important to remind the reader that models are simply abstractions of reality and are merely intended to simplify the world around us. There will never be a way to truly measure how well coaches prepare their teams for game day, manage a game, and get their teams to execute. But we can certainly try and estimate it. If you scroll up to the original theory equation, we have a way to estimate every variable so far except Game Day Factors (G). The graph on the left shows that we can explain about 3/4ths of the variation in win percentage with the five variables that we already have, and we already know what each team’s actual win percentage is. We should be able to algebraically manipulate our equation to answer what “G” is:

1. Win % = R + D + S + H + CL + G
2. G = Win % – R – D – S – H – CL

Click to enlarge.

In other words, if you look at the graph, we’re saying  is that the difference between the predicted win percentage based on the first five variables and the actual win % is due to factors related to the remaining variable, “G”—the Game Day factors. We can say this logically, because we’ve already theoretically accounted for just about everything else: talent, player development, schedule, home field impact, and luck (or performance in close games). So what’s left is the tactical stuff related to game day (execution, scheme, management, etc.). Therefore, the ranking comes from the difference between the team’s actual win percentage and the predicted based on the other variables. The lower the ranking, the better a team is on game day. Granted, this is essentially a catch-all statistic, and so other smaller issues that could affect a team’s performance will get caught in this statistic as well: things like injuries, transfers, weather, and other “random”occurrences. But we have no easy way of separating that kind of thing out beyond this type of metric.

Why don’t we use on-the-field statistics to rank a team’s performance? Because stats like offensive yards per game could easily be affected by the variables we’ve already looked at, so it’s hard to say how to filter them out—it might be impossible. So we don’t use any statistics at all when looking at on the field performance.

Game Day Factors will be looked at in greater detail in the last article in this series.

Game Day Factors from ND’s Perspective

Notre Dame ranked 49th in this metric during the Weis years. Which makes sense: the Irish fielded a very good to excellent offense three out of five years, but any offensive accomplishments were generally offset by a below average to terrible defense year-in-year-out. Despite his bluster about a schematic advantage, Notre Dame was essentially an schematically average team on game day under Weis. No matter how hard he tried, Weis couldn’t figure out how to run the team as a whole, and it showed up on game day when the offense was constantly forced to carry the entire load. The results were often disastrous when it couldn’t.

Notre Dame Scoring Offense and Defense Rankings

Cincinnati ranked 28th in Game Day Factors from 2005-2009, but that relatively high ranking might actually understate Kelly’s effect on the team, thanks to two moderate years under Dantonio that get caught in this metric. When Kelly arrived in 2007, he was able to maintain Dantonio’s defense while instantly injecting life into the offense. The scoring offense jumped from 82nd in 2006 in Dantonio’s last year to 16th in 2007. Even in a down defensive year (2009) when Cincinnati needed to replace the majority of the starters and break in a new defensive coordinator, the defense remained somewhat respectable with a 44th ranked scoring unit. It’s also worth noting that Kelly went through five different quarterbacks due to injuries in 2008, which goes a long way towards explaining the drop-off in offensive scoring that year. Kelly’s extensive experience as a head coach trumps the NFL coordinator yet again. And with far less recruited talent and a relatively comparable schedule.

Cincinnati Scoring Offense and Defense Rankings

At the end of the day…

The common thread in all of this is that Kelly clearly has at least one distinct advantage over Weis coming into the Notre Dame job: head coaching experience. Kelly has an extensive list of in-game scenarios that he’s come across throughout his head coaching career and he prepares his team for them. He clearly knows how to close out games, something that Weis mightily struggled with in 2009. Further, it was quite apparent throughout his Notre Dame career that Weis was still just an offensive coordinator at heart. He never learned how to manage the team as a whole. So while his offenses could put up numbers, Weis could never figure out how to lead the defense to similar feats. Despite his reputation as an offensive-minded guy, Kelly’s defenses at Cincinnati were never below average. He’s been quite involved in the defensive aspects of the game, while Weis preferred to hire a defensive coordinator that exclusively handled everything associated with that part of the team.

It’s important to remember that Kelly was working with a Cincinnati team that did not come close to fielding the same talent as Notre Dame. It’s also important to note that Cincinnati played very comparable schedules to Notre Dame, especially in 2008 and 2009. Yet Kelly was able to do a lot more with far less. Here’s why:

1. Kelly has a system and he sticks to it rain or shine—Notre Dame seemed to have a new identity on both sides of the ball every season under Weis; sometimes even on a game-to-game basis.
2. Kelly’s offense is relatively simple and easy to execute, whereas the pro-style offense Weis ran took time to learn. Part of the reason why 2007 occurred is because Weis tried to force-feed his playbook on a bunch of underclassmen that were still trying to learn how to play football, let alone learn a huge playbook.
3. Kelly designs his practices to be extremely fast-paced in order to maximize the limited time the coaching staff has with the players each week. Weis preferred “pro-style” practices that were slow-paced walkthroughs—it may have preserved the health of players, but it wasted valuable teaching time.

Even though the Game Day Factors ranking is theoretical in nature, I do believe that it captures these differences well. If you look at the other five variables as inputs, Cincinnati simply should not have had the output that it did (wins). The main difference is really what the coaches do with their time and how they get their teams to perform on game day, given all of the other variables. Looking at it this way, Weis was simply average: he may be a great pro offensive coordinator, but he was not an elite college head coach. It seemed like he just didn’t understand the college game well enough. Kelly has been brought up in college football. He understands that it’s less about complicated schemes and more about fundamentals and execution. We should see that difference pay off, and if history is any indicator, very soon.

The last article in this series will tackle 5-year team profiles and their applications.

Similar Posts:

• A Theory on College Football Performance: Part 1 – The Players
• A Theory on College Football Performance: Part 2 – Institutional Factors
• Clutch or Luck? Brian Kelly’s Performance the Past Five Seasons

This article is © 2007-2012 by De Veritate, LLC and was originally published at Clashmore Mike. This article may not be copied, distributed, or transmitted without attribution. Additionally, you may not use this article for commercial purposes or to generate derivative works without explicit written permission. Please contact us if you wish to license this content for your own use.

How It Works

Each team is ranked separately according to six key variables (listed below). For the most part, the lower the ranking the better—the only exception being ease of schedule, where a lower ranking simply implies an easier schedule. For example, a team ranked #1 in recruiting means that, in theory, they were the most talented team throughout the 2005-2009 time period. The rankings correlate with a team’s overall win percentage during the 2005-2009 time period, and they basically state “this is why each team had the winning percentage that they did during this time frame.” I think it will make this concept easier to digest if I only introduce two variables at a time.

The more complicated version: Team Win % = b_o + b₁R + b₂D + b₃S + b₄H + b₅CL + b₆G + ε

• R — Recruiting Ranking
• D — Player Development Ranking
• S — Ease of Schedule Ranking
• H — Home Field Impact Ranking
• CL — Clutch or Luck Factor Ranking
• G — Gameday Factors Ranking

The b‘s are parameters to be estimated, and ε is a random error term.

Ease of Schedule Ranking

Click to enlarge.

The ease of schedule ranking is determined by taking the five year average Sagarin strength of schedule ratings (2005-2009) and then ranking each team’s average schedule from easiest to hardest. The graph on the left is not showing this ranking, but rather the actual relationship between overall win percentage and the actual Sagarin SOS rankings. What jumps out at me is that the trend is essentially saying that the harder the schedule, the better a team’s expected performance. This doesn’t make any logical sense, and the low fit of the trend line with the actual data suggests that there is not a strong relationship between how many games a team wins and its strength of schedule—at least not when you’re looking at SOS by itself.

So, for example, while many bemoan that Boise State relies on their weak schedule to be successful each year, there are plenty of teams with equally weak or even weaker schedules who cannot replicate Boise State’s success. On the other end of the spectrum, Florida played the third hardest average schedule over the last five years, according to Sagarin, yet also won the national championship twice in that time. While SOS is actually statistically significant and worth looking at, it hardly seems all that deterministic. Instead, SOS needs to be looked at in conjunction with the other variables involved in this study to be of real use.

Ease of Schedule from ND’s Perspective

Think scheduling has been on the minds of ND fans lately? Former Athletic Director Kevin White’s head-scratching scheduling model is out the window, but the 2010 season still features some opponents that are not exactly going to excite many Irish fans. While the Utah game has all the appearances of being one of those under-the-radar matchups that could prove to be the most difficult game outside of USC on ND’s schedule, Western Michigan, Tulsa, and Army are essentially lay-ups. This year’s schedule should prove to be ‘easier’ than just about any recent ND schedule and will continue a general trend of easier schedules that began when Kevin White became A.D.

Click to enlarge.

But while 2007-2009 were some of the easiest schedules in recent Irish history, the results were, frankly, terrible: a 43% winning percentage during those three seasons. In that same time period, the Kelly-lead Cincinnati program played a somewhat comparable schedule with much less talent and came away with an 83% winning percentage (only Boise State, Texas, and Florida had a higher winning percentage during those three years). This, in a microcosm, supports the assertion that ease of schedule is not as important of a variable as people make it out to be—highly-rated talent and an easy schedule don’t guarantee anything. Despite the relatively easier schedule the past few seasons, Notre Dame still had the 21st hardest average schedule since 2005. With Oklahoma, Texas, and Miami all on the docket for future Irish schedules, it’s likely that the recent trend of easier schedules is about to end. But as has been shown, that’s not really anything to worry too much about as long as the other program variables are up to snuff.

Home Field Impact Ranking

Click to enlarge.

Home field impact ranking is sorted by the difference between a team’s actual five-year home game win percentage versus it’s expected five-year overall win percentage based on home field factors. The lower the ranking, the larger the home field advantage a team is considered to have (represented on the graph to the left). The intent of this ranking is to represent how much of an advantage some teams have over others in terms of the difficulty of their home stadium for opponents, as well as the leverage that program has in scheduling a large number of home games each year.

The reason we’re using a predicted overall win percentage for the comparison is because it provides a better basis for evaluating how a team is expected to perform overall given its normal performance at home as well as the number of games played at home. By doing it this way, we can use the expected overall performance to show that teams like Ohio State do indeed have a huge home field advantage; if you compare Ohio State with their actual overall win percentage, the advantage of playing at home looks minimal (about a 4% difference). But the estimated overall percentage shows that there’s actually a huge home game advantage for the program (about a 20% difference).

We don’t directly compare a team’s home win percentage with its away win percentage because doing so doesn’t take into account the previously mentioned factors (number of games at home, etc.). That, and it’s more of a measure of which teams utilize their home field as a crutch (Kansas State for example, won 71% of its home games, but only 17% of its away games) rather than teams that have a significant home advantage over the average team during each home game.

Lastly, it’s important to note that this method shows that every team has some kind of advantage from playing at home versus away. Even lowly Eastern Michigan, owner of the worst home field impact ranking, has about a 2% positive difference between their actual home winning percentage and their expected win percentage.

Beaver Stadium (Penn State)

When discussing the concepts behind this ranking with other stat-heads, a few issues were brought up: first, the definition of a home game is becoming increasingly murky (i.e. how does one classify ND vs. Washington St. in San Antonio, or ND vs. Miami at Soldier Field—are these really neutral sites?); second, some voiced that this metric was more of a scheduling thing than a true measure of a “home field advantage;” third, it’s been said that this metric seems like it could be skewed because teams that schedule a lot of cupcakes each year will appear to have a better home game record.

So, for now, let’s just define home games as games played in a team’s home stadium. As far this ranking appearing to be measuring a scheduling thing rather than a true home field advantage: if a team has more leverage than other teams to schedule an above average number of home games, that’s an advantage. The top ten home impact teams play an average of 10% more home games per year than the bottom 10 teams. Additionally, some stadiums are definitely harder for visiting teams than others. While Sagarin standardizes home field advantage to be the same for every home team across the board, I don’t think that’s a fair reflection of what the college football universe looks like. Who can say that Penn State (56% of games at home) with 100,000+ fans at each game has the same home field advantage as Eastern Michigan (37% of games at home), who cannot even fill a stadium fit for a large high school?

Finally, while one can look at Florida (ranked 5th in home field impact) and say, “Hey, they play teams like Charleston Southern at home, so of course they have a higher home winning percentage,” you’d be neglecting the fact that, again, Florida played the 3rd toughest schedule in the country over the last five years. Their 2008 schedule, for example, featured home games versus Miami, Mississippi, LSU, and South Carolina. I think that conference schedules will cancel out cupcake games in the long-run, and so it shouldn’t skew this ranking a whole lot.

Home field impact will be discussed in greater depth as it relates to college football as a whole in a later article.

Home Field Impact from ND’s Perspective

Notre Dame Stadium

To put it bluntly, Notre Dame does not have the home field impact of other elite and/or successful programs—the Irish possess a less-than-impressive ranking of 55th. It’s important to note that this metric is tracking the “boost” a team should get from its home games, not specifically its winning percentage at home. So what it’s saying is that Notre Dame only has the 55th best “boost” in its performance from home games—not much of an advantage.

From a purely qualitative standpoint, Notre Dame crowds tend to be rather tame in comparison to, say, crowds at SEC schools. Anyone that has ever been to a Notre Dame game and has received the “SIT DOWN!” treatment from a 60 year old couple knows this all too well. Notre Dame stadium-goers are often yuppy tourists that are there to experience a Notre Dame game, not actually participate in it.

Other issues: like clockwork, NBC takes a 5 minute commercial break after just about any key play which completely kills any kind of crowd involvement. Also, only playing day games contributes to a tamer crowd because:

1. The atmosphere is simply more dramatic at night games.
2. Night games allow more time to drink prior to the game, which will lead to a more raucous crowd.

An increase in alcohol consumption is the reason why Notre Dame doesn’t play night games, from my understanding, and so don’t expect that part to change (I’m not necessarily suggesting that it should, only that other programs do have night games and it helps them).

Finally, the school wants its stadium to be as family-friendly as possible and lately has gone to extreme measures to maintain that appearance—to the point that you could be removed from a game just by appearing to be intoxicated, and banned from future games if you actually were intoxicated. Even alumni were not immune from this treatment. Nobody wants a drunk guy puking on them during a football game, but having to look over your shoulder for an usher eye-balling you if you’re standing and yelling is not exactly going to encourage fans to go crazy during games.

So what can be done to make the stadium more difficult for opponents? I’m not suggesting that the Irish lose any traditions, encourage binge drinking, install a jumbotron, or pipe in crowd noise. But small things like figuring out how to encourage the crowd to actually stand throughout the game and get involved (especially on the South side of the stadium) would help. Brian Kelly and Jack Swarbrick have already gotten NBC to agree to shorter commercial breaks, but a lot more could still be done there: how about actually specifying that breaks can only be taken after touchdowns, unblocked punts, during timeouts, and other similar plays that won’t kill crowd interest? The administration could also stand to be less draconian about keeping the stadium family-friendly on game day—I thought ushers were supposed to show you to your seats, not act as a security force? With all of that said, perhaps nothing would help more than reinvigorating the energy of the program after a nearly 15-year malaise: only time will tell if Kelly can accomplish that.

At the end of the day…

Institutional factors are something that Brian Kelly has some input on (the NBC commercial situation, for one), but the onus of leadership essentially falls onto current A.D. Jack Swarbrick. There’s little-to-no evidence to show that Notre Dame needs to schedule easier games to get to the national championship, and Swarbrick should concentrate on getting opponents on the slate that excite the Notre Dame fan-base. The recent addition of traditional football powers to future schedules gives reason for optimism. Of course, it will be up to the man whom Swarbrick hired, Brian Kelly, to get the program into a condition to be able to win those games.

Of greater concern, in my opinion, is finding a way to make Notre Dame stadium a more difficult venue for opposing teams to play in. When programs like Syracuse, Navy, Air Force, and Connecticut are coming onto campus and winning games, it says a lot about the difficulty of the home field, not just the quality of those specific Charlie Weis teams. Simply put: are other teams really all that intimidated about coming to South Bend?

I’ve elaborated briefly on some things that the administration can change, but Brian Kelly can glean something from one of his predecessor’s faults. The Irish need to regain a killer instinct. Visiting teams should not feel comfortable at Notre Dame stadium, and one way to get a psychological edge is by earning a reputation for burying inferior teams. Charlie Weis, when he had the chance, often simply chose to take his foot off the gas pedal, lest we embarrass the other team. What kind of message does that send to your team? Or to the other team for that matter? This isn’t pee-wee football, sometimes you have to bully and intimidate your opposition: it’s part of college football and being an elite program. Look at the 2008 Oklahoma results (with the exception of the bowl game) for guidance.

The third article in this series will look at on-the-field factors: clutch/luck ranking, and game day factors like play-calling and execution.

Similar Posts:

• A Theory on College Football Performance: Part 3 – Game Day Performance
• A Theory on College Football Performance: Part 1 – The Players
• 2010 Elite Selection Playoff: Week Twelve

This article is © 2007-2012 by De Veritate, LLC and was originally published at Clashmore Mike. This article may not be copied, distributed, or transmitted without attribution. Additionally, you may not use this article for commercial purposes or to generate derivative works without explicit written permission. Please contact us if you wish to license this content for your own use.

This project evolved into something else in the process: a way to determine the relationship between a team’s performance and the normally hard-to-quantify college football variables (i.e. player development). So while the main objective still stands—to evaluate the Weis era and provide an outlook on ND football—we will also explore the general usefulness of this model when looking at college football from a macro perspective further down the road. There are six main variables that are plugged in: I’ll evaluate two per article.

How It Works

Each team is ranked separately according to six key variables (listed below). For the most part, the lower the ranking the better—the only exception being ease of schedule, where a lower ranking simply implies an easier schedule. For example, a team ranked #1 in recruiting means that, in theory, they were the most talented team throughout the 2005-2009 time period. The rankings correlate with a team’s overall win percentage during the 2005-2009 time period, and they basically state “this is why each team had the winning percentage that they did during this time frame.” I think it will make this concept easier to digest if I only introduce two variables at a time.

The more complicated version: Team Win % = b_o + b₁R + b₂D + b₃S + b₄H + b₅CL + b₆G + ε

• R — Recruiting Ranking
• D — Player Development Ranking
• S — Ease of Schedule Ranking
• H — Home Field Impact Ranking
• CL — Clutch or Luck Factor Ranking
• G — Gameday Factors Ranking

The b‘s are parameters to be estimated, and ε is a random error term.

Recruiting Ranking

Note: This ranking is independent of the actual rankings that are given out by Rivals.

Click to enlarge.

A brief introduction on why I’m using four-year average Rivals star ratings: it’s a statistic more correlated with win percentage than the straight-up Rivals yearly ranking. To determine recruiting ranking, the four-year average is found for each season from 2005 through 2009, and then that result is averaged out. Each team is then ranked by their overall average—the lower the ranking, the better. The recruiting ranking represents the raw talent, independent of any player development or coaching, that a team possessed in the five-year time period. This ranking does not factor in “random” occurrences such as transfers, injuries, academic/disciplinary issues, etc.

There’s a relatively decent correlation between this method of evaluating recruiting performance and a team’s overall win percentage. But most will notice that it doesn’t come close to explaining the full picture; there are plenty of teams at the lower end of the recruiting performance picture with high win percentages. So, while recruiting is obviously important, there are other team factors that must be evaluated. This seems obvious, but the other variables are harder to quantify.

Recruiting Ranking from ND’s Perspective

Click to enlarge.

Charlie Weis inherited a strange situation when he took over the Notre Dame head coaching job in 2005 in that Tyrone Willingham’s 2003 recruiting class was pretty good, however Willingham’s subsequent 2004 and 2005 recruiting efforts were, for the most part, absolutely terrible. This left the on-campus talent top heavy, and a drop-off in 2007 was expected—just not to the degree we all observed. Still, as the graph shows, Weis did his best during his tenure as head coach to rope in talent and really raise the bar on recruiting. Weis has left ND’s cupboard stocked, at least on an aggregate level (certain positions are more bare than others—QB for example), and Kelly is stepping into a much better situation, talent-wise, than Weis did.

Click to enlarge.

Despite the relative lack of talent in the beginning of the Weis era in relation to where it is today, Notre Dame was still ranked 11th overall during the 2005-2009 period in terms of recruiting ranking. This means that, theoretically, there were only 10 other teams with more raw talent than what the Irish had on campus during the 2005-2009 time frame. As shown on the graph on the right, the 2009 team had more raw talent on it than the eventual national champion, Alabama, and many of the other teams that had played in the national championship in the last five years. Notre Dame underachieved big time, especially in 2009. ND fans know the story all too well: Weis could recruit talent, but he couldn’t develop it. If there’s one thing the fans will want Kelly to maintain from the Weis regime, it’s recruiting. Here are some reasons why:

1. No team with a 4-year average Rivals star rating below 3.43 has played in the national championship game (Ohio State, 2006) in the last five years.
2. No team with a 4-year average Rivals star rating below 3.61 has won the national championship in the last five years (Alabama, 2009).

Player Development Ranking

There is an excellent correlation between long-term recruiting performance and the number of NFL draft picks a team produces in a period of time.

Click to enlarge.

A team that produces a lot of NFL draft picks may not necessarily be better than other schools at developing recruits into NFL players, but rather has better incoming talent to work with—less work is needed to turn them into draft picks. Therefore, the more a team exceeds its expected number of draft picks, the better that team is considered at developing its recruited talent—including non-NFL talent. I don’t think it’s a stretch to say that this ability to produce NFL talent can be extended down to the players who aren’t drafted—that there’s better physical strength, speed, and fundamentally-developed players on teams which produce a higher than expected number of NFL players.

This graph shows 2005-2009 recruiting performance versus draft performance. The team with the largest positive deviation away from the expected number of draft picks is Virginia Tech (Cincinnati had the second largest). The development ranking, therefore, ranks programs by how many drafted picks they produced over the expected amount.

It should be noted that because we’re looking at the 2005-2009 time frame as a whole, coaching changes are captured in this metric. So, for example, two of Mark Dantonio’s years at Cincinnati are represented here. But, you’ll see in a second that Kelly deserves most of the credit for Cincy’s developmental performance.

Player Development Ranking from ND’s Perspective

It should not come as a surprise to anyone to find that Notre Dame ranked 89th in player development during 2005-2009. Or maybe it’s surprising that Weis even did that well. Thankfully, ND appears to be bringing in one of the best developers of talent in college football. Kelly has had immediate impacts on the programs that he’s taken over. It shows up in the data, and we’re already seeing some improvements along the training front at Notre Dame in terms of weightlifting numbers after a full summer with coach Longo. But before we launch into the charts, here is some qualitative data to backup the assertion that Kelly is a great developer of talent.

• Central Michigan did not produce a single draft pick from the 1998-2004 draft classes. In Kelly’s first draft class (2005), Central Michigan had two players drafted. Two years later (Kelly’s last draft class there), CMU had another three players drafted, including a first rounder (Joe Staley) and a second rounder. Mike DeBord, the previous head coach, was 12-34 during his time at CMU, so it’s hard to believe he had a huge impact on these players.
• Staley was originally an unknown TE prior to Kelly moving him to OT and developing him into a 1st-round pick.
• Cincinnati’s 2009 draft class was the largest in school history (six players) and included second rounder DE Connor Barwin, a two-star player who was switched from TE to DE prior to his senior season and ended up leading the Big East in sacks.
• Mardy Gilyard, who wasn’t even evaluated by Rivals, was a CB under Dantonio that was switched to WR by Kelly upon arrival and ended up being an AFCA All-American and 4th-round draft pick.

The graph below shows the two period moving average for both teams in terms of number of NFL players taken. The horizontal bars over the years show when Kelly was at those programs.

Click to enlarge.

This second graph shows the trend when you combine the number of draft picks from CMU and Cincinnati. The red bars indicates Kelly’s influence, and when combined with the overall trend, shows rather definitively that Kelly’s influence can be largely attributed as a main driver in the increased number of draft picks.

Click to enlarge.

Finally, below is a graph comparing Kelly and Dantonio. Looking at both the data and the graph, I feel that Dantonio has had a negligible difference on his teams in terms of player development. First of all, while Kelly was able to ramp up the player development at both CMU and Cincinnati, Dantonio’s teams have tended to level off or even dip down. Second, given Michigan State’s relative advantage in talent level over Cincinnati (MSU 2.78 average stars for the period, Cincinnati 2.09 average stars), you’d expect a higher number of NFL picks almost regardless of Dantonio’s player development abilities: which is missing.

Click to enlarge.

At the end of the day…

There has been quite a bit of grumbling around Kelly’s recruiting thus far, mainly because that’s the only aspect of the program that the average fan really has any level of insight into at this point. This year’s squad’s four-year average star rating is above the minimum benchmarks noted earlier: 3.61 for a national championship; 3.43 for a national championship appearance. However, there are some scattered clouds on the horizon. A few late additions to the 2010 class dropped the star-rating of that individual class down to 3.44. And even though it’s early, and it’s worth noting that the rankings will likely reshuffle in the coming months, the 2011 class is carrying a 3.39 average star rating as of right now.

Recruiting appears to be on the verge of trending downward, and there aren’t a whole lot of scholarships left in this year’s class to correct it.  It’s too early to make any actual judgments, but it’s a situation worth being aware of. The real question isn’t whether or not Notre Dame could set a new minimum benchmark in the future, but rather do we feel comfortable having to be put into the position to have to set a new minimum talent benchmark for winning the national championship? Or even appearing in one?

On the flip-side, it’s not out of the question to think that Kelly could set those new benchmark if he had to. Cincinnati from 2005-2009 ranked 92nd in the recruiting metric we discussed earlier, but was 2nd in player development. Kelly has an eye for moving players into positions that take advantage of their natural strengths and it’s hard to find a better duo in the country than he and Longo when it comes to player development.

We should see that attention to development pay dividends at Notre Dame and it will be interesting to watch players like Theo Riddick, who was switched from RB to WR, along with some of the incoming freshmen that are moving over to the defense. If Notre Dame fans should feel comfortable about one thing, it’s Kelly’s ability to develop and then maximize talent.

Part 2  in this series looks at institutional factors: ease of scheduling and home field impact rankings.

Similar Posts:

• A Theory on College Football Performance: Part 3 – Game Day Performance
• A Theory on College Football Performance: Part 2 – Institutional Factors
• 670 the Score Radio Interview Audio and Reflections

This article is © 2007-2012 by De Veritate, LLC and was originally published at Clashmore Mike. This article may not be copied, distributed, or transmitted without attribution. Additionally, you may not use this article for commercial purposes or to generate derivative works without explicit written permission. Please contact us if you wish to license this content for your own use.

James’ formula does not hold up well for college football, but the principle of there being a close relationship between a team’s overall winning percentage and its season-long scoring margin does. In this sample of five seasons (2005-2009), the average scoring margin is highly predictive of a team’s winning percentage. In fact, it explains nearly 93% of the variation in winning percentage, as shown below.

The question is: what is that 7% of variation that is not explained by point margin? Have some teams just been either lucky or unlucky during the best five seasons? This would make some sense in baseball, but football is a game where team’s have a little bit more control over the outcome of each play. Think about the 2009 Notre Dame season. How many of those close losses can be pinned on luck rather than a defense that couldn’t stop anyone at the end of games? My theory is that the variation is the result of the ability or inability of some coaches being able to get their teams to perform in clutch situations.

An Example

Take the 2009 Cincinnati – Pittsburgh game. Brian Kelly has either been the third luckiest coach over the past five seasons (stemming back to his time at Central Michigan), or the third best at closing out games in his team’s favor when all else seems to be equal. Dave Wannstedt has either been the seventh unluckiest coach over the same time period, or the seventh worst at closing teams out.

Let’s move forward to 3:08 minutes left to go in the game—both teams are tied at 38 and Pitt has the ball on Cincinnati’s 32 yard line.

3:08, Pitt 1st and 10 at Cincinnati 32 yard line

Cincinnati DE Matthews has a slight grab on Pitt RB Deion Lewis’ facemask as he tackles him—no call by the referees. Wannstedt is incensed, jumping up and down and pointing at the Heinz field jumbotron. The clock keeps moving, so Pitt has to run another play.

2:44, Pitt 2nd and 12 at Cincinnati 34 yard line

Pitt runs again and Cincinnati calls timeout to preserve time. Wannstedt spends the timeout in the official’s huddle complaining about a non-call that he can’t get back, rather than in his team’s huddle.

On the next play, Pitt converts a nice third down pass to Jonathan Baldwin so it all seems like a moot point. Lewis runs into the endzone with 1:37 left to go two plays later.

1:37, Pitt extra point attempt

The Pitt holder loses control of the ball and the extra point is botched, so the score is Pitt 44 – Cincinnati 38.

Cincinnati kick return

Cincy return man Mardy Gilyard slips on the turf, or else he’s gone for a touchdown.

Pitt plays the ensuing series with a three man rush, deep prevent defense. The Cincy receivers run routes to clear up the middle for a single receiver who is running open every time because the Pitt D is so focused on preventing the homerun ball.

With 39 seconds left, Cincinnati is already at the Pitt 29 and calls timeout.

00:39, Cincinnati 1st and 10 at Pitt 29 yard line

Pitt comes out of the timeout and shows a five-man pass rush against Cincinnati, who is lined up with five receivers. This is an extremely aggressive call given the situation and forces Pitt into man-coverage with a single safety playing the middle of the field. The single receiver at the bottom of the formation is Armon Binns, who is being projected to go higher in next year’s NFL draft than his teammate Gilyard, so this seems like a pretty easy tactical decision for Cincinnati.

Pike throws a perfect pass as Pitt’s corner falls over while Binns blows past him for an easy touchdown. Cincinnati converts their extra point and wins 45 to 44.

Now, how much of a factor did luck play in this? Sure, Cincinnati lucked out that Pitt muffed their extra point. But then, Pitt lucked out when Gilyard tripped over himself on the ensuing return on what would have been a touchdown. Instead, it seems to me that you can look at Wannstedt’s temper tantrum to the officials over a play they couldn’t call back for clues about his team crumbling at the end. He was irrationally overreacting to a play that he couldn’t change rather than focusing on the present. Just as his overly-aggressive five-man rush against Cincy’s 5-WR formation on the last play was an overreaction to Cincy’s success during the previous plays. Both Wannstedt and his team lost their composure and all Cincinnati and Kelly had to do was react—however Cincinnati deserves credit for keeping their composure and recognizing opportunities and exploiting them.

An Analysis

Over the course of the last five seasons, Kelly’s teams have outperformed their expected winning percentage by an average of almost eight percent. What this means is that, on average, a Kelly-coached team has won at least one more game per season than it was expected to. One could say that he’s been lucky—and yes, some randomness plays a factor—but I think it speaks more to his ability to get his teams to perform in clutch situations. That’s not to say that anyone should expect Kelly to win every close game at Notre Dame, but rather they should feel more secure in knowing that he has a better history of doing so than almost any other NCAA football coach in the last five seasons.

The following two lists consist of coaches who were Division 1 head coaches from 2005-2009. A brief note regarding two coaches omitted from the chart below due to tenure length: Nick Saban in his three years at Alabama has a positive differential of 4.4%, while Northwestern’s Pat Fitzgerald has an impressive mark of 8.2% through four years.

Here are some individual coaches on a year-by-year basis:

Some Critiques and Responses

Critique

Omahadomer pointed out in the original thread on NDNation that this metric would make Tyrone Willingham the most “clutch” coach at ND and believes it tracks luck more than anything else.

Response

This analysis is a fairly good model of the time frame (2005-2009) and I’m tentative to insert data points from previous years. Tyrone Willingham’s performance at Washington seems to fit well with the perceived expectations (something like -4.2% per year on average). With that said, this metric will not fully explain the picture and is merely a tool. Like any effective analysis, you need to look at it from both a quantitative and qualitative viewpoint.

Critique

Some of the coaches who play in weak conferences might look better when they suffer blowout losses to bigger BCS opponents and subsequently play a fairly normal schedule in their conference.

Response

There’s probably some truth to this, although coaches like Patterson at TCU and Petersen at Boise State are pretty close to expected.

Critique

Why do some coaches have an expected winning percentage of 100%? Coaches seemed to be penalized for having a high expected winning percentage.

Response

I think Meyer and Carroll both had actual expected percentages in 2008 that went over 100% using the formula above and I just capped them at 100%. It was a bit of a reach to plug yearly data into a formula based on five year averages and I did it because the alternative was a lot more time consuming. The formula basically says that if your scoring margin averages +27 points a game or more you should be expected to win all of your games. On the flip side, if you average -27 points a game you should be expected to lose all of your games. If you look at USC and Florida’s 2008 seasons, their losses to Oregon State and Mississippi, respectively, can be considered largely unexpected upsets, especially in retrospect.

As far as being penalized for having a high expected winning percentage, I wouldn’t say that’s necessarily the case. I noticed that a lot of the coaches who’ve been in contention for the national title as of late generally perform extremely close to their expected performance based on the formula. It might have more to do with the fact that Urban Meyer, Mack Brown, Pete Carroll, etcetera spend a lot of their time blowing people out and less time playing close games. They might play one or two close games a year, win some and lose some, and it evens out over five years. Guys like Kelly at an upstart like Cincinnati will spend a lot more time playing close games (in my mind) and so there’s a lot more opportunity for swings one way or the other.

So, I wouldn’t say that Mark Richt is a better coach than Urban Meyer or even that he’s more clutch than Urban Meyer. Meyer has built Florida up to the point where he’s closing most teams out in the first half and so the point differential holds up well, where Richt and Georgia play a lot more close games. That’s a credit to Meyer’s program building. All the metric is saying is that they’ve lost some close games and that there may be a bit of an overall trend.

Note: This is only a starting point and a fun theoretical exercise. There are probably better statistical methods for analyzing this.  I used the average scoring margin for two reasons:

1. In past sets of data that I’ve worked with, the data tends to turn into a cloud. Using five years of data points rather than just combining them into a single average data point would probably show a lot more noise.
2. For simplicity, it was a lot easier and less time consuming to just use the average and work with those numbers.

At the End of the Day…

There is luck in the game of football. There are lucky plays and coaches definitely have lucky seasons. The eternal question is what can you do to create your own luck? Can you practice good habits, stay disciplined and focused, be in better condition than your opponent, and stay positive in the face of adversity? Is that really luck? I think it’s something more. It’s a state of mind. It’s one team being consistently tougher than it’s opponent. It’s one coach having his team prepared to be physically and mentally ready to perform under pressure. Toughness isn’t a word anyone would use to describe a Notre Dame football teams fielded in the Weis era, whose one lucky season (2006) is more of a credit to Brady Quinn (in addition to an amazing Jeff Samardzija play) than it is to Weis. Brian Kelly’s 2008 Cincinnati team went through five quarterbacks and still made it to a BCS game playing a schedule that wasn’t a whole lot weaker than Notre Dame’s (Sagarin schedule ratings: ND – 50; Cincinnati – 60).

One thing we can expect from Brian Kelly that we didn’t see from Charlie Weis, despite the latter’s bravado, is some toughness. I think this statistic helps bring some of that difference to light: the outlier in Kelly’s resume is the one season with a negative “clutch” rating. The outlier in the Weis resume is the one season with an overall positive rating. Positives in general have been outliers in South Bend for a while now. But there are some signs, beyond players puking during stretching, that we’ll see more Fighting and less Limp-wristed Irish football in the Kelly era.

Similar Posts:

• A Theory on College Football Performance: Part 3 – Game Day Performance
• 2009 Elite Selection Playoff: Week Fourteen and BCS Championship Predictions
• A Theory on College Football Performance: Part 2 – Institutional Factors

This article is © 2007-2012 by De Veritate, LLC and was originally published at Clashmore Mike. This article may not be copied, distributed, or transmitted without attribution. Additionally, you may not use this article for commercial purposes or to generate derivative works without explicit written permission. Please contact us if you wish to license this content for your own use.