Capabilities Enabled:
Created a quantitative business impact proof-of-concept model to increase training effectiveness.
Impact Created:
Summary:
DAU had a $216.7 million budget and they wanted to ensure they were accountable for those tax dollars by maximizing the effectiveness of their learning investments. They had commissioned a number of studies employing survey and qualitative data to examine the impact of their training investments. These prior studies indicated a positive impact on outcomes. That still did not meet the leaders need to find quantitative proof of training impact. SPP partners designed a quantitative impact measurement model as a proof-of-concept that answered the question, “What is the relationship between the number of courses completed and business outcomes? Is there a cost growth reduction or increase because of the relationship?” They also wanted to know if there was a per-course reduction or increase in costs. Training and business data were collected across multiple agencies; multiple regression models and advanced analytics were employed to answer the research questions. Additionally, the ROI of the learning measurement proof-of-concept is directly tied to the ROI of the programs measured. In this study, the agency learned that $556,913 (-4.2%) in cost-growth reduction could be attributed to training. The proof-of-concept model, process and results were recognized by the Learning & Development industry as a best measurement approach.
About the Client:
Defense Acquisition University is a federal agency with a mission to provide a global learning environment to develop qualified acquisition, requirements and contingency professionals who deliver and sustain effective and affordable warfighting capabilities.
The Challenge:
After multiple attempts to deliver quantitative proof of training impact to defend budgets and prove value, the agency was in immediate need for credible, defensible, and quantitative proof. The greatest barrier to demonstrating quantitative proof was limited access to business outcome data given the political structure. Not only was program outcome data limited, but the data that were available had limited integrity and was often inaccessible. Removing barriers to data collection and adjusting the measurement plan in real time was the greatest change effort.
The Solution:
Smarter People Planning (SPP) Partners worked with DAU to design a robust quantitative study draft the research questions and secure the necessary training and employee demographic data across nine years. Then SPP reviewed core agency documentation and proactively initiated and facilitated cross-agency meetings to identify and collect business outcome data on mission critical projects. Once all data were collected, advanced statistical techniques such as multiple regression analyses were used to answer the research questions. The greatest benefit was the fact that DAU successfully conducted the first quantitative evaluation of their training programs supporting mission-critical projects. The results have been socialized across agencies.
The Outcome:
The outcomes of the project demonstrate that a trained workforce accounts for a statistically significant portion of mission-critical program cost reduction. That translates to billions of dollars savings. This evaluation program is also an indicator that the agency’s training may be impacting business outcomes (i.e., recurring unit cost growth) in many undiscovered ways that require further exploration.