It’s probably a rule somewhere -- when you take federal funds, you have to give something back -- usually a report. When the funds are in grants that improve math and science teaching across the state of Illinois, the reports have to be sophisticated and multi-dimensional, covering quality of professional development activities and partnerships as well as changes in teachers’ content knowledge, teachers’ instructional practices, and students’ achievement.

“I have been doing the evaluations since 2007 for this grant,” said Elizabeth Oyer, the Ph.D. statistician who is director at EvalSolutions, “and this is the first year we have done a centralized evaluation.”

The state expanded on an existing data portal, the Data Driven Instruction Portal, created through an original partnership between the Illinois Area V Learning Technology Center, EvalSolutions, and RJO Technologies. Data are collected through the portal via surveys and tests. Some data are imported into the portal. EvalSolutions provides a variety of evaluation services, and for the current project, completed cross-site meta-analyses of local evaluation results and hierarchical linear modeling to analyze local and global outcomes. Data sources are quantitative and qualitative, including achievement scores, implementation data, interviews, and extant data.

In the last wave of the grant, the state shifted the program evaluation from a local focus with different strategies, tools, and analyses to a centralized, shared evaluation framework with common assessments and processes. With 14 grants servicing about 200 school districts, 380 teachers, and 6,000 students, the state faced an analytical challenge previously completed by the participating grantees -- universities, regional offices of education and school districts.

Using XLeratorDB as part of its workflow, EvalSolutions was able to automate the analysis process for grantees, which in previous years might have spent weeks working with an external evaluator cleaning and analyzing data to prepare for the federal reporting. The more efficient centralized evaluation reduced the cost of the local evaluations and allowed for more funds to go to programming. In addition, it provides the state with more consistent evaluation across the projects. Finally, this centralized approach allows for more flexibility at the state level because EvalSolutions can provide technical support regionally rather than requiring every grant to have its own evaluator to complete this reporting process.

“We needed to add to our reporting and be able to produce an affordable solution that included statistical output based on the data going into the site, which is functionality not easily available on a site that is free to users,” said Oyer. Last June she discovered XLeratorDB from Westclintech, a SQL Server based calculation engine that has the flexibility of SQL along with the power and data security of a database.

Implementation of XLeratorDB required a few tweaks to meet the demands of the education evaluation system, but Westclintech developed them quickly.

Initially the results returned by the software for t-Test functions did not include all of the statistics (like degrees of freedom) required by education industry reporting standards, said Oyer. So she emailed the company, which launched a software improvement project that delivered what she needed in weeks.

“You can’t beat that,” she said.

In past years, getting the results in and revised was a time and resource-intensive process.

“We used to have to support and remind people throughout the spring to make sure they were getting the data, then give them a deadline to submit the report, and we had 30 days to review it and we had another process to revise it.” When there were data problems, it was months before the state evaluation team knew and often too late to respond. The centralized evaluation data collection plan provided a more consistent, timely data tracking solution for the state.

“We have to tailor the report to the federal requirements.” EvalSolutions uses Telerik software to provide bar charts and descriptive tables from a SQL database. These descriptive statistics are supplemented with the inferential statistics output provided through XLeratorDB to support interpretation of the statistical significance of changes observed in the data.

“We could have figured out how to compute it with the data tables, but then it would have been an inflexible hard-wired solution. We have to be able to spit out the results in different formats, and we could have done that with Excel and a human, but that is time and resource-intensive and doesn’t provide consistency across users.”

Now, instead of a months-long process for reporting, EvalSolutions and state evaluation team members convene one-day long regional meetings with grantees to provide on-site technical support while the grant participants complete the analysis portions of their federal reports.

“We have a structured training guide where project directors complete portions of federal report before the face-to-face session. The on-site training allows state evaluators to support the directors as needed to complete the reporting requirements. We do that once a year. We have done it once and it worked beautifully. Project directors seemed very happy with it.”

No wonder.

“If they had their own evaluators and had to clean their own data and write the results, it could take them weeks.” Many directors submitted their final reports for final state review at the end of the training that day.

For university staff who want to use the data for research, they can get it from the Data Driven Instruction Portal any way they want it such as raw data, descriptive text, raw data with actual values in text or in the pre-formatted reports that align with the federal requirements.

With a centralized data portal and EvalSolution’s analytics using XLeratorDB, completing annual federal grant reports has become more efficient and less resource-intensive.

Contact Information:
Email: eoyer@evalsolutions.com
Twitter: @elizabethoyer
Blog: http://evalsolutions.wordpress.com
Website: www.evalsolutions.com