26 Nov Are We Ready for the Legislative Session? Yes!

As the Director of Institutional Research and Chief Data Analyst for the North Dakota University System (NDUS), my role involves responding to data requests from legislators, agencies, the governor’s office, and other stakeholders during the legislative session. With the 69th Legislative Assembly beginning on January 7, 2025, I am excited about the upcoming session. In the past, this task was challenging due to the manual nature of our processes—running data files, combining them, and manipulating data in spreadsheets and pivot tables. This method was not only time-consuming but also prone to errors, especially during sessions with numerous bills affecting NDUS. It often felt like trying to find a needle in a haystack, while simultaneously building the haystack and dealing with it being on fire!

However, significant advancements in the data world since the last biennium have revolutionized our processes. These improvements have made our work more efficient and insightful, allowing us to respond to legislative requests for data more quickly and accurately. This transformation preserves our role as data champions and ensures we can meet the demands of the upcoming session with greater ease.

Now, we are using Microsoft Fabric and Azure Databricks, which have transformed our data handling processes. Microsoft Fabric is an end-to-end analytics and data platform that integrates various tools into a user-friendly solution, making it easier to manage and analyze data. Azure Databricks is a unified analytics platform that helps process and analyze large amounts of data, combining the capabilities of data warehouses, data lakes, and generative AI.  Here are the key changes:

• Data Models: We have developed robust data models that streamline data integration and analysis. This has removed the work required to run and combine data files. Data models are standardized, validated, and always up to date.
• Dashboards: Using the data models, we have created a variety of dashboards to analyze our data. These dashboards are available to stakeholders who can explore and use the data independently. They provide real-time insights and are more user-friendly than spreadsheets and pivot tables.
• AI: Artificial Intelligence has been integrated into our work life through Microsoft Copilot and Databricks Genie spaces. Microsoft Copilot is incredibly helpful in many areas, from conducting research much faster than a typical web browser to writing or rewriting text and performing complex statistical analyses quickly. It’s like having a super-smart assistant who never sleeps and always has the correct answers. Databricks Genie spaces provide an AI-powered no-code interface, allowing business users to conduct self-serve data analytics using natural language. This integration has significantly enhanced our ability to handle data efficiently and accurately, making our processes more streamlined and user-friendly.

The benefits of these changes are obvious. We now have more accurate data, faster processing times, and better insights. This allows us to make more informed decisions and improve our overall efficiency. It’s like upgrading from a bicycle to a high-speed train – we’re getting to our destination faster and with much less effort. Our quality of life has improved, and we are having fun!

To sustain these improvements, however, we need more resources, primarily in terms of staff. The skills required to do what we are doing – curating data models, setting security, designing and building dashboards, and developing data governance – are not the typical list of skills embodied by a single employee. Our team includes a cloud solutions architect, data architect, security team member, business intelligence analyst, institutional researcher, and data steward, all committing a piece of their overall time to these data projects. Investing in skilled personnel, dedicated entirely to the effort of transforming how we do data, will ensure that we continue to leverage these advancements and drive further innovation. Think of it as building a dream team – with the right players, we can achieve anything.

As a data person I’m more comfortable with numbers than words, so writing a blog article is challenging for me. As mentioned above, over the past year I’ve relied on AI to assist me in my work. When I clearly convey my ideas to co-pilot, the AI produces better content faster than I can. I can then refine or edit the output to create the final product. In full disclosure, AI assisted me in creating this blog article. I began with an outline, collaborated with AI to generate content, and manually finalized it to match my vision and style. This collaboration made it easy to produce a comprehensive and (hopefully) engaging article efficiently. This is evidence of how AI has improved my skill set.

As we continue to embrace these technological advancements, we’re not just improving—we’re revolutionizing our approach to data. Imagine the endless possibilities: ability to respond to data requests within minutes, predictive analytics with pinpoint accuracy, real-time insights that drive proactive decision-making, and a dynamic, data-rich environment that continuously adapts to our needs. This is not merely a step forward; it’s a quantum leap into the future of data management. Let’s harness this momentum and transform our data-driven dreams into tangible results. The future awaits, it’s brimming with potential, and we are prepared for the challenge. Let the session commence!

================================================================

Dr. Jennifer Weber is the Director of Institutional Research and Chief Data Analyst for the North Dakota University system.  Her primary functions are to oversee the department and provide system level enrollment reporting to the State Board of Higher Education.  Jennifer also manages system-wide IR Shared Services, works closely with the State Longitudinal Data System (SLDS) and serves as the state coordinator for federal reporting.   As the NDUS-IR is also contracted through the North Dakota Department of Public Instruction (NDDPI) for data analysis and reporting, the NDUS-IR department is ultimately responsible for the data of all students attending public institutions in the state of North Dakota, pre-kindergarten through graduate school.