Data Integration Best Practices

The purpose of this article is to assist data managers with planning, building, and automating data files for their Campus Labs integration. It includes best practices learned from professional services projects conducted with member campuses by the Campus Labs Data Integration team. Specifications for each file are laid out in the Core Data Dictionary, but there are many questions unique to your campus and the SIS implementation to consider when writing scripts to create these files. Additional guidance on these issues and how to approach the integration project is included here. 

Process and policy considerations

Data mapping and scripting require collaboration between functional and technical leaders on campus. These conversations inform decisions made throughout the data integration process and may lead to changes in policy and process on campus. The questions below can help guide these discussions. 

• How is data maintained in the SIS? – Data integration may reveal gaps and other issues in your SIS data. Supporting this new use case may require you to reassess how you are collecting and maintaining your data and commit to data cleanup projects.
• When are you sending data? – Defining the scope of the project helps to focus the scripting efforts. Discuss which terms should be brought over and when, including what historical data, if any.
• What data are you sending? – Consider which unique records are needed in your Campus Labs products. Discuss whether there are types of users, courses, sections, etc. that should be excluded from the data integration.
• How will this data be used? – The products you have licensed influence the choices you make in data integration. Discuss your implementation priorities and analytic goals.

Order of build

Note that the order in which you should load data files is not necessarily the order in which you should audit and extract the data. Due to the interrelated nature of the data, writing scripts is an iterative process to ensure that logic aligns across multiple areas.

1. Review the academic terms tables and determine how terms will be structured in the Campus Labs ecosystem.
2. Review the courses and sections tables before building your organizational units. The identifiers used to assign courses and sections to organizational units must match the identifiers used to define those units; therefore, you must ensure that these codes are present in the course tables within the SIS. Use the data found in the SIS as a foundation and work with your consultant to determine the best organizational structure for your institution. You may need to do some cleanup within the SIS to assign organizational codes to individual courses.
3. Decide which courses and sections are included and how they are defined before considering section attributes, instructors, enrollments, and academic programs. The course and section identifiers and logic must align across each of these areas.
4. Determine which instructors, staff, student enrollments, and advisors (if applicable) should be included. All these individuals need accounts.

Once you have completed these steps, you will have the information you need to fully build out all necessary scripts.