Import Automation - Engage

The following documentation applies to Campus Labs Import Automation. For questions concerning this process, please contact your Campus Labs representative or email Campus Labs Support.  

Overview

Institutions with the appropriate license for Engage can use Import Automation to add or update user data. This is intended for institutions that need to update such information on a recurring basis.  

How it Works

The actual process of Import Automation is simple: your campus transfers certain files to our secure server, and we will automatically process any such files, import eligible data, and create a summary of the import so that you can retrieve and verify the results. 

Requirements

Before you can use Import Automation, your campus must first be contracted for this option. The ability to use import automation is not included in every campus’s contract.  

Next, you must obtain a transfer account with Campus Labs. This account uses secure file transfer protocol (SFTP) in order to receive and transmit files, but also can use our Transfer page to perform many of the same operations.  

In order to receive this access, we require authorization from one of your institution's Primary Contacts. This ensures that only users with express permission are granted access to the institution's SFTP site. Please have your Primary Contact send a message to [e-mail] in order to receive authorization.  

We will also need your email address, as the username and password information will be sent to you via Microsoft Secure email. Once you have that information, you should verify that you are able to connect to the site. If you are unable to do so, contact Campus Labs Support for assistance.   

Directory Structure

Once your connection has been established, you should see folders related to each application. Your institution will have Write access to all these directories but will not be able to alter existing directories or create new ones. See your product documentation for the expected folder structure.  

The file structure will only be created once your import training has been completed and your first batch of import files have been successfully processed without issue. This is so that Support can verify that the data sent is accurate, and that the correct files are being sent. 

File Transmission

For the file imports to be processed, you must place the import files into the appropriate directory. This is a key part of the Import Automation process. Without it, the system will not know the appropriate action to take upon a given file, or for which application the file is intended. If you place your files outside of the appropriate folder, an error will occur.  

You can transfer files through whatever method you choose, but this transmission must occur over Secure Shell File Transfer Protocol (SFTP). 

Any number of import files may be placed into the directory for processing, but we encourage you to consolidate their data into the smallest appropriate number of files. 

Transmission Manifest

For the import to activate, you must also include a Manifest file. This Manifest file is an XML document that indicates the names of the files to be processed and must have a DONE extension. When this is detected, it indicates that the file transfer is complete, and the files can be imported.  

A sample manifest file can be found below. The Manifest may contain as many file entries as appropriate, but there must be at least one. Also, the checksum value for each file line must either be an MD5 checksum hash or be blank. The checksum value may be left blank, or set to “none”, if you are unable to generate a checksum. Including a checksum will prevent incomplete files from being processed, so we recommend that this value be included whenever possible.  

The optional checksum hash type value must be “md5” if a checksum is included in the appropriate attribute.  

MD5 checksums for files can be generated through native commands in your operating system. Contact your operating system support vendor for assistance.  

Manifest Validation

When the manifest file is written into the appropriate directory, the system will examine the import file and validate the contents. This involves the following steps: 

1. The Manifest file is renamed to have the PROCESSING extension. If the system cannot rename the file it will make a second attempt and, if it fails again, move on to any additional .done files in the directory. 
2. The Manifest file’s contents are validated, and we determine if the XML is valid. If it is not, the file will be renamed to have the INVALIDSCHEMA extension. 
3. If the Manifest is valid, we’ll validate each file entry in the Manifest. This reference validation includes the following steps, which will be repeated for each referenced file: 
   1. Verify that the filename exists in the directory. If not, the Manifest file will be renamed to CONTAINSINVALIDFILES. 
   2. If a matching file name is found in the directory, we’ll generate an MD5 checksum for the file and compare it to the checksum listed in the file line. If they do not match, the manifest file will be renamed to CONTAINSINVALIDFILES. 
   3. If the two checksums do match, we’ll check to see if automatic imports have been enabled for your campus. If it has not, the manifest file will be renamed to IMPORTCONFIGMISSING. If this occurs, please contact [e-mail] for further information. 

After all these checks are completed successfully, the import will start processing.  

Data Processing

If your file passes the listed test, it will start processing. The reference file is copied to an outside location so that it can be processed without being overwritten, and we’ll track the results of the action performed for each record. This will produce the following four output files: 

• Errors.csv: a list of any records that could not be processed, along with the corresponding details. This file will be overwritten each time processing occurs, so that it always represents the errors that resulted from the most recent occurrence of import automation. 
• Errors.csv copy:  a duplicate copy of the above error file above will be written into the “errors” folder of the directory. The file’s name will be a timestamp of when the file was created. This copy will act as the archival record of errors. 
• Results.txt: a summary of the import process. This will be overwritten each time processing occurs, so that it always represents the latest result summary from the most recent occurrence of import automation. 
• Results.txt copy: a duplicate copy of the summary file will be written into the “results” folder. The file’s name will be a timestamp of when the file was created. This will act as the archival record since the aforementioned “results.txt” will only contain information on the most recent occurrence of import automation. 

For security purposes, after processing is complete the data file will be permanently removed from the directory into which it was transmitted. The manifest file will be removed as well, but only if none of the file references included in it contained errors that an institution would need to see.