SFTP Data Exchange Instructions

Secure File Transfer Protocol (sFTP)

Step 1: Connecting to the Server

Step 2: Request to send data via sFTP

Step 3: Importing Your Data

    Directory Structure

    File Transmissions & Manifest

    Data Processing

Secure File Transfer Protocol (sFTP)

Institutions with the appropriate license for certain Campus Labs products may utilize a special process known as Import Automation for automating the transfer and import of "flat-files"(i.e., comma-separated or "CSV" spreadsheets containing data).  Import Automation is primarily intended for institutions that wish to import or update user information into select Campus Labs products on a recurring basis. Campus Labs provides a file server that can act as a temporary storage location for data files that institutions wish to securely transfer to Campus Labs for processing.

Once your campus has created your files for import you can then send them to us using the sFTP option.

Step 1: Connecting to the Server

In order to connect to the Campus Labs file server, an institution must utilize a file transfer client capable of supporting SFTP.

Use the following settings:

• Host: ftp.campuslabs.com (US)
• Port: 22
• Protocol: SFTP – SSH File Transfer Protocol
• Logon type: Normal
• User and password are provided via phone
• Do not enter anything for “Account”

Step 2: Request to send data via sFTP

An individual appropriately-designated by an institutional contract known to Campus Labs must provide an electronic communication to Campus Labs requesting the creation of an account on the Campus Labs secure file server. If you are going through the implementation process your technical consultant will walk you through this step during the process. The following details should be included in the account request that can be sent to [e-mail].

• First name
• Last name
• Title/position
• Campus email address
• Office telephone number

Step 3: Importing Your Data

Campus Labs will set up a sFTP file directory for you.  Within that directory there will be a folder for each of the different files you will be sending.  Drop the completed file into the corresponding folder along with a manifest file(.done file) which will indicate the data file is complete and ready to be imported.  

Directory Structure

Once a secure file server connection has been successfully established, an institution will see a folder hierarchy (n.b., some of the folders shown below may or may not appear in your directory based on the products/features your institution licenses or has implemented).  An institution will have write access to their directories but will not be able to alter existing directories or create new ones.

Beacon Specific

\Beacon\BeaconImports	Student/Advisor connections into Beacon
\Accounts\Photo	Institution-uploaded photo of student/faculty/staff in Beacon

Engage Specific

\CollegiateLink\Users

Accounts and demographics into Engage

Core Data

File path	Description	Max File Size (MB)
\Accounts\AccountImports	Creates and updates account	40
\Courses\OrgUnit	Organizational Units	10
\Courses\AcademicTerm	Academic terms(e.g., Fall 2017, Spring 2017)	10
\Courses\Course	Courses (e.g., BIO 101)	100
\Courses\Section	Course sections	100
\Courses\SectionAttribute	Course section attributes	100
\Courses\Enrollment	Course section student enrollment records	100
\Courses\Instructor	Course section instructor assignment records	100
\Courses\RemoveInstructor	Removing instructor assignments from course sections	100
\Courses\AcademicProgram	Academic programs(collection of course IDs)	10
\Demographics\StudentImport	Student demographic data for existing accounts	20
\Demographics\FacultyImport	Faculty demographic data for existing accounts	20

Validation & Testing Imports

File path	Description	Max File Size (MB)
\ValidationAccounts\AccountImports	Creates and updates account.  Will also accept bulk photos	40
\ValidationCourses\OrgUnit	Organizational Units	10
\ValidationCourses\AcademicTerm	Academic terms(e.g., Fall 2017, Spring 2017)	10
\ValidationCourses\Course	Courses (e.g., BIO 101)	100
\ValidationCourses\Section	Course sections	100
\ValidationCourses\SectionAttribute	Course section attributes	100
\ValidationCourses\Enrollment	Course section student enrollment records	100
\ValidationCourses\Instructor	Course section instructor assignment records	100
\ValidationCourses\RemoveInstructor	Removing instructor assignments from course sections	100
\ValidationCourses\AcademicProgram	Academic programs(collection of course IDs)	10

Please note if a file is inadvertently placed outside of the appropriate folder (e.g, institution transmits a file into the root directory instead of a specific sub-folder) then an error will occur.

File Transmissions & Manifest

A file of any name with the extension ".done" must be placed into the appropriate directory in order to signify that the transmissions of the file from the institution is complete and that the import of any files referenced inside the manifest should begin.  The ".done" file is an XML document that must conform to the following layout:

"accounts" would be replaced with the name of the file you are sending.

 

• The .done file may contain as many <file...> lines as appropriate.  At least one file name is required.
• The checksum value for each <file...> line must either be an MD5 checksum hash/string or completely blank.  The option to leave this value blank is there for the convenience of institutions that do not have the technical capability to generate a checksum.  
  • It is highly recommend that a checksum be included however, to increase the integrity of the verification process which is meant to guard against the chance that data file corruption occurred in the process of transmitting the data file to the secure file server.
• The checksum hash type value must be "md5" (n.b., letter case of this string does not matter) if a checksum is included in the appropriate attribute of the <file...> line.  If a checksum is not included, then the checksum hash type value must be set to "none"(n.b., letter case of this string does not matter) or simply left empty (e.g., <filename="accounts.csv"checksum=""checksumhashtype=""/>).

When the manifest file is written into the appropriate directory the system will examine the file and attempt to perform numerous validations on its contents.

1. The system renames the .done file to "{filename}.PROCESSING" so that it is clear that Import Automation has begun.  If the system cannot rename the file it will make a second attempt and upon further failure, move on to any additional .done files in the directory.
2. The system will read the contents of the "{filesname}.PROCESSING" file and ensure that the structure of the XML is valid.  If the XML is invalid, the file will be renamed to "{filename}.INVALIDSCHEMA".
3. If the structure of the file is determined to be valid, the system will begin validating the references within each <file...> line.  This reference validation includes the following steps, which will be repeated for each referenced file:
   1. The system will verify a file matching the name listed exists in the directory.  If a matching file is not found an "INVALID" attribute will be added to the <file...> line indicating that the reference is invalid and the manifest file will be renamed to "{filename}.CONTAINSINVALIDFILES".
   2. If a matching file is found in the directory, the system will generate an MDS checksum for the file and compare it to the checksum listed in the <file...> line.  If the two checksums do not match an "INVALID" attribute will be added to the <file...> line indicating that the reference is invalid and the manifest file will be renamed to "{filename}.CONTAINSINVALIDFILES".
   3. If the two checksums match, the system will check the configuration information managed by Campus Labs, which defines what actions should be taken on the file given the directory it was placed into.  If configuration information is not available or is inaccessible (i.e., if a database access issue occurs) the manifest file will be renamed to "{filename}.IMPORTCONFIGMISSING".  If this occurs, please contact [e-mail] for further information.
   4. If a vaild configuration for your institution's import is found, the system will proceed with the data processing steps required for importing the records contained in the actual data file.

When importing multiple types of data, the data transfer script must first create the .csv file, followed by the .done file and then pause for 15-30 minutes to allow adequate time for the previous file to process before creating the next .csv file.

Data Processing

When a single <file...> line within the manifest file has been validated, data processing will occur. Data processing involves the following steps:

1. The system will copy the reference file to a location outside the directory to which it was transmitted so that further action can occur without concern for the file being over-written during data processing.
2. For security purposes, after processing is complete the data file will be permanently removed from the directory into which it was transmitted.  The manifest file will be removed as well, but only if none of the file references included in it contained errors that an institution would need to see.

Imports are not instantaneous and can take at least 15-30 minutes before completing.

Campus Data Managers can go into the Core Data Management site to the see the status of the Core Data sFTP files. The status for these files will show as complete, incomplete, or failed. The failed files will come with an error attachment.