Local API - Assessment Concepts

Overview

What do we mean by Assessments?

George got a ‘B’ in his maths test on the website of Online Test Provider. His teacher, Miss Beale, wants to enter the marks for George and the rest of the class in SIMS. 

Miss Beale would start by checking if there is an appropriate maths test Aspect.  She will need to bear in mind that everyone else in the maths department also defines a maths test and these are usually very different as the students pass through the school.  For completeness, we will assume that this doesn’t exist.

Miss Beale would have to first decide if there was an appropriate Grade Set available. If none is available, she would create one.

• Grade Set Name:  Year 4 Autumn Term Maths Test Grades (Mrs Beale)
• Date Range From: Now until <For ever>
• Grades:
  • A                  10 points
  • B                   8 points
  • C                   6 points
  • F                   0 points.

NB: The names need to be specific and unique to avoid confusion.

She would then need to define an Aspect (mark definition) to collect the grades.

• Aspect Name: Year 4 Autumn Term Maths Test (Mrs Beale)
• Aspect Type: Grade
• Grade Set: Year 4 Autumn Term Maths Test Grades (Mrs Beale)

Miss Beale would need to create a Template, which is a list of column headings for a mark sheet.

• Each column defines an Aspect and optional result set (a purpose for the mark) and other optional items.  We only need to set the Aspect to the one created above.

Once this has been done, Miss Beale would create a Mark Sheet for her group and then she would be able to enter George’s mark and those of his classmates.

• At the point of entry the marks are validated and errors can easily be seen and corrected.

Once the marks are in SIMS, there is a significant range of analysis tools and reporting options available to the MIS users.

Importing the marks from external systems automatically

Any expectation that steps above can be left out or avoided are likely to be disappointed.  Quite the reverse is true!

Stage 1: Apples and pears

We need to match up the mark definitions that we have in the VLE with the mark definitions in SIMS.  Given that the key validation point must be when the mark is entered, the VLE needs to validate the entry against the set of valid marks in SIMS before the mark is entered.

There are 2 ways of doing this.

Option A: Using School-defined Aspects and Grade Sets.

• If the marks collected are based on existing standards within the school then the correct route is to export:
  • All Aspects
  • All Grade Sets
  • [All result sets / Assessment Periods]
• From SIMS to the VLE.
• The VLE then (before marks are collected ideally) links their mark definitions to the SIMS mark definitions.

Marks are then validated on entry and are easily transferable to SIMS.

Option B: Using VLE supplier defined Aspects and Grade Sets.

• If the VLE provided offers standard tests typically across more than one school then the owner of the mark definition should be the VLE provider.  The VLE provider would also own and manage their own grade sets and possibly result sets.
• When data is exported to SIMS, it will come complete with the VLE provider’s definitions and hence can be imported without issue.
• Each provider needs to decide which is most appropriate for them.  Typically a mix of owners is likely to be more problematic unless great care is taken.
• It is critical that if SIMS is used to generate Aspects, Grade Sets and other artefacts on behalf of a Technical Integrator then the SIMS training database MUST be reset to have unique GUID for supplier and other artefacts. 

Technical Integrators MUST contact the Partner Support Team before generating these from training data.  Externally generated artefacts should be fine!

Stage 2 – Moving the marks from A to SIMS.

There a many ways to move the data by data we usually mean results along with their definitions.  Usually, the simplest one is to put an application on a server in the school / data centre and schedule an export/import daily. 

The key thing is that the school controls what runs and there are many excellent examples of Technical Integrator systems populating their systems in this way and returning data to SIMS.

NB: Trying to push data into SIMS where the external provider is in control will usually fall foul of firewalls and security rules.

Please note that data centres may charge for hardware and software maintenance on Technical Integrator data extraction utilities. This is because it actually costs time and resources to upgrade these systems and support them within the data centre.  On a school server, this effort is usually fairly minimal and the resources are internal to the school benefitting from the application.

If the data transfer is divorced from the data entry error messages are likely to lack meaning and be difficult to get back to the member of staff who can fix the fault.    That is why there is an emphasis on validation at point of entry.

There is a check on the supplier name in SIMS assessment imports.  The customer needs to create an agency with the same as the supplier or SIMS will refuse to import the data.

 

Stage 3 - Customers using the imported data in SIMS

At this point, there are results in SIMS.  There will be no templates or mark sheets which are the typical way that SIMS users would view results in SIMS.  In other words, the transferred data is invisible in SIMS.

If the VLE is creating a set of aspects and they have some form of coherence when grouped in sets then it might be worth them creating templates for their aspects in a copy of SIMS modified to ensure that supplier and other identifiers are unique.  These can then be exported to an XML file and simply imported by the schools.  This is an easy task but there is no simple automation process for template imports.

If there is no coherent grouping then templates can be created by staff as in the manual mechanism described above.

In both cases, once the template has been created, it is a simple task to make mark sheets from the template.

If the aspects, grade sets and result sets are exported from SIMS to the VLE, we assume that schools would have already created suitable templates and mark sheets.

Interim Conclusion

The value of the automation process which moves marks from a VLE (or similar) to the MIS is significant and a key step forward identified by BECTA before their demise.

It is also critical that SIMS Technical Integrators engaging in providing a facility to export data to SIMS consider this guidance carefully so that we can ensure that the quality of the data is maintained across the transaction.

Detail Dive for the Assessment System

​

Introduction

Assessment Manager is area of SIMS used to record and manage results of any kind which need to be stored against a student. 

Assessment is one of the key areas of data that BECTA has identified for publication to parents and one of the areas where data is typically generated and stored in diverse systems such as VLEs in addition to the MIS. 

It is therefore a key area for data exchange if we are to provide a holistic view of a student’s achievements to parents, staff and students alike.

Whilst technically it is possible to code against the same business objects and processes that are used in the SIMS UI, we have provided some simple XML formats for exchanging key assessment data in a format similar to the SIF (System Interoperability Framework) formats.

Assessment Structures

Aspect Types

The following types of aspect are supported in SIMS.

Type	Supported in SIMS	Supported in SIF
Grade	Yes	Yes
Decimal	Yes	Yes
Integer	Yes	No – Use Decimal
Comment	Yes	Yes
Age	Yes	No

 

Grade Sets

Grade sets are required elements for Aspects of type Grade.  A grade set may have multiple value sets.  A simple example of this is Formula 1

Pre 1/1/2010		Post 1/1/2010
Grade	Value	Grade	Value
First	10	First	25
Second	8	Second	18
Third	6	Third	15
Fourth	5	Fourth	12
Fifth	4	Fifth	10
Sixth	3	Sixth	8
Seventh	2	Seventh	6
Eighth	1	Eighth	4
		Ninth	2
		Tenth	1

 

So a grade set defines the set of allowable grades for an aspect at a given point in time and the relative values of each grade.

Within SIMS Grade Set definitions you would see

Or click on the version 01/01/2010

Supplier

Some assessment artefacts have the concept of an owner.  When SIMS is installed, a unique identifier based on the school name and number is generated and this is used to define who created the ‘Aspect’.  Whilst this may seem trivial, it has a large impact on the Assessment System.  If you want to be able to provide a standard assessment to multiple schools then it needs to have the same supplier and identity.  Within a SIMS installation, artefacts that were created by a supplier other than the local system are not editable which significantly restricts what the school can do with them!  Partners need to define a suitably unique supplier id for themselves (25 characters) and ensure that all of their artefacts are from them as a supplier. 

ESS would expect partners not to distribute artefacts with supplier ids based on ‘Green Abbey’, ‘Water’s Edge’, ‘CCS’, ESS or ‘SIMS’ for obvious reasons.

Aspects

The term aspect is used to define a mark or result within SIMS.  An Aspect is comparable to a LEARNERRESULTCOMPONENT within the SIF model. 

Aspects now have unique GUIDs which are used within our Partner interfaces to identify them.  Aspects also have an external id which should be unique and is used through older import routines to match up. Partners should assume that their Aspects will be exchanged between schools using any means available.

Where aspects are of type decimal or integer the following additional information is required.

Where aspects are of type grade they require the following additional information

Comments require no further information and grades require the following:

Whilst there are ‘classifications’ that can be applied to aspects these are local to the school and not supported by the Partner Interfaces.

Aspects have the following set of data populated when they are saved.

Becomes

for our example.  The module will always be CES Assessment Services and this is outside of Partner control.   

Result Set

A result set is a device used to give a specific result a purpose.  For example and end of term assessment ‘result set’ may be created and applied to say English, maths and science effort grades.  The teacher may collect weekly effort grades but there would only be one grade used on the report.

The only data entered by the user is the basic section.  Similar issues would apply to suppliers and IDs as they do for aspects.

Results

Results can be recorded as follows:

With a Result Set

One mark can be recorded for any aspect, student and result set.  Any further mark for this combination will overwrite the previous mark.

Without a Result Set

One mark can be recorded for any aspect, student and calendar day.  Any further mark for this combination will overwrite the previous mark.   Hence a historical performance record is maintained for ‘standard’ results.

Results are validated against their aspect and grade set where appropriate. 

Result validation is date based because the grade sets and value ranges are editable over time.

Results dates are critical – using the current date as the date for the result will create incorrect results for students.  For example if we were to create a program that saves resends results on a daily basis to SIMS

Result date        Who      Result  

10/01/2010       Fred      Grade B

10/01/2010       Jane      Grade C

If the application used the current date for results then it might create an apparent history as follows:

10/01/2010       Fred      Grade B

11/01/2010       Fred      Grade B

12/01/2010       Fred      Grade B

13/01/2010       Fred      Grade B

14/01/2010       Fred      Grade B

 SIMS does provide a mechanism for Fred to be given the same grade for an aspect on another day, but the data would suggest that Fred had been assessed 5 times when this was not the case.  Worse if the data was corrected to a grade C on 15/01 then the data would be completely wrong

10/01/2010       Fred      Grade B

…

14/01/2010       Fred      Grade B

15/01/2010       Fred      Grade C

WARNING

The use of today to save results is usually an error unless the marking process posts the result back to SIMS without delay.

Template

A template is a device used to group a set of Aspects for collection within a mark sheet.  This is where an aspect is tied to a result set within SIMS for collection.

 

We have added the F1 result to the template and made it a result tied to a result set ‘End of Term Assessment’.

Mark sheets can then be created for most types of groups within SIMS, simply by pressing add on marks sheets and choosing the applicable group.

Mark Sheet

Once saved, a mark sheet can then be opened as in the figure above.

Saving and then right click on a result will allow the editing and display of the mark history.

Partner Interfaces

There are two key partner designed interfaces one for import and export. These interfaces allow for the following:

Artifact	Import	Export
Grade Set	Yes*	Yes
Aspect	Yes*	Yes
Results	Yes	Yes#
Result Sets	No	Yes

# ESS cannot conceive of a valid reason to export every single mark ever held for a student.  A full export would also be slow.  We have made a filter compulsory on export for these reasons.

* Aspects and Grade Set scan only be imported if a supplier is created in Agencies and the name matches the supplier name within the import.

Only the 2 required fields need be filled in but this is deemed to be an agreement that artifacts from that provider may be added to the SIMS system.  Users should only add the agency if they trust the provider and believe that they will provide imports within the terms of their assessment policy.

Export / Import Process

Both interfaces are XML based and similar to standard SIF formats.  This does mean that you cannot use integer or age aspects at this time.

The divergence from the SIF format is in the support for Result Sets which have no SIF equivalence.

The basic rules for import are that any artifact that your import is dependent on must exist before the import can succeed.

So for the following result import to work

Result	Grade Set	Result Set	Student	Date
A	1	2	3	29/02/2010

 

• Grade set 1 must exist
• Result set 2 must exist
• Student 3 must exist
• The date must be valid
• The result must be a valid grade on the import date.

Real time versus scheduled transfers

 There must be a balance between real time and annual transfers of data.  It is possible to approach real time exchange but this may add significant loads at both ends if not done well.

A model where a scheduled application asks say a VLE service ‘have you got data transfer requests’ is likely to be optimal. 

Scheduled application runs every 5 minutes on a suitable server.

Scheduled application checks to see whether it is already running and exits if it is.

Schedule application makes a web service call to the VLE – what requests do you have.

Scheduled application gets a list of jobs

              For each job

                           Scheduled application invokes a web service to request the job details.

                           Scheduled application runs job.

                           Scheduled application invokes web service with data returned and /or errors

This mechanism minimizes data exchange to that demanded solely by the task requested. It also allows for the sender to understand in basic terms if a job has succeeded or possibly why it failed.

WARNING

Partners need to be aware that validation messages are unlikely to be passed to the user responsible in real time and in a way that is clear and unambiguous to the user.  It is therefore critical that marks are validated at the point of entry and hence reduce errors (or at least restrict errors to those that can be fixed automatically – i.e. by retransmission.

When can marks be exchanged (and what shouldn’t be)?

It is important that Partners consider the traceability of marks.  If we assume for example that the PE department had a grade set:

Grade	Meaning
A
B
C
M	Excused Swimming (Period)
I	Excused Medical
P	Pregnant Pupil
Z	Forgot kit

 

Whilst it could be argued that these are inappropriate, nothing stops a school from creating something similar.

Publishing the fact that a pupil is/was pregnant to parents would breach the data protection act and be highly risky from an ESS / Partner perspective.  In this case, SIMS has done nothing wrong and SIMS Learning Gateway will only publish to parents, results marked for publication.

It is the opinion of ESS that Partners should exercise caution when exporting marks from SIMS.  We would recommend that marks are only exported at the request of a member of staff and that all such requests are logged.  In that way the erroneous display of this type of information can be shown to be at the user’s specific request which then clearly renders the fault with the school rather than the computer systems.  

Partners are also advised to make schools fully aware and in control of any data exported from SIMS.  Particular care must be taken with data intended to be shown to students or parents.  Partners are advised to make the school and their users fully aware that they need to manage the export and publication of MIS data and that they are responsible (demonstrably so) for any such export. Consideration should be given to including warnings on screen, in documentation and in any EULAs.

 

During training ESS staff will remind partners of these potential issues and the need to attribute exports to known users.  By virtue of ESS’s market position, we are unable to ‘prevent’ data export and/or record why it was shared.

Further Guidance for Schools

The mechanisms provided for partner data exchange in the assessment area do not include mark sheets or templates.  In SIMS the only way that marks are effectively used is via the mark sheet paradigm.  Partners are therefore advised to remind schools of the need to create templates and mark sheets for results imported from partners. 

Whilst it is theoretically possible to use SIMS business objects to create templates and mark sheets, they are relatively complex and rely upon the provider having all of the additional information available to enable creation and update.  We would recommend that schools assessment coordinators manage the creation of templates and mark sheets because they should be aware of the data being imported.

​