SIMS 7 - Assessment Concepts in Detail

Warning

This document was first published in 2010 and applies to SIMS 7 only.  Supported partners may wish to contact us if additional help is needed.

Background

Assessment Manager is area of SIMS used to record and manage results of any kind which need to be stored against a student. 

Assessment is one of the key areas of data that BECTA has identified for publication to parents and one of the areas where data is typically generated and stored in diverse systems such as VLEs in addition to the MIS. 

It is therefore a key area for data exchange if we are to provide a holistic view of a student’s achievements to parents, staff and students alike.

Whilst technically it is possible to code against the same business objects and processes that are used in the SIMS UI, we have provided some simple XML formats for exchanging key assessment data in a format similar to the SIF (System Interoperability Framework) formats.

Assessment Structures

Aspect Types

The following types of aspect are supported in SIMS.

Type	Supported in SIMS	Supported in SIF
Grade	Yes	Yes
Decimal	Yes	Yes
Integer	Yes	No – Use Decimal
Comment	Yes	Yes
Age	Yes	No

 

Grade Sets

Grade sets are required elements for Aspects of type Grade.  A grade set may have multiple value sets.  A simple example of this is Formula 1

Pre 1/1/2010		Post 1/1/2010
Grade	Value	Grade	Value
First	10	First	25
Second	8	Second	18
Third	6	Third	15
Fourth	5	Fourth	12
Fifth	4	Fifth	10
Sixth	3	Sixth	8
Seventh	2	Seventh	6
Eighth	1	Eighth	4
		Ninth	2
		Tenth	1

 

So a grade set defines the set of allowable grades for an aspect at a given point in time and the relative values of each grade.

Within SIMS Grade Set definitions you would see

Or click on the version 01/01/2010

Supplier

Some assessment artefacts have the concept of an owner.  When SIMS is installed, a unique identifier based on the school name and number is generated and this is used to define who created the ‘Aspect’.  Whilst this may seem trivial, it has a large impact on the Assessment System.  If you want to be able to provide a standard assessment to multiple schools then it needs to have the same supplier and identity.  Within a SIMS installation, artefacts that were created by a supplier other than the local system are not editable which significantly restricts what the school can do with them!  Partners need to define a suitably unique supplier id for themselves (25 characters) and ensure that all of their artefacts are from them as a supplier. 

ESS would expect partners not to distribute artefacts with supplier ids based on ‘Green Abbey’, ‘Water’s Edge’, ‘CCS’, ESS or ‘SIMS’ for obvious reasons.

Aspects

The term aspect is used to define a mark or result within SIMS.  An Aspect is comparable to a LEARNERRESULTCOMPONENT within the SIF model. 

Aspects now have unique GUIDs which are used within our Partner interfaces to identify them.  Aspects also have an external id which should be unique and is used through older import routines to match up. Partners should assume that their Aspects will be exchanged between schools using any means available.

Where aspects are of type decimal or integer the following additional information is required.

Where aspects are of type grade they require the following additional information

Comments require no further information and grades require the following:

Whilst there are ‘classifications’ that can be applied to aspects these are local to the school and not supported by the Partner Interfaces.

Aspects have the following set of data populated when they are saved.

Becomes

for our example.  The module will always be CES Assessment Services and this is outside of Partner control.   

Result Set

A result set is a device used to give a specific result a purpose.  For example and end of term assessment ‘result set’ may be created and applied to say English, maths and science effort grades.  The teacher may collect weekly effort grades but there would only be one grade used on the report.

The only data entered by the user is the basic section.  Similar issues would apply to suppliers and IDs as they do for aspects.

Results

Results can be recorded as follows:

With a Result Set

One mark can be recorded for any aspect, student and result set.  Any further mark for this combination will overwrite the previous mark.

Without a Result Set

One mark can be recorded for any aspect, student and calendar day.  Any further mark for this combination will overwrite the previous mark.   Hence a historical performance record is maintained for ‘standard’ results.

Results are validated against their aspect and grade set where appropriate. 

Result validation is date based because the grade sets and value ranges are editable over time.

Results dates are critical – using the current date as the date for the result will create incorrect results for students.  For example if we were to create a program that saves resends results on a daily basis to SIMS

Result date        Who      Result  

10/01/2010       Fred      Grade B

10/01/2010       Jane      Grade C

If the application used the current date for results then it might create an apparent history as follows:

10/01/2010       Fred      Grade B

11/01/2010       Fred      Grade B

12/01/2010       Fred      Grade B

13/01/2010       Fred      Grade B

14/01/2010       Fred      Grade B

 SIMS does provide a mechanism for Fred to be given the same grade for an aspect on another day, but the data would suggest that Fred had been assessed 5 times when this was not the case.  Worse if the data was corrected to a grade C on 15/01 then the data would be completely wrong

10/01/2010       Fred      Grade B

…

14/01/2010       Fred      Grade B

15/01/2010       Fred      Grade C

WARNING

The use of today to save results is usually an error unless the marking process posts the result back to SIMS without delay.

Template

A template is a device used to group a set of Aspects for collection within a mark sheet.  This is where an aspect is tied to a result set within SIMS for collection.

 

We have added the F1 result to the template and made it a result tied to a result set ‘End of Term Assessment’.

Mark sheets can then be created for most types of groups within SIMS, simply by pressing add on marks sheets and choosing the applicable group.

Mark Sheet

Once saved, a mark sheet can then be opened as in the figure above.

Saving and then right click on a result will allow the editing and display of the mark history.

Partner Interfaces

There are two key partner designed interfaces one for import and export. These interfaces allow for the following:

Artifact	Import	Export
Grade Set	Yes*	Yes
Aspect	Yes*	Yes
Results	Yes	Yes#
Result Sets	No	Yes

# ESS cannot conceive of a valid reason to export every single mark ever held for a student.  A full export would also be slow.  We have made a filter compulsory on export for these reasons.

* Aspects and Grade Set scan only be imported if a supplier is created in Agencies and the name matches the supplier name within the import.

Only the 2 required fields need be filled in but this is deemed to be an agreement that artifacts from that provider may be added to the SIMS system.  Users should only add the agency if they trust the provider and believe that they will provide imports within the terms of their assessment policy.

Export / Import Process

Both interfaces are XML based and similar to standard SIF formats.  This does mean that you cannot use integer or age aspects at this time.

The divergence from the SIF format is in the support for Result Sets which have no SIF equivalence.

The basic rules for import are that any artifact that your import is dependent on must exist before the import can succeed.

So for the following result import to work

Result	Grade Set	Result Set	Student	Date
A	1	2	3	29/02/2010

 

• Grade set 1 must exist
• Result set 2 must exist
• Student 3 must exist
• The date must be valid
• The result must be a valid grade on the import date.

Real time versus scheduled transfers

 There must be a balance between real time and annual transfers of data.  It is possible to approach real time exchange but this may add significant loads at both ends if not done well.

A model where a scheduled application asks say a VLE service ‘have you got data transfer requests’ is likely to be optimal. 

Scheduled application runs every 5 minutes on a suitable server.

Scheduled application checks to see whether it is already running and exits if it is.

Schedule application makes a web service call to the VLE – what requests do you have.

Scheduled application gets a list of jobs

              For each job

                           Scheduled application invokes a web service to request the job details.

                           Scheduled application runs job.

                           Scheduled application invokes web service with data returned and /or errors

This mechanism minimizes data exchange to that demanded solely by the task requested. It also allows for the sender to understand in basic terms if a job has succeeded or possibly why it failed.

WARNING

Partners need to be aware that validation messages are unlikely to be passed to the user responsible in real time and in a way that is clear and unambiguous to the user.  It is therefore critical that marks are validated at the point of entry and hence reduce errors (or at least restrict errors to those that can be fixed automatically – i.e. by retransmission.

When can marks be exchanged (and what shouldn’t be)?

It is important that Partners consider the traceability of marks.  If we assume for example that the PE department had a grade set:

Grade	Meaning
A
B
C
M	Excused Swimming (Period)
I	Excused Medical
P	Pregnant Pupil
Z	Forgot kit

 

Whilst it could be argued that these are inappropriate, nothing stops a school from creating something similar.

Publishing the fact that a pupil is/was pregnant to parents would breach the data protection act and be highly risky from an ESS / Partner perspective.  In this case, SIMS has done nothing wrong and SIMS Learning Gateway will only publish to parents, results marked for publication.

It is the opinion of ESS that Partners should exercise caution when exporting marks from SIMS.  We would recommend that marks are only exported at the request of a member of staff and that all such requests are logged.  In that way the erroneous display of this type of information can be shown to be at the user’s specific request which then clearly renders the fault with the school rather than the computer systems.  

Partners are also advised to make schools fully aware and in control of any data exported from SIMS.  Particular care must be taken with data intended to be shown to students or parents.  Partners are advised to make the school and their users fully aware that they need to manage the export and publication of MIS data and that they are responsible (demonstrably so) for any such export. Consideration should be given to including warnings on screen, in documentation and in any EULAs.

 

During training ESS staff will remind partners of these potential issues and the need to attribute exports to known users.  By virtue of ESS’s market position, we are unable to ‘prevent’ data export and/or record why it was shared.

Further Guidance for Schools

The mechanisms provided for partner data exchange in the assessment area do not include mark sheets or templates.  In SIMS the only way that marks are effectively used is via the mark sheet paradigm.  Partners are therefore advised to remind schools of the need to create templates and mark sheets for results imported from partners. 

Whilst it is theoretically possible to use SIMS business objects to create templates and mark sheets, they are relatively complex and rely upon the provider having all of the additional information available to enable creation and update.  We would recommend that schools assessment coordinators manage the creation of templates and mark sheets because they should be aware of the data being imported.