We’ve been supporting a lot of RTOs lately with completing their assessment validation. As RTO consultants and RTO experts, we’ve conducted literally hundreds of validation sessions. Running an effective validation session can take a bit of practice.
The Standards for RTO’s 2015 require all RTOs to ensure that at least 50% of its Scope of Registration is validated in the first three years of RTO registration and 100% every five years.
It’s often seen as a painful process that may feel a bit like a box-ticking exercise; however, a well-conducted validation session can truly add value to quality outcomes in your RTO. It can help develop the skills of your trainers/assessors, help them feel more confident in their roles and your RTO’s processes, and provide valuable feedback and insights into what’s happening ‘on-the-ground’ once your assessment system and tools have been put in place.
So before we dive in too deep, let’s look at what validation is.
The process of validation requires an RTO to confirm:
- The RTO’s assessment system and processes produce valid results
- The RTO’s assessment practices are valid
- That students who have been assessed as Competent have the required skills and knowledge described in the relevant training product.
You may have heard around the sector that there are two types of validation.
- A process to confirm that an assessment tool meets training package requirements conducted prior to its use is often referred to as Pre-Validation; and
- Post-Assessment Validation is where the process confirms, through the review of student evidence, that valid assessment decisions have been made.
However, although two processes are often involved in a thorough validation session, these terms have probably added even more confusion about what a validation process is and what shouldn’t be counted as validation.
Because validation involves checking that your RTO’s tools, practices and system have produced valid assessment results, ‘Pre-Validation’ which doesn’t include the review of how assessment tools are used and interpreted, cannot be considered a complete validation process.
We like to refer to what has been considered Pre-Validation as an Assessment Tool Quality Review. This is where you confirm that an assessment tool covers all requirements of the unit/s of competency, and the tool, if used consistently, will support the Principles of Assessment to be followed and the collection of evidence that meets the Rules of Evidence.
Post-Assessment Validation – which is what this article will focus on and what should be considered Validation – is a process of reviewing the assessment tool AFTER its use. Validation incorporates a review of students’ completed assessments as well as a review of the assessment tool and system.
The purpose is to confirm that students who have been assessed using the RTO’s assessment system, processes and tool, do have the required skills and knowledge, as outlined in the training package requirements.
In this process, we compare the evidence produced by completed student assessments against each other and against the components of the assessment tool, in line with strategic questions – to determine that the assessment process is working.
We are looking at:
- Whether the assessment tool covers the requirements of the unit of competency/ies
- Whether the assessments have been conducted in line with Principles of Assessment and Rules of Evidence in practice.
This involves:
- Looking at whether student responses are lining up to the assessor/marking guide
- Looking at whether assessors are marking different students and between assessors accurately
- Confirming each unit of competency requirement is covered
When done properly, the validation process is a great way to check in on students, your assessors and your assessment tools to see where things are going well and where improvement might be needed.
So, let’s get into our top tips for effective validation in your RTO.
1. Plan What to Validate and When
- You must have a Validation Plan to ensure that 50% of the items on your RTO’s Scope of Registration are validated in the first 3 years of each registration period, and all scope items are validated every 5 years.
- Your plan should include WHO, WHAT, WHEN & HOW validation will be conducted.
2. Use a Good Validation Tool
A good validation tool will guide your validation participants through the process. It will ask the right kinds of questions and guide the validation team on comparing and contrasting evidence.
Some useful questions to guide a validation session include:
- Does the Assessment Tool cover the unit of competency/ies requirements sufficiently?
- Have students provided the evidence required by the assessment tool and is the evidence appropriate?
- Are consistent assessment decisions being made – from cohort to cohort and from assessor to assessor?
- Are students consistently interpreting assessment instructions?
- Are assessors consistently interpreting assessment instructions and filling in the tool correctly?
- Have any students been marked as Competent when they should not have been?
- Does the evidence demonstrate for each student that they have the required skills and knowledge of the unit?
Your validation tool should also capture:
- The unit/s being validated.
- What documents were reviewed as part of the validation, including version numbers.
- Who was involved in the validation, their role in the validation and their credentials.
- The sample size, how it was calculated and how the samples were selected.
- Outcomes of the validation and actions to be taken.
3. Get the Right People Involved in Validation
In addition to the specifications in the Standards for RTOs 2015 , it is important that you make sure the people involved are familiar with the assessment tool and/or have current industry experience so they can advise whether student responses meet industry standards.
It is also beneficial to involve the trainers and assessors who are involved in the delivery and assessment of the unit/s as it gives them an opportunity to see things from a different perspective. In many of the validations we have conducted, trainers have found the process incredibly valuable to their practice as a trainer & assessor and have taken their ‘assessing game’ to another level.
Note: the assessor who made the assessment judgements being reviewed is not able to determine the validation outcomes. This is the role of the lead validator. In this way, bringing in an RTO Quality Consultant to lead your training and assessment team through the process can be useful.
4. Ensure you Have a Statically Valid Sample
For validation to be compliant, you must be conducting validation on a statistically valid sample. This means you must validate a certain number of assessment judgements and you must select the students to be included randomly.
To determine your sample size, run a report on the students who have completed the relevant unit within the past 6 months. Then identify a valid sample size (ASQA provide a calculator here to help you work this out https://www.asqa.gov.au/resources/tools/validation-sample-size-calculator ).
Once you have the number of students, use a random selection method to identify which students will be included in this validation.
5. Ensure you Have Required Documents
As well as the validation tool and student evidence to be reviewed, there are a number of documents its useful to have available at a validation session for reference.
This includes:
Complete Assessment Tool. This includes:
- Marking Guide, including any and all assessor checklists or observation checklists.
- Assessment Task Mapping
- Student Assessment Instructions
- Any record-keeping tools (e.g. assessment task cover sheet, assessment record, workplace visit reports)
- Any placement or workplace documentation, third-party reports, simulation documents, supporting documents etc.
It is also recommended to have on hand:
- Your RTOs Training and Assessment Strategy
- RTO Training and Assessment policies and procedures
- Your RTOs Validation Schedule and register
6. Plan your Time
Depending on your sample size, there can be a lot of evidence and content to look at during a validation session. Therefore, planning your time before the session so you can make sure you get through what you need to is a good idea. This might mean looking at how many students you need to review and dividing your time between the number of students that need to be looked at. You will also need time to explain the process to everyone involved and discuss findings and outcomes.
7. Conduct an Assessment Tool Quality Review Prior
It can help the process to do an Assessment Tool Quality Review prior to reviewing the assessment judgements.
This ensures you are familiar with the tool before reviewing student files, which will support understanding if students are supplying the right type of responses/evidence.
8. Common Things to Look Out for in a Validation Session
When reviewing each assessment judgement, review each student’s evidence against the marking guide and comment in your validation tool regarding the quality of the student responses in line with the marking guide, as well as the assessor’s recordkeeping (or lack thereof).
Key things to look for are:
- Does the student response match the marking guide? Is it less than? More than? Is there a different response?Note: In some instances, you might find the student has provided a correct response that isn’t in the marking guide – this would indicate that the marking guide needs updating to allow for a range of responses (if appropriate) or that the assessment needs to be updated to ensure it asks the question in a non-ambiguous way.
- Have all questions been answered?
- Has the assessor provided sufficient feedback?
- Is it clear where a student has been marked NS and why? In these instances, is it also clear that there has been a re-submission?
Note: We see many assessors marking students’ work as incorrect without formerly providing an NS response in the record-keeping tool but rather allowing the student to amend their responses and resubmit without following formal channels. This creates a lot of ambiguity around the assessment process, is usually not in line with the organisation’s policies and procedures and can make it difficult to determine the process followed. - Are there questions that students have consistently answered incorrectly? And if so – could the question be re-worded / changed to support students in understanding what the question means and provide a correct answer?
- Are assessors marking to the same standard? Is this in line with the marking guide?
- Are the assessment conditions of the unit being met?
- Is there adequate evidence to support the decision about competency?
- Has the assessment been carried out in line with the RTO’s assessment system/policies and procedures?
Ultimately, when reviewing the student assessment evidence, are you confident that the student has the skills and knowledge to go into the workplace and carry out the workplace functions as required in the Unit of Competency?
9. Document Findings
Write all of the feedback and comments in your validation tool in the relevant sections.
Ensure you keep track of which completed student judgements had inconsistencies or areas to be improved. A good validation tool will have a section summarising all required actions following the validation session.
The lead validator must also decide whether there are any critical issues that make the assessment system non-compliant with the principles of assessment, rules of evidence or the training package requirements.
If so, decide whether or not the training product needs to be validated further (extending the sample size) or again soon in the future. This may be required in order to identify whether the issues pertain to other units or other students and/or have been remedied with the actions undertaken as a result of the validation. If yes, the validation schedule should be updated accordingly and communicated with the validation team.
10. Make Improvements
In line with your organisation’s continuous improvement processes, carry out and act upon the suggested recommendations.
This step is an integral part of the validation process, it is not enough just to carry out the validation. You are required to act upon the findings. This step completes the validation process.
11. Retain evidence.
You must retain evidence of everything gathered and reviewed during the assessment validation process.
For example:
- Student outcome report indicating completed students within the last six months
- Statistically valid sample size
- Random selection method used, and the resulting list of students
- Validation session participants
- Assessment tools reviewed
- Student assessment outcomes reviewed
- Policies and procedures
- Completed validation tool
- Register of outcomes and actions taken
- Any other evidence supplied/viewed during the validation process. This may also include a new assessment tool (if changes were required to be made), re-assessed student work, and other such evidence.
Not feeling confident to leading your own assessment validation, or simply too strapped for time?
Engage our team of highly experienced Consultants today.
Phone: 1300 761 141
