Trial Review

Technical difficulties have been reported by some users of the search function and is being investigated by technical staff. Thank you for your patience and apologies for any inconvenience caused.

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been endorsed by the ANZCTR. Before participating in a study, talk to your health care provider and refer to this information for consumers
Trial registered on ANZCTR


Registration number
ACTRN12618000437268
Ethics application status
Approved
Date submitted
15/02/2018
Date registered
27/03/2018
Date last updated
27/03/2018
Type of registration
Retrospectively registered

Titles & IDs
Public title
Evaluating the Thinking Maths program for building the capacity of middle-school mathematics teachers
Scientific title
Evaluation of the impact of the South Australian Department of Education and Child Development Thinking Maths Program on middle-school students (Years 6-9) mathematics achievement
Secondary ID [1] 294034 0
Nil known
Universal Trial Number (UTN)
Trial acronym
Thinking Maths
Linked study record

Health condition
Health condition(s) or problem(s) studied:
Mathematics anxiety 306586 0
Condition category
Condition code
Other 305680 305680 0 0
Research that is not of generic health relevance and not applicable to specific health categories listed above

Intervention/exposure
Study type
Interventional
Description of intervention(s) / exposure
The Thinking Maths program has been developed by the South Australian Department of Education and Child Development (SA DECD), based on its Teaching for Effective Learning Framework. The program aims to address a significant drop in mathematics performance in NAPLAN from Year 7 to Year 9.
The program supports Year 6-9 teachers in the deep learning of mathematical content as outlined in the Australian Curriculum Mathematics. In particular, the project focuses on quality task design, sequencing of conceptual development, and research-informed effective pedagogies.
It involves five days of teacher professional learning (workshops), run over 6 months, by facilitators who model rigorous teaching and learning processes, undertaking tasks with multiple entry and exit points to differentiate the curriculum to cater for students with a wide range of mathematical experience and dispositions.
The full-day, face-to-face workshops are delivered by two expert facilitators to groups of approximately 30 Years 6-9 mathematics teachers. The facilitators, who administer the intervention, are DECD personnel and the original developers of the intervention.
The topics covered on each day have a content focus and a pedagogical focus: Day 1 - Patterns and Generalisation & Differentiating Learning; Day 2 - Space and Measurement & Effective Questioning; Day 3 – Geometry & Active and Collaborative Learning; Day 4 – Statistics & Personalising and Connecting Learning; Day 5 - Location, Directed Number and Geometry & Teaching for Understanding.
Between each of the five workshops (periods of 4-5 weeks), teachers make a commitment to implement high-gain strategies to improve student achievement and engagement. Between sessions, telephone, email and online platforms support teachers’ improvement efforts in a professional learning community. There is an expectation that teachers access support and the online professional community of practice, however, it is at their discretion. The Thinking Maths facilitators are proactive and reach-out if teachers are slow to engage.
The facilitators monitored adherence to the intervention on several levels. Workshop attendance is closely monitored and if a participant cant make a particular day, they are encouraged to attend a parallel session on another day or send a colleague. Between workshops, they correspond regularly and work closely with participants to provide support, advice and to value-add to the workshop learnings. The study staff monitored adherence to the intervention through the completion of Workshop feedback forms, which included questions about how teachers were implementing strategies back in their classroom.
The evaluation timeline spans a year, commencing with the mandated completion of a state-wide student mathematics test (PAT Maths) in September the previous year. Once schools, teachers and their respective maths classrooms have been recruited, student’s mathematics achievement (primary outcome) will be retrospectively provided by DECD to the study team. The intervention commences in Term 1 (pre-surveys administered; schools randomised to control/treatment; PL workshops commence) and finishes in September Term 3, two-weeks after the last PL workshop (post-surveys administered; PAT Maths administered).
Given that a decline in mathematics performance is also evident in Australian students’ results more widely, the findings from this trial may be of interest to other Australian practitioners and policy makers. To date, there has been no rigorous evaluation of the impact this intervention has on student achievement in mathematics.
Intervention code [1] 300311 0
Treatment: Other
Comparator / control treatment
The trial will be structured as a randomised control trial (RCT), with pairs of Year 6-9 Maths teachers recruited from 167 South Australian government schools. 120 recruited teachers (57 pairs and 6 individuals in 63 schools) will receive the intervention and the other 198 teachers (94 pairs and 10 individuals in the remaining 104 schools) will be on a wait-list and act as a business-as-usual control. The control group will be offered the intervention in Term 4 2017, as soon as the evaluation period has finished and post-tests have been collected.
Control group
Active

Outcomes
Primary outcome [1] 304773 0
The primary outcome in this evaluation is student achievement in mathematics, as measured by the standardised PATMaths test routinely completed by all students in South Australian government schools since 2015.
Timepoint [1] 304773 0
T1 - Pre-test (September 2016, 5 months prior to intervention commencement in Term 1 2017)
T2 - Post-test (September 2017, 2 weeks after intervention completion)
Secondary outcome [1] 343135 0
Students’ mathematics anxiety and low self-concept (SASE)
Designed for study based on existing validated items and scales (eg PISA)
Mean score of 10 items measured on a five-point Likert scale of Strongly disagree (1) to Strongly agree (5), with high internal reliability (a=0.89).
• I get nervous doing maths problems
• I get worried when I have to do maths homework
• Maths is hard to understand so I just try to learn the steps
• I worry that I will get poor grades in maths
• I don’t like it when the teacher asks me questions in maths
• I usually do well in maths (reverse scored)
• I am just not good at maths
• I learn things quickly in maths (reverse scored)
• I am good at working out difficult maths problems (reverse scored)
• Maths is harder for me than any other subject
Timepoint [1] 343135 0
T1 - Pre-survey (2 weeks prior to commencement of intervention)
T2 - Post-survey (2 weeks after completion of intervention)
Secondary outcome [2] 343136 0
Students’ cognitive engagement (SCOG)
Designed for study based on existing validated items and scales (eg PISA)
Mean score of five items measured on a five-point Likert scale of Strongly disagree (1) to Strongly agree (5), with high internal reliability (a=0.81).
• I know what my teacher expects me to do
• I am interested in what my teacher says
• My teacher is easy to understand
• My teacher believes all students can be good at maths
• My teacher gives me interesting things to do in maths
Timepoint [2] 343136 0
T1 - Pre-survey (2 weeks prior to commencement of intervention)
T2 - Post-survey (2 weeks after completion of intervention)
Secondary outcome [3] 344631 0
Students’ metacognitive strategies (SMET)
Designed for study based on existing validated items and scales (eg PISA)
Mean score of five items measured on a five-point Likert scale of Strongly disagree (1) to Strongly agree (5), with moderate internal reliability (a=0.76).
• I check my maths school work for mistakes
• I try to connect the things I am learning in maths with what I already know
• I like to try and use what I learn in maths, in real life
• When I become confused about something in maths, I go back and try to figure it out
• I make up my own maths problems to test my understanding
Timepoint [3] 344631 0
T1 - Pre-survey (2 weeks prior to commencement of intervention)
T2 - Post-survey (2 weeks after completion of intervention)
Secondary outcome [4] 344632 0
Students’ learning through effective teaching practice (SETL)
Designed for study based on existing validated items and scales (eg PISA)
Mean score of 16 items measured on a five-point Likert scale of Never (1) to Always (5), with high internal reliability (a=0.89).
• My teacher asks me or my classmates to present our mathematical thinking
• My teacher asks questions to check whether we have understood what was taught
• At the beginning of a maths lesson, the teacher reminds us of what we did in the previous lesson
• My teacher asks me to explain my answers
• Our maths assignments and homework make me think hard about what I'm learning
• My teacher makes sure I understand before we move on to the next topic
• My teacher gives different work to classmates depending on their ability (e.g. to those who have difficulties learning or to those who finish quickly)
• We work in groups to come up with joint solutions to a problem
• My teacher asks us to guess or estimate the answer to a problem before we calculate it
• My teacher asks me questions in maths that challenge my thinking
• My teacher tells me about how well I am doing in my maths class
• My teacher talks to me about what I need to do to become better in maths
• My teacher shows an interest in every student’s learning
• My teacher gives extra help when students need it
• My teacher continues teaching until we understand
• My teacher gives students an opportunity to express opinions and ask questions
Timepoint [4] 344632 0
T1 - Pre-survey (2 weeks prior to commencement of intervention)
T2 - Post-survey (2 weeks after completion of intervention)
Secondary outcome [5] 344633 0
Teacher professional identity and self-efficacy (TPID)
Designed for study based on existing validated items and scales (eg TALIS)
Mean score of seven items measured on a five-point Likert scale of Not at all (1) to A great deal (5), with high internal reliability (a=0.89).
When teaching maths, to what extent can you do the following:
• Engage all students
• Help your students think critically
• Improve the understanding of a student who is failing
• Motivate students who show low interest in maths
• Help your students value maths learning
• Help students to believe they can do well in maths
• Create opportunities for all students to experience productive struggle
Timepoint [5] 344633 0
T1 - Pre-survey (2 weeks prior to commencement of intervention)
T2 - Post-survey (2 weeks after completion of intervention)
Secondary outcome [6] 344634 0
Teacher pedagogical and content knowledge (TPCK)
Designed for study based on existing validated items and scales (eg TALIS)
Mean score of 10 items measured on a five-point Likert scale of Not at all (1) to A great deal (5), with high internal reliability (a=0.91).
How confident are you in the following areas:
• Designing learning with the Australian Curriculum mathematics proficiencies
• Designing learning with the Australian Curriculum mathematics content
• Knowing the mathematics developmental learning progression across Years 6 and 9
• Differentiating your teaching of the Australian Curriculum Mathematics
• Creating and maintaining a mathematical learning environment that challenges all students
• Creating and maintaining a mathematical learning environment that supports creative and critical thinking
• Using questioning to develop students' conceptual understanding
• Using questioning to diagnose students' conceptual misunderstandings
• Identifying students' learning challenges
• Providing timely feedback to students
Timepoint [6] 344634 0
T1 - Pre-survey (2 weeks prior to commencement of intervention)
T2 - Post-survey (2 weeks after completion of intervention)
Secondary outcome [7] 344635 0
Teacher beliefs about mathematics learning (TBEL)
Designed for study based on existing validated items and scales (eg TALIS)
Mean score of three items measured on a five-point Likert scale of Strongly disagree (1) to Strongly agree (5), with low internal reliability (a=0.68).
• I deeply believe that everyone can learn maths
• You are either good at maths or you’re not (reverse scored)
• Some students are probably never going to be good at maths (reverse scored)
Timepoint [7] 344635 0
T1 - Pre-survey (2 weeks prior to commencement of intervention)
T2 - Post-survey (2 weeks after completion of intervention)

Eligibility
Key inclusion criteria
Eligible schools need to meet the following criteria:
• Government school located in South Australia
• Cater for students in Years 6-7 and/or Years 8-9 (K-12 Area schools are counted as one site)
• School have not previously received the Thinking Maths (or equivalent) intervention
• Teachers teach a Year 6, 7, 8 and/or 9 class in mathematics and have not previously participated in the Thinking Maths (or equivalent) intervention (mixed classes or small schools may also include Year 5 and Year 10 students)
Minimum age
8 Years
Maximum age
16 Years
Sex
Both males and females
Can healthy volunteers participate?
Yes
Key exclusion criteria
• Non-government schools in South Australia
• Early Years teachers and teachers of Years 1-4 or Years 11-12
• Teachers who have previously participated in the Thinking Maths intervention

Study design
Purpose of the study
Educational / counselling / training
Allocation to intervention
Randomised controlled trial
Procedure for enrolling a subject and allocating the treatment (allocation concealment procedures)
Schools are recruited through an expression of interest and their random allocation to Control or Treatment is concealed until after T1 assessments have been completed. Schools are de-identified and given an ID number by a researcher independent of the study. Allocation is undertaken using this de-identified sampling frame, which only includes the number of teachers from each school (ie 1 or 2). Allocation is done by the study statistician, aware of which group is treatment (n=120 teachers) and which group is control (remaining teachers), but blind to where schools/teachers have been allocated.
Methods used to generate the sequence in which subjects will be randomised (sequence generation)
Simple random sample using SPSS function - select random sample
Masking / blinding
Open (masking not used)
Who is / are masked / blinded?



Intervention assignment
Parallel
Other design features
Phase
Not Applicable
Type of endpoint/s
Efficacy
Statistical methods / analysis
The primary student outcome measure, student mathematics achievement (PATM17), will be analysed using a hierarchical linear model (HLM) to reflect the nested nature of the data and the method of assignment, with students nested within classes, within schools. The student model will include individual student’s prior PATM16 score. Intervention and control groups will be compared by including an intervention indicator at the school level, where Intervention = 1 and Control = 0.

Recruitment
Recruitment status
Completed
Date of first participant enrolment
Anticipated
Actual
5/09/2016
Date of last participant enrolment
Anticipated
Actual
7/10/2016
Date of last data collection
Anticipated
Actual
30/11/2017
Sample size
Target
6000
Accrual to date
Final
7367
Recruitment in Australia
Recruitment state(s)
SA

Funding & Sponsors
Funding source category [1] 298660 0
Other Collaborative groups
Name [1] 298660 0
Social Ventures Australia
Country [1] 298660 0
Australia
Primary sponsor type
Commercial sector/Industry
Name
Australian Council for Educational Research
Address
19 Prospect Hill Road, Camberwell, VIC 3124
Country
Australia
Secondary sponsor category [1] 297828 0
None
Name [1] 297828 0
Address [1] 297828 0
Country [1] 297828 0

Ethics approval
Ethics application status
Approved
Ethics committee name [1] 299614 0
Australian Council for Educational Research
Ethics committee address [1] 299614 0
19 Prospect Hill Road, Camberwell, VIC 3124
Ethics committee country [1] 299614 0
Australia
Date submitted for ethics approval [1] 299614 0
22/11/2016
Approval date [1] 299614 0
20/12/2016
Ethics approval number [1] 299614 0
CS1600075-1.16

Summary
Brief summary
Thinking Maths has been developed by the South Australian Department of Education and Child Development (SA DECD), based on its Teaching for Effective Learning Framework. The program aims to address a significant drop in mathematics performance in NAPLAN from Year 7 to Year 9.
The program supports Year 7 and Year 8 teachers in the deep learning of mathematical content as outlined in the Australian Curriculum Mathematics. The project focuses on the following three areas for better teaching and learning of mathematics:
• Quality task design;
• Sequencing of conceptual development and;
• Research-informed effective pedagogies
Five professional learning days are run over two terms by facilitators who model rigorous teaching and learning processes, undertaking tasks with multiple entry and exit points to differentiate the curriculum to cater for students with a wide range of mathematical experience and dispositions.
After each session teachers make a commitment to implement high gain strategies to improve student achievement and engagement. Between sessions, telephone, email and online platforms support teachers’ improvement efforts in a professional learning community. At each session after the first, there are three to four presentations from participants to share their experiences, successes and challenges.
The SA DECD has identified the drop in mathematics performance between Year 7 and Year 9 as a critical state-wide issue and has designed this program in response to it.
In order to inform the Department’s understanding of the effect and cost-effectiveness of the program, a rigorous and independently-funded trial is useful. As a decline in mathematics performance is also evident in Australian students’ results more widely, the findings from this trial may be of interest to other Australian practitioners and policy makers, as well.
To date, there has been no rigorous evaluation of the impact this program has on student achievement in mathematics. The program however uses approaches which have a basis in research evidence,
Teachers in the program receive explicit feedback from peers on their instruction and feedback, on average, yields 8 months’ learning progress for students. Further, the professional learning has been designed with key findings from Timperly et al’s Teacher Professional Learning Best Evidence Synthesis (2007) in mind.
The program evaluator, the Australian Council for Educational Research (ACER), is the leading independent Australian organisation in educational research. The trial is structured as a clustered randomised control trial (RCT), with pairs of Year 7 and Year 8 Maths teachers recruited from 167 South Australian government schools. 120 recruited teachers (57 pairs and 6 individuals in 63 schools) will receive the intervention and the other 198 teachers (94 pairs and 10 individuals in the remaining 104 schools) will as act as a business-as-usual control.
Trial website
http://evidenceforlearning.org.au/lif/current-projects/thinkingmaths/
Trial related presentations / publications
Hollingsworth H & Dix, K (2016). Evaluation Protocol: Thinking Maths. Evidence for Learning, Sydney. http://evidenceforlearning.org.au/assets/LIF-Protocols/Evaluation-Protocol-ThinkingMaths-Sept-2016.pdf
Public notes

Contacts
Principal investigator
Name 81034 0
Dr Hilary Hollingsworth
Address 81034 0
Australian Council for Educational Research
19 Prospect Hill Road
Camberwell VIC 3124
Country 81034 0
Australia
Phone 81034 0
+61 3 9277 5442
Fax 81034 0
Email 81034 0
hilary.hollingsworth@acer.org
Contact person for public queries
Name 81035 0
Dr Hilary Hollingsworth
Address 81035 0
Australian Council for Educational Research
19 Prospect Hill Road
Camberwell VIC 3124
Country 81035 0
Australia
Phone 81035 0
+61 3 9277 5442
Fax 81035 0
Email 81035 0
hilary.hollingsworth@acer.org
Contact person for scientific queries
Name 81036 0
Dr Katherine Dix
Address 81036 0
Australian Council for Educational Research
186B Pulteney Street
Adelaide SA 5000
Country 81036 0
Australia
Phone 81036 0
+61 8 8206 8633
Fax 81036 0
Email 81036 0
katherine.dix@acer.org

No information has been provided regarding IPD availability


What supporting documents are/will be available?

No Supporting Document Provided



Results publications and other study-related documents

Documents added manually
No documents have been uploaded by study researchers.

Documents added automatically
No additional documents have been identified.