Supplementary data for the paper: 
De Winter, J. C. F., Dodou, D., Eisma, Y. B. (2024). System 2 thinking in OpenAI’s o1-preview model: Near-perfect performance on a mathematics exam. Preprint.
https://www.researchgate.net/publication/384071542_System_2_thinking_in_OpenAI's_o1-preview_model_Near-perfect_performance_on_a_mathematics_exam

* Connect_API.m				MATLAB script that inputs the exam questions into the OpenAI API.

* Figure1.m				MATLAB script that produces Figure 1 of the paper.

* Figure2.m				MATLAB script that produces Figure 2 of the paper. It reads the files "Exam 2024 Q9 responses - 250 times.txt" and "Exam 2024 Q9 responses - 250 times - correctness.xlsx".


* Folder "Exam 2023/2024 material"	(1) PDF file with the exam questions (in Dutch), as taken from College voor Toetsen en Examens.
					(2) PDF file with the answer model (in Dutch), as taken from College voor Toetsen en Examens.
					(3) Exam questions with highlighted modifications, for prompting.

* Folder "Exam 2023 responses" 		Files with model output (responses, time spent, and tokens per exam question).
					(1) Exam 2023 - attempt 1.docx: Output of the 1st attempt of both models (o1-preview and GPT-4o) for all 19 exam questions.
					(2) Exam 2023 - attempt 2.docx: Output of the 2nd attempt of both models (o1-preview and GPT-4o) for all 19 exam questions.
					(3) Exam 2023 - Q3 - twenty times.docx: Output for Question 3, run 20 times with the o1-preview model.
					(4) Exam 2023 - Q11 - twenty times.docx: Output for Question 11, run 20 times with the o1-preview model.
					(5) Exam 2023 - Q16 - twenty times.docx: Output for Question 16, run 20 times with the o1-preview model.
					(6) Exam 2023 - Scores and assessment.xlsx: Excel file containing the evaluation of the outputs of the two OpenAI models.

* Folder "Exam 2024 responses" 		Files with model output (responses, time spent, and tokens per exam question).
					(1) Exam 2024 - o1-mini: Output of o1-mini for all 18 exam questions.
					(2) Exam 2024 - o1-preview and GPT-4o.docx: Output of the 2nd attempt of both models (o1-preview and GPT-4o) for all 19 exam questions.
					(3) Exam 2024 Q9 responses - 250 times.txt: Text file with all o1-preview output for Question 9 of the exam.
					(4) Exam 2024 Q9 responses - 250 times - correctness.xlsx: Manual scoring of all 250 answers for Question 9 of the exam.
					(5) Exam 2024 - Scores and assessment.xlsx: Excel file containing the evaluation of the outputs of the three OpenAI models.


References
College voor Toetsen en Examens. (2023a). Wiskunde B VWO 2023, tijdvak 1 [Mathematics B VWO 2023, period 1]. https://www.examenblad.nl/system/files/exam-document/2024-06/vw-1025-a-23-1-o.pdf
College voor Toetsen en Examens. (2023b). Correctievoorschrift, Wiskunde B VWO 2023, tijdvak 1 [Correction guidelines, Mathematics B VWO 2023, period 1]. https://www.examenblad.nl/system/files/2023/ex2023/VW-1025-a-23-1-c.pdf
College voor Toetsen en Examens. (2024a). Wiskunde B VWO 2024, tijdvak 1 [Mathematics B VWO 2024, period 1]. https://www.examenblad.nl/system/files/exam-document/2024-07/vw-1025-a-24-1-o-spr.pdf
College voor Toetsen en Examens. (2024b). Correctievoorschrift, Wiskunde B VWO 2024, tijdvak 1 [Correction guidelines, Mathematics B VWO 2024, period 1]. https://www.examenblad.nl/system/files/exam-document/2024-05/vw-1025-a-24-1-c.pdf

Other useful sources:

Cito. (2023a). Toets en item analyse VWO Wiskunde B 2023 tijdvak 1 [Test and item analysis VWO Mathematics B 2023 first exam period]. https://www2.cito.nl/vo/ex2023/VW-1025-a-23-1-TIA.docx
Cito. (2023b). Cito betekenis cijfers 2023 [Cito meaning of grades 2023]. https://cito.nl/media/kw0jow0k/cito_betekenis_cijfers_2023.xlsm
Cito. (2024). Frequentieverdeling [Frequency distribution]. https://cito.nl/media/0wugu4a5/24116-frequentieverdeling.xlsx (accessed on 18 October 2024).
OpenAI. (2024). API reference: Chat. https://platform.openai.com/docs/api-reference/chat









