17. chatGPT marking essays

Anyone who has been involved in marking exam essays knows that it takes a little while to adapt so that you are marking to the same standard as everyone else. It also requires constant checks to ensure that standard is maintained. Large Language Models such as chatGPT are trained on a vast amount of textual material but it hasn't been trained on large numbers of Higher Philosophy essays. My guess is that some such system is already possible but chatGPT isn't it. Every time the bot marks a Higher Philosophy essay it as if it is marking its very first Higher Philosophy essay. Each new chat is a new experience. It cannot remember from one chat to the next. It is not surprising, therefore, that its marking can be somewhat unreliable and inconsistent. What I do find surprising is that it can actually manage to do it at all.

Even if it is unreliable and inconsistent that doesn't mean asking it to mark an essay is a complete waste of time. Sometimes deciding whether you agree or disagree with somebody else's marking, and why that is, can help you to think about the essay in a way you might not have done without that input.

So, with all the caveats about reliability and consistency how do you get chatGPT to mark an essay?

If you simply ask it to mark the essay it will draw on all the examples on which it has been trained and give some kind of mark but that isn't what we want. Higher Philosophy essays are marked holistically according to given criteria. The bot needs to be told what criteria to use. My experiments suggest it is best to ignore the detailed marking instructions for the question and to just use the holistic marking instructions. There are slightly different instructions depending on whether they are for an essay on Descartes or Hume, or whether they are for a Moral Philosophy response to a quotation or to a scenario.

First exemplar essay

Read the following exam question, the student response and the marking criteria. What mark out of 30 would you give this essay?

Give reasons for the mark you have awarded.

This is the exam question:

Read the following: ‘Late one night a parent discovers that their child has a very severe headache and decides to take the child to hospital by car. They come to a junction where the traffic lights are at red. Seeing that there are no other cars at the junction, the parent decides to cross the red light. This is seen by police officers who stop the parent, question and breathalyse them. This is standard procedure for any driver stopped for committing a traffic offence. After issuing the parent with a fine for crossing the red light, they allow them to go on their way to hospital. Because they were stopped, they arrived later than they would have done if they had waited for the lights to change.’

Analyse and evaluate how utilitarians might respond to the parent’s decision to cross the red light.

In your answer you should make clear the difference between act and rule utilitarianism.

Pupil response:
[Here I pasted in the essay which the SQA had awarded 16/30]

The following is the marking criteria I want you to use:
[Here I pasted in the moral philosophy situation holistic marking criteria]

*** chatGPT says ***
I would give this essay a mark of 15 out of 30.

While the student provides a good definition of utilitarianism and explains the difference between act and rule utilitarianism, the essay is not very relevant to the given scenario. There is no clear application of the theory to the situation, and the discussion is quite general and abstract.

The student does not evaluate how utilitarians might respond to the parent’s decision to cross the red light, and the analysis provided is not well connected to the scenario. The essay includes some relevant information about utilitarianism and the hedonic calculus, but this information is not used to analyse the scenario in a meaningful way.

The essay also includes several errors and inconsistencies. For example, the essay states that utilitarianism is based on happiness, but then contradicts itself by saying that utilitarianism contradicts Kantian ethics because it is not based on duty and responsibility. There are also some grammatical errors and spelling mistakes throughout the essay.

Overall, the essay provides some relevant information about utilitarianism, but it is not well applied to the given scenario, and there are several errors and inconsistencies.

As already noted, each time you ask the bot to mark an essay it is as if they are marking a Higher Philosophy essay for the first time. So if you ask the same question several times you are likely to get different marks.

The SQA commentary on this essay says:

The candidate was awarded 16 marks.

They have given a good introduction which describes some of the background to Utilitarianism. They describe Act Utilitarianism competently with good use of the quote from Bentham. They get a little bit muddled when describing Rule Utilitarianism. Their first sentence in paragraph 3 is correct, but then they wrongly associate the idea of Mill’s Higher and Lower Pleasures with Rule Utilitarianism. The rest of this paragraph is irrelevant to the scenario. Their reference to the situation shows some understanding of the application of the theories but does not go in to depth.

They give several evaluative comments that are quite superficial and not all directly relevant to the scenario. They miss the opportunity in their discussion of the scenario to point out that the predicted consequences were different from the actual consequences. They give a personal view on issues raised by the situation but with no real support to their position.

Second exemplar essay

Read the following exam question, the student response and the marking criteria. What mark out of 30 would you give this essay?

Give reasons for the mark you have awarded.

This is the exam question:

Read the following quotation and answer the question that follows.

‘Kant’s moral theory is cold and detached from real life.’

To what extent do you consider this criticism to be fair?
In your answer you could include the following

  • a description of Kant’s theory
  • an explanation of how Kant says we should make moral decisions
  • discussion of how the criticism applies to Kantian ethics
  • evaluation of Kantian ethics in relation to this criticism.


Pupil response:
[Here I pasted in the essay which the SQA had awarded 30/30]

The following is the marking criteria I want you to use:
[Here I pasted in the moral philosophy quotation holistic marking criteria]


*** chatGPT says ***
I would give this essay a mark of 26 out of 30.

The student has demonstrated a thorough understanding of Kant's moral theory and has effectively explained the key elements of the theory, including deontology, the good will, and the categorical imperative. The student also effectively addresses the criticism that Kant's theory is cold and detached from real life, providing arguments and explanations to counter this criticism. The essay is well organized and clearly structured, with the ideas presented in a logical and coherent manner.

However, there are a few areas for improvement. The language used in the essay is sometimes complex and may be difficult for someone unfamiliar with Kant's theory to understand. The student could also benefit from providing more specific examples or real-life scenarios to illustrate the key points made in the essay. Additionally, the conclusion could be stronger and provide a more conclusive evaluation of Kantian ethics in relation to the criticism, rather than just acknowledging the criticism and leaving it at that.

Again, all the caveats about unreliability and inconsistency apply

The SQA commentary on this essay says:

The candidate was awarded 30/30 marks.

The candidate showed a thorough, in depth and accurate knowledge and understanding of the theory, application and reasonable interpretation of Kantian Ethics. They presented a detailed, methodical and sophisticated response to the issues raised by the quotation with evaluative comments woven throughout their essay in response to the points of theory they described. Their line of argument directly addresses the question and unpacks the different elements of the quotation, addressing the criticism that Kantian ethics is cold and also that it is detached from real life separately.

There is considerable evaluative discussion through the course of the essay and a well-supported personal position on the issues raised by the quotation is shown by the candidate. This is an excellent essay.