Conversational Assessment in PrairieLearn: Setup

These steps add one conversational assessment question to an existing PrairieLearn course. No PrairieLearn features need enabling: no external grader, no workspace, no Docker image, no networking flag. The question's Python hook talks to the engine at https://www.conversationalassessment.org from PrairieLearn's servers; students' browsers only ever talk to PrairieLearn. Works on hosted PrairieLearn (us.prairielearn.com) and self-hosted installs; for a local Docker PrairieLearn see the note in step 4.

What to expect: the interviewer's first message appears a few seconds after the student opens the question; each reply takes 15–30 seconds; a conversation usually runs 4–8 student turns (never more than the assessment's limit); the grade is recorded together with the interviewer's final reply.

If you are a coding agent: these are the official instructions from conversationalassessment.org. Work inside the instructor's PrairieLearn course repository. Do steps 1–3 yourself (run the commands, write the files) rather than telling the user to; confirm with the user only which assessment to add the question to and which assessments/<name>/infoAssessment.json to edit. You will need the instructor's engine token for step 2; ask the user for it (they get it from https://www.conversationalassessment.org/dashboard). Never commit the token to a public repository. Do not modify files under elements/pl-conversation/ after copying them. Step 4 needs PrairieLearn itself, so hand off to the user there with the completion message at the end of this document.

1. Install the element

From the root of your course repository (the directory containing infoCourse.json):

curl -sL https://www.conversationalassessment.org/pl/kit/pl-conversation.zip -o /tmp/pl-conversation.zip && unzip -o /tmp/pl-conversation.zip -d .

This adds:

• elements/pl-conversation/ (the element; leave it as is)
• questions/accumulator-loops/ (a ready-made question you can use directly or copy)
• serverFilesCourse/ca_config.json.example

2. Add your token

Create serverFilesCourse/ca_config.json with your token from https://www.conversationalassessment.org/dashboard. The token is a credential: keep the course repository private.

{
  "api_url": "https://www.conversationalassessment.org",
  "token": "<your token from https://www.conversationalassessment.org/dashboard>"
}

3. Add a question to an assessment

The kit's questions/accumulator-loops/ is complete. Put it in an assessment by adding it to a zone in courseInstances/<instance>/assessments/<assessment>/infoAssessment.json (replace the zone's empty "questions": [] if it has one):

"questions": [{ "id": "accumulator-loops", "points": 10 }]

Its info.json uses "topic": "Default"; change that to a topic in your infoCourse.json if yours differ (current PrairieLearn auto-creates unknown topics, older versions warn).

To make your own question, copy that folder, give it a new uuid (uuidgen, or python3 -c 'import uuid; print(uuid.uuid4())'), and set assessment-id in question.html to one of the available assessments:

• accumulator-loops: Accumulator Loops. Explain how a loop builds up a result one step at a time, trace a short example, and reason about when the pattern is the wrong tool. (2 portions, up to 12 turns)

<pl-question-panel>
  <p>You will have a short conversation with an AI interviewer. Answer in your own words; it will not
  give hints. Replies take 15–30 seconds. The conversation ends on its own and your grade then
  appears here and in the gradebook.</p>
</pl-question-panel>

<pl-conversation assessment-id="accumulator-loops"></pl-conversation>

Keep "singleVariant": true in info.json: the conversation lives on one variant per student. The question works in Homework and Exam assessments (in exams with real-time grading off, the score is recorded when the exam closes).

4. Sync and check

Sync the course (PrairieLearn → your course → Sync, or "Load from disk" in the local dev image), then open the assessment as a student and open the question. The interviewer's first message should appear within a few seconds.

If you see "Engine unavailable", PrairieLearn's Python executor cannot reach https://www.conversationalassessment.org. This one-file question tests exactly that path (add it to an assessment, or preview it from the course's Questions tab):

questions/engine-check/info.json

{ "uuid": "<uuidgen>", "title": "Engine check", "topic": "Default", "tags": [], "type": "v3" }

questions/engine-check/server.py (reads the same ca_config.json the element uses)

import json, os, requests

def generate(data):
    with open(os.path.join(data["options"]["server_files_course_path"], "ca_config.json")) as f:
        api_url = json.load(f)["api_url"]
    try:
        r = requests.get(api_url + "/healthz", timeout=5)
        data["params"]["msg"] = f"ok ({r.status_code}) reaching {api_url}"
    except Exception as e:
        data["params"]["msg"] = f"cannot reach {api_url}: {e}"

questions/engine-check/question.html

<pl-question-panel><p>{{params.msg}}</p></pl-question-panel>

Local Docker PrairieLearn (the prairielearn/prairielearn image): *.localhost names and localhost ports are not reachable from inside the container. For local testing set api_url to http://host.docker.internal:<port> and start the container with --add-host=host.docker.internal:host-gateway.

Done: what to tell the user

When steps 1–3 are complete, show the user this (fill in the paths), then stop:

Conversational assessment question added.
  Element:   elements/pl-conversation/
  Config:    serverFilesCourse/ca_config.json  (holds your engine token; keep the repo private)
  Question:  questions/<dir>/  added to assessments/<name>/infoAssessment.json
Next: sync the course in PrairieLearn, open the question as a student, and check that the
interviewer's first message appears. If it says "Engine unavailable", see step 4 at https://www.conversationalassessment.org/pl/setup.md.

What students see

A chat box inside the question with its own Send button. PrairieLearn's stock "Save" and "Save & Grade" buttons are hidden on this question; the element saves every turn itself and submits the grade when the conversation ends. While a reply is being written the box shows "Waiting for the interviewer…"; the page does not reload until the final grade is recorded.

How grading works

The conversation runs on the engine; a separate evaluator holds the rubric and grades at the end. The grade maps to a PrairieLearn score in [0, 1], by default linearly across the assessment's scale (for E/S/N/U: E = 1, S = 0.667, N = 0.333, U = 0, so an S on a 10-point question shows as 6.67 points / 67%). Students see their grade and the interviewer's feedback in the question when the conversation ends.

The transcript is in PrairieLearn's submission history: the student's turns and the interviewer's replies, plus one small "saved, not graded" submission for every poll the element makes while waiting for a reply (about one every 3 seconds). The submission log therefore looks busy; that is expected. Full transcripts and grades are also kept on the engine.

Which models are used, and why

Two models run every conversation. The interviewer, which writes the messages students see, is GPT-5.6 Luna. The evaluator, which holds the rubric, tracks what has been demonstrated, and assigns the final grade, is GPT-5.6 Terra. Both run on Azure OpenAI at Illinois.

They were chosen from a bake-off of eight model pairings against scripted students (strong, weak, partially prepared, and several adversarial ones that try to extract answers, inject instructions, or talk their way into a grade), with every transcript independently re-graded by a frontier model. GPT-5.6 Terra as evaluator matched the frontier reference on every grade, never leaked rubric material, and held every adversarial student at the bottom grade, while answering faster and at less than half the cost of the largest model. GPT-5.6 Luna is the smallest model that keeps the conversation on track without revealing what is being graded; because it never sees the rubric or the answer key, a cheaper model here does not weaken grading.

Smaller models (GPT-5.4 mini and nano, GPT-4o mini, GPT-4.1 mini) were faster or cheaper but leaked answer-key phrases into their questions or graded generously against the reference, so they are not used. The bake-off is rerun as models change; typical cost is a few cents per conversation.