Getting Started

Everything from creating an account to a graded conversation in your PrairieLearn course, one click at a time. About an hour of your own time, plus two short waits for approvals.

Have Your AI Walk You Through It

If you have an AI agent on your computer, such as the Claude desktop app, Codex, or Cursor, it can take you through these steps itself and check each one as you go. You paste one line, and it does the file work.

Get your line

Or follow the guide below, which has the same steps with videos and a browser-only route.

What you are building · 1:12

By the end, one question in your PrairieLearn course will hold a short, graded conversation with each student about a topic you chose, with criteria you set. Students talk to an AI interviewer that never sees the answer key. A separate evaluator grades against your rubric. The grade lands in the PrairieLearn gradebook like any other question.

You do not need to write code or use a terminal. Where files have to be created, an AI assistant does it for you, and there is a browser-only route for every step.

Before You Start

Have these ready. Each is quick to get, but two involve waiting for someone else.

1. Create Your Account

Signing in and requesting access · 1:44

Your account on this site is where you get the two credentials the rest of the guide uses, and where your assessments and your students' results appear. Creating it is the same as signing in for the first time.

  1. Go to www.conversationalassessment.org and click Login in the top right.
  2. Illinois: click Sign in with Illinois. Microsoft asks for your NetID, password, and Duo, then sends you back here.
    Everyone else: type your institutional email address in the box and click Email me a link. The page says to check your email.
    1. Open the email from conversationalassessment.org. If it has not arrived in two minutes, look in Junk or Spam.
    2. Click the link. A page titled Confirm sign-in opens.
    3. Click the Sign in button. This extra click exists because some university mail systems open every link to scan it, and that would use up the link before you saw it.
    The link works once and expires after an hour. If it has expired, request another from the sign-in page.
  3. You land on Request Access. An AI assistant asks a few short questions: your role, where you teach, and what you want to assess. Your answers fill in a form beside the chat, which you can also edit yourself. When the required fields are filled, click Send request.
  4. Your Dashboard now says Waiting for approval. We approve requests by hand and email you, usually within a few days. There is nothing to do here until then. Meanwhile, request your PrairieLearn course if you have not, and read on.
  5. When the approval email arrives, sign in again the same way. The dashboard now has sections called Your Token, Set Up With Your Coding Agent, Your Assessments, and Author With Your AI. Accounts outside Illinois also see a red notice saying students cannot start yet, and a section called Your AI Endpoint. That is Part 2. Illinois accounts skip to Part 3.

2. Connect Your Own AI

Connecting and testing your AI endpoint · 1:10

Illinois accounts skip this part. Everyone else runs conversations on an AI account they control. That keeps the cost on your bill, a few cents per conversation, and means student text never passes through a provider you did not choose.

  1. Get an API key. The key is a long string that lets this site call the AI on your behalf. Copy it the moment it is shown; providers show it once.
    • OpenAI: platform.openai.com, then API keys, then Create new secret key. The base URL is https://api.openai.com/v1.
    • Anthropic: console.anthropic.com, then API Keys, then Create Key. The base URL is https://api.anthropic.com.
    • Azure OpenAI: ask your IT department for the resource's URL and a key. The URL looks like https://your-resource.openai.azure.com.
  2. On your dashboard, scroll to Your AI Endpoint. Choose the provider type (OpenAI-compatible, Azure OpenAI, or Anthropic), paste the base URL into the first box and the key into the second, then click Save and find models. The key is stored encrypted and is never shown again.
  3. The page lists every model your endpoint offers, with its price and, where we have tested it, how it did in our bake-off. Choose an evaluator (the model that holds the rubric and grades) and an interviewer (the model that talks to students). Prefer models the list marks as tested. Click Save models and test. A short test call runs and reports.
  4. Under Check These Models, click the button to run three simulated students against your endpoint: a strong one, a weak one, and one that tries to extract the answers. This takes a few minutes and costs a few cents on your account. The verdict appears on the dashboard (pass, warn, or fail) with the transcripts. Do this before real students arrive.

Costs: a page listing every model we know, with list prices and bake-off results, is at Model Evaluations. The dashboard estimates what a conversation will cost on the models you chose.

3. Install Your AI and Connect It

Adding the authoring connector · 1:11 · preliminary cut: parts that happen outside the browser are shown as slides for now

You write assessments by talking to your own AI assistant, which is connected to this site so it can save, check, and test what you write together. This guide walks through Claude and Codex. Any assistant that supports custom connectors (the standard is called MCP) works the same way. The videos show Claude. Codex works the same way, and the differences are in the text.

First, generate your connector. On your dashboard, scroll to Author With Your AI and click Generate authoring connector. A box appears with a prompt that contains a web address of the form https://api.conversationalassessment.org/mcp/caa_…. That address is your private connector. It is shown once. Keep this tab open until you have added it below.

Using Claude

  1. Install Claude Desktop. Download it from claude.com/download (Mac or Windows), install it like any program, open it, and sign in with your Claude account. You can do Parts 3 and 4 in the browser at claude.ai instead; the desktop app is what Part 5 uses.
  2. Add the connector. In Claude, open Settings, choose Connectors (under Customize), and click Add, then Add custom connector. Name it Conversational Assessment, paste the address from the dashboard into the MCP server URL field (just the address, from https to the end of the long code), and click Continue. On the second step choose No sign-in (the address itself is the key) and click Add. Connectors belong to your Claude account, so one added at claude.ai also appears in the desktop app, and one added in the desktop app appears on the web.
  3. Start a new chat and make sure the connector is switched on for it (the tools menu next to the message box lists it).

Using Codex

  1. Install the ChatGPT desktop app. Download it from OpenAI's desktop app page, install it like any program, open it, and sign in with your ChatGPT account. Codex is part of this app. The ChatGPT website does not read the connector you add here, so stay in the app for Parts 3 to 6.
  2. Add the connector. Open Settings, choose MCP servers, and click Add server. Name it conversational-assessment, choose Streamable HTTP, and paste the address from the dashboard as the server's URL (just the address, from https to the end of the long code). Save it, then click Restart.
  3. Start a new chat and choose Codex in the message box.

Then test it. In that new chat, type:

You: Using the Conversational Assessment connector, call get_guide and then list the assessments in the library.

Your AI reads the guide and lists the shared library, which includes an intro-programming assessment called accumulator-loops. If it says it has no such tool, check the connector is enabled for this chat. If the address was mistyped, click Rotate authoring connector on the dashboard to get a new one and add it again; the old one stops working.

The connector can read and write your assessments and read your students' transcripts. It cannot run student attempts, and it is separate from the course token in Part 5. Rotating one does not affect the other.

4. Write Your First Assessment

Writing an assessment with your AI · 1:31 · preliminary cut: parts that happen outside the browser are shown as slides for now

An assessment is a topic broken into two or three portions, each a thing you want the student to talk through. Each portion has criteria (what a student must demonstrate), a rubric for each criterion, a private answer key, and a few follow-up angles. There is also a short description of who the interviewer is. Your AI knows the format and the rules; you bring the topic and the judgement about what counts. Budget thirty to sixty minutes for the first one.

  1. On the dashboard, in Author With Your AI, click Copy prompt and paste it into a new chat. If the box has gone (it is shown once), type this instead:

    You: Using the Conversational Assessment connector, call get_guide and help me write a conversational assessment for my course.

  2. Your AI reads the guide and asks about your topic, what a student who understands it should be able to explain, and how you want to grade (the default is four levels, E, S, N, U, which record in PrairieLearn as 100, 67, 33, and 0 percent). Answer in plain language. For example:

    You: Second week of intro Python. I want to know whether they understand how an if/elif/else chain picks one branch, and whether they can trace a short example by hand.

    Your AI: Two portions, then. First, explain how a chain chooses a branch, with criteria for describing the top-to-bottom check and for what happens when no condition is true. Second, trace a five-line example. Here is a draft of the criteria and rubrics; tell me what you would change.

  3. Read the criteria it proposes. A good criterion names what the student does ("traces a concrete example step by step"), not the answer. The answer lives in a private answer key the interviewer never sees. Push back where the pedagogy is wrong; your AI will not know your course.
  4. It checks the draft (you will see it call validate_assessment and lint_assessment) and fixes what the checks flag, such as a criterion that gives the answer away. Then it saves the assessment (create_assessment) under a short name called a slug, for example conditionals. Write the slug down. Part 5 uses it.
  5. Simulate students. Ask your AI to run three simulated students against it: a good one (expect the top grade), a weak one (expect a low grade), and one that tries to get the interviewer to reveal the answers (expect the bottom grade and no leaks). Each run takes a minute or two and costs a few cents. Read the transcripts it shows you and ask yourself: does the interviewer ask what I intended, and is the grade the one I would give? Tell it what to change. It updates the assessment and simulates again.
  6. On your dashboard, Your Assessments now lists it, marked yours. Click the slug to see the document and any remaining lint notes. Nothing reaches students until a PrairieLearn question refers to it, which is Part 5.

Two habits that pay off: keep the first assessment small (two portions, two or three criteria each), and always read the simulated transcripts before students see it. Publishing is optional and separate: a published assessment lets other instructors run it by slug, and shares the interviewer's persona and structure with them, but never the answer key.

5. Put the Question in Your Course

Adding the question to PrairieLearn · 2:14 · preliminary cut: parts that happen outside the browser are shown as slides for now

PrairieLearn stores every course as a set of files. Adding the question means putting a small folder of files into your course, adding a file that holds your course token, and listing the question in an assessment. There are two ways to do that. Path A has your AI do the file work in a copy of your course on your computer, and is what we recommend. Path B installs nothing and uses the GitHub and PrairieLearn websites only.

Path A: Let Your AI Do the File Work

  1. Get a copy of your course on your computer. Install GitHub Desktop and sign in with the GitHub account that PrairieLearn gave access to your course repository. Choose File › Clone repository, find your course (its name starts with pl-, and PrairieLearn shows it on your course's Settings page), and click Clone. Note the folder it chose, usually under Documents.
  2. Open that folder in your AI.
    • Claude: open Claude Desktop and click the Code tab at the top. Click + New Session. In the box where you type, set the Project folder to the folder GitHub Desktop created. Leave the permission mode on the setting that asks before running commands.
    • Codex: in the ChatGPT desktop app, open the folder GitHub Desktop created and choose Codex in the message box. Leave the permissions on Auto. Codex normally cannot use the internet or change the .agents and .git folders, so it will ask you to approve downloading the kit, installing it, and committing.
    Either way, your AI explains each step and waits for you to approve it.
  3. Generate your course token. Back on your dashboard, under Your Token, click Generate token. The token appears for fifteen minutes, and the box called Set Up With Your Coding Agent now shows a prompt whose link carries the token. Click Copy prompt. Do this after step 2, not before: the link stops working after fifteen minutes.
  4. Paste the prompt into that session and press Enter. If you wrote your own assessment in Part 4, add one sentence first: Make the question use my assessment your-slug. Your AI fetches the instructions, downloads the kit into your course folder, writes the token file, and asks which homework or exam to add the question to. Name one, or say create a new homework called Conversation practice. Approve each action when asked. It finishes with a short summary that starts Conversational assessment question added.
  5. Send the changes to GitHub. Your AI asks whether to commit. Say yes and ask it to push. Or, in GitHub Desktop, type a summary such as Add conversational assessment question, click Commit to main, then Push origin. Course repositories are private, which is what keeps your token private.
  6. Sync in PrairieLearn. Open your course, click Sync in the sidebar, and click Pull from remote git repository. Wait for the green result. A red result names the file that is wrong; paste the message to your AI.

Path B: Browser Only, No Installs

  1. Download the kit: kit.zip. Unzip it (double-click on a Mac; right-click and Extract All on Windows). Inside are folders called elements, questions, and serverFilesCourse. Ignore any others.
  2. Upload the folders to GitHub. Open github.com, go to your course repository, click Add file, then Upload files. Drag the elements folder and the questions folder from the unzipped kit onto the page (drag the folders themselves; GitHub keeps the folder names). Click Commit changes.
  3. Get your token file. On your dashboard, under Your Token, click Generate token, then Download ca_config.json. Your browser saves a small file with that name. The download works for fifteen minutes after generating.
  4. Upload the token file to PrairieLearn. In PrairieLearn, open your course, click Files in the sidebar, click Add new server files, choose the downloaded ca_config.json, and upload it. PrairieLearn places it in the right folder and sends it to GitHub for you.
  5. Sync. Click Sync in the sidebar, then Pull from remote git repository, and wait for the green result.
  6. Add the question to an assessment. In the course instance, open Assessments and open one (or click Add assessment and create a Homework called Conversation practice). On its Questions tab click Edit, Add zone, then Add question, search for accumulator-loops, pick it, and click Save.
  7. If you wrote your own assessment in Part 4: in the course sidebar open Questions, open accumulator-loops, go to its Files tab, click Edit next to question.html, change assessment-id="accumulator-loops" to your slug, adjust the sentence above it that introduces the topic, and click Save.

Either way, the question's folder and the element are copies from the kit and should not be edited by hand: when the kit is updated, your dashboard tells you, and updating overwrites those files.

6. Try It as a Student and Read the Results

A conversation, a grade, and the transcript · 1:24 · preliminary cut: parts that happen outside the browser are shown as slides for now
  1. In PrairieLearn, open the user menu in the top right and choose Student view without access restrictions. Open the assessment, then the question. The interviewer's first message appears after a few seconds.
  2. Answer in your own words. Each reply takes fifteen to thirty seconds; while you wait the box says Waiting for the interviewer. The conversation ends on its own after a handful of turns. Your grade and the interviewer's feedback appear in the question, and the score appears in the gradebook: on the default scale an E is 100 percent and an S is 67.
  3. If you later change which questions the assessment holds, click Regenerate your assessment instance (the yellow banner only staff see) before trying again. Switch back to staff view from the same menu when you are done.
  4. Where the record lives. The gradebook has the score. The question's submission history has every turn of the transcript. It also holds many small "saved, not graded" rows: those are the question checking for a reply every few seconds while the student waits, and are expected.
  5. Ask your AI how students did. In a chat with the connector on, ask:

    You: How did students do on conditionals? Show me the grade distribution and read me the weakest conversation.

    It lists the attempts, summarizes the grades, and walks through transcripts with you. Your dashboard also counts attempts per assessment. To change a grade, edit the score in PrairieLearn as you would for any manually graded question.
  6. Tell students what to expect. Replies take fifteen to thirty seconds. The interviewer does not give hints and does not confirm answers. The conversation ends by itself. The grade appears when it does. Everything they type is treated as an answer, including requests to the AI. The question works in exams too; in an exam with real-time grading off, the score is recorded when the exam closes.

7. When Something Goes Wrong

The question shows "Engine unavailable".
PrairieLearn's servers could not reach this site. On us.prairielearn.com this is rare and usually passes; try again in a minute. On a PrairieLearn your institution hosts, ask its administrators to allow outgoing connections to api.conversationalassessment.org. The PrairieLearn page has a one-file test question that checks this.
The question fails to render and mentions an unknown attribute.
The question was written for an older kit. Your dashboard will also say the kit is out of date. In a Claude Code or Codex session in your course folder, type update the conversational assessment kit; or repeat the upload in Path B. Then sync.
The sign-in link says it no longer works.
Links work once and expire after an hour. Request a new one from the sign-in page.
Still "Waiting for approval" after a few days.
Email challen@illinois.edu. Say which address you signed in with.
Your AI says it has no such tool, or the connector shows as disconnected.
Check the connector is switched on for the chat. In Codex, check you restarted after adding it and chose Codex in the message box. If the address was mistyped or lost, click Rotate authoring connector on the dashboard and add the new address; the old one stops working immediately.
Codex says it cannot reach the internet or write to a folder.
That is its sandbox. Approve the step when it asks. If it does not ask, type /permissions in the message box and allow it for this session. Do not turn off the sandbox entirely.
The setup prompt's link has expired.
Links carry the token for fifteen minutes. Click Rotate token to get a fresh prompt. The old token stops working, so if the question was already installed with it, the new token has to replace the old one in serverFilesCourse/ca_config.json; your AI will do that if you ask.
Students see "Waiting for the interviewer" for more than a minute.
The AI endpoint is slow or down. Accounts with their own endpoint: click Test again under Your AI Endpoint. Illinois accounts: email us. The student's conversation is kept, and they can reload the page and continue when the endpoint returns.
The dashboard says the course kit is out of date.
Update as for the unknown-attribute problem above. Your token and questions are untouched by an update.

Anything else: challen@illinois.edu. Questions about PrairieLearn itself are answered fastest on the PrairieLearn Slack.