PennGrader

Welcome to the PennGrader!

Here at PennGrader, we believe that learning comes from lots of practice...and from making lots of mistakes.

After many years as a student I found myself very frustrated in the following homework timeline: struggle on a homework assignment for weeks, submit something that may or may not be right and then wait a few more weeks to receive any type of feedback, at which point I had forgotten all about the homework. After many years as a TA, I also found myself very frustrated with the common auto-grading tools, the hours and hours of manual grading and the onslaught of re-grade requests that came thereafter.

From these frustrations, the PennGrader was born!

The PennGrader was built to allow students to get instant feedback and many opportunities for re-submission. After all, programming is about making mistakes and learning from feedback! Moreover, we wanted to allow TAs and Instructors to write their homework in any way they pleased, without having to worry about the structure of a specific auto-grader. The examples below are done using Jupyter Notebooks which is the most common use case, but you can use this for normal Python homework as well.

Here is what a student sees in his Homework Notebook. All a student has to do is write her solution and run the auto-grading cell.

Through the magic of AWS Lambdas, the student's answer (in this case the addition_function object) is packaged and shipped to the backend where it is checked against the teacher-defined test case. Finally, a score is returned. All students' scores are saved in the backend and are easily accessible to the instructors. If at first they don't succeed, they can learn from their mistakes and try again. (Yes, if you'd like, can set a maximum number of daily submissions to incentivize students to start early). This "grader" function can easily be used from any Jupyter notebook (even Google Colab), all you have to do is to pip install penngrader. See templates below.

Ok, ok, you might be saying to yourself: "That looks easy enough, but what about us TAs, we want something that simple too!" Well, look no further. The TAs/Instructors' experience is just as seamless. All TAs will share a Teacher_Backend notebook, which contains all the test case functions. The logic of how testing is done is simple: whatever Python object gets passed through the answer field in the grade(...) function (see above) will be the input to the test case function (see below). In the above example, addition_function is passed as the answer to a test case named "test_case_1". Therefore, the TAs will need to write a test_case_1(addition_function) function in the Teacher_Backend notebook, as follows:

As you can see, this function tests that addition_function(1,2) == 3, if correct it adds 5 points to the student_score. The test must then return an integer tuple (student_score, max_score), which is what will be displayed to the student. As you can see this type of test function gives the Instructor complete flexibility on what to test and how much partial credit to give. Remember that the answer that gets passed to the test case could be anything... a function, a class, a DataFrame, a list, a picture... anything! The PennGrader automatically serializes it and all its dependencies and ships to AWS for grading.

To create homework for your class you will need a course SECRET_KEY. We will open to the public soon, however for now contact me at leonardo.murri1995@gmail.com if you are interested in using the PennGrader for your class.

PennGrader_Homework_Template.ipynb

PennGrader_TeacherBackend.ipynb

Download these two notebooks and launch them via Jupyter. They will show you how to add grading cells in your homework notebook and add write/update test cases via the teacher backend, as well as view student's grades.

Behind the scenes...

In the following section, I will go into detail about the system implementation. Below is the system design overview we will go into.

Clients

There are two pip installable clients, one for students and one for instructors. You can install these two clients by running pip install penngrader in your favorite terminal. When creating a new homework download the Homework_Template.ipynb and the TeacherBackend.ipynb notebooks and follow the instructions. More details are presented below.

Student's Client: PennGrader

The student's client will be embedded in the homework release notebook. Its main purpose will be to interface the student's homework with the AWS backend. This client is represented by the PennGrader class which needs to be initialized by the instructor when writing the homework as follows. Note: the HOMEWORK_ID needs to be filled in before releasing the notebook. The student should only enter his or her's student id.

import penngrader.grader
grader = PennGrader(homework_id = HOMEWORK_ID, student_id = STUDENT_ID)

The HOMEWORK_ID is the string obtained when creating new homework via the teacher backend, see below.

STUDENT_ID is the student defined variable representing their 8-digit PennID. The student will need to run this cell at the beginning of the notebook to initialize the grader. After every question, the Instructor will also need to write a grading cell which the student will run to invoke the grader. A grading cell looks as follows:

grader.grade(test_case_id = TEST_CASE_NAME, answer = ANSWER) 

TEST_CASE_NAME is the string name of the test case function that will grader the given question.

ANSWER is the object that needs to be graded.

For example, you might have a question where you instruct the student to create a DataFrame called answer_df. You will need to write answer_df as the input answer parameter of the grader function as follows:

grader.grade(test_case_id = TEST_CASE_NAME, answer = answer_df) 

That way, when the student runs this cell, the grader will automatically find the test function named TEST_CASE_NAME, serialize the answer_df python variable and ship it to AWS. The cool thing about the PennGrader is that you can pass pretty much anything as the answer.

Teacher's Client: PennGraderBackend

The teacher client allows instructors to create and edit the test cases function mentioned earlier, as well as define multiple homework metadata parameters. As shown in the template notebooks linked above, you first need to initialize the PennGraderBackend for a specific homework as follows:

backend = PennGraderBackend(secret_key = SECRET_KEY, homework_number = HOMEWORK_NUMBER)

SECRET_KEY is the string variable obtained when creating a course.

HOMEWORK_NUMBER identifies which homework number you are planning to write/edit.

After running the above cell in a Jupyter Notebook, given a correct SECRET_KEY, the assigned HOMEWORK_ID string will be printed out. This HOMEWORK_ID needs to be copied into the initialization of the PennGrader student client for release. Just the homework id, without the secret key, will not allow students to see any of the test cases, so make sure the secret key does not get out.

After initialization, the backend can be used to 1) update metadata 2) edit/write test cases.

You can edit the following metadata parmaters by runnin the following code:

TOTAL_SCORE = 100
DEADLINE = '2019-12-05 11:59 PM'
MAX_DAILY_TEST_CASE_SUBMISSIONS = 100

backend.update_metadata(DEADLINE, TOTAL_SCORE, MAX_DAILY_TEST_CASE_SUBMISSIONS)

TOTAL_SCORE represents the total number of points the homework is worth and should equal the sum of all test cases weights.

DEADLINE represents the deadline of the homework, with format: YYYY-MM-DD HH:MM A.

MAX_DAILY_TEST_CASE_SUBMISSIONS represents the number of allowed submissions per test case per day.

Writing test cases is also just as simple. In a Jupyter Notebook (see Teacher Backend template above), you just need to write test case functions for each gradable question. Each test case will be identified by a test_case_id which is the name of the test case function. A test case functions needs to follow the following format:

def test_case_name(answer_object_name):
	student_score = 0 # Current student score
	max_score = 5 # Number of points this question is worth

	# some kind of grading that adds or subtracts from the student_score #

	return (student_score, max_score)

The test case function needs to return an integer tuple representing the student score for their answer and max number of points that can be earned in the given question.

After writing all the test cases you need, simply run the following code in a cell and the PennGraderBackend class will automatically extract all user-defined functions in the current notebook and upload them to AWS.

backend.update_test_cases()

A success message will print once the operation has succeeded. If loading a lot of external libraries this might take a few minutes.

Lambdas

Grader

The Grader lambda gets triggered from an API Gateway URL from the student's PennGrader client. The student's client as defined above will serialize its answer and make a POST request to the lambda with the following body parameters:

{'homework_id' : ______, 'student_id' : ________, 'test_case_id' : ________, 'answer' : _______ }

The lambda will proceed by downloading the correct serialized test_case's and libraries