In order to collect good statistical data, I will need to have a double-blind experiment to determine how Copilot compares to other methods. Here are the methodologies I have considered so far.
I have professors give me an assignment from the class. I have Copilot complete the assignment, find one answer on StackOverflow, and post the question on Reddit and copy/paste the first code response I receive, whether or not it works. I also complete the assignment myself and have two other students complete the assignment with me. I submit these 6 entries to the professor and have them grade them.
- Assignments are properly anonymized - professors will have no context to determine which assignments belong to students and which are from Copilot/online
- More work for professors - they now have to grade 6 more assignments
- More work for me and my friends - I need to get 3 people to complete each assignment I want to test
I have professors give me a dummy account in each of their classes. In each of these assignments, I will post one of three answers: one from Copilot, one from the internet (StackOverflow or Reddit), or one I do manually. If professors download all of these assignments and grade them as normal, then I can ask for the average grade/other statistics, plus I can see my individual grades
- Easier for professors - they only have to grade one additional assignment
- Easier for me - I only need to write one answer per assignment
- I still have to write a program by hand for 1/3 of the classes
- I need to network with professors' IT departments to get the dummy accounts. Not all schools may support this
- Some professors may not grade anonymously
Have a student submit an assignment as their own. After getting the professor's approval, get a list of the students in the class. Chose one or more of the students and contact them. Instead of submitting the assignment they write themselves, I'll randomly give them an assignment from Copilot, from the internet, or tell them to just submit their own work (as a control group). I'll have them email me their assignment which I will send to the professor myself (so that the student can still be graded normally). After the students receive back their grades, I'll let the professor know which students were the test subjects so that the professor can correct their grades.
Also, I'd as the professors for statistics like average, standard deviation, and so on in order to see if my assignment is statistically different from the other assignments.
- Less work for professors - they only have to grade one (or potentially a few more) additional assignments, but they'll be grading assignments of this type already (since the assignments are from a class they're teaching)
- Easier to gather data - I can submit three assignments to each class via three random students rather than being limited to one assignment per class
- Doesn't require as much anonymity - even if professors don't grade anonymously, they won't know which student submitted my assignments, so the data won't be as biased.
- I don't have to do any assignments myself - we use a student's work as a comparison, which should be a better indicator of the class anyway
- I'll need a list of all the students in the class, which brings up privacy issues. Does the professor need to get each of the students' permission to give me their names/email addresses? This is a major privacy issue.
- Some students may not respond to my attempts to contact them. I could choose a different student if they don't respond quick enough, but this could lead to biased data
- Students will need to be careful not to hint to the professor that they are in the experimental group
- If a student has certain conventions they usually use, the professor can see if these are or are not present and use this to determine which students are in the experimental group
- If a student usually does very poorly and they're chosen as the test subject, the professor may be suspicious if they suddenly do well