Adrienne Decker
Department of Computer Science & Engineering
University at Buffalo
adrienne@cse.buffalo.edu
I Want to be a Computer Scientist When I Grow Up: Evaluating the Skills
Necessary for Computer Science
Various metrics have been used to assess a student's ability to succeed in introductory programming courses. Current work at SUNY Buffalo demonstrates that these traditional metrics do not hold for the Objects-First approach to teaching CS1.[7] Part of this work has involved using the Cornell Critical Thinking Test[2] as a metric for success in the Objects-First course. Being involved with CS1 at the University, I had the ability to take the Cornell Critical Thinking Test myself. It seemed that the material it was testing, student's skills in critical thinking and logic were not all that were needed to have success in the discipline. The test itself and many other similar metrics lack the ability to analyze one or more of the fundamental skills that are important in the field of computer science. The work seeks to identify the breadth of skills needed in computer science, and to identify a way to tell if a student has aptitude in those areas.
Many have argued that mathematical ability is what a computer scientist truly possesses. Leeper and Silver [5] looked at many rankings to predict success and found that SAT Math scores were the second strongest predictor of overall success in the course. The test was fairly good at identifying possible failures. Out of the potential failures identified, over 2/3rds did fail. However, the test also miscast some students as "passing" and then ended up failing. Furthermore, they identified that SAT Verbal was a stronger predictor than SAT Math for success in programming. Another standardized test that was used to try to predict success was the IBM PAT (Programmer's Aptitude Test), which Mazlack [6] found also not to be a good predictor of overall success. The IBM PAT test has three parts, one of which is arithmetic reasoning, or mathematical word problems.
Switching away from pure mathematical ability were Evans and Simkin [3], who looked at many factors to indicate success, among them being prior computing experience, academic factors, and the Myer's-Briggs Cognitive Style Indicator. This too did not have a highly successful predictive power. Cantwell Wilson and Shrock [1] found that student's comfort level in the course was the highest predictor of success. This was followed by mathematical background, which they recommended when advising students on choosing the curriculum.
Kurtz [4] administered a formal reasoning test that included questions on logic, reasoning, probability, and proportions. In this work, the rating scheme for students is not clearly defined, nor validated.
The ultimate goals of the research are to first identify the breadth of skills needed for a student's success in the field of computer science and then to create a test that can be administered to incoming students that can accurately assess their current level of ability in computer science. The test should incorporate a way to analyze all of the skills needed to be a good computer scientist, not just for preliminary success with syntax and a compiler.
The current hypotheses are:
At the time of this writing, only preliminary background information has been gathered. Much analysis has yet to be done on the current metrics for looking at student success in computer science, as well as looking at what elements are missing from standardized testing metrics that would give a better indication of possible success in this field.
The first impression of most of this work with predictors is that most of it is looking at students of many years ago. Students of that time were not exposed to computers in their young academic careers as many of today's students have been. This comfort level with the computer as a machine common to their everyday lives could be a significant factor in a student's success. This idea of comfort is not one with the class or classroom environment, but the actual idea of working with a computer, which would impact far beyond one course into a student's career.
From personal observation and interactions with students, I have discovered that there are many majors that are gifted in the field that could be categorized as "math phobic" or at least "math hesitant". It would seem that for these students, SAT math scores or any focus on math would not lend to a true predictor for their success in the field.
Previous programming experience can indicate that students have jumped the first hurdle of getting their hands dirty with syntax and have learned how to beat the compiler, but not that they will be able to solve the more complex problems that computer scientists are frequently presented with. Many "self-learned" programmers do not last much more than a few weeks into the major before disliking the courses and the content. Therefore, it seems that being able to write a working program does not seem to be a good enough predictor.
Lastly, there is my own personal experience with the Cornell Critical Thinking Test. I found that while it is fairly strong at testing logical thinking ability along with the ability to recognize contradictions and false statements, it seems to lack the more in depth questions about problem solving that successful programmers should be able to answer. The ability to understand logic and the logic of programs will only get a computer scientist so far. The ability to design a solution to a problem and see the relationships between entities in a program will suit a student much more. This test does not seem to test that ability to the degree necessary.
Open issues for this research include
I am currently pre-proposal stages. I am hoping to defend my proposal within the next year.
I hope to gain input and feedback about my research ideas. I am also hoping for informed guidance on the approach I am taking towards my research and suggestions on how to proceed forward.
[1] Cantwell Wilson, B., & Shrock, S. (2001). Contributing to success in an introductory computer science course: A study of twelve factors. ACM SIGCSE Bulletin, Proceedings of the thirty second SIGCSE technical symposium on Computer Science Education, 33(1), 184-188.
[2] Ennis, R. H. and Millman, J. (1985). Cornell critical thinking test, level Z. Pacific Grove, CA: Midwest Publications.
[3]Evans, G. E., & Simkin, M. G. (1989). What best predicts computer proficiency? Communications of the ACM, 32(11), 1322-1327.
[4]Kurtz, B. L. (1980). Investigating the relationship between the development of abstract reasoning and performance in an introductory programming class. The papers of the eleventh SIGCSE technical symposium on Computer science education : SIGCSE bulletin, 110 - 117.
[5] Leeper, R. R., & Silver J. L. (1982). Predicting success in a first year programming course. Paper presented at the Thirteenth SIGCSE Technical Symposium on Computer Science Education, Indianapolis, Indiana.
[6] Mazlack, L. J. (1980). Identifying potential to acquire programming skill. Communications of the ACM, 23(1), 14-17.
[7] Ventura, P. R. (2002). Objects-first CS1 not considered harmful: An empirical investigation of CS1. Paper presented at the SIGCSE Doctoral Consortium, Convington, Kentucky.