Video

Logistics

We begin with the logistics of deadlines and the various things that have to be submitted. Later we will expand on what each milestone entails.

Deadlines

For your convenience, below are all the relevant deadlines. The deliverables are due by 11:00am of the due date.

Deliverable	Due Date
Team Composition and Case Study	February 28
Video link	April 10 April 20
Survey	April 13 April 22

Submission Deadlines

You form groups of size exactly three (3) for the mini-project. Below are the various logistics:

• A list of members of your team and your case study is due by 11:00am on Friday, February 28.

  Note

  Your group needs to have exactly THREE (3) members. In particular, I will not assign groups and it is your responsibility to make sure you form a group before the deadline. If you do not know many people in class, feel free to use piazza to look for group members. If you cannot form a group of size three by the deadline, then you get a zero for the entire mini-project.

• Every group has to pick a specific case study that shows the ethical impact of an algorithm (or a class of algorithms). While two groups can pick the same (class of) algorithm that solve (similar) problems, the ethical impacts have to be different for different group. A list of case studies already chosen can be found online. For the purposes of the mini project a case study is a real life occurrence that highlights the ethical impact of algorithms. You will need to submit the following information as part of your case study:
  • Your chosen algorithm,
  • Your case study (i.e. the application of your chosen algorithm), and
  • URLs for your main reference(s).

  Submission Information

  Please submit the required information for you team via this Google form . If you do not see your submitted information on the chosen case studies page within a week of you filling in the form, please send Erdem an email.

• The group will submit a link to a YouTube video that the group has created. The link is due by 11:00am, Friday, April 10 11:00am, Monday, April 20.
• Each group member will individually fill in a survey by 11:00am, Monday, April 13 10:59am, Wednesday, April 22. (See below for submission instructions.)

Grading

The video and the survey will each be worth $4\%$ and $1\%$ of the grade respectively. All members of the group will receive the same grade for the video submission. The survey grades will be individual grades: the actual survey grade will depend on the group video grade as well as the individual survey responses of members in the group.

Note

If a group does not submit their choice via the Google form then they will get a zero for the entire mini project.

• For the mini project there is no restriction on what sources you can use.

Details on your submissions

Next we present what is expected from each of the video and survey as well as their grading guidelines.

The video

Some more remarks before we go into the details:

• If you prefer, you can submit an unlisted YouTube video if you would not like your video be public.

The full details on the video part are below.

1. Brief description of the case study along with a reference for your case study.
   • This should include a brief description of the problem, and
   • A brief description of how the chosen algorithm(s) work: i.e. a brief algorithm idea.
2. Brief description of the ethical problem(s) in the scenario considered in the case stud. Your video should identify the primary and any secondary ethical issues (if any) and explain why these are ethical issues.

   "Independence" of ethical scenarios

   Note that these issues would be independent of your chosen algorithm.

3. Brief description of the stakeholder(s) in the case study. The video should identify all of the stakeholders and correctly explain their positions on the issues relevant to the case study.

   Who is a stakeholder?

   Any individual or group of people or organization that is impacted by your chosen algorithm(s) is a stakeholder in the case study.

4. Brief description of the the potential biases in the data or algorithms used in the case study. The video should explain the all of the biases resulting from using these data/algorithms.

   What is the definition of bias?

   Great Question! There is actually no agreement on one definition of bias. For example, here is a tutorial on twenty-one(!) definitions of fairness:

   Unlike most of 331, we will not insist on a formal definition of bias for your videos: any reasonable interpretation of bias will do. E.g. one common definition of bias is with respect to dis-proportionate representation. At a high level, "bias" is present in data when certain groups of people are under-represented or over-represented in a data (as compared to say the general population). At a high level, an algorithm is "biased" if its outcome dis-proportionately favors or dis-favors a group of people.

5. Brief description of the ethical impact(s) of your chosen algorithm in your case study. The video should describe the result(s) of your chosen algorithm, how stakeholders were affected by the algorithm(s)' outcomes, and ways to minimize or negate any negative effects of the chosen algorithm in the case study.

   Mitigating negative affects of the chosen algorithm

   This is the only part of the video where you can be speculative. Of course if you can find a reference that talk about the mitigation of the negative effects of the chosen algorithm, then go ahead and use it. If not, you can put in your thoughts for this part. However, note that for every other part of the video, any claim made must be backed up by references.

6. Bonus! Brief description of other possible algorithms that solve the same problem as your chosen algorithm and compare their ethical impacts. The video should describe another possible algorithm, describe the impacts of this other algorithm on the stakeholders, and compare it with your chosen algorithm based on the ethical and performance tradeoffs.

   Comparison needs a reference!

   Any claims on the other algorithm have to be backed up by references! Any speculative claims/comparisons will get a zero on this part.

Below we list some relevant reminders and clarifications.

Submission

You need to submit one PDF file to Autolab. The only thing the PDF needs to have is the link to your video.

Grading Guidelines

We will follow the usual grading guidelines for non-programming questions. Here is a high level grading rubric specific to the video:

1. Overall video quality: 10 points. The video does not need to be of production quality but it needs to be engaging. Make sure that your video is audible!
2. Case study description: 10 points. Out of this description of the problem being solved is worth 5 points while the algorithm idea is worth 5 points.
3. Ethical Problems: 20 points. To get full credit your claimed ethical problems must be backed up by a reference.
4. Stakeholders: 10 points. To get full credit your claims on stakeholders must be backed up by a reference.
5. Potential biases: 25 points. To get full credit your claimed impact must be backed up by a reference.
6. Ethical Impact: 25 points. To get full credit your claimed impact must be backed up a reference except for the part on mitigating negative effects of your chosen algorithm. The mitigation part is worth 10 points.
7. Other algorithms: 10 points. To get full credit your claims on the other algorithm must be backed up by a reference. Also since this is a bonus part of the project, it will be graded much much more strictly than the other parts of the video.

Survey

Each group member will fill in a survey rating their own and their other group member's contribution to the mini project under the categories of team role, leadership, participation, professionalism and quality of work (on scale of $0-3$ on each). These scores will then be used to divide the team’s points so that individual students’ survey grades reflect how well they contributed to the overall result. The table below explains what the different numerical values for various categories mean.

Category	0 points (Unsatisfactory)	1 points (Developing)	2 points (Satisfactory)	3 points (Exemplary)
Role	Does not willingly assume team roles; Rarely completes assigned work	Usually accepts assigned team roles; Occasionally completes assigned work	Accepts assigned team roles; Mostly completes assigned work	Accepts all assigned team roles; Always completes assigned work
Leadership	Rarely takes leadership role; Does not collaborate; Sometime willing to assist teammates	Occasionally shows leadership; Mostly collaborates; Generally willing to assist teammates	Shows an ability to lead when necessary; Willing to collaborate; Willing to assist teammates	Takes leadership role; Is a good collaborator; Always willing to assist teammates
Participation	Often misses meetings; Routinely unprepared for meetings; Rarely participates in meetings and does not share ideas	Occasionally misses or does not participate in meetings; Somewhat prepared for meetings; Offers unclear or unhelpful ideas in meetings	Attends and participates in most meetings; Comes prepared to meetings; Offers useful ideas in meetings	Attends and participates in all meetings; Comes prepared to meetings; Clearly expresses well-developed ideas in meetings
Professionalism	Often discourteous and/or openly critical of teammates; Does not want to listen to any alternate perspectives	Not always considerate or courteous towards teammates; Usually appreciates teammates' perspectives, but often unwilling to consider them	Mostly courteous to teammates; Values teammates' perspectives and often willing to consider them	Always courteous to teammates; Values teammates' perspectives, knowledge, and experiences, and always willing to consider them
Quality	Rarely contributes to preparing and making of the video; Others often required to revise, debug, or fix their work	Occasionally contributes to preparing and making of the video; Others sometimes needed to revise, debug, or fix their work	Often contributes to preparing and making of the video; Others occasionally needed to revise, debug, or fix their work	Frequently contributes to preparing and making of the video; Others rarely needed to revise, debug, or fix their work

Submitting the survey

The peer evaluation survey will have to be filled on https://cse.buffalo.edu/teamwork . You will evaluate yourself and your groupmates in all the five categories.

Other Comments

• Unlike other aspects of the course, for the video component of mini project you can refer to any source you want as long as (i) you explicitly refer to your source and (ii) the video is your own.
• Once a case study has been chosen by a group, the choice cannot be changed. Hence, make your choice carefully, keeping in mind that you need to demonstrate the ethical impact of algorithms in your chosen case study.

Resources

Reading material

This reading list on societal impacts of algorithms has a lot of pointers. Happy reading!