Course News

• Feb 26Follow Piazza for more recent course updates
• Feb 3 Our calendar has been updated with all OH (including Friends of the Course OH)
• Feb 3 We are proud that some of the brightest students from the previous semesters will join our Instructors team as Friends of Course. Please welcome - Mudita, Weijin and Nathan! They are willing to give you additional help and assist you in your success throughout the course.
• Jan 27 Movie about Alpha Go, we have talked about today (Trailer)
• Jan 27 Please complete the beginning of the semester survey here
• Jan 27 Welcome to all returning & new students. We have a great semester ahead!

Our Group

This was a great although challenging semester! We have encountered a great challenge in transitioning from the off-line to on-line mode of learning. I am so grateful that you successfully rose to the challenge of this transition and that we were able to gain even more than expected.

Selected Final Projects Presentations

• Augmented Random Search (MuJoCo) Gautam Suryawanshi, Prajit Krisshna Kumar
• Group Equivariant Q-networks Kiran Vaddi
• Ship Docker Problem (MARL) Bhavin Jawade, Vamshi Krishna
• DeepRacer Car Xingtong Li
• City On Fire: Multi Agent Reinforcement Learning Srisai Karthik Neelamraju, Anantha Srinath Sedimbi
• UAV Learning to Fly Rugved Srinath Hattekar, Chen Zeng, Nanditha
• AWS DeepRacer Raunaq Jain, Bipul Kumar, Joseph Naro
• Halite III: Multi-agent Learning Victor Lee
• Cracking PHYRE with a World Model Sheng Liu
• Connected Autonomous Driving (Macad-Gym) Salil Dabholkar
• Building Goal Oriented Chatbots Charan Nama Arunkumar
• Multi-Agent RL Hoan Tran
• HTTP Streaming & Optimization Areas Bekir Oguzhan Turkkan
• Multi Agent Reinforcement Learning Peter M. VanNostrand
• Chase Tag Game (MARL) Gaurav Vijay Vergiya

Prerequisites

• CSE 410: CSE 250, MTH 309 and one of the following: EAS 305 or MTH 411 or STA 301.
• CSE 4/510: CSE4/574 or CSE4/555 or CSE4/573 or CSE4/568 is recommended to be either completed or taken during the same semester.

Redeem your Candy Question bonus.

[Bonus request should be submitted within two days after the lecture, otherwise they will expire.]

Key Topics

• RL task formulation (action space, state space, environment definition)
• Tabular based solutions (dynamic programming, Monte Carlo, temporal-difference)
• Function approximation solutions (Deep Q-networks)
• Policy gradient from basic (REINFORCE) towards advanced topics (Trust Regions Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), etc.)
• Model-based reinforcement learning
• Imitation learning (behavioral cloning, inverse RL)
• Meta-learning
• Multi-agent reinforcement learning (MARL), partial observable environments

Grading Rubrics

Course Component	% of grade
Assignments [3 assignments: 15% + 15% + 10%]	40%
Final Project	20%
Weekly Quizzes	10%
Midterm 1	15%
Midterm 2	15%

Bonus Points

• Piazza Rockstar
• Jupyter Demo Time
• Candy Questions
• Poster Session Partiipation
• Other activities to be released as the course goes

Late Day Policy

• Students can use up to 5 free late days throughout the course that can be applied towards the assignments and/or a course project (some assignments may have a hard deadline)
• A late day extends the deadline by 24 hours
• If there is more than 5 days after the deadline, a penalty of 25% for one day will be applied to any work submitted after that time
• For the group work [course project] max late days left among teammates can be applied

Weekly Quizes - How does it work?

• Released every Monday 9:00am, due by Sunday 11:59pm
• Can be found at UBlearns > Assignments
• Each quiz will contain 3-4 problems on topics covered that week
• The quiz can be either multiple choise questions or written
• 11 quizzes in total, only 10 quizzes with the highest scores will be counted toward the final grade
• If the quizz is multi-choice questions - three attempts allowed, only the highest score will be kept

Generic requirements:

• Experience in Python

All class assignments are composed in Python (using numpy / Tensorflow / Keras / PyTorch). If student has extensive programming experience in a different language (e.g. C/ C++/ Matlab/ Javascript) this will be acceptable.

• Calculus, Linear Algebra

Student should be comfortable taking derivatives and understanding matrix vector operations and notations.

• Basic Probability and Statistics

Student should know basics of probabilities, conditional probability distributions, conditional expectations, Gaussian distributions, mean, standard deviation, etc.

• Foundations of Machine Learning and Deep Learning

Some knowledge of machine learning and deep learning will help to better understand the cost functions, approximation methods, NN and CNN structures.

Reference Materials

There is no official textbook for the class, but a number of the supporting readings will come from:

• Richard S. Sutton and Andrew G. Barto, "Reinforcement learning: An introduction", Second Edition, MIT Press, 2019 - is a classical book and covers all the basics
• Lecture slides, relevant papers, and other materials will be added in the table above

Additional references, that can be useful:

• Li, Yuxi. "Deep reinforcement learning." arXiv preprint arXiv:1810.06339 (2018). - an overview of the latest algorythms and applications in reinfocment learning
• David Silver's course on Reiforcement Learning

Useful RL Materials

• MDP Cheatsheet Reference (2 pages) by John Schulman (pdf)
• Neural Network tutorial Article + Sample code

Usefull Tools:

• Overleaf (LaTex online document generator) - great tool for creating reports (referral link)
• Google Colab (online Jupyter Notebook with free GPU) (link)

Academic Integrity Policy

Academic integrity is a fundamental university value. No collaboration, cheating, and plagiarism is allowed in projects, quizes, and the exam. Those found violating academic integrity will get an immediate F in the course.

1. All the submissions will be checked using SafeAssign (integrated tool into UBlearns). This tool checks the submissions for originality. After each submission it returns similarity score along with the original source. It is based on the submitted works for the past semesters as well the current submissions. If there is a high similarity between the submissions from the same class, ALL students involved will be recorded though the Academic Dishonesty Report.
2. The submissions should include all the references. Kindly note that referencing the source does not mean you can copy/paste it fully and submit as your original work. Updating the hyperparameters or modifying the existing code is a subject to plagiarism. Your work has to be original.
3. Submitting a material that has been previously submitted, in whole or in substantial part is a subject for further investigation.
4. You work has to be done on your personally owned device or university provided equipment. Completing the assignment on your friend’s / neighbor's device is a subject for further investigation for both parties involved. Any cheating during the midterms will be subject to immediate 0.
5. In any suspicious case that includes a student or a group of students will be immediately passed to the undergraduate/graduate adviser and the case will be recorded using the Academic Dishonesty Report Form.
6. Please read more about UB Academic Integrity Policy 2019-20.

We want you to demonstrate your own achievements and showcase your own abilities during the course! From the course instructors side, we are glad to provide you all the help needed for you to succeed in the course. Here is some of the free resources provided by the University:

1. If you need help with English, check UB Writing Center (http://www.buffalo.edu/writing/make-an-appointment-old.html)
2. If you have issues with your device, the University provides access to computers, as well as equipment loans.
3. Your well-being is highly important, if you have any concerns, make sure to check Counseling Service.

Academic Integrity is a very high priority not only for our Department, but the University as a whole. We are glad to provide you help to ensure you achieve great results during the course, however we are not tolerate any kind of cheating.

Accessibility Resources

If you have any disability which requires reasonable accommodations to enable you to participate in this course, please contact the Office of Accessibility Resources, 60 Capen Hall, 645-2608, and also the instructor of this course. The office will provide you with information and review appropriate arrangements for reasonable accommodations. More details.

Diversity

The UB School of Engineering and Applied Sciences considers the diversity of its students, faculty, and staff to be a strength, critical to our success. We are committed to providing a safe space and a culture of mutual respect and inclusiveness for all. We believe a community of faculty, students, and staff who bring diverse life experiences and perspectives leads to a superior working environment, and we welcome differences in race, ethnicity, gender, age, religion, language, intellectual and physical ability, sexual orientation, gender identity, socioeconomic status, and veteran status.

FAQ

Is there any GPU available to use for our projects?

CCR is supporting our course with accessing to powefull GPU servers. If you need access to GPU, create an account at CCR and let me know, so I will add you to the resources.

I am in the waiting list, can you help me to enrol?

Unfortunately there is nothing we can do at this time. I would suggest to keep an eye at the enrollment. Typically some students drop the course right before the drop-date deadline (Feb 4), so if your are in the waiting list, there is a high chance you will get enrolled, so I would strongly suggest to visit the lectures, before the enrolment is finilized, even if you are not registered at this time.

Can this course satisfy breadth/depth requirement?

Yes, the course can be used to satisfy the depth requirement for the AI focus area for graduate level (CSE 510).

What programming language will be used?

We will be using Python as the programming language for the projects, also familarity with Keras/Tensorfow/PyTorch will help.

Is attendance required?

Attendance is not required but is encouraged. Sometimes we may do in class exercises or discussions related to quizes or projects and these are harder to do and benefit from by yourself

I am highly interested in the course, can audit it?

Typically I welcome students interested in the topics to audit the course. Unfortunately this Spring our scheduled room is not big enough to fill all people interested. You are welcome to drop me an email one week after the class begins, I will give you updates if there is some space available.

Any suggestions or comments?

I would be glad to get a feedback from you, just send me an email.