Machine Learning in Production (17-445/17-645/17-745)

Fall 2022

<em>Formerly Software Engineering for AI-Enabled Systems (SEAI) and also taught as AI Engineering (11-695), CMU course that covers how to build, deploy, assure, and maintain products with machine-learned models. Covers also responsible AI (safety, security, fairness, explainability) and MLOps. The course is crosslisted both as Machine Learning in Production and AI Engineering. For earlier offerings see websites for Fall 2019, Summer 2020, Fall 2020, Spring 2021 and Spring 2022. This Fall 2022 offering is designed for students with some data science experience (e.g., has taken a machine learning course, has used sklearn) and basic programming skills, but will not expect a software engineering background (i.e., experience with testing, requirements, architecture, process, or teams is not required). Going forward we expect to offer this course at least every spring semester and possibly some fall semesters (not summer semesters).</em>

---

Note for Spring 2023: We have a fairly long waitlist on all sections for master students, but we are optimistic that we will be able to enroll most students within the first week of the semester. Spring 2023 website

For researchers, educators, or others interested in this topic, we share all course material, including slides and assignments, under a creative commons license on GitHub (https://github.com/ckaestne/seai/) and have also published an article describing the rationale and the initial design of this course: Teaching Software Engineering for AI-Enabled Systems. A textbook is emerging. Video recordings of the Summer 2020 offering are online on the course page. We would be happy to see this course or a similar version taught at other universities. See also an annotated bibliography on research in this field.

Course Description

This is a course for those who want to build applications and products with machine learning. Assume you can learn a model to make predictions, what does it take to turn the model into a product and actually deploy it, have confidence in its quality, and successfully operate and maintain it?

The course is designed to establish a working relationship between software engineers and data scientists: both contribute to building AI-enabled systems but have different expertise and focuses. To work together they need a mutual understanding of their roles, tasks, concerns, and goals and build a working relationship. This course is aimed at software engineers who want to build robust and responsible systems meeting the specific challenges of working with AI components and at data scientists who want to understand the requirements of the model for production use and want to facilitate getting a prototype model into production; it facilitates communication and collaboration between both roles. The course is a good fit for student looking at a career as an ML engineer. The course focuses on all the steps needed to turn a model into a production system in a responsible and reliable manner.

It covers topics such as:

• How to design for wrong predictions the model may make? How to assure safety and security despite possible mistakes? How to design the user interface and the entire system to operate in the real world?
• How to reliably deploy and update models in production? How can we test the entire machine learning pipeline? How can MLOps tools help to automate and scale the deployment process? How can we experiment in production (A/B testing, canary releases)? How do we detect data quality issues, concept drift, and feedback loops in production?
• How do we scale production ML systems? How do we design a system to process huge amounts of training data, telemetry data, and user requests? Should we use stream processing, batch processing, lambda architecture, or data lakes?
• How to we test and debug production ML systems? How can we evaluate the quality of a model’s predictions in production? How can we test the entire AI-enabled system, not just the model? What lessons can we learn from software testing, automated test case generation, simulation, and continuous integration for testing for production machine learning?
• Which qualities matter beyond a model’s prediction accuracy? How can we identify and measure important quality requirements, including learning and inference latency, operating cost, scalability, explainablity, fairness, privacy, robustness, and safety? Does the application need to be able to operate offline and how often do we need to update the models? How do we identify what’s important in a AI-enabled product in a production setting for a business? How do we resolve conflicts and tradeoffs?
• How do we build effective interdisciplinary teams? How can we bring data scientists, software engineers, UI designers, managers, domain experts, big data specialists, operators, legal council, and other roles together and develop a shared understanding and team culture?

Examples and case studies of ML-driven products we discuss include automated audio transcription; distributed detection of missing children on webcams and instant translation in augmented reality; cancer detection, fall detection, COVID diagnosis, and other smart medical and health services; automated slide layout in Powerpoint; semi-automated college admissions; inventory management; smart playlists and movie recommendations; ad fraud detection; delivery robots and smart driving features; and many others.

An extended group project focuses on building, deploying, evaluating, and maintaining a robust and scalable movie recommendation service under somewhat realistic “production” conditions.

Learning Outcomes

After taking this course, among others, students should be able to

• analyze tradeoffs for designing production systems with AI-components, analyzing various qualities beyond accuracy such as operation cost, latency, updateability, and explainability
• plan for mistakes in AI components and implement production-quality systems that are robust to those mistakes
• design fault-tolerant and scalable data infrastructure for learning models, serving models, versioning, and experimentation
• ensure quality of the entire machine learning pipeline with test automation and other quality assurance techniques, including automated checks for data quality, data drift, feedback loops, and model quality
• build systems that can be tested and monitored in production and build robust deployment pipelines
• consider system-level requirements such as safety, security, privacy, fairness, and usability when building complex AI-enabled products
• communicate effectively in interdisciplinary teams

In addition, students will gain familiarity with production-quality infrastructure tools, including stream processing with Apache Kafka, test automation with Jenkins, monitoring with Prometheus and Grafana, and deployment with Docker and various MLOps tools.

Logistics and People

17-445/17-645/17-745, 12 Units

The course is the same under all course numbers, with the exception of the PhD-level 17-745 which replaces two homework assignments with a mandatory research project.

Open to undergraduate and graduate students meeting the prerequisites.

Fall 2022

Lectures Monday/Wednesday 1:25-2:45pm, in person, TEP 1308

Recitations Friday 10:10-11:00am in Wean 5409 and 1:25-2:55pm in GHC 5222

Instructors: Christian Kaestner

Coordination

We are happy to answer questions by email, over Slack, over Canvas, meet in person, and will jump on a quick Zoom call if you ask us. We also always arrive 5 to 10 min early to class and stay longer for discussions and questions.

Course content

The general course content has been fairly stable over the last few years, though specific topics and tools are constantly updated with new research and tooling. Our list of learning goals under Learning Goals describes what we aim to cover. Below is a table of a preliminary schedule. This is subject to change and will be updated as the semester progresses, especially to help focus on requested topics or support learning.

Schedule

Course Syllabus and Policies

The course uses Canvas and Gradescope for homework submission, grading, discussion, questions, announcements, and supplementary documents; slides will be posted here; Slack is used for communication around homeworks and projects; Github is used to coordinate group work. All public course material (assignments, slides, syllabus) can be found in the course’s GitHub repository; announcements and all private material (e.g., grades, passwords) will be shared through Canvas.

Prerequisites: The course does not have formal prerequesites, but we describe background knowledge that will help you be successful in the course. In a nutshell, we expect basic exposure to machine learning and basic programming skills, but do not require software engineering experience.

Machine learning (some experience recommended): We suggest that you have basic familiarity with