CAS CS 591, Programmable Smart Machines

Schedule

Thu -- 6:00pm-9:00pm

Course Outline

A longstanding goal in computer science has been to build machines capable of automatically improving based on their experience and size. In his 1968 paper, 'Memo' Functions and Machine Learning (Nature, Apr. '68), Donald Michie challenged computer scientists with the following goal:

"It would be useful if computers could learn from experience and thus automatically improve the efficiency of their own programs during execution... When I write a clumsy program for a contemporary computer a thousand runs on the machine do not re-educate my handiwork. On every execution, each time-wasting blemish and crudity, each needless test and redundant evaluation, is meticulously reproduced."

The course will be predominantly structured as a seminar in which each week we will be looking at one or two research papers. You will be required to submit weekly reviews and actively participate in the discussions. Each student will conduct and a term long project.

Instructors

Jonathan Appavoo: jappavoo@bu.edu

Steve Homer: homer@cs.bu.edu

Amos Waterland: apw@seas.harvard.edu

Target audience

This course is targeted towards graduate and advanced undergraduate students. We encourage enrollment by both theory and systems students, especially those of you willing to push the boundaries of what you have been thinking about. You are encouraged to incorporate your personal interests and background into the class material.

Prerequisites

Students taking this class must have permision from the instructors, be a either a senior or graduate student, and be proficient in the core material of systems and theory.

Workload

Each week we will be covering on average one to two research papers that you will be expected to read, review and discuss. These papers will likely require that you find and read additional material as necessary to ensure your comprehension. Do not underestimate the amount of work this can be. You will be required to submit a written review of the papers prior to class. You will also be expected to actively participate in the in-class discussion. Each student is expected to lead one of the class discussions by summarizing the paper and seeding conversation with questions and observations from the paper.

In addition to the weekly paper reviews there will be a final project in which you will explore in detail an aspect of Programmable Smart Machines. Projects can range from a theoretical exploration to an applied experimental evaluation. The topic of your project must be approved by the instructors. You will be expected to present a poster about your project and to submit a brief written report by the end of the term. As part of the project you will be expected to establish in conjunction with the instructors the goals and criteria for its evalution. Each week you will provide a brief update on the progess of your project.

The project is due by the end of the lecture period. The project presentations will be given in the form of a final poster during the scheduled final exam slot.

Students are expected to work individually on the weekly reviews. With instructor approval the final project may be done in groups of up to two. There will be no final exam other than the poster presentations.

Grading

Project Poster and Report: 60%

Tentative schedule

Papers & Project Details

List of some possible papers:

Theory
1. Kushilevitz Mansour learning algorithm via Fourier coefficient analysis. link KUSHILEVITZ, E., and MANSOUR, Y., "Learning decision trees using the Fourier spectrum", SIAM Journal on Computing, Vol. 22, No. 6, pp. 1331-1348, 1993. If you want to dig deeper you can look up the by paper Goldreich and Levin which is referenced in the KUSHILEVITZ, E., and MANSOUR, Y paper and forms the basis for their method.
2. Fourier analysis on the cube."A Brief Introduction to Fourier Analysis on the Boolean Cube" By Ron de Wolf. link This is a good intro to some of the ideas underlying the learning theory above.
3. Paul Vitanyi et al on compression, relative compressibility, normalized compression distance and various thing related to Kolmogorov complexity: "Clustering by compression", by Vitanyi and Cilibrasi (link) and follow the backward and forward references to many related papers. Other papers by Vitanyi and others include: Nonapproximablity of the Normalized Information Distance, New applications of the incompressibility method: Part II Harry Buhrman, Tao Jiang, Ming Li, Paul Vitanyi, Philosophical Issues in Kolmogorov Complexity, Normalized Information Distance is Not Semicomputable, On Prediction by Data Compression
4. "Memo" functions and Machine Learning
   D. Michie
   Nature, 1968
5. A bridging model for parallel computation
   L. Valiant
   CACM, 1990
6. Limits to Parallel Computation (Chapter 2)
   R. Greenlaw
   1991
7. Systems
8. Learning
9. Neuromorphic

Project Ideas

Fourier Analysis
1. Study a set of functions using Fourier Analysis
ASC
2. Add Input output support
3. Break on PC values of read write open extend for IO
4. Build OS calls into ASC
5. Visualization
6. Speculative execution with IO
7. Study / explore a class of programs in asc

Neueral Network Hardware
8. Add a UART TX/RX wrapper interface to a version of DANA that can be used to communication an NN Transaction to DANA
9. Currently, the modified FANN configuration only supports an all-to-all connectivity subset of FANN configurations. This needs to be relaxed to enable more limited connection strategies. This is a precursor to enabling Convolutional Neural Network (CNN) support. Note, that this specific issue does not allow for connections that skip layers.
10. There exists the potential for some fault-tolerance within neural networks as evidenced by previous work with modular redundancy and no voting, see BARC 2015. A more advanced approach involves dynamically splitting "important" nodes or ignoring "un-important" nodes while an NN is executing (on an NN accelerator like DANA or simply in software). Much of this work is TBD, but this should be at least one publishable unit of work, here.
11. The FANN should be generic enough to support Convolutional Neural Networks (CNNs). However, this support doesn't already exist. At the software level, we need to add support for this. What is primarily needed: 1) Add max-pooling activation function 2) Possibly some support for describing a CNN and then mapping this to a FANN configuration 3) Evaluation and testing
12. The current version of DANA (at least at the level of gem5 integration) uses an x86 ISA interface. This needs to be moved to the RISC-V ISA. Note, that the RISC-V ISA should have support for generic "accelerator" extensions. However, we may need more support than this currently offers.

Collaboration and Academic Honesty

All course participants must adhere to the CAS Academic Conduct Code. Instances of academic dishonesty will be reported to the academic conduct committee.