Big Data Analytics (EECS E6893) and Advanced Big Data Analytics (EECS E6895)





E6893 Student List and Group Assignments
Big Data Public Dataset Information


COURSE BENEFITS:

  • Students will gain knowledge on analyzing Big Data. It serves as an introductory course for graduate students who are expecting to face Big Data storage, processing, analysis, visualization, and application issues on both workplaces and research environments.

  • Gain knowledge on this fast-changing technological direction. Big Data Analytics is probably the fastest evolving issue in the IT world now. New tools and algorithms are being created and adopted swiftly. Get insight on what tools, algorithms, and platforms to use on which types of real world use cases.

  • Get hands-on experience on Analytics, Mobile, Social and Security issues on Big Data through homeworks and final project

  • Final Project Reports will be published as Proceedings and Final Project Software will become Open Source. (Sapphirine Big Data Analytics Open Source Applications first release: Dec 22, 2014)


PROFESSOR CHING-YUNG LIN:


    Dr. Ching-Yung Lin is the IBM Chief Scientist, Graph Computing Research and an IBM Distinguished Researcher. He is also an IEEE Fellow and IEEE Distinguished Lecturer. He has been also an Adjunct Professor in Columbia University since 2005 and New York University since 2014. His interest is mainly on fundamental research of large-scale multimodality signal understanding, network graph computing, and computational social & cognitive sciences, and applied research on security, commerce, and collaboration. Since 2011, he has been leading a team of more than 40 Ph.D. researchers in worldwide IBM Research Labs and more than 20 professors and researchers in 9 universities (Northeastern, Northwestern, Columbia, Minnesota, Rutgers, CMU, New Mexico, USC, and UC Berkeley). He is currently the Principal Investigator of three major Big Data projects: DARPA Anomaly Detection at Multiple Scales (ADAMS), DARPA Social Media in Strategic Communications (SMISC), and ARL Social and Cognitive Network Academic Research Center (SCNARC). He leads a major IBM R&D initiative on Linked Big Data called IBM System G. Dr. Lin was the first IEEE fellow elected for contributions to Network Science. His team recently earned the Best Paper Awards on ACM CIKM 2012 and IEEE BigData 2013.

 


APPLICABLE DEGREE PROGRAMS:

  • Recommended for MS or Ph.D. students in Electrical Engineering, Computer Science or any discipline requires big data analytics.
  • Most courses 4000-level and above can be credited to all degree programs.  All courses are subject to advisor approval.

COURSE FEES:

  • None


EECS E6893: Big Data Analytics

Lecturer/Manager:

Ching-Yung Lin

 

Office Hours:

Thursday 9:40 - 10:10pm or by appointment

Office Location/Phone:

SIPA 417

Email Address:

c {dot} lin {at} columbia {dot} edu

 

Day & Time Class 
Meets on Campus:

Thursday 7:10pm - 9:40pm

Location:

International Affairs Building (SIPA building) 417

Credits for course:

3

Class Type:

Lecture


Prerequisites:

This will be a hands-on course. Students need to know at least one or more programming languages: C, C++, Java, Perl, Python, and/or Javascript to finish homeworks and final project.


Description:

With the advance of IT storage, pcoressing, computation, and sensing technologies, Big Data has become a novel norm of life. Only until recently, computers are able to capture and analysis all sorts of large-scale data from all kinds of fields -- people, behavior, information, devices, sensors, biological signals, finance, vehicles, astronology, neurology, etc. Almost all industries are bracing into the challenge of Big Data and want to dig out valuable information to get insight to solve their challenges.


This course shall provide the fundamental knowledge to equip students being able to handle those challenges. This discipline inherently invoves many fields. Because of its importance and broad impact, new software and hardware tools and algorithms are quickly emerging. A data scientist needs to keep up with this ever changing trends to be able to create a state-of-the-art solution for real-world challenges.


This Big Data Analytics course shall first introduce the overview applications, market trend, and the things to learn. Then, I will introduce the fundamental platforms, such as Hadoop, Spark, and other tools, such as IBM System G for Linked Big Data. Afterwards, the course will introduce several data storage methods and how to upload, distribute, and process them. This shall include HDFS, HBase, KV stores, document database, and graph database. The course will go on to introduce different ways of handling analytics algorithms on different platforms. Then, I will introduce visualization issues and mobile issues on Big Data Analytics. Students will then have fundamental knowledge on Big Data Analytics to handle various real-world challenges.


Afterwards, the course will zoom in to discuss large-scale machine learning methods that are foundations for artificial intelligence and cognitive networks. The course will discuss several methods to optimize the analytics based on different hardware platforms, such as Intel & Power chips, GPU, FPGA, etc. The lectures will conclude with introduction of the future challenges of Big Data, especially on the onging Linked Big Data issues which involves graphs, graphical models, spatio-temporal analysis, cognitive analytics, etc.


Students will choose the topics of their own for a final project. The application domain can be based on the students' own interest. This will be a good opportunity for students to apply what's learned in the class for their needs, either for the future work requirements or for the research problems at hand.


TAs (Graders):

Eric Johnson (efj2106), Munan Cheng (munan.cheng), Kushwanth Shantharam (kk3098), Rohan Kulkarni (rohan.kulkarni), Gautam Sihag (gautam.sihag), Peiran Zhou (pz2210), Emily Yao (dy2307), Chuwen Xu (cx2178), and TBA.

Office Hours:

Peiran Zhou: Monday 9am - 11am; Gautam Sihag: Monday 2pm - 4pm

Emily Yao: Tuesday 10am - noon; Kushwanth Shantharam: Tuesday 1pm - 3pm

Chuwen Xu: Wednesday 9am - 11am; Munan Cheng: Wednesday 4pm - 6pm

Rohan Kulkarni: Friday 11:30am - 1:30pm

TBA

Office Location/Phone:

CS TA room

 

Required Textbook(s):

None

Reference Textbook(s):

class notes, and reference books or papers

Homework(s):

Three assignments (HW#1 - HW#3) including programming and written reports.

Project(s):

Final project in which students conduct research and hands-on implementation for self-selected topic on Big Data Analytics. Team collaboration of up to 3 students is encouraged.

Paper(s):

Report for each homework, the final project proposal, and the final project result. oral presentation of the final project results required. Remote student will use web conferecing tool to present the final project.

Midterm Exam:

None

Final Exam:

None

Grading:

Three homework assignments: 50%, Final Project (proposal, presentation, and report): 50%

Hardware
requirements:

PC with Internet access. 

Software
requirements:

Students may use their preferred software (C, C++, Java, Python, Perl, and/or Javascript) on their computers to complete homework assignments. 

Homework
submission:



by submission through Columbia CourseWorks

Course Outline

Class Date

Class 
Number

Topics Covered

Assignment

Due

09/08/16

1

Introduction to Big Data Analytics

 

09/15/16

2

Big Data Platforms

HW #1 Data Store & Processing using Hadoop

09/22/16

3

Big Data Storage and Analytics

 

09/29/16

4

Big Data Analytics ML Algorithms

 

HW #1

10/06/16

5

Big Data Analytics ML Algorithms II

HW #2 Recommendation, Clustering, and Classification using Mahout/Spark

10/13/16

6

Machine Learning, Streams, and Database on Spark

10/20/16

7

Linked Big Data -- Graph Computing

HW #2

10/27/16

8

Linked Big Data -- Graph Analytics

HW #3 Graph Database & Analysis and Machine Reasoning using System G

11/03/16

9

Graphical Models and Bayesian Networks

11/10/16

10

Big Data Visualization

HW #3

11/17/16

11

Final Project Proposal Presentations

 

Proposal Slides

11/24/16

 

NO CLASS -- Thanksgiving Holiday

 

 

12/01/16

12

Project Proposals II

 

 

12/08/16

13

Cognitive Mobile Analytics

 

 

12/15/16

14

Big Data Analytics Workshop

 

Final Project Slides

 

 


EECS E6895: Advanced Big Data Analytics

Lecturer:

Ching-Yung Lin

 

Office Hours:

Thursday 9:30 - 10:00pm or by appointment

Office Location/Phone:

Mudd 535

Email Address:

c {dot} lin {at} columbia {dot} edu

 

Day & Time Class 
Meets on Campus:

Thursday 7:00pm - 9:30pm

Location:

IAB 417

Credits for course:

3

Class Type:

Lecture


Prerequisites:

This will be a hands-on course. Students need to know at least one or more programming languages: C++, Java, Perl, Python, and/or Javascript to finish task milestones and final project.


Description:

The Big Data Analytics area evolves in a speed that was seldom seen in the history. New Software and Hardware tools are emerging and disruptive. Furthermore, its boundary with Artificial Intelligence becomes blurring. We may no longer find a clear distinction on what is a Big Data Analytics problem and what is an AI problem.


In this Advanced Big Data Analytics course, we will devote to something new -- "How far could we achieve to build a brain that mimics human functions through the state-of-the-art computer science and electrical engineering technologies?" What we would like to discuss is not machines that play Games (Chess, Question & Answering quiz, or Go) or recognize voice and face, but how machines could possibly achieve what are unique to the human beings. Our brains can reason, can associate, and can memorize. We have feeling, emotions, ethics and morality, arts, and consciousness. We dream during the night.


In this course, students will conduct Research and Development on the tasks that shall collectively contribute to building intelligent machines that are like human, or more knowledgable than human, through analyzing Big Data. Most lectures will be divided by two parts. The first part will be the presentations by Prof. Lin or guest speakers to explain the potential Computer Science / Electrical Engineering technologies for building such machines. The second part will be students' presentations on their progress in the 4 areas: (1) Cognitive Robot, (2) Finance Robo-Advisor, (3) Knowledge Graphs, and (4) Software for Next-Generation Hardware Platforms.


TAs (Graders):

TBD

Office Hours:

TBD

Office Location/Phone:

CS TA room, Mudd building

 

Required Textbook(s):

None

Reference Textbook(s):

class notes, and reference books or papers

Homework(s):

None


Task:

Each student will need to sign in a task in one of these four areas: (1) Cognitive Robot, (2) Finance Robo-Advisor, (3) Knowledge Graphs, and (4) Software for Next-Generation Hardwares. Each area will have 15 tasks. Task lists will be announced in Lecture 1. Each task will have three milestones. Each milestone includes programming, presentation and a written report.


Project:

Final project in which students define a Big Data Analytics application and apply the software built in any combination of the 60 tasks in the class to accomplish the project. Team collaboration of 2 students per project is encouraged.


Paper(s):

Report for each milestone and the final project results. Oral presentations of each milestone and the final project. Source codes will be submitted to a course Github repository. Final project will also include a video presentation.


Midterm Exam:

None

Final Exam:

None

Grading:

Task Milestones: 60% (3 milestones, each milestone: 20%), Final Project (presentation and report): 30%, Class Participation: 10%

Hardware
requirements:

PC with Internet access. 

Software
requirements:

Depending on the task, students will need to use appropriate software (C++, Java, Python, Perl, and/or Javascript) on their computers to complete the task milestones and the final project. 

Task
submission:



by submission through a course website.

Class Date

Class 
Number

Lecture Topics

Student Presentations

01/19/17

1

Advanced Big Data Analytics to Build a Brain

01/26/17

2

Foundations toward Human-Like Artificial Intelligence

 

02/02/17

3

Next-Generation Hardwares (I) -- GPU

Cognitive Robot (I)

02/09/17

4

Social Machines

Finance Robo-Advisor (I)

02/16/17

5

New Chips for AI & Big Data

Knowledge Graphs (I)

02/23/17

6

Ethics Machines (I)

SW for Next-Generation Hardwares (I)

03/02/17

7

Ethics Machines (II)

Cognitive Robot (II)

03/09/17

8

Feeling Machines (I)

Finance Robo-Advisor (II)

03/23/17

9

Feeling Machines (II)

Knowledge Graphs (II)

03/30/17

10

Arts-Aware Machines

SW for Next-Generation Hardwares (II)

04/06/17

11

Reasoning Machines (I)

Cognitive Robot (III)

04/13/17

12

Reasoning Machines (II)

Finance Robo-Advisor (III)

04/20/17

13

Consciousness Machines

Knowledge Graphs (III)

04/27/17

14

Dreaming Machines

SW for Next-Generation Hardwares (III)

05/11/17

15

Final Project Presentations