Access to doc 10 5o docs googleusercontent com was denied

Big Data

Compare and contrast five clustering algorithms on your own. Provide real-world examples to explain any one of the clustering algorithm. In other words, how is an algorithm beneficial for a process, industry or organization.
What clustering Algorithms are good for big data? Explain your rationale?
Please locate and review an article relevant to Chapter 4. The review is between 200-to-250 words and should summarize the article. Please include how it applies to our topic, and why you found it interesting.

Data Science & Big Data Analytics

Discovering, Analyzing, Visualizing and Presenting Data

EMC Education Services

WILEY

'

Data Science & Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data

Published by John Wiley & Sons, Inc. 10475 Crosspoint Boulevard Indianapolis, IN 46256 www. wiley. com

Copyright© 2015 by John Wiley & Sons, Inc., Indianapolis, Indiana

Published simultaneously in Canada

ISBN: 978-1-118-87613-8 ISBN: 978-1-118-87622-0 (ebk) ISBN: 978-1-118-87605-3 (ebk)

Manufactured in the United States of America

10987654321

No part ofthis publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permis- sion of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http: I /www. wiley. com/ go/permissions.

limit ofliability/DisclaimerofWarranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Web site is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or website may provide or recommendations it may make. Further, readers should be aware that Internet websites listed in this work may have changed or disappeared between when this work was written and when it is read.

For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 orfax (317) 572-4002.

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand.lf this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http: I /book support. wiley. com. For more information about Wiley products, visit www. wiley. com.

library of Congress Control Number: 2014946681

Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other coun- tries, and may not be used without written permission. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.

Credits

Executive Editor

Carol Long

Project Editor

Kelly Talbot Production Manager

Kathleen Wisor Copy Editor

Karen Gill Manager of Content Development

and Assembly

Mary Beth Wakefield Marketing Director

David Mayhew

Marketing Manager

Carrie Sherrill

Professional Technology and Strategy Director

Barry Pruett

Business Manager

Amy Knies Associate Publisher

Jim Minatel Project Coordinator, Cover

Patrick Redmond Proofreader

Nancy Carrasco Indexer

Johnna Van Hoose Dinse Cover Designer

Mallesh Gurram

About the Key Contributors

David Dietrich heads the data science education team within EMC Education Services, where he leads the

curriculum, strategy and course development related to Big Data Analytics and Data Science. He co-au- thored the first course in EMC's Data Science curriculum, two additional EMC courses focused on teaching leaders and executives about Big Data and data science, and is a contributing author and editor of this

book. He has filed 14 patents in the areas of data science, data privacy, and cloud computing. David has been an advisor to severa l universities looking to develop academic programs related to data

analytics, and has been a frequent speaker at conferences and industry events. He also has been a a guest lecturer at universi- ties in the Boston area. His work has been featured in major publications including Forbes, Harvard Business Review, and the 2014 Massachusetts Big Data Report, commissioned by Governor Deval Patrick.

Involved with analytics and technology for nearly 20 years, David has worked with many Fortune 500 companies over his career, holding multiple roles involving analytics, including managing analytics and operations teams, delivering analytic con-

sulting engagements, managing a line of analytical software products for regulating the US banking industry, and developing Sohware-as-a-Service and BI-as-a-Service offerings. Additionally, David collaborated with the U.S. Federal Reserve in develop-

ing predictive models for monitoring mortgage portfolios. Barry Heller is an advisory technical education consultant at EMC Education Services. Barry is a course developer and cur-

riculum advisor in the emerging technology areas of Big Data and data science. Prior to his current role, Barry was a consul- tant research scientist leading numerous analytical initiatives within EMC's Total Customer Experience organization. Early in his EMC career, he managed the statistical engineering group as well as led the

data warehousing efforts in an Enterprise Resource Planning (ERP) implementation. Prior to joining EMC,

Barry held managerial and analytical roles in reliability engineering functions at medical diagnostic and technology companies. During his career, he has applied his quantitative skill set to a myriad of business applications in the Customer Service, Engineering, Manufacturing, Sales/Marketing, Finance, and Legal

arenas. Underscoring the importance of strong executive stakeholder engagement, many of his successes

have resulted from not only focusing on the technical details of an analysis, but on the decisions that will be resulting from the analysis. Barry earned a B.S. in Computational Mathematics from the Rochester Institute ofTechnology and an M.A. in

Mathematics from the State University of New York (SUNY) New Paltz. Beibei Yang is a Technical Education Consultant of EMC Education Services, responsible for developing severa l open courses

at EMC related to Data Science and Big Data Analytics. Beibei has seven years of experience in the IT industry. Prior to EMC she worked as a sohware engineer, systems manager, and network manager for a Fortune 500 company where she introduced

new technologies to improve efficiency and encourage collaboration. Beibei has published papers to

prestigious conferences and has filed multiple patents. She received her Ph.D. in computer science from the University of Massachusetts Lowell. She has a passion toward natural language processing and data

mining, especially using various tools and techniques to find hidden patterns and tell stories with data. Data Science and Big Data Analytics is an exciting domain where the potential of digital information is maximized for making intelligent business decisions. We believe that this is an area that will attract a lot of talented students and professionals in the short, mid, and long term.

Acknowledgments

EMC Education Services embarked on learning this subject with the intent to develop an "open" curriculum and certification. It was a challenging journey at the time as not many understood what it would take to be a true

data scientist. After initial research (and struggle), we were able to define what was needed and attract very talented professionals to work on the project. The course, "Data Science and Big Data Analytics," has become

well accepted across academia and the industry. Led by EMC Education Services, this book is the result of efforts and contributions from a number of key EMC organizations and supported by the office of the CTO, IT, Global Services, and Engineering. Many sincere

thanks to many key contributors and subject matter experts David Dietrich, Barry Heller, and Beibei Yang for their work developing content and graphics for the chapters. A special thanks to subject matter experts John Cardente and Ganesh Rajaratnam for their active involvement reviewing multiple book chapters and

providing valuable feedback throughout the project.

We are also grateful to the fol lowing experts from EMC and Pivotal for their support in reviewing and improving the content in this book:

Aidan O'Brien Joe Kambourakis

Alexander Nunes Joe Milardo

Bryan Miletich John Sopka

Dan Baskette Kathryn Stiles

Daniel Mepham Ken Taylor

Dave Reiner Lanette Wells

Deborah Stokes Michael Hancock

Ellis Kriesberg Michael Vander Donk

Frank Coleman Narayanan Krishnakumar

Hisham Arafat Richard Moore

Ira Schild Ron Glick

Jack Harwood Stephen Maloney

Jim McGroddy Steve Todd

Jody Goncalves Suresh Thankappan

Joe Dery Tom McGowan

We also thank Ira Schild and Shane Goodrich for coordinating this project, Mallesh Gurram for the cover design, Chris Conroy and Rob Bradley for graphics, and the publisher, John Wiley and Sons, for timely support in bringing this book to the

industry.

Nancy Gessler

Director, Education Services, EMC Corporation

Alok Shrivastava

Sr. Director, Education Services, EMC Corporation

Contents Introduction ................ . .. . .....• . •.. ... .... •..... .. .. . .. . .......... .. ... . ..................... •.•...... xvii

Chapter 1 • Introduction to Big Data Analytics ................... . . . ....................... 1

1.1 Big Data Overview ..................... ....... .....•... • ...... . . . ........ • .. ... . . ... ....... ....... 2 1.1.1 Data Structures .. . .. . . . .. ................ ... ... . .. . ...... . .. .. .... . .................... ..... . .. . . . .. 5 1.1.2 Analyst Perspective on Data Repositories . ............................. . .......... .......•. ... ... .. .. 9

1.2 State of the Practice in Analytics ................................................................. . 11 1.2.1 Bl Versus Data Science .............. .... ....... . .. . ........... . . . .... . ....................... .. .... 12 1.2.2 Current Analytical Architecture ... . .... .• . . ................ .... .............. .... .... ...... •.. . ..... 13 1.2.3 Drivers of Big Data .................................................... . . . .. ................. .. ... . . 15 1.2.4 Emerging Big Data Ecosystem and a New Approach to Analytics .. ....... ...... . ............ .. ....... 16

1.3 Key Roles for the New Big Data Ecosystem ....... ..... ......... . ....... . ..... .. .................... 19 1.4 Examples of Big Data Analytics ... .... .......... .... . ... ....... ... .... . ...... . .................... 22 Summary .............. ............ ... ... ......... .... • ... •....... ........ .. • ..•... . ................ 23 Exercises ..................... .... ..... .. ...... . ......•......... .. .. . ... .... . ..•.................... 23 Bibliography ........................... .... .. ... ... ... •................... .. • ...... ..... ..... ....... 24

Chapter 2 • Data Analytics Lifecycle ..................................................... . 25 2.1 Data Analytics Lifecycle Overview ... ..... . ............. • ...... •.. ..... ...... • ... •............. . . . 26

2.1.1 Key Roles for a Successful Anolytics Project .... . .. . .... .... . ........ . .. .. . ..•......... •. •....... . .. . . 26 2.1.2 Background and Overview of Data Analytics Lifecyc/e .......................... . .......•... . ..... ... 28

2.2 Phase 1: Discovery ..... .. .. .. . ............................. . ..•..................... •........... . 30 2.2.1 Learning the Business Domain .. . ....... ... ..•.•. •.... . .. ..... . . .. . ...................•........... .30 2.2.2 Resources . . ... . ................... . ...... . ......................... ..... ............. •.......•.... 31 2.2.3 Framing the Problem ............•.... . ...................................•......... •.•.... . . ...... 32 2.2.41dentifying Key Stakeholders ... .. ....................... ... . ... ......... .... . ....... •. . .......... . . 33 2.2.51nterviewing the Analytics Sponsor ...... ........ ...... .. .......... .... ... .. ... ..... .. ........... ... 33 2.2.6 Developing Initial Hypotheses ................. .. . . . .. . . . .. . . . . ... .... .. ........... . . •............ . . 35 2.2.71dentifying Potential Data Sources . ... ...•. •.. .... . . .. . ......•. •.......... . ....... . ..... . ... . .. .. . . 35

2.3 Phase 2: Data Preparation ...........................................................•...•..•..... 36 2.3.1 Preparing the Analytic Sandbox ............... . ...................... ... •. •.......•.......... .. .... 37 2.3.2 Performing ETLT ..................................................................•.•.......•... .. . 38 2.3.3 Learning About the Data .. ..... . .............. .. ........................•.•.......•.•........ ..... . 39 2.3.4 Data Conditioning ....... .. ....•.......... . ....................... .. . .. . . . ......•. •............. .. .40 2.3.5 Survey and Visualize . . . ... .. .... .. .. ...... . . ..... .. . .................. . . •. ...... . .•.. .. .. .. . . . ..... 41 2.3.6 Common Tools for the Data Preparation Phase . . . .... .. ..... ....... . •......... •.• .•.. .. ..... .. .. . . .42

2.4 Phase 3: Model Planning ............................•................. . ... . .. •..... .....•........ 42 2.4.1 Data Exploration and Variable Selection . . ... . . .. . ......... •... . ... . . ........ . .............. .. .. . . . .44 2.4.2 Model Selection . ... ................ . .. . . . ................ •.......•...•.......................... . .45 2.4.3 Common Tools for the Model Planning Phase . ...........•....... . . •. ........................... . . . .45

CONTENTS

2.5 Phase 4: Model Building ...... .................. ...... •. ... ..... .... • ... •. . •. .. •.........•...•.... 46 2.5.1 Common Tools for the Mode/Building Phase ...... .. .. . ..... .. ..... . ....... . .. . . .. . . .. . .... . . .. . .... 48

2.6 Phase 5: Communicate Results ......... .... ...... . ... •........ ........ ... . •..... .....•. ..... •.... 49 2.7 Phase 6: Operationalize ... ... ....... ... . .. ........ ....... ... ........... •. . •. . ... ....... .......... SO 2.8 Case Study: Global Innovation Network and Analysis (GINA) ................. •...................... 53

2.8.1 Phase 1: Discovery ................................................................................. 54 2.8.2 Phase 2: Data Preparation .... •........ . ...................................................... . .... 55 2.8.3 Phase 3: Model Planning . . . ...•.•. . . .. . . ..... .. . . .. . ..... .. .. ... ...... . . . ................... . . . .. . . 56 2.8.4 Phase 4: Mode/Building ..... . ....•.. .. .. .......... . .............. . . .. . ... . . ....... .. . .... ... . .. . . . 56 2.8.5 Phase 5: Communicate Results .. . . ..... . ...... .. ...... ... .. . .. . . ..................... ...... ........ 58 2.8.6 Phase 6: Operationalize . . ... ......•..... ..• .. . . . .. . . ..............•................................ 59

Summary ................................. • ................. •..•.. •.......•.....••........ . ....•.... 60 Exercises .................................•.... .. ..............•. . •....................... . . . . . •.... 61 Bibliography ....• . .••...................................•.... . . • ..... .. ............................. 61

Chapter 3 • Review of Basic Data Analytic Methods Using R . . . . . . .. . ... . .. .. . ... . . . . . .. ... . 63

3.1 Introduction toR ............................ ... .................................... ..... ......... 64 3.1.1 R Graphical User Interfaces . ............ . ............................... ...... . .. ... . . . ... ....... ... 67 3.1.2 Data Import and Export. . ......... . .. ............. ........... ........... .................... ....... 69 3.1.3 Attribute and Data Types . .......... .. ...... . ....................................................... 71 3.1.4 Descriptive Statistics ....................... . . . ..................................................... 79

3.2 Exploratory Data Analysis .............. • ... . .• •.............•........... . .................... .... 80 3.2.1 Visualization Before Analysis ........ . ..................................................•........... 82 3.2.2 Dirty Data ............ .. ................................................ . ........... ...•...... .... . 85 3.2.3 Visualizing a Single Variable ........ •.. . ................ .. .. . . ........... . .... ....... •.. . . . .... .. . . 88 3.2.4 Examining Multiple Variables . .... .... ....• . .. . ... .......... .............. ...... . .. .. .............. 91 3.2.5 Data Exploration Versus Presentation ...... . ........ •. . . . .. . . ..... ...... ................... ...... .. 99

3.3 Statistical Methods for Evaluation .................... . .. .• ......... ... . .. .................... . .. 101 3.3.1 Hypothesis Testing ........ ........ .......... .... ............................ . .. . ...... .. ...... . ... 102 3.3.2 Difference of Means ...... . .... .. . .... ..... . ..................................................... 704 3.3.3 Wilcoxon Rank-Sum Test ................•........................ ... .. . ... . .................. •... 108 3.3.4 Type I and Type II Errors ... . ...... . .. . .................. . ........ . .. .... .. ......................... 109 3.3.5 Power and Sample Size .....•.. . . .. . ... ...... . ........ ....... .............. ....... .. .... .......... 110 3.3.6 ANOVA ................. . .. ......... . . .... .. . . ... .... ........ . . .. ..... . ... .. .. .... . •. •.......•... . 110

Summary ...... ............. • ....... ...... ....• .. •... • ............................... •......•...... 114 Exercises ...... ......... ......................... . ............... ...... . ... ... ....... •............. 114 Bibliography ................................... . . . ................. .................. •.... . . .. . .... 11 5

Chapter 4 • Advanced Analytical Theory and Methods: Clustering .. . . .. . .. . ... . .. . . . ... . .. 117

4.1 Overview of Clustering ........ ...... ......... .. ................................................. 118 4.2 K-means ............... ....... ... ....................... .. ........ . ... . .......... . .... . .... .... 118

4.2.1 Use Cases ..... .. ............. . •.....• ... ... .. ..... ........ .......... . . .. ........ ...... ... .. . ...... 119 4.2.2 Overview of the Method . ............ ....... ... . .. ........ ................... ... ... .. . .•. ..... . .. . 120 4.2.3 Determining the Number of Clusters . . . .. .. •. •...................... . .......... ..... .. ... ...... . ... 123 4.2.4 Diagnostics .. ......................... ...•.... ........... ..... ....................... .. .. ....... . 128

CONTENTS

4.2.5 Reasons to Choose and Cautions .. . .. . . . . . . .. . . . . . .. ... . ..... ... .. .. . . •. •. •. . ...•. • .•. ... . ..... ... 730 4.3 Additional Algorithms .............. ... . . . . .. . ...... . ... . ........ .• .. .. . .. ................ .. .... 134 Summary ......... .... ........................ .. . ....................... . . . ..•.. . .................. 135 Exercises ........... ..................... . . ..... . ............................... . .......... .. ..... . 135 Bibliography ............................. ....... ................................ . .................. 136

Chapter 5 • Advanced Analytical Theory and Methods: Association Rules .................. 137

5.1 Overview .... . . ... ........................................ .. . .. . ..... . .. .................. .. .... 138 5.2 A priori Algorithm ........... . ............... . . ....... ... . . .... . . ..... .......... .. ......... ... ... 140 5.3 Evaluation of Candidate Rules ....................... . ... .. . .. ..... •....... . ................ ..... 141 5.4 Applications of Association Rules ............ ... ..... . ..... . . . ... ..... . . .. . . . ...... .............. 143 5.5 An Example: Transactions in a Grocery Store ... . .................... .... . . ... .......... ........... 143

5.5.1 The Groceries Dataset ................... . . .. .............. •........... •... . .......•............... 144 5.5.2 Frequent ltemset Generation . . ........................... .. ......... . . •. •......... •............... 146 5.5.3 Rule Generation and Visualization ...... . ... . ......................... . .•. •.... .•. •........... . .. . 752

5.6 Validation and Testing ........... . ... .... . . ............................................. . ....... 157 5.7 Diagnostics .. .... ..................... . .. . . ..... . ............ . ... . . ... . ...... . ......... .. .... . . . 158 Summary ....... .. ................ . ..... ... . . .. . . ...... .... .... . ........ . . .... ..... .............. . . 158 Exercises ................................ ... . . . ........ . ................. . .... ....... ......... . .... 159 Bibliography ................................ . .. .... ..... ............ ..... . ... ........... ... . ...... . 160

Chapter 6 • Advanced Analytical Theory and Methods: Regression .................. . ..... 161

6.1 Linear Regression .......... . .......... . .. . .. .. ...... . ............ .... . . . ....... ........... ...... 162 6.1.1 UseCases . . . ... . . . .. . ...... ..... ......................... .. . ....... .... .... .. ...... . .......... . .. . /62 6.1.2 Model Description .. ... .. . .. . ..... . ........... . .. . .. .... . . •. ..... . •.•.• . ...... . .•............. . .. . 163 6.1.3 Diagnostics ....................... . .... .. . . . . . . ....... •.•.• .....•. •.•...... .• . •.•.. . .. . .... . . . . . . . 773

6.2 Logistic Regression ............ ........ . ..... ................................ . ......... .. .. . .. .. 178 6.2.1 Use Cases ...... . ....................................... .... ................ .... ................... 179 6.2.2 Model Description ........ .. .... ... •..... . .... ........ .. .. •. ..... ... . .•. •...• .•................... 179 6.2.3 Diagnostics ................. ..... ...... . . .. ............•. •. ........•. ..... .• .•................... 181

6.3 Reasons to Choose and Cautions ....... . . .... .. .... ............ ........... ......... ....... ..... . 188 6.4 Additional Regression Models ............ ... .. ...... . ... . ............. . ... ........ ........... ... 189 Summary ........... .... . ............ . ....... . .........•... . ...... . ...... ... . .. . . ... .. ........... . . 190 Exercises ............ .. .......... .. . .. ................ .. .. .. ............ . . .. .......... . . . .. .. .... . . 190

Chapter 7 • Advanced Analytical Theory and Methods: Classification ...... . .......... . .... 191

7.1 Decision Trees ... .. ............... ...... ............ ............. .......... .............. ... .... 192 7.1.1 Overview of a Decision Tree ...... . .................... .. . ........................ .. .... ..... . ...... 193 7.1.2 The General Algorithm . .............. .............. ... ..•. ... .............. .• .. .. ........ .... . .. . . 197 7.1.3 Decision Tree Algorithms ............. .. . .... .. ......•. . .•.. ... •. •... .... . .... ... . .............. .. 203 7.1.4 Evaluating a Decision Tree ............. . . •... . ... . ...•... .... . ....... . .................... . ... . . . . 204 7.1.5 Decision Trees in R . . . .. ................ ...... .. .. ..... ..... .... .................. . ..... ........ .. 206

7.2 Na'lve Bayes . .... ... ................ . ..... . ...... . .......... . .. . ... . ..... .. ..... ......... . ...... 211 7.2.1 Bayes' Theorem . . .. . ........................ . ..................................................... 212 7.2.2 Nai've Bayes Classifier ................... •... . ... ..... .......•.................................. .. . 214

CONTENTS

7.2.3 Smoothing . ............... .................... . .. . ........ . .. . ...... .. •. .. .......... .. .......... . 277 7.2.4 Diagnostics .. . ........... . ..................... .. .... . .•......... •.•.....•...•........ . . . ......... 217 7.2.5 Nai've Bayes in R ............... . . .. . .....•... .. . ...•.•.........•.•.. .. . .. •. •.•.... ........ . .. .... . 278

7.3 Diagnostics of Classifiers ............ •...... ........... •.......... ...•...• .. •... •. .... ........... 224 7.4 Additional Classification Methods .... • ... • ...... • ............. • .................•... .... ......... 228 Summary ................. ..... ............ • ......•.............. .. ..........................•..... 229 Exercises .................. ... ......... .... .........................•.... . . . .................•..... 230 Bibliography ...... . ..........•......... .... ........... . ... . .............. ... ...•................... 231

Chapter 8 • Advanced Analytical Theory and Methods: Time Series Analysis . . .. ... . ... . .. . 233

8.1 Overview of Time Series Analysis ....... ....... ................ ......................... .... ..... 234 8.1.1 Box-Jenkins Methodology ................... . .. .... ...... . .................... . .. ..... ............ 235

8.2 ARIMA Model. ................ . .. . ....... •..•..... .. ...... . ... •................. • ... . ..•........ 236 8.2.1 Autocorrelation Function (ACF) .. ......... ...................... ... ........ . ......... . .. ..... ..... 236 8.2.2 Autoregressive Models . ...... ... ............ . . . .. •. ... ..... ... . .. ... ... . ......... . ....... .. . . .... 238 8.2.3 Moving Average Models . .. .. . .................................... .................... •..... . .... . 239 8.2.4 ARMA and ARIMA Models ............. . .................................•...........•.....•....... 241 8.2.5 Building and Evaluating an ARIMA Model ............................. . .•.........•. •. . ... •...... 244 8.2.6 Reasons to Choose and Cautions .. ................ . .. . ........ .. . . .. . ....... . .... .•.•. •.. . •. . .... . 252

8.3 Additional Methods ........ ... . ... ....... ... .. ...... ...... .. ....... ....... .. ... . .... . ... . ...... . 253 Summary ........................ ... ... ...... .. ............ • ......... ......... ..• .. .......• ... ..... 254 Exercises .............. . .......... ... ......... . •. .. .............................• .. . . .. • . .• ... ..... 254

Chapter 9 • Advanced Analytical Theory and Methods: Text Analysis ...... . ... . .. .. .. . . ... 255 9.1 Text Analysis Steps .......... . .... ......... ...... ... .................... . ...... . ...... . . .•....... 257 9.2 A Text Analysis Example ..... •.... .... ............................ .. ............ ...... • .... ...... 259 9.3 Collecting Raw Text ........ .. .............. 00 00 00 00 ••••• 00 ••• ••• ••• ••••• 00 ••••• 00 ••••• •• ••• 00 ••• 260 9.4 Representing Text .......................... ... .................. . ...................•.. ...... .. 264 9.5 Term Frequency-Inverse Document Frequency (TFIDF) ...... • .......... • ..... .•. ...... . ......... 269 9.6 Categorizing Documents by Topics .... ................... .. .•..... . . ... • ...... •.. . . .. . . ......... 274 9.7 Determining Sentiments ............... . ...... . ......•...•..•.... .. .. .. •.. •... •.. . . .. ........... 277 9.8 Gaining Insights ................ .. ....................... •..•....... .. ........•... . ..... . ....... 283 Summary ............... . ........... . ......... •.................... • ..... . . . ......... •..... . ....... 290 Exercises ...............•... . ..... . . .. ........ •..•... . ............. • ................. . ..... . ....... 290 Bibliography ............ •. ..•... . ..... . ....... ... . ....... . .. . ................ . ............ . ........ 291

Chapter 10 • Advanced Analytics-Technology and Tools: MapReduce and Hadoop . . . ..... 295

10.1 Analytics for Unstructured Data . 00 .. .... .. 00 ••••• 00. 00 ••• 00 00 .......... 00 ......... 00 •• 00 .. . .... 296 10.1.1 UseCasesoo .. 00.00 00 ••••• 00.00 00 •••••• 00 ••••••• 00 ••• 00 • • 00.00 .. . .................... 00 . .... .. 00. 296 10.1.2 MapReduce . .. .... ......... .. ............... . .......... •......... •.•....... •.•. •....... . ....... 298 70.7.3 Apache Hadoop ......... ... ........... . ......... . . .. ....... .. . . .. . .... ... . .• ...•.... .. . •....... 300

10.2 The Hadoop Ecosystem ....•... . ........... ..... ... . •... .. .............• . •. .. .. ....... . •• ...... 306 70.2.1 Pig . ....... ..... ........ . ......................................... . .. . . .......•... . ..... •.•..... 306 70.2.2 Hive ............... . ............•................ . ... •.•...........•.......•. . .. . .. . ..... . .. . .. 308 70.2.3 HBase ...... .. 00 .. ... . . 00 ••••••••••••••• 00 •••••• 00 . .... . ..... .. ...... 00 .. .. . . . 00 ••• 00 00 ... 00 •••• • 317 10.2.4 Mahout .. 00 • •• • ••••••••••• 00 ............ . . . .. . ... . .... .... . ..... .. .......... .. .. 00 • • • 00 .. . .. . .. • 319

CONTENTS

10.3 NoSOL ...............•........................ • ................. •..................... • ....... 322 Summary .............•...•........................•................. •.....................•....... 323 Exercises .................•........................ • ..................... •...... ................... 324 Bibliography ....... •...... • ................. •...... • .................•......... .... ........ • ..•.... 324

Chapter 11 • Advanced Analytics-Technology and Tools: In-Database Analytics ........ . . . 327

11.1 SOL Essentials ............................................................. .. . . ........ • ..•.... 328 77.1.1 Joins .. . .. . . .. .. . .. ... .. . ......... ... ............. . .. .. . ...... .. .. ... . ....... ... .... . .. ...... . .. . 330 77.1.2 Set Operations ................ . .. . ...................... . ...... ... ........................... . ... 332 11.1.3 Grouping Extensions ......... .. .. .. . . . . .. ........................ ............. .. ................ . 334

11.2 In-Database Text Analysis ............... •... . .............•......•......... . . .. ... .• . . . •..•.... 338 11 .3 Advanced SOL ... .. ......................... •.. • .................•........... . .........•....... 343

71.3.1 Window Functions . . . . ............................... ... .. .... .. . . . ..... . ....................... 343 11.3.2 User-Defined Functions and Aggregates ............................•. •. •............... .. ... .... .347 11.3.3 Ordered Aggregates ............. ..... .... ..... ....... .... .. ..................................... 351 11.3.4 MADiib ...................................... ............•. ....... . . .... . .... •. •................ .352

Summary ..........•.. • ... • .......................................................... .. . . .......... 356 Exercises ......... . .............................. ........ ............................ .. . . .......... 356 Bibliography .......•...... •. .• • .................... • ... .. ........... . •. ..• . ......... .... .. . ........ 357

Chapter 12 • The Endgame, or Putting It All Together ..................................... 359

12.1 Communicating and Operationalizing an Analytics Project. ........ . .....................•....... 360 12.2 Creating the Final Deliverables ......................... ..... . .. .. .. .•.......................... 362

12.2.1 Developing Core Material for Multiple Audiences ........................ •..... .. •.•.............. 364 12.2.2 Project Goals . . . . . .. . . ............ . ............ . ..... . ........ ..... . .. . . ..... . . . ................ 365 12.2.3 Main Findings ....... . ... . . ... . ....................... . .. ... . .. . ... ....• . . . ... . •. •........... . .. .367 12.2.4 Approach ... . .. . . .. . . ............................................................ .... .... ...... 369 12.2.5 Model Description ... . .. . .................................... .. ......... . .... . ...•..... . ..... .... 371 12.2.6 Key Points Supported with Data . .......................... . . . . . ....... . . . ..... .. .. .. . ..... . ..... .372 12.2.7 Model Details .. . . .. ................................................. . ....... •.•....... . ........ .372 12.2.8 Recommendations ........ ... .... ....... .... ........... .......... . .... . . ...... •.•.• .. .... ..... . . 374 12.2.9 Additional Tips on Final Presentation ......... . .. . ............ .. . . . . .. . .. . ..... . •. •.............. .375 12.2.10 Providing Technica15pecificarions and Code ................................... . ................ . 376

12.3 Data Visua lization Basics .......... .... ... .... ....................•.......... . .... . ............. 377 12.3.1 Key Points Supported with Data ............... . ... . . . .................. . ............... ... ...... .378 12.3.2 Evolution of a Graph ................ ..... .... ............. ...... . ...... •.•... •. •.•......... •.... 380 12.3.3 Common Representation Methods .............. .. ............ .. . . . •. •.. . .... •. . ................ 386 12.3.4 How to Clean Up a Graphic ................... •. . . .... . ..... . .......... . . . ..... . ... .......... ... .387 12.3.5 Additional Considerations ..... ................. .... ... . ..... .. . . . . •.•. .. ... . •.• ...... . ...... ... . 392

Summary ............ .. .........................•...... • ... • . ... .........•... •..................... 393 Exercises ........... . . .... . ................. .. .. . . . .... • ................. . . .. . .. • .......... . ....... 394 References and Further Reading ... .. ............ .... ...... ..... ......... . .... . . .................... 394 Bibliography .... . . ... ......... .... . ........................ • ................. .. . .. .. . ... . . ... ...... 394

Index .. . .............. . .. . .. . .. . . .. . . ........... . . . .. . .. . . . ....... . . . ... . . .. . .. .. . .. . . . ... .. . . ............... . 397

Foreword

Technological advances and the associated changes in practical daily life have produced a rapidly expanding "parallel universe" of new content, new data, and new information sources all around us. Regardless of how one defines it, the phenomenon of Big Data is ever more present, ever more pervasive, and ever more important. There is enormous value potential in Big Data: innovative insights, improved understanding of problems, and countless opportunities to predict-and even to shape-the future. Data Science is the principal means to discover and tap that potential. Data Science provides ways to deal with and benefit from Big Data: to see patterns, to discover relationships, and to make sense of stunningly varied images and information.

Not everyone has studied statistical analysis at a deep level. People with advanced degrees in applied math- ematics are not a commodity. Relatively few organizations have committed resources to large collections of data gathered primarily for the purpose of exploratory analysis. And yet, while applying the practices of Data Science to Big Data is a valuable differentiating strategy at present, it will be a standard core competency in the not so distant future.

How does an organization operationalize quickly to take advantage of this trend? We've created this book for that exact purpose.

EMC Education Services has been listening to the industry and organizations, observing the multi-faceted transformation of the technology landscape, and doing direct research in order to create curriculum and con- tent to help individuals and organizations transform themselves. For the domain of Data Science and Big Data Analytics, our educational strategy balances three things: people-especially in the context of data science teams, processes-such as the analytic lifecycle approach presented in this book, and tools and technologies-in this case with the emphasis on proven analytic tools.

So let us help you capitalize on this new "parallel universe" that surrounds us. We invite you to learn about Data Science and Big Data Analytics through this book and hope it significantly accelerates your efforts in the transformational process.

Introduction

Big Data is creating significant new opportunities for organizations to derive new value and create competitive advantage from their most valuable asset: information. For businesses, Big Data helps drive efficiency, quality, and personalized products and services, producing improved levels of customer satisfaction and profit. For scientific efforts, Big Data analytics enable new avenues of investigation with potentially richer results and deeper insights than previously available. In many cases, Big Data analytics integrate structured and unstructured data with real- time feeds and queries, opening new paths to innovation and insight.

This book provides a practitioner's approach to some of the key techniques and tools used in Big Data analytics. Knowledge ofthese methods will help people become active contributors to Big Data analytics projects. The book's content is designed to assist multiple stakeholders: business and data analysts looking to add Big Data analytics skills to their portfolio; database professionals and managers of business intelligence, analytics, or Big Data groups looking to enrich their analytic skills; and college graduates investigating data science as a career field.

The content is structured in twelve chapters. The first chapter introduces the reader to the domain of Big Data, the drivers for advanced analytics, and the role of the data scientist. The second chapter presents an analytic project lifecycle designed for the particular characteristics and challenges of hypothesis-driven analysis with Big Data.

Chapter 3 examines fundamental statistical techniques in the context of the open source R analytic software environment. This chapter also highlights the importance of exploratory data analysis via visualizations and reviews the key notions of hypothesis development and testing.

Chapters 4 through 9 discuss a range of advanced analytical methods, including clustering, classification, regression analysis, time series and text analysis.

Chapters 10 and 11 focus on specific technologies and tools that support advanced analytics with Big Data. In particular, the Map Reduce paradigm and its instantiation in the Hadoop ecosystem, as well as advanced topics in SOL and in-database text analytics form the focus of these chapters.

XVIII ! INTRODUCTION

Chapter 12 provides guidance on operationalizing Big Data analytics projects. This chapter focuses on creat· ing the final deliverables, converting an analytics project to an ongoing asset of an organization's operation, and creating clear, useful visual outputs based on the data.

EMC Academic Alliance University and college faculties are invited to join the Academic Alliance program to access unique "open" curriculum-based education on the following topics:

• Data Science and Big Data Analytics

• Information Storage and Management

• Cloud Infrastructure and Services

• Backup Recovery Systems and Architecture

The program provides faculty with course resources to prepare students for opportunities that exist in today's

evolving IT industry at no cost. For more information, visit http: // education . EMC . com/ academicalliance.

EMC Proven Professional Certification EMC Proven Professional is a leading education and certification program in the IT industry, providing compre-

hensive coverage of information storage technologies, virtualization, cloud computing, data science/Big Data analytics, and more.

Being proven means investing in yourself and formally validating your expertise.

This book prepares you for Data Science Associate (EMCDSA) certification. Visit http : I I educat i on . EMC . com for details.

INTRODUCTION TO BIG DATA ANAL YTICS

Much has been written about Big Data and the need for advanced analytics within industry, academia,

and government. Availability of new data sources and the rise of more complex analytical opportunities

have created a need to rethink existing data architectures to enable analytics that take advantage of Big Data. In addition, significant debate exists about what Big Data is and what kinds of skil ls are required to make best use of it. This chapter explains several key concepts to clarify what is meant by Big Data, why

advanced analyt ics are needed, how Data Science differs from Business Intelligence (BI), and what new

roles are needed for the new Big Data ecosystem.

1.1 Big Data Overview Data is created constantly, and at an ever-increasing rate. Mobile phones, social media, imaging technologies

to determine a medical diagnosis-all these and more create new data, and that must be stored somewhere for some purpose. Devices and sensors automatically generate diagnostic information that needs to be stored and processed in real time. Merely keeping up with this huge influx of data is difficult, but substan-

tially more cha llenging is analyzing vast amounts of it, especially when it does not conform to traditional

notions of data structure, to identify meaningful patterns and extract useful information. These challenges of the data deluge present the opportunity to transform business, government, science, and everyday life.

Several industries have led the way in developing their ability to gather and exploit data:

• Credit card companies monitor every purchase their customers make and can identify fraudulent purchases with a high degree of accuracy using rules derived by processing billions of transactions.

• Mobile phone companies analyze subscribers' calling patterns to determine, for example, whether a caller's frequent contacts are on a rival network. If that rival network is offering an attractive promo- tion that might cause the subscriber to defect, the mobile phone company can proactively offer the subscriber an incentive to remain in her contract.

• For companies such as Linked In and Facebook, data itself is their primary product. The valuations of these companies are heavi ly derived from the data they gather and host, which contains more and more intrinsic va lue as the data grows.

Three attributes stand out as defining Big Data characteristics:

• Huge volume of data: Rather than thousands or millions of rows, Big Data can be billions of rows and millions of columns.

• Complexity of data t ypes and structures: Big Data reflects the variety of new data sources, formats, and structures, including digital traces being left on the web and other digital repositories for subse- quent analysis.

• Speed of new data creation and growth: Big Data can describe high velocity data, with rapid data ingestion and near real time analysis.

Although the volume of Big Data tends to attract the most attention, generally the variety and veloc-

ity of the data provide a more apt definition of Big Data. (Big Data is sometimes described as having 3 Vs: volume, variety, and velocity.) Due to its size or structure, Big Data cannot be efficiently analyzed using only

traditional databases or methods. Big Data problems require new tools and technologies to store, manage, and realize the business benefit. These new tools and technologies enable creation, manipulation, and