Bi Data Aalitics * Big Oa > Data which i¢ extremly huge in sige, dt Grows expoaental "Trad DBMS connet stow tH efficiently. Nee C4TBy date evoy day) Social Media. C 500+ Tes everg data) > Flight Oata Co+ TB in 30 minutes) Petabytes in one day # Sve of Big Data © Volume Wy wikh time cole of date. 7B, 0B, ER @ Velocity How fost can we Generate dala Analysis of data, ® Voriety DPfeven forms { data File formate, ate Quality or Correctness of Data ) © Vevacity © Value C+ eth useful needs to be exbracte ofTyfes_q Dats © Qrucured ata — Bored, accessed § (ivocessed 89 @ Greed format > Rat Sige Gees upte Zeta bashes > Roms > Examples > Atdfoe, ‘veatory, Ses ® Unstruchuced data > Unknown dom or Shructuce 7 Difecutt t» devive value Heterogeneous, dota Conkeing a combination of Simple tect files, images P dene @ emi - Sheuchwed > tke an xm fle 2 structured but not deltned lhe a table * Traditional Date > Jolornal Data Gnstde enterprise) — Etemal 4 enterprise oma deme rtrion—* — Predefined Struchore > 4-1 = Ceatvalfged. 7 High end Hi componedts vegutce ol Can wun 0a Coraonadtty Ho > unkoo SL Liable & tater veldafonstip reed cea > octal medfa, Imex 2, aud?a, vPdeo > Binary Data i 8 Sensors, all data* Concest Ace| of Hodoop @ escing large amts of ~ Open source frame cork dor stovtgg 6 procesciag data CUreientiy ~ Commodity Hardeoove * Cove Components Hid Hadoop © Vadoop HOFS ™ Hadoor dishabuted file system ™ Date %s replicated @ Sragmerted 44> Blockg ~ Too Components —s Narnenode (6; Stoyag @ B mgmt metadata) Datanode CBboves data) ® Man Reduce — Stace oery large & dtshPbuted data, fravaltel Processing. required > Map Phase 5 Sort B og Fo Groups Jol Frachey 8 Genevede qotermedfate Nay-Ualue Pears Tetetonctar — Veang — Reduce Phase ' del Histon Server O date wa Ke V pats ave grouped 8 reduced acc to alge pvovided: by der® yaen Cet another vesouce Negotfatec) — ole = Resource Man ager > Tob Scheduling > bes 9 mon veduce ave Yaan > Seperated by vesource managey | Master Name Node YARN Master Slave Architecure ? Metadda. >| Namenode |- HOFS Architecture.* Hodoop Distibuded File System ee nr | v w (Dena Data Blocks in HOFS (= OtstRbute Fle Systems axe Gasfeally de use of Slandalone computas Bombing them to cpewrcte Yodepenclently > Compute nodes — Rochs C864) — Nodes on a Stogle rach ave connected by Grate ethernet — Maultal ule Racke are connected. dheough Audtches, 7 'o auca goflure oaon. done. % generally set bo & > Computatfon 98 ali ko didided fate ‘tasbs such that % ‘@ can vestact * Large Ycate Fle Ayctorn Orgaised on => gFs » HDFS one fails — Files ave dtueded gato chunks C64 MB) — Repitcat on To fead chunks g dil, Master node/ data node - nh ake tel. replicated.* [a Renuce| Procescfa > tia Programing Model cps loge datasets ep Fabos chunks 8 conver dhom Gato hey value paies Collects § sorts ther office) sorted hes axe dftded $ soagaed. among odhec | reducers combines game hous 6 flerforrs operation ace to oat Example ‘ re 7 bis cav Teva a oe US Car Teaka Teen plane cay 5 Trai (lone Cavs re 83 8 co Gx us Rane 7° Be OH MS INTE fecal — |ra way |. beh ey Mepgng Fg @ord) Count 2a fog % Conabfnors, > MPa? veduce , be ‘prove ferppenance ~ $ besteally oggregales, valnes lecally before Sending fo Me weducors Ly 5: For a sales faroblem % we have mul tiple vecords of Q Sing ode gory 4 Products, CoMBINE THEM befoce sending fo Reducer% Mabie Maltacaton using Map Reduce ah | Examale : Oo wv a 3 e of 4 a of Q 1 x | 8 4 I Rs 0 dl . A d 88 Two matrices can only multe lool cob A = vous ¢ B ® Mon Tash 2 Coavet dota ante For mohs 1A cn) > Ag, “| Fo matvixc 2-5 g [6.9 C4, ay Ney Value Posvg Lo no using those. formule Jor maltx A @ Q O For Ay =4 970, §=0 jhe §0,43 [Co.0, (no, 9] [04) , (A,o, ] — For Agus 4 1=Og-4, b= fo,43 [Co,0), (a, 4,2] [C0,4), (A, 4,9)]> For Ay, = 3 til p. £0, h-f0,33 [ C4, 0) , (A, 0, 3)] loi (4,0 ,3)] ™ Fox Au~\4 4, $4, bsfo,a} Cea, °) | CA, 4, at Peta) oa, 1,9] Now or madrfoc 8 > B Boor 2 = Bye 4 JO. k=O, {0,19 $-O,hed 7204 1C0,9) C4, 0,09] [Co,4) , C8, 0,4) Cc4,M , (8, 0,4 [ (4,0) , (80,3) 1 ) > Bao=S om lee Jz4,b-0, 820.4 joka oa [c0, 0), (C8, 1, S)) Eco, 4), (8,4, 1) C4, 0) , ¢8,4,5)] [4,9 , €8)4,0)© Comber Tosh Combine all some heye (0,0) ASD, cages, (4) CA0,4) Ca,4, 5), (49) €8,0,.9), (8,4,5) (4,013) CAr4,4) (8,0,5) (8, 4,5) 4D Cod (Ady (809 (G41 — / Poovey, - ’ ® Reducer Foch ( SAME S VALWE ware v ¥ Foe (0,0) @o,4) CA,4,9) 8,0,2) (Rag i ) dea Dass Ot ae '0 —> 12 Cfadex 0,0) For C04) C00, 4) (A, 4,9) C8, 0,4) CB,4, 4) dy @ yg SFr 9) (40,3) (A, 1,4) (8,0, 2) (84,9 Gay 2 — 26 For C44) (Ayo, 3) CA, 4, 4) (8,0,4) CB, 44) Bg 3s 4? [2 J