Deep  Learning  for  NLP                                               (without  Magic)   Richard  Socher,  Yoshua  Bengio  and  Chris  Manning   ACL  2012 Deep Learning NER      WordNet   Most  current  machine  learning  works       well  because  of  human-­‐designed     SRL            Parser     representa>ons  and  input  features   Machine  learning  becomes  just  op>mizing   weights  to  best  make  a  final  predic>on   Representa>on  learning  aCempts  to     automa>cally  learn  good  features  or  representa>ons   Deep  learning  algorithms  aCempt  to  learn  mul>ple  levels  of   representa>on  of  increasing  complexity/abstrac>on   2 A Deep Architecture Mainly,  work  has  explored  deep  belief  networks  (DBNs),  Markov   Random  Fields  with  mul>ple  layers,  and  various  types  of   mul>ple-­‐layer  neural  networks   Output  layer   Here  predic>ng  a  supervised  target     Hidden  layers   These  learn  more  abstract     representa>ons  as  you  head  up   Input  layer   Raw  sensory  inputs  (roughly)   3 Part  1.1:  The  Basics   Five Reasons to Explore Deep Learning 4 #1 Learning representations Handcra\ing  features  is  >me-­‐consuming   The  features  are  o\en  both  over-­‐specified  and  incomplete   The  work  has  to  be  done  again  for  each  task/domain/…   We  must  move  beyond  handcra\ed  features  and  simple  ML   Humans  develop  representa>ons  for  learning  and  reasoning    Our  computers  should  do  the  same   Deep  learning  provides  a  way  of  doing  this   5 #2 The need for distributed representations Current  NLP  systems  are  incredibly  fragile  because  of   their  atomic  symbol  representa>ons   Crazy  sentenAal   complement,  such  as  for   “likes  [(being)  crazy]”   6 #2 The need for distributed representations Learned  word  representa>ons  that  model  similari>es     help  enormously  in  NLP   Distribu>onal  similarity  based  word  clusters  greatly  help  most   applica>ons   +1.4%  F1  Dependency  Parsing  15.2%  error  reduc>on    (Koo  &   Collins  2008,  Brown  clustering)   +3.4%  F1  Named  En>ty  Recogni>on  23.7%  error  reduc>on   (Stanford  NER,  exchange  clustering)   7 #2 The need for distributed C1   C2   C3   representations Mul>-­‐   input   Clustering   Clustering   Learning  features  that  are  not  mutually  exclusive  can  be  exponen>ally   more  efficient  than  nearest-­‐neighbor-­‐like  or  clustering-­‐like  models   8 Distributed representations deal with the curse of dimensionality Generalizing  locally  (e.g.,  nearest   neighbors)  requires  representa>ve   examples  for  all  relevant  varia>ons!   Classic  solu>ons:   •  Manual  feature  design   •  Assuming  a  smooth  target   func>on  (e.g.,  linear  models)   •  Kernel  methods  (linear  in  terms   of  kernel  based  on  data  points)   Neural  networks  parameterize  and   learn  a  “similarity”  kernel     9 #3 Unsupervised feature and weight learning Today,  most  prac>cal,  good  NLP&  ML  methods  require   labeled  training  data  (i.e.,  supervised  learning)   But  almost  all  data  is  unlabeled   Most  informa>on  must  be  acquired  unsupervised   Fortunately,  a  good  model  of  observed  data  can  really  help  you   learn  classifica>on  decisions   10