A  Shallow  Introduc0on  to  Deep   Learning   Anil  Anthony  Bharath   [email protected]   This  Version:  23-­‐Jan-­‐2014   Imperial  College  London  &  Cortexica  Vision  Systems Disclaimer     •  These  slides  were  created  with  the  purpose  of  explaining   biological  recep0ve  fields  and  the  manner  in  which  similar   spa0al  paOerns  can  be  “learned”  in  a  DL  context.    These   slides  are  not  intended  to  be  a  full  review  or  exposi0on  on   DL.   •  These  are  first-­‐genera0on  notes;  although  many  elements   have  been  presented  previously,  the  order  of  presenta0on   is  new.   •  I  welcome  feedback  on  how  this  lecture  might  be   improved:  I  may  not  agree,  but  I  do  not  bite.   •  An  accompanying  bibliography  in  DL  is  available  from  Woo-­‐ Sup  Han  (BICV  Group). Deep  Learning:  Defini0on   •  An  unambiguous  defini0on  is  debatable,  but   some  characteris0cs  are  clearly  iden0fiable:     –  A  Deep  Learning  (DL)  system  contains  mul0ple  hidden   layers  of  network  between  input  and  output  layers;     –  DL  systems  are  reminiscent  of  mul0-­‐layer   perceptrons,  BUT:   •  DL  systems  are  oZen  trained  by  exploi0ng  both  supervised   and  unsupervised  training  in  the  same  network;   •  DL  systems  may  mixed  architectures  from  layer  to  layer,  that   oZen  conform  to  hybrid  probabilis0c  models;   •  Training  regimes  may  differ  depending  on  layer,  reflec0ng  a   balance  between  the  need  to  represent  and  to  classify  data. Acronyms   •  DLN:  Deep  Learning  Network   •  DL:  Deep  Learning  (refers  to  the  field  of  study)   •  DBN:  Deep  Belief  Network   •  DBM:  Deep  Boltzmann  Machine   •  RBM:  Restricted  Boltzmann  Machine   •  CNN:  Convolu0onal  Neural  Network   •  HTM:  Hierarchical  Temporal  Memory   OZen  not     Included   •  MSC:  Map  Seeking  Circuit   Architecture  is   In  DL  literature   mul0-­‐layered,   but  training  is   not  in  the   usual  manner In  order  to  appreciate  DLNs…   •  We’ll  look  first  at  two  examples  of  sensory   processing  networks;   •  One  is  technological,  the  other  is  biological;   •  Both  handle  visual  informa0on. Ramón  y  Cajal’s  Re0na   Mammalian  re0nal  structure  –  note     the  apparent  layers  that  are     very  clearly  depicted.    Different  anatomical     Layers  are  associated  with  different   func’onal  proper0es  of  cells.     The  layered  structure  is  typical  of  neural     circuitry  involved  in  sensory  percep0on.   c.1900  By  San’ago  Ramón  y  Cajal.   c.   1896 The  Complexity  of  Vision   Some  “sta0s0cs”  on  the  human  re0na:     –  120  million  photoreceptors  in  each  human  re0na.   –  Approximately  5  dis0nct  layers  of  cells.   –  Various  areas  of  the  re0na:   •  Central  Fovea   •  Fovea   •  Periphery   –  Varia0on  of  proper0es/ra0os  of  photoreceptors/ganglion   cells  across  the  re0na.     7 A  “Handle”  to  Complexity   The  Recep0ve  Field:   –  A  “nearly  definable”*  concept  to  describe  the  effect  that  a   visual  s0mulus  has  on  a  ganglion  cell  exhibi0ng  centre   surround  organisa0on.   •  Change  from  sustained  to  transient  centre  response  as    s’muli  are   displaced  from  centre.     •  Excitatory  responses  evoked  to  opposite  s’mulus  phase  with   surround  s’mula’on.   •  Ac’ve  inhibi’on  of  the  centre  response  with  surround  s’mula’on.   •  A  maximal  response  can  be  evoked  only  with  an  op’mally  sized   spot.     *The  term  “nearly  definable”  is  aOributed  to  Hartline.   8 Recep0ve  fields   •  Spike-­‐triggered  averaging  (STA)  –  also  known   as  reverse  “correla0on”:  a  probabilis0c   characterisa0on  of  recep0ve  fields.   •  Run  through  a  random  series  of  spa0al   paOerns,  and  empirically  es0mate:   –  P(spike  |  spa0al  paOern  s0mulus;  t)   •  Provides  a  systema0c  way  of  capturing   response  characteris0cs  of  cells. The  “Op0mal”  S0mulus   •  A  visual  Interpreta0on  of  a  re0nal  recep0ve  Field   –  The  “op0mal  visual  s0muli”:  probability  of  firing  of   two  classes  of  re0nal  ganglion  cells  might  be  highest   for  these  s0muli*.   –  Examples:   OFF  Centre   ON  Centre   *This  is  a  simplified,  but  “on  average”  response.   10