I
nte
rna
t
io
na
l J
o
urna
l o
f
E
lect
rica
l a
nd
Co
m
pu
t
er
E
ng
ineering
(
I
J
E
CE
)
Vo
l.
1
6
,
No
.
2
,
A
p
r
il
20
2
6
,
p
p
.
945
~
963
I
SS
N:
2088
-
8
7
0
8
,
DOI
: 1
0
.
1
1
5
9
1
/ijece.
v
1
6
i
2
.
pp
9
4
5
-
9
6
3
945
J
o
ur
na
l ho
m
ep
a
g
e
:
h
ttp
:
//ij
ec
e.
ia
esco
r
e.
co
m
O
ptimi
zing
neu
ra
l net
wo
rks
:
a
com
pa
ra
tive study
of
a
ctiva
tion
funct
io
ns i
n dee
p
lea
rning
Ahm
ed
M
o
ba
rk
i,
Abdu
lla
h
Sh
eik
h
D
e
p
a
r
t
me
n
t
o
f
C
o
mp
u
t
e
r
S
c
i
e
n
c
e
,
Ta
i
f
U
n
i
v
e
r
s
i
t
y
,
Ta
i
f
,
S
a
u
d
i
A
r
a
b
i
a
Art
icle
I
nfo
AB
S
T
RAC
T
A
r
ticle
his
to
r
y:
R
ec
eiv
ed
Ma
y
5
,
2
0
2
5
R
ev
is
ed
No
v
2
4
,
2
0
2
5
Acc
ep
ted
J
an
1
5
,
2
0
2
6
Ac
ti
v
a
ti
o
n
f
u
n
c
ti
o
n
s
p
la
y
a
p
iv
o
t
a
l
ro
le
in
d
e
e
p
lea
rn
in
g
(DL)
m
o
d
e
ls,
th
u
s
sh
a
p
in
g
th
e
ir
lea
rn
in
g
c
a
p
a
b
il
it
ies
,
c
o
n
v
e
r
g
e
n
c
e
b
e
h
a
v
i
o
r,
a
n
d
g
e
n
e
ra
li
z
a
ti
o
n
p
e
rfo
rm
a
n
c
e
.
Ho
we
v
e
r,
th
e
se
lec
ti
o
n
o
f
a
c
ti
v
a
ti
o
n
fu
n
c
ti
o
n
s
with
o
u
t
sy
ste
m
a
ti
c
e
v
a
lu
a
ti
o
n
in
m
a
n
y
a
p
p
li
c
a
ti
o
n
s
h
a
s
l
imited
th
e
m
o
d
e
l'
s
p
e
rfo
rm
a
n
c
e
.
In
a
p
p
r
o
p
riate
a
c
ti
v
a
ti
o
n
f
u
n
c
ti
o
n
s
m
a
y
c
a
u
se
g
r
a
d
ien
ts
t
o
sh
rin
k
o
r
b
l
o
w
-
up
d
u
rin
g
b
a
c
k
p
r
o
p
a
g
a
ti
o
n
,
th
e
re
b
y
a
ffe
c
ti
n
g
e
ffe
c
ti
v
e
lea
rn
in
g
.
T
o
c
o
n
q
u
e
r
t
h
is
p
ro
b
lem
,
t
h
is
p
a
p
e
r
p
ro
v
id
e
s
a
n
o
v
e
l
c
o
m
p
re
h
e
n
siv
e
e
m
p
ir
ica
l
i
n
v
e
stig
a
ti
o
n
o
f
n
i
n
e
a
c
ti
v
a
ti
o
n
fu
n
c
ti
o
n
s,
in
c
lu
d
in
g
trad
it
io
n
a
l
fu
n
c
ti
o
n
s
li
k
e
re
c
ti
fied
li
n
e
a
r
u
n
it
(Re
LU),
S
ig
m
o
i
d
,
Tan
h
,
a
n
d
e
x
p
o
n
e
n
ti
a
l
li
n
e
a
r
u
n
it
(
ELU
)
,
a
n
d
m
o
d
e
rn
n
o
n
li
n
e
a
rit
ies
li
k
e
S
wish
,
M
ish
,
G
a
u
ss
ian
e
rro
r
l
in
e
a
r
u
n
i
t
(G
EL
U)
,
a
n
d
sm
o
o
th
m
a
x
imu
m
u
n
it
(S
M
U)
.
I
n
t
h
e
p
ro
p
o
se
d
m
e
th
o
d
o
lo
g
y
,
t
h
e
se
n
i
n
e
a
c
ti
v
a
ti
o
n
fu
n
c
ti
o
n
s
a
re
e
v
a
lu
a
ted
wit
h
in
two
p
r
o
m
in
e
n
t
n
e
u
ra
l
n
e
two
rk
a
rc
h
it
e
c
tu
re
s,
n
a
m
e
ly
c
o
n
v
o
lu
ti
o
n
a
l
n
e
u
ra
l
n
e
two
rk
s
(C
NN
s)
a
n
d
m
u
lt
i
-
lay
e
r
p
e
rc
e
p
tr
o
n
s
(M
LP
s),
a
c
ro
ss
b
e
n
c
h
m
a
rk
d
a
tas
e
ts,
n
a
m
e
ly
CIF
AR
-
1
0
,
CIF
AR
-
1
0
0
,
a
n
d
M
NIST
.
Th
e
e
v
a
lu
a
ti
o
n
c
rit
e
ria
in
c
l
u
d
e
v
a
li
d
a
ti
o
n
a
c
c
u
ra
c
y
,
l
o
ss
,
train
i
n
g
ti
m
e
,
a
n
d
g
ra
d
ien
t
sta
b
il
it
y
.
E
x
p
e
rime
n
tal
re
su
lt
s
p
r
o
v
e
d
t
h
a
t
G
EL
U
a
c
ti
v
a
ti
o
n
fu
n
c
ti
o
n
imp
ro
v
e
d
M
LP
a
c
c
u
ra
c
y
t
o
9
8
.
0
3
%
a
n
d
CNN
a
c
c
u
ra
c
y
to
9
3
.
8
2
%
wh
il
e
m
a
in
tain
i
n
g
sta
b
le
g
ra
d
ien
ts
a
n
d
lo
w
l
o
ss
v
a
lu
e
s
o
f
0
.
0
8
8
a
n
d
0
.
2
2
1
,
re
sp
e
c
ti
v
e
ly
.
Th
e
se
fin
d
in
g
s
p
r
o
v
i
d
e
d
p
ra
c
ti
c
a
l
g
u
i
d
e
li
n
e
s
fo
r
se
lec
ti
n
g
a
c
ti
v
a
ti
o
n
fu
n
c
ti
o
n
s
su
it
e
d
to
s
p
e
c
ifi
c
tas
k
c
o
m
p
lex
i
ti
e
s
a
n
d
m
o
d
e
l
d
e
p
th
s,
c
o
n
tri
b
u
ti
n
g
to
th
e
d
e
sig
n
o
f
m
o
re
e
f
ficie
n
t
a
n
d
a
c
c
u
ra
te DL
sy
ste
m
s.
K
ey
w
o
r
d
s
:
Activ
atio
n
f
u
n
ctio
n
s
Dee
p
lear
n
in
g
Gau
s
s
ian
er
r
o
r
lin
ea
r
u
n
it
Mish
Neu
r
al
n
etwo
r
k
s
R
ec
tifie
d
lin
ea
r
u
n
it
Swis
h
T
h
is i
s
a
n
o
p
e
n
a
c
c
e
ss
a
rticle
u
n
d
e
r th
e
CC B
Y
-
SA
li
c
e
n
se
.
C
o
r
r
e
s
p
o
nd
ing
A
uth
o
r
:
Ah
m
ed
Mo
b
a
r
k
i
Dep
ar
tm
en
t o
f
C
o
m
p
u
ter
a
n
d
I
n
f
o
r
m
atio
n
T
ec
h
n
o
lo
g
y
,
T
ai
f
Un
iv
er
s
ity
T
aif
,
Kin
g
d
o
m
o
f
Sau
d
i A
r
ab
i
a.
E
m
ail: a
h
m
ed
.
m
o
b
ar
k
i
9
1
@
g
m
ail.
co
m
1.
I
NT
RO
D
UCT
I
O
N
Dee
p
lear
n
in
g
(
DL
)
h
as
em
er
g
ed
as
a
p
ar
ad
ig
m
s
h
if
t
in
ar
t
if
icial
in
tellig
en
ce
(
AI
)
in
r
ec
en
t
y
ea
r
s
,
s
ig
n
if
ican
tly
im
p
r
o
v
in
g
ca
p
a
b
ilit
ies
in
f
ield
s
lik
e
m
ed
ical
d
iag
n
o
s
tics
,
im
ag
e
r
ec
o
g
n
itio
n
,
r
ec
o
m
m
e
n
d
atio
n
s
y
s
tem
s
,
au
to
n
o
m
o
u
s
d
r
iv
in
g
,
an
d
n
atu
r
al
lan
g
u
ag
e
p
r
o
ce
s
s
in
g
.
Neu
r
al
n
etwo
r
k
s
,
a
s
tr
atif
ied
ar
ch
itectu
r
e
d
esig
n
ed
to
m
im
ic
th
e
c
o
g
n
i
tiv
e
f
u
n
ctio
n
s
o
f
t
h
e
h
u
m
an
b
r
ain
,
ar
e
ess
en
tial
f
o
r
DL
m
o
d
els.
Activ
atio
n
f
u
n
ctio
n
s
in
f
lu
en
ce
n
ea
r
ly
all
f
ac
ets
o
f
a
DL
m
o
d
el’
s
tr
ai
n
in
g
a
n
d
p
er
f
o
r
m
an
ce
.
T
h
ese
p
r
o
ce
s
s
es
g
o
v
e
r
n
n
eu
r
al
ac
tiv
atio
n
an
d
af
f
ec
t
in
f
o
r
m
atio
n
p
r
o
p
ag
atio
n
d
u
r
in
g
in
f
er
en
ce
.
Als
o
,
ac
tiv
atio
n
f
u
n
ctio
n
s
h
av
e
th
e
r
ev
er
s
e
ef
f
ec
t
d
u
r
in
g
g
r
ad
ien
t
-
b
ased
o
p
tim
izatio
n
.
Din
g
et
a
l.
[
1
]
ass
is
ted
in
co
n
tr
o
llin
g
th
e
g
r
ad
ien
t
f
l
o
w,
f
ac
ilit
atin
g
th
e
tr
ain
in
g
o
f
d
ee
p
er
m
o
d
els.
T
h
e
ac
tiv
atio
n
f
u
n
ctio
n
,
wh
ich
a
d
d
s
n
o
n
lin
ea
r
it
y
to
th
e
m
o
d
el,
is
a
cr
u
cial
co
m
p
o
n
e
n
t
o
f
th
is
ar
ch
itectu
r
e.
W
ith
o
u
t
it,
a
n
eu
r
al
n
etwo
r
k
,
n
o
m
atter
h
o
w
d
e
ep
,
b
eh
av
es
lik
e
a
lin
ea
r
r
eg
r
ess
io
n
m
o
d
el,
wh
ic
h
s
ev
er
ely
lim
its
its
ab
ilit
y
to
d
ep
ict
co
m
p
le
x
d
ata
r
elatio
n
s
h
ip
s
[
2
]
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
0
8
8
-
8
7
0
8
I
n
t J E
lec
&
C
o
m
p
E
n
g
,
Vo
l.
1
6
,
No
.
2
,
Ap
r
il
20
2
6
:
9
4
5
-
963
946
T
h
e
in
f
lu
e
n
ce
o
f
t
h
e
ac
tiv
ati
o
n
f
u
n
ctio
n
also
e
x
ten
d
s
to
th
e
ac
cu
r
ac
y
a
n
d
g
en
er
aliza
ti
o
n
o
f
th
e
n
etwo
r
k
,
al
o
n
g
with
its
co
n
v
e
r
g
en
ce
r
ate
an
d
lear
n
i
n
g
s
tab
i
lity
[
3
]
.
Ov
er
t
h
e
p
ast
d
e
ca
d
e
s
,
n
eu
r
al
n
etwo
r
k
s
h
av
e
u
s
ed
T
a
n
h
an
d
Sig
m
o
id
ac
tiv
atio
n
f
u
n
ctio
n
s
.
T
an
h
o
u
tp
u
ts
in
th
e
r
a
n
g
e
o
f
(
1
,
-
1
)
h
elp
to
m
ain
tain
t
h
e
d
ata
ce
n
ter
ed
a
r
o
u
n
d
ze
r
o
,
an
d
Sig
m
o
id
,
wh
ic
h
tr
an
s
f
er
s
th
e
in
p
u
t
v
al
u
es
to
th
e
r
a
n
g
e
[
0
,
1
]
,
ca
n
b
e
u
s
ed
in
b
in
ar
y
class
if
icatio
n
p
r
o
b
lem
s
.
T
h
e
Sig
m
o
id
an
d
T
a
n
h
ac
tiv
atio
n
with
its
f
u
n
ctio
n
(
)
an
d
th
e
in
p
u
t
(
)
ar
e
g
iv
en
as
(
1
)
,
(
2
)
,
(
)
=
1
/
(
1
+
−
)
(
1
)
(
)
=
ℎ
(
)
(
2
)
wh
er
e
,
(
)
is
th
e
ex
p
o
n
en
tial
f
a
cto
r
.
Alth
o
u
g
h
b
o
th
m
eth
o
d
s
ar
e
in
itially
p
o
p
u
lar
,
th
ey
ar
e
b
eset
with
th
e
p
r
o
b
lem
o
f
v
an
is
h
in
g
g
r
ad
i
en
ts
,
th
u
s
h
in
d
er
in
g
ef
f
ec
tiv
e
weig
h
t
u
p
d
atin
g
in
d
ee
p
er
n
etwo
r
k
s
[
4
]
.
A
b
r
ea
k
t
h
r
o
u
g
h
in
DL
co
m
es
with
th
e
in
tr
o
d
u
ctio
n
o
f
th
e
r
e
ctif
ied
lin
ea
r
u
n
it
(
R
eL
U)
,
g
i
v
en
b
y
th
e
f
u
n
ctio
n
(
)
as
(
3
)
,
(
)
=
(
0
,
)
(
3
)
B
y
h
av
in
g
n
o
n
-
ze
r
o
g
r
ad
ien
t
s
,
th
e
R
eL
U
ac
tiv
atio
n
f
u
n
ct
io
n
allev
iates
th
e
v
an
is
h
in
g
g
r
ad
ien
t
is
s
u
e
f
o
r
p
o
s
itiv
e
in
p
u
ts
an
d
allo
ws d
ee
p
er
n
etwo
r
k
s
to
co
n
v
er
g
e
f
aster
[
5
]
.
R
eL
U
in
tr
o
d
u
ce
s
th
e
d
y
i
n
g
R
eL
U
p
r
o
b
lem
in
wh
ic
h
n
eu
r
o
n
s
with
n
eg
ativ
e
i
n
p
u
ts
al
way
s
o
u
tp
u
t
ze
r
o
an
d
a
r
e
in
ac
tiv
e
d
u
r
in
g
tr
ain
in
g
[
6
]
.
Par
am
etr
ic
R
e
L
U
(
PR
eL
U)
an
d
L
ea
k
y
R
e
L
U
ar
e
cr
ea
ted
to
r
ed
u
ce
th
ese
lim
itatio
n
s
,
o
f
f
e
r
in
g
a
n
o
n
-
ze
r
o
g
r
ad
ien
t
f
o
r
n
eg
ativ
e
in
p
u
ts
.
Fu
r
th
er
,
ex
p
o
n
en
tial
lin
ea
r
u
n
its
(
E
L
Us)
an
d
s
ca
led
E
L
Us
(
SE
L
U)
ar
e
p
r
esen
ted
t
o
im
p
r
o
v
e
g
r
ad
ie
n
t
f
lo
w
an
d
en
ab
le
s
elf
-
n
o
r
m
alizin
g
ac
tio
n
[
7
]
,
[
8
]
.
(
)
=
{
>
0
(
−
1
)
(
4
)
(
)
=
{
>
0
(
−
1
)
(
5
)
W
h
er
e,
(
)
is
th
e
f
ac
to
r
u
tili
ze
d
to
co
n
tr
o
l
th
e
ac
tiv
atio
n
f
u
n
ct
io
n
r
eg
ar
d
i
n
g
n
eg
ativ
e
im
p
ac
t,
an
d
(
)
is
th
e
s
ca
lin
g
f
ac
to
r
.
R
ec
en
t
r
esear
ch
h
as
lo
o
k
ed
at
a
d
ju
s
tab
le,
n
o
n
-
m
o
n
o
t
o
n
ic
ac
tiv
atio
n
f
u
n
ctio
n
s
.
R
eg
ar
d
in
g
Go
o
g
le
ac
ad
em
ics an
d
th
e
in
p
u
t
(
)
with
its
f
u
n
ctio
n
(
)
,
Swis
h
is
ch
ar
ac
ter
ized
as
(
6
)
,
(
)
=
∗
s
ig
m
o
id
(
)
(
6
)
Swis
h
co
m
b
in
es
th
e
s
m
o
o
th
n
ess
o
f
Sig
m
o
id
with
th
e
u
n
b
o
u
n
d
ed
n
atu
r
e
o
f
R
eL
U,
th
u
s
i
m
p
r
o
v
i
n
g
g
r
ad
ien
t
p
r
o
p
a
g
atio
n
an
d
ac
h
i
ev
in
g
s
tr
o
n
g
p
er
f
o
r
m
an
ce
in
i
m
ag
e
class
if
icatio
n
task
s
[
9
]
.
B
u
ild
in
g
u
p
o
n
th
is
,
Mish
is
d
ef
in
ed
as
(
7
)
,
(
)
=
∗
ℎ
[
(
1
+
)
]
(
7
)
Mish
is
in
tr
o
d
u
ce
d
to
f
u
r
t
h
er
en
h
a
n
ce
lear
n
in
g
d
y
n
am
i
cs
an
d
s
m
o
o
th
n
ess
.
Mish
h
as
s
h
o
wn
s
u
p
er
io
r
p
er
f
o
r
m
an
ce
,
p
ar
ticu
lar
l
y
in
co
n
v
o
l
u
tio
n
al
a
r
ch
itectu
r
es
[
1
0
]
,
[
1
1
]
.
I
n
ad
d
itio
n
t
o
Swis
h
an
d
Mish
,
o
th
er
m
o
d
er
n
ac
tiv
atio
n
f
u
n
ctio
n
s
h
av
e
g
ain
ed
atten
tio
n
.
Gau
s
s
ian
er
r
o
r
lin
ea
r
u
n
it
(
GE
L
U)
,
f
r
eq
u
en
tly
u
s
ed
in
m
o
d
els
lik
e
b
id
ir
ec
tio
n
al
en
co
d
er
r
ep
r
esen
tatio
n
s
f
r
o
m
t
r
an
s
f
o
r
m
er
s
(
B
E
R
T
)
an
d
g
e
n
er
ativ
e
p
r
e
-
tr
ai
n
ed
tr
an
s
f
o
r
m
er
(
GPT)
,
is
d
ef
in
ed
as
(
8
)
,
(
)
=
∗
(
)
(
8
)
Her
e,
th
e
cu
m
u
l
at
iv
e
d
i
s
t
r
ib
u
t
io
n
f
u
n
c
t
io
n
o
f
th
e
s
t
an
d
a
r
d
n
o
r
m
al
d
i
s
tr
ib
u
t
io
n
i
s
r
ep
r
e
s
en
t
ed
a
s
(
)
.
GE
L
U
o
f
f
er
s
b
o
th
s
m
o
o
th
t
r
an
s
i
t
io
n
s
an
d
s
t
o
ch
a
s
ti
c
r
eg
u
lar
iz
at
io
n
,
th
er
eb
y
im
p
r
o
v
in
g
co
n
v
er
g
en
ce
in
t
r
an
s
f
o
r
m
er
s
an
d
d
e
ep
m
o
d
e
ls
[
1
2
]
,
[
1
3
]
.
An
o
th
er
r
e
ce
n
t
p
r
o
p
o
s
al,
n
a
m
ed
s
m
o
o
th
m
ax
im
u
m
u
n
i
t
(
SM
U)
,
i
s
d
ef
in
ed
as
(
9
)
,
(
)
=
∗
ℎ
[
s
o
f
tp
lu
s
(
)
]
(
9
)
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J E
lec
&
C
o
m
p
E
n
g
I
SS
N:
2088
-
8
7
0
8
Op
timiz
in
g
n
eu
r
a
l n
etw
o
r
ks:
a
co
mp
a
r
a
tive
s
tu
d
y
o
f
a
ctiva
tio
n
fu
n
ctio
n
s
in
…
(
A
b
d
u
lla
h
S
h
eikh
)
947
Ma
n
y
DL
s
y
s
tem
s
s
till
u
s
e
R
eL
U
as
th
e
d
ef
a
u
lt
ac
tiv
atio
n
f
u
n
ctio
n
,
ty
p
ically
ch
o
s
en
w
ith
o
u
t
a
n
y
test
in
g
d
esp
ite
th
ese
d
ev
elo
p
m
en
ts
.
Nev
er
th
eless
,
th
e
n
et
wo
r
k
ar
ch
itectu
r
e,
d
e
p
th
,
d
at
aset
co
m
p
lex
ity
,
an
d
tr
ain
in
g
d
y
n
am
ics
h
a
v
e
a
s
ig
n
if
ican
t
im
p
ac
t
o
n
th
e
b
est
ac
tiv
atio
n
f
u
n
ctio
n
[
1
4
]
,
[
1
5
]
.
As
an
illu
s
tr
atio
n
,
a
s
h
allo
w
m
u
lti
-
lay
er
p
er
ce
p
tr
o
n
(
ML
P)
ca
n
g
ain
a
d
v
an
ta
g
es
f
r
o
m
u
tili
zin
g
a
s
in
g
le
ac
tiv
atio
n
f
u
n
ctio
n
,
wh
er
ea
s
a
d
ee
p
co
n
v
o
lu
tio
n
al
n
eu
r
al
n
etwo
r
k
(
C
NN)
ca
n
no
t.
A
co
m
p
r
eh
en
s
iv
e
em
p
ir
ical
e
x
am
in
atio
n
o
f
n
i
n
e
ac
tiv
atio
n
f
u
n
ctio
n
s
,
in
clu
d
i
n
g
R
eL
U,
Sig
m
o
id
,
T
an
h
,
E
L
U,
SEL
U,
Swis
h
,
Mish
,
GE
L
U,
an
d
SMU,
is
ca
r
r
ied
o
u
t
i
n
th
e
cu
r
r
en
t
wo
r
k
.
T
h
is
e
v
alu
atio
n
is
d
r
iv
e
n
b
y
th
e
co
n
clu
s
io
n
p
r
esen
ted
b
ef
o
r
e.
I
n
o
r
d
er
to
ca
r
r
y
o
u
t
th
e
ev
alu
atio
n
,
two
s
tan
d
ar
d
ar
ch
itectu
r
es,
n
am
el
y
ML
P
an
d
C
NN,
as
wel
l
as
two
b
en
ch
m
ar
k
d
atasets
,
n
am
ely
C
an
ad
ian
I
n
s
titu
te
f
o
r
Ad
v
an
ce
d
R
esear
ch
-
1
0
(
C
I
FAR
-
1
0
)
f
o
r
n
atu
r
al
p
ictu
r
e
class
if
icatio
n
an
d
Mo
d
if
ied
Natio
n
al
I
n
s
titu
te
o
f
Stan
d
ar
d
s
an
d
T
ec
h
n
o
lo
g
y
(
MN
I
ST)
f
o
r
h
an
d
wr
itten
d
ig
it
class
if
icat
io
n
,
ar
e
u
tili
ze
d
.
Valid
atio
n
ac
cu
r
ac
y
,
tr
ain
in
g
lo
s
s
,
tr
ain
i
n
g
tim
e,
an
d
g
r
ad
ien
t
s
tab
ilit
y
ar
e
th
e
f
o
u
r
p
r
im
ar
y
p
er
f
o
r
m
an
ce
in
d
icato
r
s
th
at
a
r
e
tak
en
in
to
c
o
n
s
id
er
atio
n
with
in
th
is
f
r
am
ewo
r
k
.
T
h
ese
in
d
icato
r
s
o
f
f
e
r
a
m
u
lti
-
d
im
en
s
io
n
al
p
e
r
s
p
ec
tiv
e
o
n
th
e
p
er
f
o
r
m
a
n
ce
o
f
th
e
f
u
n
ctio
n
.
T
h
e
p
u
r
p
o
s
e
o
f
th
is
r
e
s
ea
r
ch
is
to
r
en
d
e
r
ev
id
en
ce
-
ce
n
t
r
ic
r
ec
o
m
m
e
n
d
a
tio
n
s
f
o
r
s
elec
tin
g
ac
tiv
atio
n
f
u
n
ctio
n
s
b
ased
o
n
th
e
s
p
ec
if
ic
m
o
d
el
an
d
th
e
r
eq
u
ir
em
e
n
ts
o
f
th
e
wo
r
k
.
B
y
way
o
f
its
d
is
co
v
er
ies,
th
is
wo
r
k
b
r
id
g
es
th
e
g
ap
b
etwix
t
th
eo
r
etica
l
ad
v
an
ce
m
e
n
ts
in
ac
tiv
atio
n
f
u
n
ctio
n
d
esig
n
an
d
th
eir
p
r
ac
ti
ca
l
im
p
lem
en
tatio
n
in
ac
tu
al
DL
p
r
o
ce
s
s
es.
T
h
is
wo
r
k
is
a
s
ig
n
if
ican
t
c
o
n
tr
ib
u
t
io
n
to
th
e
f
ield
.
T
h
e
p
r
o
b
lem
s
in
ex
is
tin
g
r
esear
ch
wo
r
k
s
r
e
g
ar
d
in
g
ac
tiv
atio
n
f
u
n
ctio
n
i
n
DL
ar
e
lis
ted
as,
a.
Mo
s
t
o
f
th
e
ex
is
tin
g
wo
r
k
s
co
m
p
ar
ed
o
n
ly
a
f
ew
ac
tiv
atio
n
s
(
i.e
.
,
Sig
m
o
id
,
T
an
h
,
an
d
R
eL
U)
an
d
ig
n
o
r
ed
th
e
n
ewe
r
ac
tiv
atio
n
s
,
s
u
ch
as GE
L
U,
Mish
,
Swis
h
,
o
r
SMU.
b.
T
h
e
ex
is
tin
g
wo
r
k
s
f
ailed
t
o
m
ea
s
u
r
e
th
e
t
r
ain
in
g
s
tab
ilit
y
,
co
n
v
er
g
en
ce
s
p
ee
d
,
g
r
a
d
ien
t
f
lo
w,
an
d
co
m
p
u
tatio
n
al
e
f
f
icien
cy
.
c.
I
n
m
an
y
co
n
v
en
tio
n
al
wo
r
k
s
,
th
e
co
m
p
ar
is
o
n
s
wer
e
r
estric
ted
to
a
s
in
g
le
m
o
d
el
t
y
p
e,
t
h
u
s
lim
itin
g
th
e
g
en
er
aliza
tio
n
o
f
f
in
d
in
g
s
.
B
ased
o
n
th
e
p
r
o
b
lem
s
tatem
en
ts
,
th
e
r
esear
ch
q
u
esti
o
n
s
o
f
th
e
p
r
o
p
o
s
ed
m
o
d
el
ar
e
d
e
v
elo
p
ed
a
n
d
ar
e
lis
ted
:
R
Q1
:
Ho
w
d
o
es
th
e
p
er
f
o
r
m
a
n
ce
o
f
d
if
f
e
r
en
t
ac
tiv
atio
n
f
u
n
ctio
n
s
v
ar
y
ac
r
o
s
s
n
eu
r
al
n
e
two
r
k
ar
c
h
itectu
r
es
lik
e
ML
P a
n
d
C
NN?
R
Q2
:
W
h
ich
ac
tiv
atio
n
f
u
n
ct
io
n
s
(
i.e
.
,
class
ical
o
r
m
o
d
er
n
)
d
eliv
er
th
e
b
est
b
alan
ce
b
etwe
en
ac
cu
r
ac
y
,
co
n
v
er
g
en
ce
s
p
ee
d
,
an
d
tr
ain
i
n
g
s
tab
ilit
y
in
DL
m
o
d
els?
R
Q3
:
Ho
w
d
o
ac
tiv
atio
n
f
u
n
ctio
n
s
d
if
f
er
r
eg
ar
d
in
g
t
h
e
g
r
ad
ien
t
f
lo
w
a
n
d
s
u
s
ce
p
tib
ilit
y
to
v
a
n
is
h
in
g
o
r
ex
p
lo
d
in
g
g
r
a
d
ien
ts
d
u
r
i
n
g
tr
a
in
in
g
?
B
y
co
n
s
id
er
in
g
th
e
r
esear
ch
q
u
esti
o
n
s
an
d
p
r
o
b
lem
s
tatem
en
ts
,
s
o
m
e
co
n
tr
ib
u
tio
n
s
ar
e
d
ev
elo
p
ed
an
d
ex
p
lain
ed
b
el
o
w,
a.
I
n
th
e
p
r
o
p
o
s
ed
m
eth
o
d
o
lo
g
y
,
ac
tiv
atio
n
f
u
n
ctio
n
s
,
n
a
m
ely
Sig
m
o
id
,
R
eL
U
,
T
an
h
,
E
L
U
,
Swis
h
,
Mish
,
GE
L
U,
an
d
SMU,
ar
e
co
m
p
ar
ed
f
o
r
i
m
p
r
o
v
ed
p
er
f
o
r
m
an
ce
.
b.
T
h
e
p
r
o
p
o
s
ed
m
o
d
el
m
ea
s
u
r
es
th
e
tr
ain
in
g
s
tab
ilit
y
,
co
n
v
e
r
g
en
ce
s
p
ee
d
,
g
r
a
d
ien
t
f
lo
w,
an
d
co
m
p
u
tatio
n
al
ef
f
icien
cy
d
u
r
in
g
p
er
f
o
r
m
a
n
ce
an
aly
s
is
.
c.
T
h
e
co
m
p
a
r
is
o
n
is
d
o
n
e
f
o
r
th
e
C
NN
an
d
ML
P n
eu
r
al
n
etwo
r
k
s
to
im
p
r
o
v
e
th
e
g
en
er
aliza
ti
o
n
o
f
f
in
d
in
g
s
.
T
h
e
p
ap
er
is
s
tr
u
ctu
r
ed
as:
T
h
e
liter
atu
r
e
s
u
r
v
ey
is
co
n
v
ey
e
d
in
s
ec
tio
n
2
,
th
e
p
r
o
p
o
s
ed
m
e
th
o
d
o
lo
g
y
is
an
aly
ze
d
in
s
ec
tio
n
3
,
th
e
r
e
s
u
lts
an
d
d
is
cu
s
s
io
n
ar
e
p
r
esen
ted
in
s
ec
tio
n
4
,
an
d
last
ly
,
t
h
e
p
r
o
p
o
s
ed
wo
r
k
is
win
d
ed
u
p
in
s
ec
tio
n
5
with
f
u
tu
r
e
r
ec
o
m
m
en
d
atio
n
s
.
2.
RE
L
AT
E
D
WO
RK
Ma
n
y
r
ese
ar
c
h
e
r
s
lo
o
k
e
d
at
ac
ti
v
at
io
n
f
u
n
c
ti
o
n
s
i
n
d
ee
p
n
eu
r
a
l
n
e
tw
o
r
k
s
,
in
v
esti
g
a
tin
g
b
o
t
h
co
n
v
e
n
ti
o
n
a
l
a
n
d
co
n
t
em
p
o
r
a
r
y
s
u
b
s
tit
u
t
es.
T
h
o
u
g
h
th
ei
r
d
r
awb
ac
k
s
,
i
n
c
lu
d
i
n
g
v
a
n
is
h
i
n
g
g
r
a
d
ie
n
ts
a
n
d
d
e
ad
n
e
u
r
o
n
p
r
o
b
le
m
s
,
we
r
e
we
ll
d
o
c
u
m
en
te
d
[
1
]
,
th
ese
p
r
o
b
le
m
s
h
a
p
p
e
n
e
d
b
ec
a
u
s
e
s
o
m
e
ac
ti
v
ati
o
n
s
c
r
e
at
ed
g
r
a
d
ie
n
ts
t
o
s
h
r
an
k
e
x
p
o
n
e
n
t
iall
y
d
u
r
i
n
g
b
ac
k
p
r
o
p
a
g
a
ti
o
n
i
n
d
ee
p
e
r
a
r
c
h
i
te
ct
u
r
es.
A
ls
o
,
t
h
ese
d
r
aw
b
ac
k
s
s
p
o
ile
d
t
h
e
n
et
wo
r
k
’
s
p
o
te
n
t
i
al
to
l
ea
r
n
l
o
n
g
e
r
-
r
an
g
e
d
e
p
e
n
d
e
n
ci
es.
L
i
k
ewis
e,
i
n
R
e
L
U
-
lik
e
ac
t
iv
ati
o
n
s
,
t
h
e
d
e
ad
n
eu
r
o
n
is
s
u
e
w
as
a
co
m
m
o
n
p
r
o
b
l
em
,
w
h
e
r
e
n
eu
r
o
n
s
w
er
e
i
n
a
cti
v
e
o
w
in
g
to
ze
r
o
g
r
a
d
ie
n
ts
.
C
lass
ic
al
f
u
n
cti
o
n
s
,
n
am
el
y
S
ig
m
o
i
d
,
R
eL
U
,
a
n
d
T
a
n
h
,
w
er
e
ess
e
n
t
ia
l
f
o
r
n
et
wo
r
k
c
o
n
s
tr
u
c
ti
o
n
[
2
]
,
[
3
]
o
wi
n
g
t
o
t
h
e
ir
s
im
p
li
cit
y
,
ea
s
ie
r
i
m
p
le
m
e
n
ta
t
io
n
,
a
n
d
s
u
c
ce
s
s
ac
r
o
s
s
a
v
a
r
i
e
ty
o
f
tas
k
s
.
L
i
k
e
wis
e
,
R
e
L
U
w
as
a
d
e
f
a
u
lt
c
h
o
ic
e
in
s
e
v
e
r
a
l
co
m
p
u
te
r
v
is
io
n
t
ask
s
d
u
e
to
i
ts
co
m
p
u
tat
io
n
a
l
e
f
f
ic
ie
n
c
y
.
S
im
i
la
r
l
y
,
Si
g
m
o
i
d
a
n
d
T
a
n
h
w
er
e
f
o
u
n
d
at
io
n
al
i
n
ea
r
l
y
n
e
u
r
al
n
etw
o
r
k
m
o
d
e
ls
o
wi
n
g
t
o
t
h
eir
s
m
o
o
t
h
a
n
d
b
o
u
n
d
e
d
o
u
t
p
u
ts
.
R
esea
r
c
h
e
r
s
c
r
e
at
ed
f
u
n
cti
o
n
s
li
k
e
L
e
a
k
y
R
eL
U
a
n
d
PR
e
L
U
t
o
s
o
l
v
e
th
ese
p
r
o
b
le
m
s
.
H
o
w
e
v
e
r
,
L
e
a
k
y
R
eL
U
a
n
d
PR
e
L
U
e
n
h
a
n
ce
d
g
r
a
d
ie
n
t
f
lo
w
i
n
d
e
e
p
a
r
c
h
i
te
ct
u
r
es
an
d
we
r
e
es
p
e
cial
ly
b
e
n
e
f
ic
ial
i
n
a
c
o
u
s
tic
m
o
d
els
a
n
d
R
esNet
-
s
t
y
l
e
C
NNs
[
4
]
,
[
5
]
.
T
h
ese
m
o
d
i
f
ie
d
v
ar
i
a
n
ts
all
o
w
ed
s
m
all
a
n
d
n
o
n
-
z
e
r
o
g
r
ad
ie
n
ts
e
v
e
n
w
h
en
t
h
e
i
n
p
u
t
w
as
n
e
g
a
ti
v
e
,
p
r
e
v
e
n
ti
n
g
n
eu
r
o
n
s
f
r
o
m
b
e
c
o
m
i
n
g
i
n
a
cti
v
e
an
d
m
i
ti
g
at
in
g
t
h
ei
r
h
a
r
s
h
c
u
t
-
o
f
f
at
z
er
o
[
6
]
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
0
8
8
-
8
7
0
8
I
n
t J E
lec
&
C
o
m
p
E
n
g
,
Vo
l.
1
6
,
No
.
2
,
Ap
r
il
20
2
6
:
9
4
5
-
963
948
T
h
e
p
a
p
e
r
[
7
]
s
u
g
g
este
d
th
e
e
x
p
o
n
e
n
t
ial
li
n
e
ar
u
n
it
(
E
L
U)
,
w
h
i
ch
a
cc
e
le
r
at
e
d
c
o
n
v
e
r
g
e
n
ce
b
y
k
e
ep
in
g
m
ea
n
ac
ti
v
ati
o
n
s
cl
o
s
e
t
o
ze
r
o
a
n
d
d
im
in
is
h
in
g
t
h
e
“in
te
r
n
al
c
o
v
a
r
ia
te
s
h
if
t”
a
n
d
e
n
a
b
li
n
g
n
etw
o
r
k
s
t
o
tr
a
in
m
o
r
e
ef
f
e
cti
v
el
y
.
Als
o
,
n
eg
ati
v
e
s
a
tu
r
ati
o
n
o
f
E
L
Us
h
e
lp
e
d
p
u
s
h
m
ea
n
a
cti
v
a
ti
o
n
s
t
o
war
d
z
er
o
wi
th
o
u
t
s
ac
r
if
ici
n
g
ca
p
a
cit
y
.
T
h
e
s
c
ale
d
ex
p
o
n
e
n
ti
al
li
n
ea
r
u
n
its
(
S
E
L
U)
b
r
o
a
d
en
ed
t
h
is
wit
h
s
el
f
-
n
o
r
m
a
liz
in
g
f
ea
t
u
r
es
.
T
h
u
s
,
th
e
n
etw
o
r
k
s
a
u
t
o
m
a
tica
ll
y
r
eg
u
l
ate
d
t
h
eir
ac
t
iv
ati
o
n
s
.
SE
L
U
r
e
m
o
v
ed
th
e
n
ee
d
f
o
r
e
x
p
li
cit
n
o
r
m
al
iz
ati
o
n
l
a
y
e
r
s
i
n
s
o
m
e
ar
ch
ite
ct
u
r
es
[
8
]
.
Als
o
,
s
wis
h
a
n
d
its
n
e
ar
c
o
u
s
i
n
Si
g
m
o
i
d
li
n
e
ar
u
n
it
(
Si
L
U
)
p
r
o
v
i
d
e
d
s
m
o
o
t
h
,
n
o
n
-
m
o
n
o
t
o
n
ic
b
e
h
a
v
i
o
r
a
n
d
w
er
e
d
em
o
n
s
tr
a
te
d
to
b
e
at
R
eL
U
i
n
i
m
a
g
e
c
lass
i
f
i
ca
t
io
n
a
n
d
r
ei
n
f
o
r
ce
m
e
n
t
le
ar
n
i
n
g
ac
ti
v
iti
es.
T
h
ei
r
g
r
a
d
u
a
l
t
r
an
s
iti
o
n
f
o
r
n
e
g
ati
v
e
i
n
p
u
ts
a
n
d
n
o
n
-
m
o
n
o
t
o
n
icit
y
e
n
a
b
l
e
d
r
ic
h
er
f
e
at
u
r
e
r
ep
r
es
e
n
tat
io
n
a
n
d
s
m
o
o
th
er
g
r
a
d
i
en
t f
l
o
w
.
Si
m
il
ar
l
y
,
t
h
e
m
o
d
e
l p
r
o
v
id
e
d
e
n
h
a
n
c
e
d
o
p
ti
m
i
za
t
io
n
lan
d
s
ca
p
es
f
o
r
t
as
k
s
wi
th
h
i
g
h
-
d
im
en
s
io
n
al
in
p
u
t
s
p
ac
es
[
9
]
,
[
1
0
]
.
A
f
u
r
t
h
e
r
e
x
te
n
s
i
o
n
o
f
t
h
is
te
n
d
e
n
c
y
,
M
is
h
,
en
h
a
n
ce
d
l
ea
r
n
in
g
d
y
n
a
m
ics a
n
d
g
e
n
e
r
al
iza
ti
o
n
b
y
m
e
a
n
s
o
f
its
s
el
f
-
r
e
g
u
la
r
iz
in
g
s
tr
u
c
tu
r
e
,
t
h
e
r
e
b
y
i
n
t
r
o
d
u
ci
n
g
an
im
p
lici
t
b
i
as
t
o
w
ar
d
s
m
o
r
e
s
ta
b
l
e
ac
t
iv
ati
o
n
s
o
v
e
r
t
h
e
c
o
u
r
s
e
o
f
tr
ai
n
i
n
g
[
1
1
]
.
M
is
h
p
r
o
v
id
e
d
b
et
te
r
o
u
tc
o
m
es
i
n
o
b
je
ct
d
et
ec
t
io
n
,
n
at
u
r
a
l
la
n
g
u
a
g
e
p
r
o
ce
s
s
i
n
g
(
NL
P)
,
a
n
d
o
t
h
e
r
d
o
m
a
in
s
o
wi
n
g
to
its
c
a
p
a
b
il
it
y
t
o
b
al
a
n
ce
s
m
o
o
th
n
ess
an
d
g
r
a
d
i
en
t
s
t
r
en
g
t
h
.
I
n
t
r
a
n
s
f
o
r
m
e
r
-
b
ase
d
m
o
d
els
li
k
e
B
E
R
T
,
GE
L
U
w
as
p
ar
tic
u
l
ar
ly
s
tr
o
n
g
in
NL
P
a
n
d
cl
ass
i
f
ic
ati
o
n
j
o
b
s
s
i
n
c
e
it
in
cl
u
d
e
d
s
t
o
c
h
ast
ic
r
e
g
u
l
ar
iza
ti
o
n
a
n
d
s
m
o
o
t
h
n
ess
.
T
h
is
ai
d
e
d
tr
a
n
s
f
o
r
m
e
r
la
y
e
r
s
i
n
h
a
n
d
li
n
g
n
o
is
y
,
h
i
g
h
-
d
im
en
s
io
n
a
l
la
n
g
u
ag
e
e
m
b
e
d
d
in
g
s
m
o
r
e
e
f
f
ec
ti
v
el
y
,
l
ea
d
i
n
g
to
s
u
p
e
r
i
o
r
p
e
r
f
o
r
m
a
n
ce
[
1
2
]
,
[
1
3
]
.
B
ased
o
n
t
h
ese
f
i
n
d
in
g
s
,
s
e
v
e
r
al
a
cti
v
a
ti
o
n
f
u
n
c
ti
o
n
s
we
r
e
m
et
h
o
d
ic
all
y
co
n
t
r
aste
d
ac
r
o
s
s
d
i
f
f
e
r
e
n
t
ar
ch
ite
ct
u
r
es
a
n
d
d
atas
ets
i
n
t
h
is
w
o
r
k
.
Sp
ec
if
ic
r
ec
o
m
m
en
d
ati
o
n
s
we
r
e
p
r
o
v
i
d
e
d
o
n
h
o
w
t
o
c
h
o
o
s
e
ac
ti
v
at
io
n
f
u
n
cti
o
n
s
ap
p
r
o
p
r
i
ate
f
o
r
ce
r
t
a
in
DL
t
ask
s
b
y
m
ea
n
s
o
f
t
h
ei
r
in
f
l
u
e
n
c
e
o
n
ac
cu
r
a
c
y
,
t
r
ai
n
i
n
g
tim
e,
a
n
d
g
r
a
d
i
en
t
s
tab
ilit
y
,
s
h
o
w
in
g
t
h
a
t
ac
ti
v
ati
o
n
s
we
r
e
c
o
m
p
u
t
ati
o
n
a
ll
y
b
u
d
g
e
t
-
co
n
s
t
r
a
i
n
e
d
[
1
4
]
.
C
o
m
p
ar
ati
v
e
r
es
ea
r
c
h
v
e
r
i
f
ie
d
t
h
a
t
th
ese
co
n
t
em
p
o
r
a
r
y
ac
t
iv
ati
o
n
s
,
s
u
c
h
as
Sw
is
h
,
Mish
,
a
n
d
GE
L
U,
s
h
o
we
d
n
o
t
ab
l
e
g
ai
n
s
i
n
b
o
t
h
C
NNs
an
d
ML
Ps
o
v
e
r
cl
ass
ic
a
l
f
u
n
cti
o
n
s
,
p
ar
tic
u
l
ar
ly
in
te
r
m
s
o
f
a
cc
u
r
a
cy
an
d
co
n
v
e
r
g
e
n
ce
,
th
u
s
d
i
m
i
n
is
h
i
n
g
o
v
e
r
f
itti
n
g
p
r
o
b
le
m
s
an
d
im
p
r
o
v
i
n
g
t
h
e
s
ta
b
ilit
y
i
n
d
e
e
p
e
r
ar
ch
i
te
ct
u
r
es
[
1
5
]
,
[
1
6
]
.
A
u
t
o
m
a
ted
s
e
a
r
c
h
m
e
th
o
d
s
wer
e
u
s
ed
to
f
i
n
d
n
ew
ac
ti
v
ati
o
n
f
u
n
cti
o
n
s
,
s
u
p
p
o
r
ti
n
g
t
h
e
p
o
we
r
o
f
Sw
is
h
an
d
e
n
co
u
r
a
g
i
n
g
d
ata
-
d
r
iv
e
n
ac
ti
v
ati
o
n
d
is
c
o
v
e
r
y
.
He
r
e,
g
e
n
et
ic
p
r
o
g
r
a
m
m
i
n
g
a
n
d
n
e
u
r
al
a
r
c
h
i
te
ct
u
r
e
s
ea
r
ch
f
r
a
m
ew
o
r
k
s
a
n
a
ly
ze
d
m
ass
i
v
e
d
esi
g
n
s
p
a
ce
s
o
f
ac
ti
v
ati
o
n
,
o
f
te
n
s
ea
r
c
h
i
n
g
s
wis
h
p
at
t
er
n
s
as
a
p
r
i
m
a
r
y
ch
o
i
ce
[
1
7
]
,
[
1
8
]
.
Mis
h
r
a'
s
in
v
est
ig
ati
o
n
o
f
n
o
n
-
m
o
n
o
t
o
n
ic
f
u
n
c
ti
o
n
s
f
u
r
t
h
e
r
u
n
d
e
r
li
n
e
d
Mis
h
'
s
b
ett
er
s
ta
b
ilit
y
o
v
e
r
tr
ai
n
i
n
g
s
t
ag
es,
d
e
m
o
n
s
tr
ati
n
g
t
h
at
th
ese
f
u
n
cti
o
n
s
m
ai
n
t
ai
n
e
d
b
ala
n
c
e
d
g
r
a
d
i
en
t
m
a
g
n
it
u
d
es
t
h
r
o
u
g
h
o
u
t
t
h
e
o
p
t
im
i
za
ti
o
n
p
r
o
ce
s
s
a
n
d
r
es
u
lt
in
g
i
n
b
ette
r
g
e
n
e
r
a
liz
ati
o
n
a
n
d
c
o
n
s
is
t
e
n
t
l
ea
r
n
in
g
r
ates
.
I
n
r
e
in
f
o
r
ce
m
e
n
t
le
ar
n
i
n
g
a
n
d
f
u
n
ct
io
n
a
p
p
r
o
x
i
m
a
ti
o
n
,
Si
L
U/S
wis
h
al
lo
we
d
m
o
r
e
c
o
n
s
is
t
e
n
t
co
n
v
e
r
g
e
n
c
e
a
n
d
s
m
o
o
th
er
g
r
ad
i
e
n
t
p
r
o
p
ag
ati
o
n
t
h
an
c
o
n
v
en
t
io
n
al
ap
p
r
o
ac
h
es,
t
h
e
r
e
b
y
m
a
k
i
n
g
t
h
e
m
wel
l
-
s
u
it
ed
f
o
r
r
e
al
-
ti
m
e
d
ec
is
i
o
n
-
m
a
k
i
n
g
a
g
en
ts
[
1
9
]
.
O
n
t
h
e
o
t
h
e
r
h
a
n
d
,
s
o
m
e
w
o
r
k
s
c
o
n
t
r
i
b
u
te
d
t
o
d
e
f
i
n
i
n
g
t
h
e
lin
k
b
etw
ee
n
DL
s
ta
b
il
it
y
,
wei
g
h
t
in
iti
ali
za
t
io
n
,
a
n
d
a
ct
iv
a
ti
o
n
f
u
n
cti
o
n
s
,
t
h
u
s
est
ab
l
is
h
i
n
g
i
n
iti
ali
za
t
io
n
tec
h
n
i
q
u
es
li
k
e
H
e
an
d
X
av
ie
r
f
o
r
ali
g
n
i
n
g
ac
ti
v
ati
o
n
v
a
r
ia
n
c
e
w
it
h
wei
g
h
t
s
c
ali
n
g
.
T
h
is
i
m
p
r
o
v
e
d
t
h
e
t
r
a
in
in
g
d
e
p
t
h
lim
its
[
2
0
]
.
T
h
eir
i
d
e
as
s
p
u
r
r
ed
t
h
e
wi
d
es
p
r
ea
d
u
s
e
o
f
m
et
h
o
d
s
li
k
e
Xa
v
i
er
i
n
iti
al
izat
io
n
,
w
h
i
ch
we
r
e
ess
e
n
ti
al
f
o
r
t
r
ai
n
i
n
g
d
e
ep
er
m
o
d
els
.
C
e
r
ta
in
s
t
u
d
ies
u
n
d
e
r
li
n
ed
h
o
w
ac
t
i
v
ati
o
n
d
ec
i
s
io
n
s
a
f
f
ec
t
ed
d
e
ep
co
n
v
o
lu
ti
o
n
al
n
etw
o
r
k
s
a
n
d
r
ec
o
m
m
e
n
d
e
d
a
co
m
p
r
o
m
is
e
b
et
we
en
s
m
o
o
t
h
n
ess
a
n
d
p
r
o
ce
s
s
i
n
g
e
f
f
ic
ie
n
c
y
,
s
h
o
wi
n
g
th
at
t
h
e
ac
ti
v
at
io
n
s
’
m
at
h
e
m
a
tic
al
co
m
p
le
x
i
ty
w
as
wei
g
h
t
ed
ag
ai
n
s
t
t
h
eir
r
u
n
ti
m
e
p
e
r
f
o
r
m
a
n
c
e
wh
i
le
d
e
p
l
o
y
i
n
g
lat
e
n
c
y
r
e
q
u
ir
em
e
n
t
e
n
v
ir
o
n
m
e
n
ts
[
2
1
]
,
[
2
2
]
.
R
a
n
d
o
m
a
n
d
r
e
g
u
la
r
iz
in
g
f
e
a
tu
r
es
o
f
ac
ti
v
at
io
n
f
u
n
cti
o
n
s
w
er
e
ad
d
r
ess
ed
i
n
o
t
h
e
r
s
tu
d
i
es.
S
o
m
e
s
t
u
d
ies
a
d
d
r
ess
e
d
t
h
e
i
n
t
er
ac
ti
o
n
o
f
s
t
o
c
h
as
tic
el
em
e
n
ts
,
s
u
c
h
as
d
r
o
p
o
u
t
a
n
d
a
cti
v
a
ti
o
n
s
p
a
r
s
it
y
,
r
e
v
e
ali
n
g
t
h
e
ir
in
f
lu
e
n
ce
o
n
g
e
n
e
r
a
liz
ati
o
n
a
n
d
o
p
ti
m
iz
ati
o
n
.
T
h
is
s
h
o
we
d
t
h
at
s
o
m
e
ac
t
iv
ati
o
n
s
s
y
n
e
r
g
iz
e
d
b
e
tte
r
wit
h
d
r
o
p
o
u
t,
p
r
o
v
i
d
i
n
g
e
n
h
a
n
ce
d
r
o
b
u
s
t
n
ess
a
g
ai
n
s
t
o
v
e
r
f
itti
n
g
[
2
3
]
.
I
n
ce
r
t
ai
n
s
t
u
d
i
es,
th
e
SM
U
was
m
ea
n
t
t
o
m
ai
n
t
ai
n
t
r
ai
n
i
n
g
s
ta
b
i
lit
y
,
t
h
e
r
e
b
y
e
li
m
i
n
at
in
g
s
h
a
r
p
tr
a
n
s
i
ti
o
n
s
an
d
s
at
u
r
ate
d
g
r
a
d
i
en
ts
.
SM
U
p
r
o
v
id
e
d
a
b
al
a
n
ce
d
t
r
a
d
e
-
o
f
f
b
et
wee
n
R
e
L
U
’
s
s
p
a
r
s
it
y
a
n
d
T
a
n
h
’
s
s
m
o
o
t
h
n
ess
[
2
4
]
.
E
v
e
n
t
u
al
ly
,
s
o
m
e
r
ese
ar
ch
er
s
lo
o
k
e
d
at
h
o
w
a
cti
v
a
ti
o
n
f
u
n
cti
o
n
s
i
n
te
r
a
ct
ed
wi
th
o
p
ti
m
iz
ati
o
n
tec
h
n
i
q
u
es
a
n
d
s
u
p
p
o
r
te
d
s
ta
b
le
l
ea
r
n
i
n
g
i
n
p
o
ly
n
o
m
ial
r
e
g
r
ess
i
o
n
an
d
p
ic
tu
r
e
c
lass
i
f
i
c
ati
o
n
s
etti
n
g
s
,
t
h
u
s
d
e
m
o
n
s
tr
ati
n
g
th
at
a
cti
v
a
ti
o
n
s
ele
cti
o
n
in
f
l
u
e
n
ce
d
o
p
t
im
i
ze
r
p
e
r
f
o
r
m
an
ce
a
n
d
s
e
n
s
it
iv
it
y
to
h
y
p
e
r
p
a
r
a
m
e
te
r
s
etti
n
g
s
[
2
5
]
.
R
ec
en
t
s
tu
d
i
es
ill
u
s
t
r
at
e
d
th
e
u
s
e
o
f
m
a
c
h
i
n
e
l
ea
r
n
in
g
r
e
g
a
r
d
i
n
g
n
o
n
li
n
ea
r
m
o
d
eli
n
g
an
d
c
o
m
p
le
x
d
e
cisi
o
n
-
m
a
k
i
n
g
a
p
p
lic
ati
o
n
s
[
2
6
]
.
T
h
e
d
at
a
-
d
r
i
v
e
n
in
tel
li
g
e
n
ce
s
y
s
t
em
w
as
co
m
b
i
n
e
d
wit
h
a
p
r
a
cti
ca
l
s
y
s
t
em
f
o
r
p
h
y
s
ic
al
-
g
u
id
ed
l
ea
r
n
in
g
[
2
7
]
.
Me
a
n
w
h
i
le
,
s
ca
l
ab
le
s
o
l
u
ti
o
n
s
w
e
r
e
p
er
f
o
r
m
e
d
t
o
e
n
h
a
n
c
e
th
e
r
es
o
u
r
ce
-
co
n
s
t
r
a
in
ed
p
er
f
o
r
m
a
n
c
e
[
2
8
]
.
R
ec
e
n
t
e
c
o
n
o
m
ic
m
o
d
eli
n
g
s
t
u
d
ies
als
o
f
o
r
e
ca
s
t
ed
m
a
r
k
et
u
n
ce
r
tai
n
t
y
an
d
p
r
ice
d
y
n
am
ics
[
2
9
]
,
[
3
0
]
.
Am
o
n
g
t
h
e
r
e
ce
n
t
s
t
u
d
i
es,
t
h
e
Ga
u
s
s
i
an
p
r
o
ce
s
s
r
e
g
r
ess
i
o
n
s
e
r
v
e
d
as
a
s
t
r
o
n
g
b
as
eli
n
e
f
o
r
u
n
ce
r
t
ai
n
t
y
-
awa
r
e
m
o
d
eli
n
g
[
3
1
]
,
[
3
2
]
.
I
n
a
d
d
it
io
n
,
t
h
e
g
r
a
p
h
ic
al
an
d
c
au
s
alit
y
-
b
ase
d
m
o
d
els
h
a
d
g
a
in
e
d
t
h
e
in
t
er
p
r
et
a
b
le
m
o
d
e
li
n
g
o
f
m
u
l
tiv
ar
iat
e
d
ep
en
d
en
ci
es
[
3
3
]
,
[
3
4
]
.
Fi
n
al
ly
,
t
h
e
e
n
s
e
m
b
le
a
n
d
co
m
p
o
s
ite
l
ea
r
n
in
g
d
e
m
o
n
s
tr
ate
d
s
u
p
er
i
o
r
p
er
f
o
r
m
an
ce
o
v
er
t
h
e
f
in
a
n
ci
al
a
n
d
ec
o
n
o
m
i
c
p
r
e
d
i
cti
o
n
s
[
3
5
]
,
[
3
6
]
.
I
n
s
u
m
m
a
r
y
,
t
h
e
e
ar
lie
r
s
t
u
d
ies
o
n
t
h
e
ac
ti
v
at
io
n
f
u
n
c
ti
o
n
s
m
a
in
l
y
f
o
c
u
s
ed
o
n
t
h
e
t
r
a
d
i
ti
o
n
al
f
u
n
cti
o
n
s
lik
e
S
ig
m
o
i
d
,
T
a
n
h
,
a
n
d
R
e
L
U,
wit
h
li
m
it
ed
e
v
al
u
at
io
n
o
n
Swis
h
,
Mis
h
,
G
E
L
U
,
a
n
d
S
M
U.
T
h
u
s
,
t
h
e
g
r
a
d
i
e
n
t
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J E
lec
&
C
o
m
p
E
n
g
I
SS
N:
2088
-
8
7
0
8
Op
timiz
in
g
n
eu
r
a
l n
etw
o
r
ks:
a
co
mp
a
r
a
tive
s
tu
d
y
o
f
a
ctiva
tio
n
fu
n
ctio
n
s
in
…
(
A
b
d
u
lla
h
S
h
eikh
)
949
b
e
h
a
v
i
o
r
,
c
o
n
v
e
r
g
e
n
c
e,
a
n
d
c
o
m
p
u
t
ati
o
n
al
e
f
f
ici
en
c
y
we
r
e
af
f
e
cte
d
.
Als
o
,
in
c
o
n
s
is
te
n
t
tr
ai
n
in
g
s
et
ti
n
g
s
li
m
it
ed
th
e
g
e
n
e
r
al
iz
ati
o
n
.
He
n
c
e,
t
h
is
s
tu
d
y
r
e
n
d
e
r
s
a
c
o
m
p
r
eh
en
s
i
v
e
a
n
al
y
s
is
o
f
n
i
n
e
a
cti
v
ati
o
n
f
u
n
cti
o
n
s
r
e
g
a
r
d
i
n
g
ML
P
a
n
d
C
NN
m
o
d
e
ls
o
n
th
e
C
I
FAR
-
1
0
,
MN
I
S
T
,
a
n
d
C
I
FA
R
-
1
0
0
d
a
tas
ets.
3.
M
E
T
H
O
D
T
h
e
ap
p
r
o
ac
h
f
o
llo
we
d
in
th
is
s
tu
d
y
attem
p
ts
to
g
iv
e
a
c
o
m
p
r
eh
en
s
iv
e
a
n
d
b
alan
ce
d
in
v
es
tig
atio
n
o
f
d
if
f
er
en
t a
ctiv
atio
n
f
u
n
ctio
n
s
in
v
ar
io
u
s
n
eu
r
al
n
etwo
r
k
ar
c
h
i
tectu
r
es a
n
d
d
ata
s
ets.
T
h
e
m
aj
o
r
o
b
jectiv
e
o
f
th
e
p
r
o
p
o
s
ed
m
o
d
el
is
to
ex
a
m
in
e
h
o
w
th
e
ch
o
ice
o
f
ac
tiv
atio
n
af
f
ec
ts
th
e
m
o
d
el
ac
cu
r
ac
y
,
co
n
v
er
g
en
ce
s
p
ee
d
,
tr
ain
in
g
s
tab
ilit
y
,
an
d
r
o
b
u
s
tn
ess
.
Her
e,
two
wel
l
-
k
n
o
wn
b
en
ch
m
ar
k
d
atasets
,
s
u
ch
as
C
I
FAR
-
1
0
f
o
r
o
b
ject
r
ec
o
g
n
itio
n
an
d
MN
I
ST
f
o
r
h
an
d
wr
itten
d
ig
it
r
ec
o
g
n
itio
n
,
ar
e
s
elec
ted
to
en
s
u
r
e
a
h
ete
r
o
g
en
eo
u
s
ev
alu
atio
n
en
v
ir
o
n
m
en
t.
T
h
ese
d
atasets
d
if
f
er
in
co
m
p
lex
ity
.
Her
e,
MN
I
ST
is
s
im
p
ler
a
n
d
g
r
ay
s
ca
le,
wh
er
ea
s
C
I
FAR
-
10
is
co
lo
r
ed
im
ag
es
with
h
ig
h
er
in
tr
a
-
class
v
ar
iab
ilit
y
,
th
u
s
allo
win
g
f
o
r
p
er
f
o
r
m
an
ce
ass
ess
m
en
t
u
n
d
e
r
v
ar
io
u
s
co
n
d
itio
n
s
.
I
n
th
e
p
r
o
p
o
s
ed
m
eth
o
d
o
l
o
g
y
,
two
m
o
d
el
ar
c
h
itectu
r
es
ar
e
d
esig
n
ed
:
an
ML
P
f
o
r
th
e
MN
I
ST
d
ataset
an
d
a
C
NN
f
o
r
th
e
C
I
FAR
-
1
0
.
W
ith
th
e
n
in
e
ac
tiv
atio
n
f
u
n
ctio
n
s
(
i.e
.
,
R
eL
U
,
Sig
m
o
id
,
T
a
n
h
,
E
L
U,
SEL
U,
Swis
h
,
Mish
,
GE
L
U,
an
d
SMU)
,
ML
P
an
d
C
NN
ar
e
tr
ain
ed
s
ep
a
r
ately
.
Fo
r
en
s
u
r
in
g
f
ai
r
n
ess
in
ev
alu
atio
n
,
all
h
y
p
er
p
ar
am
eter
s
,
in
clu
d
in
g
th
e
n
u
m
b
er
o
f
e
p
o
ch
s
,
b
atc
h
s
ize,
a
n
d
o
p
tim
ize
r
,
ar
e
k
e
p
t
co
n
s
tan
t
ac
r
o
s
s
ex
p
er
im
en
ts
.
Als
o
,
th
e
tr
ain
in
g
is
d
o
n
e
b
ased
o
n
th
e
Ad
am
o
p
tim
izer
with
its
d
ef
au
lt
lear
n
in
g
r
ate
o
f
0
.
0
0
1
.
All in
p
u
t
f
ea
tu
r
es a
r
e
n
o
r
m
alize
d
to
th
e
r
an
g
e
o
f
[
0
,
1
]
,
[
0
,
1
]
,
an
d
[
0
,
1
]
.
3
.
1
.
Da
t
a
s
et
s
elec
t
io
n
T
h
e
two
b
en
ch
m
a
r
k
s
ets
ar
e
ch
o
s
en
in
th
is
r
esear
ch
t
o
g
a
u
g
e
th
e
m
o
d
el’
s
p
er
f
o
r
m
an
ce
a
t
d
if
f
er
e
n
t
lev
els
o
f
co
m
p
lex
ity
.
T
h
e
MN
I
ST
d
ataset
[
2
0
]
,
co
m
p
r
i
s
in
g
7
0
,
0
0
0
im
ag
es
o
f
g
r
ay
-
s
ca
le
d
ig
its
wr
i
tten
(
2
8
×2
8
p
ix
els
ea
ch
)
b
y
h
an
d
,
6
0
,
0
0
0
s
am
p
les
f
o
r
tr
ain
in
g
p
u
r
p
o
s
es,
an
d
1
0
,
0
0
0
s
am
p
les
f
o
r
test
p
u
r
p
o
s
es,
is
u
s
ed
to
s
tu
d
y
ML
Ps
.
T
h
is
d
a
taset
'
s
co
m
p
ar
ativ
ely
l
o
w
r
es
o
lu
tio
n
an
d
lim
ited
v
ar
iab
ilit
y
m
ak
e
it
t
h
e
b
est
ch
o
ice
f
o
r
ass
ess
in
g
s
im
p
ler
ar
ch
itectu
r
es
lik
e
ML
Ps
.
Als
o
,
MN
I
ST
v
alid
ates
th
e
f
u
n
d
am
en
tal
asp
ec
ts
o
f
m
o
d
el
d
esig
n
,
i
n
clu
d
in
g
ac
ti
v
atio
n
f
u
n
ctio
n
s
in
f
u
lly
co
n
n
ec
ted
la
y
er
s
.
T
o
test
p
er
f
o
r
m
an
ce
in
a
m
o
r
e
co
m
p
lex
f
r
am
ewo
r
k
,
C
NNs
ar
e
test
ed
u
s
in
g
th
e
C
I
FAR
-
1
0
d
ataset
[
3
0
]
.
T
h
is
d
ataset
co
n
s
is
ts
o
f
6
0
,
0
0
0
co
l
o
r
im
ag
es
(
3
2
×
3
2
p
ix
els
ea
ch
)
f
r
o
m
1
0
d
if
f
er
en
t
ca
teg
o
r
ies,
wi
th
5
0
,
0
0
0
f
o
r
tr
ain
i
n
g
a
n
d
1
0
,
0
0
0
f
o
r
test
in
g
.
T
h
e
C
I
FAR
-
1
0
d
ataset
co
n
s
is
ts
o
f
h
ig
h
e
r
v
is
u
al
c
o
m
p
lex
ity
o
w
in
g
to
co
lo
r
ch
an
n
els,
in
tr
a
-
cl
ass
v
ar
iatio
n
s
,
an
d
clu
tter
ed
b
ac
k
g
r
o
u
n
d
s
,
m
ak
in
g
it
s
u
itab
le
f
o
r
ev
alu
at
in
g
ac
tiv
atio
n
p
er
f
o
r
m
a
n
ce
in
co
n
v
o
lu
tio
n
al
ar
ch
itectu
r
es.
T
o
in
cr
ea
s
e
th
e
g
en
er
aliza
b
ilit
y
,
th
e
C
I
FAR
-
1
0
0
d
ataset
is
u
tili
ze
d
.
T
h
e
C
I
FAR
-
1
0
0
d
ataset
co
n
s
is
ts
o
f
6
0
,
0
0
0
im
ag
es
(
3
2
×3
2
p
ix
els
ea
ch
)
with
1
0
0
cla
s
s
es
g
r
o
u
p
ed
in
to
2
0
s
u
p
e
r
cla
s
s
es
.
Fro
m
th
e
to
tal
im
ag
es,
4
5
,
0
0
0
,
5
,
0
0
0
,
an
d
1
0
,
0
0
0
im
ag
es
ar
e
u
tili
ze
d
f
o
r
tr
ain
in
g
,
v
alid
atio
n
,
a
n
d
t
esti
n
g
th
e
class
if
ier
m
o
d
el,
r
esp
ec
tiv
ely
.
3
.
2
.
Net
w
o
rk
a
rc
hite
ct
ures
T
wo
d
if
f
er
en
t
ty
p
es
o
f
n
e
u
r
al
n
etwo
r
k
ar
ch
itectu
r
es,
s
u
ch
as
ML
P
an
d
C
NN,
ar
e
u
s
ed
.
T
h
e
s
p
ec
if
icatio
n
s
o
f
ML
P a
n
d
C
N
N
ar
e
ex
p
lain
ed
as f
o
llo
ws,
3
.
2
.
1
.
M
ulti
-
la
y
er
perc
ept
ro
n
T
h
e
ML
P
d
esig
n
ed
f
o
r
MN
I
ST
class
if
icatio
n
co
m
p
r
is
es
a
n
o
u
t
p
u
t
la
y
er
,
a
n
in
p
u
t
la
y
e
r
,
an
d
two
d
en
s
e
h
id
d
e
n
lay
er
s
.
T
h
e
ML
P
is
ch
o
s
en
b
ec
au
s
e
it
p
er
f
o
r
m
s
well
o
n
lo
w
-
d
im
e
n
s
io
n
al
t
ask
s
.
Her
e,
th
e
in
p
u
t
lay
er
(
)
ac
ce
p
ts
th
e
f
latten
ed
2
8
×2
8
g
r
ay
s
ca
le
p
ix
el
v
alu
es.
Her
e,
th
e
in
p
u
t
(
ℎ
)
is
f
ed
in
to
M
L
P,
an
d
th
e
p
r
o
ce
s
s
in
g
o
f
(
)
is
ca
r
r
ied
o
u
t a
s
(
1
0
)
,
=
[
ℎ
+
]
∗
(
1
0
)
W
h
er
e,
(
,
)
ar
e
th
e
weig
h
t
a
n
d
b
ias
v
alu
es
o
f
th
e
in
p
u
t
(
ℎ
)
,
r
esp
ec
tiv
ely
.
Als
o
,
th
e
two
d
en
s
e
h
id
d
en
lay
er
s
co
n
s
is
t
o
f
1
2
8
an
d
6
4
n
eu
r
o
n
s
.
L
ik
ewise,
d
r
o
p
o
u
t
r
e
g
u
lar
izatio
n
is
u
s
ed
in
ea
ch
la
y
er
to
p
r
ev
en
t
a
n
y
in
s
tan
ce
s
o
f
o
v
er
f
itti
n
g
.
So
f
tm
ax
ac
tiv
atio
n
(
)
is
u
tili
ze
d
in
th
e
f
in
al
o
u
tp
u
t la
y
er
(
)
f
o
r
s
itu
atio
n
s
in
v
o
lv
in
g
m
u
lti
-
class
clas
s
if
icatio
n
is
s
u
e
s
.
=
{
[
∑
+
]
∗
}
×
(
1
1
)
Her
e,
th
e
h
id
d
e
n
lay
e
r
s
ize
ch
o
ices
ar
e
m
o
tiv
ated
b
y
t
h
e
n
e
ed
to
b
alan
ce
co
m
p
u
tatio
n
al
e
f
f
icien
cy
.
T
h
e
ML
P
class
if
ier
d
iag
r
am
is
d
is
p
lay
e
d
in
Fig
u
r
e
1
.
Als
o
,
all
ac
ti
v
atio
n
f
u
n
ctio
n
s
ar
e
a
p
p
lied
s
ep
ar
ately
to
t
h
ese
h
id
d
en
la
y
er
s
f
o
r
ev
alu
atin
g
th
eir
im
p
ac
t o
n
m
o
d
el
p
er
f
o
r
m
a
n
ce
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
0
8
8
-
8
7
0
8
I
n
t J E
lec
&
C
o
m
p
E
n
g
,
Vo
l.
1
6
,
No
.
2
,
Ap
r
il
20
2
6
:
9
4
5
-
963
950
Fig
u
r
e
1
.
ML
P c
lass
if
ier
d
iag
r
am
3
.
2
.
2
.
Co
nv
o
lutio
na
l
neura
l
net
wo
rk
T
h
e
C
NN
m
o
d
el
f
o
r
C
I
FAR
-
1
0
an
d
C
I
FAR
-
1
0
0
class
if
icatio
n
co
n
s
is
ts
o
f
two
co
n
v
o
lu
tio
n
al
lay
er
s
,
a
f
latten
lay
er
,
a
d
e
n
s
e
h
id
d
e
n
lay
er
,
an
d
a
n
o
u
tp
u
t
la
y
er
.
T
h
e
C
NN
is
s
elec
ted
b
ec
au
s
e
it
lear
n
s
th
e
s
p
atial
an
d
lo
ca
l
f
ea
tu
r
es
i
n
th
e
im
ag
es
au
to
m
atica
lly
.
Her
e,
th
e
tw
o
co
n
v
o
lu
tio
n
al
lay
er
s
c
o
n
tai
n
3
2
an
d
6
4
f
ilter
s
.
T
h
e
co
n
v
o
lu
tio
n
al
la
y
er
(
)
is
u
s
ed
f
o
r
m
ap
p
i
n
g
th
e
f
ea
tu
r
es,
a
n
d
it is
ex
p
r
ess
ed
as
(
1
2
)
,
=
[
+
]
×
(
1
2
)
W
h
er
e,
(
)
is
th
e
in
p
u
t d
ata
o
f
C
NN,
an
d
(
,
)
ar
e
th
e
weig
h
t
a
n
d
b
ias
v
alu
es
o
f
th
e
in
p
u
t
(
)
,
r
esp
ec
tiv
ely
.
Fu
r
th
er
,
th
e
m
ax
-
p
o
o
lin
g
is
ca
r
r
ied
o
u
t.
I
t
is
u
tili
ze
d
to
d
i
m
in
is
h
th
e
s
p
atial
d
im
en
s
io
n
ality
o
f
th
e
s
am
p
les.
Af
ter
p
o
o
lin
g
,
th
e
m
ax
-
p
o
o
le
d
d
ata
is
co
n
v
er
ted
in
to
a
s
in
g
le
v
ec
to
r
in
th
e
f
latten
lay
er
.
T
h
e
o
u
tp
u
t la
y
er
(
)
is
r
esp
o
n
s
ib
le
f
o
r
class
if
y
in
g
th
e
o
u
tp
u
ts
u
tili
zin
g
So
f
tMa
x
ac
tiv
atio
n
(
)
.
=
[
(
∑
+
)
×
]
∗
(
1
3
)
Ad
d
itio
n
ally
,
a
d
r
o
p
o
u
t
is
in
clu
d
ed
in
o
r
d
er
to
g
et
a
h
i
g
h
er
l
ev
el
o
f
g
en
e
r
aliza
tio
n
.
T
r
ain
in
g
is
p
er
f
o
r
m
ed
o
n
b
o
th
m
o
d
els
b
y
u
s
in
g
th
e
Ad
a
m
o
p
tim
izer
with
its
d
ef
au
lt
p
ar
am
eter
s
,
wh
ich
in
clu
d
e
a
lear
n
in
g
r
ate
o
f
0
.
0
0
1
.
B
ef
o
r
e
tr
ain
in
g
,
it
is
n
ec
ess
a
r
y
to
s
tan
d
ar
d
ize
all
o
f
t
h
e
i
n
p
u
t
d
ata
t
o
th
e
r
a
n
g
e
[
0
,
1
]
.
I
n
Fig
u
r
e
2
,
th
e
d
iag
r
am
m
atic
r
ep
r
esen
tatio
n
o
f
th
e
C
NN
class
if
ier
is
d
is
p
lay
ed
.
Her
e,
in
ea
ch
e
x
p
er
im
en
t,
th
e
ac
tiv
atio
n
f
u
n
ctio
n
s
ar
e
a
p
p
lied
s
ep
ar
atel
y
ac
r
o
s
s
all
h
id
d
en
lay
e
r
s
.
L
ik
ewise,
th
e
m
o
d
els
ar
e
tr
ain
ed
f
o
r
a
f
ix
e
d
n
u
m
b
er
o
f
ep
o
c
h
s
with
id
en
tical
b
atch
s
izes f
o
r
m
ain
tain
in
g
c
o
m
p
ar
a
b
ilit
y
.
Fig
u
r
e
2
.
Diag
r
a
m
m
atic
r
ep
r
esen
tatio
n
o
f
t
h
e
C
NN
class
if
ier
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J E
lec
&
C
o
m
p
E
n
g
I
SS
N:
2088
-
8
7
0
8
Op
timiz
in
g
n
eu
r
a
l n
etw
o
r
ks:
a
co
mp
a
r
a
tive
s
tu
d
y
o
f
a
ctiva
tio
n
fu
n
ctio
n
s
in
…
(
A
b
d
u
lla
h
S
h
eikh
)
951
4.
RE
SU
L
T
S AN
D
D
I
SCU
SS
I
O
N
4
.
1
.
Dif
f
er
ent
a
ct
iv
a
t
io
n f
un
ct
io
ns
I
n
o
r
d
er
to
co
m
p
r
e
h
en
s
iv
ely
e
x
am
in
e,
ca
teg
o
r
ize,
an
d
r
an
k
th
e
im
p
ac
t
th
at
ac
tiv
atio
n
f
u
n
ct
io
n
s
h
av
e
o
n
lear
n
in
g
d
y
n
am
ics
an
d
to
ca
lcu
late
th
e
o
v
er
all
p
er
f
o
r
m
an
ce
o
f
n
eu
r
al
n
etwo
r
k
m
o
d
els,
n
in
e
d
is
tin
ct
ac
tiv
atio
n
f
u
n
ctio
n
s
ar
e
s
u
b
j
ec
ted
to
e
x
ten
s
iv
e
test
in
g
a
n
d
co
m
p
ar
is
o
n
.
T
h
e
s
elec
tio
n
o
f
th
o
s
e
ac
tiv
atio
n
f
u
n
ctio
n
s
is
m
ad
e
with
th
e
in
ten
tio
n
o
f
b
ein
g
r
ep
r
esen
tativ
e
o
f
b
o
t
h
m
o
r
e
co
n
v
en
tio
n
al
m
eth
o
d
s
th
at
h
av
e
b
ee
n
u
tili
ze
d
f
o
r
a
co
n
s
id
er
ab
l
e
am
o
u
n
t
o
f
tim
e
an
d
m
o
r
e
co
n
tem
p
o
r
a
r
y
m
eth
o
d
s
th
at
h
av
e
em
er
g
e
d
i
n
m
o
r
e
r
ec
en
t
y
ea
r
s
.
T
h
e
R
eL
U,
Sig
m
o
id
,
a
n
d
T
an
h
f
u
n
ctio
n
s
ar
e
ex
am
p
les
o
f
th
e
class
ical
f
u
n
ctio
n
s
th
at
ar
e
d
is
cu
s
s
ed
in
th
is
ar
t
icle.
T
h
es
e
f
u
n
ctio
n
s
h
av
e
b
ee
n
u
s
ed
i
n
th
e
p
ast
f
o
r
d
ev
elo
p
in
g
,
tr
a
in
in
g
,
an
d
u
tili
zin
g
n
eu
r
al
n
etwo
r
k
s
.
H
o
wev
er
,
it
is
d
is
co
v
er
ed
t
h
at
th
ey
h
a
v
e
s
o
m
e
lim
its
in
th
e
s
en
s
e
(
i.e
.
,
t
h
ey
ar
e
s
u
s
ce
p
tib
le
to
s
o
m
e
o
f
t
h
eir
o
wn
p
itfa
lls
)
.
T
h
ese
p
itfa
lls
in
clu
d
e
p
r
o
b
le
m
s
,
s
u
ch
as
v
an
is
h
in
g
g
r
ad
ien
ts
an
d
d
ea
d
n
eu
r
o
n
s
,
d
u
r
in
g
t
h
e
lear
n
in
g
p
r
o
ce
s
s
.
W
h
ile
th
e
s
tu
d
y
is
s
till
g
o
in
g
o
v
er
in
ter
m
e
d
iate
v
ar
ian
ts
,
s
o
m
e
co
n
s
id
er
ati
o
n
is
g
iv
en
to
th
e
p
o
s
s
ib
ilit
y
o
f
u
tili
zin
g
ELU
an
d
SEL
U
ac
tiv
atio
n
f
u
n
ctio
n
s
.
T
h
ese
ac
tiv
atio
n
f
u
n
ctio
n
s
ar
e
s
elec
ted
to
ad
d
r
ess
s
o
m
e
o
f
th
e
m
o
s
t
tr
o
u
b
leso
m
e
r
estrictio
n
s
o
f
R
eL
U
.
T
h
e
u
til
izatio
n
o
f
a
n
eg
ativ
e
in
teg
er
a
s
in
p
u
t,
alo
n
g
with
th
e
in
tr
o
d
u
ctio
n
o
f
a
n
o
n
-
ze
r
o
v
alu
e,
alb
eit
m
in
im
al,
f
ac
ilit
ates th
is
o
u
tco
m
e.
T
h
e
f
ea
tu
r
es
f
ac
ilit
ate
en
h
a
n
ce
d
f
lo
w
th
r
o
u
g
h
g
r
a
d
ien
t
u
tili
za
tio
n
an
d
im
p
r
o
v
e
c
o
n
v
er
g
e
n
ce
,
esp
ec
ially
in
d
ee
p
er
n
eu
r
al
n
etwo
r
k
a
r
ch
itectu
r
es.
E
v
e
r
y
s
in
g
le
o
n
e
o
f
t
h
ese
ac
tiv
atio
n
f
u
n
ctio
n
s
is
in
co
r
p
o
r
ated
in
to
t
h
e
ML
P
an
d
C
NN
m
o
d
els
th
at
ar
e
b
ein
g
ev
alu
ated
in
s
tr
ateg
ic
lo
ca
tio
n
s
.
I
t
is
im
p
o
r
tan
t
to
k
ee
p
in
m
i
n
d
th
at
th
e
u
s
e
o
f
an
o
u
tp
u
t
lay
er
r
em
ain
s
th
e
s
am
e
ac
r
o
s
s
all
o
f
th
e
d
i
f
f
er
en
t
co
n
f
ig
u
r
atio
n
s
th
at
ar
e
ev
alu
ated
in
t
h
is
in
v
esti
g
a
tio
n
.
T
h
is
p
r
o
ce
s
s
is
d
o
n
e
b
y
em
p
lo
y
in
g
th
e
well
-
k
n
o
wn
So
f
tMa
x
alg
o
r
ith
m
to
f
ac
ilit
ate
an
ev
en
co
m
p
ar
is
o
n
.
T
ab
le
1
o
f
f
er
s
an
ex
h
au
s
tiv
e
class
if
icatio
n
o
f
ac
tiv
atio
n
f
u
n
ctio
n
s
in
ter
m
s
o
f
th
eir
u
n
iq
u
e
d
esig
n
class
es,
wh
ich
h
av
e
b
ee
n
ca
t
eg
o
r
ized
as
class
ical,
in
ter
m
e
d
iate,
an
d
m
o
d
er
n
.
T
h
is
tab
le
also
r
ef
lects
th
eir
p
r
o
p
er
ties
,
in
clu
d
in
g
th
eir
m
o
n
o
to
n
icity
an
d
s
m
o
o
t
h
n
ess
,
as
well
as
th
eir
r
esp
ec
ti
v
e
s
im
p
le
alg
eb
r
aic
eq
u
atio
n
s
.
T
h
is
g
u
id
e
is
u
s
ed
to
s
u
p
p
o
r
t
an
d
r
ei
n
f
o
r
ce
th
e
th
e
o
r
etica
l
in
ter
p
r
etatio
n
o
f
r
esu
lts
f
r
o
m
ex
p
er
im
en
ts
.
T
ab
le
1
.
Su
m
m
a
r
y
o
f
ac
tiv
atio
n
f
u
n
ctio
n
s
u
s
ed
in
th
e
s
tu
d
y
A
c
t
i
v
a
t
i
o
n
Ty
p
e
M
o
n
o
t
o
n
i
c
S
mo
o
t
h
S
i
mp
l
i
f
i
e
d
F
o
r
m
u
l
a
R
e
LU
C
l
a
s
si
c
a
l
Y
e
s
No
(
)
=
(
0
,
)
S
i
g
m
o
i
d
C
l
a
s
si
c
a
l
Y
e
s
Y
e
s
(
)
=
1
/
(
1
+
−
)
Ta
n
h
C
l
a
s
si
c
a
l
Y
e
s
Y
e
s
(
)
=
ℎ
(
)
ELU
I
n
t
e
r
med
i
a
t
e
Y
e
s
Y
e
s
(
)
=
{
>
0
(
−
1
)
S
ELU
I
n
t
e
r
med
i
a
t
e
Y
e
s
Y
e
s
(
)
=
{
>
0
(
−
1
)
S
w
i
sh
M
o
d
e
r
n
No
Y
e
s
(
)
=
∗
si
gmo
i
d
(
)
M
i
s
h
M
o
d
e
r
n
No
Y
e
s
(
)
=
∗
ℎ
[
(
1
+
)
]
G
ELU
M
o
d
e
r
n
No
Y
e
s
(
)
=
∗
(
)
S
M
U
M
o
d
e
r
n
No
Y
e
s
(
)
=
∗
ℎ
[
so
f
t
p
lus
(
)
]
4
.
2
.
T
he
ex
perim
ent
a
l set
up
I
n
th
is
p
ap
er
,
th
e
Py
th
o
n
s
o
f
twar
e
to
o
l
is
u
s
ed
f
o
r
im
p
le
m
en
tatio
n
.
E
ac
h
m
o
d
el
is
p
a
in
s
tak
in
g
ly
d
ev
elo
p
e
d
u
s
in
g
h
i
g
h
-
p
er
f
o
r
m
an
ce
to
o
lb
o
x
es
in
T
en
s
o
r
Flo
w
an
d
Ker
as,
w
h
ich
h
av
e
b
e
co
m
e
wid
ely
k
n
o
wn
f
o
r
t
h
eir
h
i
g
h
s
u
cc
ess
in
s
o
lv
in
g
co
m
p
le
x
an
d
c
o
m
p
u
tatio
n
ally
d
em
a
n
d
in
g
m
ac
h
in
e
lear
n
in
g
task
s
.
Dev
elo
p
m
en
t
f
o
r
all
m
o
d
els
o
cc
u
r
s
in
th
e
cl
o
u
d
-
b
ased
c
o
m
p
u
tatio
n
al
en
v
ir
o
n
m
e
n
t
f
u
r
n
is
h
ed
b
y
Go
o
g
le
C
o
lab
,
wh
er
e
ac
ce
ler
atio
n
b
y
GPU
is
u
s
ed
to
co
n
s
id
er
ab
ly
in
cr
ea
s
e
co
m
p
u
tatio
n
al
s
p
ee
d
an
d
o
v
er
all
p
er
f
o
r
m
an
ce
.
T
h
e
h
ar
d
war
e
r
eq
u
ir
em
en
ts
in
clu
d
e
an
NVI
D
I
A
T
4
GPU
with
1
6
GB
R
AM
.
I
n
all
in
d
i
v
id
u
al
m
o
d
els
d
ev
elo
p
ed
,
s
ee
d
s
o
f
1
0
,
3
0
,
an
d
5
0
ep
o
c
h
s
in
tr
ain
i
n
g
ar
e
u
tili
ze
d
in
c
o
m
b
in
atio
n
with
a
b
atc
h
th
at
co
n
s
is
ts
o
f
3
2
s
am
p
les
f
o
r
a
ll
iter
atio
n
s
p
er
f
o
r
m
ed
.
Fo
r
all
ac
tiv
atio
n
f
u
n
ctio
n
s
,
t
r
ain
in
g
is
d
o
n
e
u
n
d
e
r
p
r
ec
is
ely
id
en
tical
en
v
ir
o
n
m
e
n
tal
s
ettin
g
s
to
allo
w
f
o
r
u
n
b
i
ased
an
d
f
ai
r
co
m
p
ar
is
o
n
b
et
wee
n
th
em
.
I
n
o
r
d
er
to
ac
h
iev
e
u
n
if
o
r
m
ity
f
o
r
all
ex
p
er
im
e
n
ts
,
th
e
r
an
d
o
m
s
e
ed
s
o
f
1
0
,
3
0
,
an
d
5
0
ep
o
ch
s
ar
e
m
ain
tain
ed
to
p
r
ev
en
t
v
ar
ian
ce
th
at
m
ay
r
es
u
lt
f
r
o
m
v
ar
y
in
g
i
n
itializatio
n
s
.
E
ac
h
m
o
d
el
is
d
ev
el
o
p
ed
s
ep
ar
ately
u
s
in
g
an
id
en
tical
in
itializatio
n
ap
p
r
o
ac
h
with
th
e
Ad
am
o
p
tim
izer
to
en
s
u
r
e
th
at
th
e
r
esu
lts
d
er
iv
e
d
ar
e
r
eliab
le
as we
ll
as
co
m
p
ar
a
b
le.
T
h
e
h
y
p
e
r
p
ar
a
m
eter
s
o
f
ML
P
an
d
C
NN
ar
e
tu
n
ed
m
an
u
ally
,
a
n
d
t
h
eir
d
et
ails
ar
e
p
r
o
v
id
e
d
in
T
ab
les
2
an
d
3
.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
0
8
8
-
8
7
0
8
I
n
t J E
lec
&
C
o
m
p
E
n
g
,
Vo
l.
1
6
,
No
.
2
,
Ap
r
il
20
2
6
:
9
4
5
-
963
952
T
ab
le
2
.
Hy
p
er
p
ar
a
m
eter
s
o
f
ML
P
C
o
m
p
o
n
e
n
t
S
e
t
t
i
n
g
/
V
a
l
u
e
D
e
scri
p
t
i
o
n
A
c
t
i
v
a
t
i
o
n
(
h
i
d
d
e
n
l
a
y
e
r
s)
R
e
LU
/
S
i
g
mo
i
d
/
T
a
n
h
/
ELU
/
S
ELU
/
S
w
i
sh
/
M
i
s
h
/
G
ELU
/
S
M
U
Ev
a
l
u
a
t
e
d
i
n
d
i
v
i
d
u
a
l
l
y
O
u
t
p
u
t
a
c
t
i
v
a
t
i
o
n
S
o
f
t
M
a
x
M
u
l
t
i
‐
c
l
a
ss
c
l
a
ssi
f
i
c
a
t
i
o
n
I
n
p
u
t
sh
a
p
e
2
8
×
2
8
(
f
l
a
t
t
e
n
e
d
t
o
7
8
4
)
N
o
r
mal
i
z
e
d
g
r
a
y
s
c
a
l
e
O
p
t
i
mi
z
e
r
A
d
a
m
A
d
a
p
t
i
v
e
g
r
a
d
i
e
n
t
o
p
t
i
m
i
z
a
t
i
o
n
Le
a
r
n
i
n
g
r
a
t
e
0
.
0
0
1
D
e
f
a
u
l
t
u
n
l
e
ss s
p
e
c
i
f
i
e
d
i
n
t
h
e
l
e
a
r
n
i
n
g
r
a
t
e
(
L
R
)
s
e
n
s
i
t
i
v
i
t
y
e
x
p
e
r
i
me
n
t
B
a
t
c
h
si
z
e
32
F
i
x
e
d
f
o
r
c
o
n
si
s
t
e
n
c
y
Ep
o
c
h
s
1
0
,
3
0
,
a
n
d
5
0
F
i
x
e
d
c
o
mp
a
r
i
s
o
n
se
t
u
p
Lo
ss f
u
n
c
t
i
o
n
C
a
t
e
g
o
r
i
c
a
l
C
r
o
ss‐
E
n
t
r
o
p
y
S
t
a
n
d
a
r
d
m
u
l
t
i
‐
c
l
a
ss
l
o
ss
W
e
i
g
h
t
i
n
i
t
i
a
l
i
z
a
t
i
o
n
H
e
/
X
a
v
i
e
r
(
K
e
r
a
s
d
e
f
a
u
l
t
)
D
e
p
e
n
d
s
o
n
a
c
t
i
v
a
t
i
o
n
R
e
g
u
l
a
r
i
z
a
t
i
o
n
D
r
o
p
o
u
t
(
0
.
2
)
A
v
o
i
d
s
o
v
e
r
f
i
t
t
i
n
g
T
ab
le
3
.
Hy
p
er
p
ar
a
m
eter
s
o
f
C
NN
C
o
m
p
o
n
e
n
t
S
e
t
t
i
n
g
/
V
a
l
u
e
D
e
scri
p
t
i
o
n
A
c
t
i
v
a
t
i
o
n
(
h
i
d
d
e
n
l
a
y
e
r
s)
R
e
LU
/
S
i
g
mo
i
d
/
T
a
n
h
/
ELU
/
S
ELU
/
S
w
i
sh
/
M
i
s
h
/
G
ELU
/
S
M
U
Te
st
e
d
i
n
d
i
v
i
d
u
a
l
l
y
O
u
t
p
u
t
a
c
t
i
v
a
t
i
o
n
S
o
f
t
ma
x
M
u
l
t
i
‐
c
l
a
ss
c
l
a
ssi
f
i
c
a
t
i
o
n
I
n
p
u
t
sh
a
p
e
3
2
×
3
2
×
3
R
G
B
i
ma
g
e
s
O
p
t
i
mi
z
e
r
A
d
a
m
A
d
a
p
t
i
v
e
o
p
t
i
m
i
z
a
t
i
o
n
Le
a
r
n
i
n
g
r
a
t
e
0
.
0
0
1
D
e
f
a
u
l
t
u
n
l
e
ss
t
u
n
e
d
B
a
t
c
h
si
z
e
32
C
o
n
si
s
t
e
n
t
a
c
r
o
ss
t
e
s
t
s
Ep
o
c
h
s
1
0
,
3
0
,
a
n
d
5
0
F
a
i
r
p
e
r
f
o
r
m
a
n
c
e
c
o
mp
a
r
i
s
o
n
Lo
ss f
u
n
c
t
i
o
n
C
a
t
e
g
o
r
i
c
a
l
C
r
o
ss‐
E
n
t
r
o
p
y
S
t
a
n
d
a
r
d
f
o
r
c
l
a
ss
i
f
i
c
a
t
i
o
n
W
e
i
g
h
t
i
n
i
t
i
a
l
i
z
a
t
i
o
n
H
e
/
X
a
v
i
e
r
(
K
e
r
a
s
d
e
f
a
u
l
t
)
B
a
se
d
o
n
l
a
y
e
r
t
y
p
e
P
o
o
l
i
n
g
M
a
x
P
o
o
l
i
n
g
(
2
×
2
)
S
p
a
t
i
a
l
d
o
w
n
sa
mp
l
i
n
g
D
r
o
p
o
u
t
0
.
3
I
mp
r
o
v
e
g
e
n
e
r
a
l
i
z
a
t
i
o
n
4
.
3
.
E
v
a
lua
t
i
o
n
m
et
rics
I
n
a
s
in
ce
r
e
attem
p
t
t
o
en
s
u
r
e
an
ex
h
a
u
s
tiv
e
an
d
u
n
b
iased
as
s
ess
m
en
t
o
f
all
th
e
ac
tiv
atio
n
f
u
n
ctio
n
s
o
n
th
e
r
ad
ar
in
t
h
is
r
ev
iew,
th
e
s
p
ec
if
ic
r
esear
ch
h
as
d
ec
id
e
d
to
a
d
o
p
t
an
o
r
d
er
l
y
ap
p
r
o
ac
h
th
at
in
v
o
lv
es
f
o
u
r
cr
u
cial
m
ea
s
u
r
em
en
t in
d
icato
r
s
.
T
h
e
in
d
icato
r
s
in
clu
d
e
v
alid
atio
n
ac
cu
r
ac
y
,
v
alid
atio
n
lo
s
s
,
tr
ain
in
g
tim
e,
an
d
g
r
ad
ien
t
s
tab
ilit
y
.
T
h
r
o
u
g
h
th
ese
p
ar
ticu
lar
p
ar
am
eter
s
,
th
i
s
r
esear
ch
is
wo
r
k
in
g
h
ar
d
to
m
ak
e
an
i
n
-
d
e
p
th
an
aly
s
is
,
f
ea
tu
r
in
g
b
o
th
q
u
an
ti
tativ
e
m
ea
s
u
r
es a
n
d
q
u
alitativ
e
f
ea
tu
r
es,
to
attem
p
t to
d
elv
e
in
d
ep
th
in
to
all
th
e
d
y
n
am
ics ass
o
ciate
d
with
ef
f
i
ca
cy
an
d
e
f
f
icien
cy
.
4
.
3
.
1
.
Va
lid
a
t
io
n
a
cc
ura
cy
Valid
atio
n
ac
cu
r
ac
y
is
th
e
p
r
o
p
o
r
tio
n
o
f
co
r
r
ec
tly
class
if
ied
in
s
tan
ce
s
in
th
e
test
d
ata
s
et
an
d
is
u
s
ed
as th
e
m
ain
m
ea
s
u
r
e
f
o
r
d
ete
r
m
in
in
g
if
a
m
o
d
el
h
as lea
r
n
ed
to
g
en
er
alize
.
I
t is co
m
p
u
te
d
a
s
(
1
4
)
,
A
c
c
ura
c
y
=
N
um
b
e
ro
f
C
o
rre
c
t
Pre
d
i
c
t
ion
s
T
o
t
a
l
N
u
m
b
e
r
o
f
Pre
d
ic
t
ion
s
(
1
4
)
Gr
ea
ter
ac
cu
r
ac
y
in
d
icate
s
m
o
r
e
ef
f
ec
tiv
en
ess
in
co
r
r
ec
tly
id
en
tify
in
g
p
atter
n
s
in
n
ew,
n
ev
er
-
b
e
f
o
r
e
-
s
ee
n
d
ata.
4
.
3
.
2
.
Va
lid
a
t
io
n
l
o
s
s
Valid
atio
n
lo
s
s
est
im
ates
th
e
d
eg
r
ee
to
wh
ich
th
e
esti
m
ated
p
r
o
b
ab
ilit
ies
d
iv
er
g
e
f
r
o
m
ac
t
u
al
lab
els.
I
n
m
u
lti
-
class
class
if
icat
io
n
task
s
,
v
alid
atio
n
lo
s
s
is
u
s
ed
in
co
m
b
in
atio
n
with
th
e
ca
teg
o
r
ical
cr
o
s
s
-
en
tr
o
p
y
lo
s
s
f
u
n
ctio
n
(
)
,
an
d
it is
ca
lcu
la
ted
as
(
1
5
)
,
=
−
∑
(
̂
)
=
1
(
1
5
)
Her
e,
th
e
lab
el
is
im
p
lied
as
(
)
,
th
e
p
r
ed
icted
p
r
o
b
ab
ilit
y
f
o
r
class
(
)
is
n
o
tated
as
(
̂
)
,
an
d
th
e
to
tal
n
u
m
b
er
o
f
class
es
is
ex
em
p
lifie
d
as
(
)
.
A
lo
wer
v
alu
e
o
f
lo
s
s
in
d
icate
s
a
h
ig
h
er
d
eg
r
ee
o
f
co
n
f
id
en
ce
a
n
d
ac
cu
r
ac
y
in
p
r
ed
ictio
n
s
.
4
.
3
.
3
.
T
ra
ini
ng
t
im
e
T
h
e
tr
ain
in
g
tim
e
r
ef
er
s
to
th
e
to
tal
tim
e
tak
en
to
co
m
p
lete
th
e
tr
ain
in
g
p
r
o
ce
s
s
o
v
er
s
o
m
e
s
p
ec
if
ied
n
u
m
b
er
o
f
ep
o
c
h
s
.
T
h
is
m
ea
s
u
r
e
aid
s
in
r
ep
r
esen
tin
g
b
o
th
th
e
co
m
p
u
tatio
n
al
c
o
s
t
an
d
t
h
e
p
er
f
o
r
m
a
n
ce
o
f
d
if
f
er
en
t a
ctiv
atio
n
f
u
n
ctio
n
s
i
n
ter
m
s
o
f
s
ec
o
n
d
s
p
er
m
o
d
el
wh
ile
k
ee
p
in
g
h
ar
d
war
e
s
ettin
g
s
co
n
s
tan
t.
Evaluation Warning : The document was created with Spire.PDF for Python.
I
n
t J E
lec
&
C
o
m
p
E
n
g
I
SS
N:
2088
-
8
7
0
8
Op
timiz
in
g
n
eu
r
a
l n
etw
o
r
ks:
a
co
mp
a
r
a
tive
s
tu
d
y
o
f
a
ctiva
tio
n
fu
n
ctio
n
s
in
…
(
A
b
d
u
lla
h
S
h
eikh
)
953
4
.
3
.
4
.
G
ra
dient
s
t
a
bil
it
y
Gr
ad
ien
t
s
tab
ilit
y
r
ef
e
r
s
to
th
e
u
n
if
o
r
m
ity
a
n
d
p
er
s
is
ten
ce
o
f
g
r
ad
ien
ts
p
r
o
p
a
g
ated
in
th
e
co
u
r
s
e
o
f
tr
ain
in
g
in
th
e
b
ac
k
war
d
p
h
ase.
I
n
th
is
s
tu
d
y
,
g
r
ad
ien
t stab
ilit
y
is
n
o
t q
u
an
titativ
ely
d
eter
m
in
ed
u
s
in
g
g
r
ad
ie
n
t
n
o
r
m
s
b
u
t
i
n
f
er
r
e
d
in
d
u
ctiv
e
ly
f
r
o
m
p
atter
n
s
in
tr
ain
i
n
g
lo
s
s
cu
r
v
es
as
well
as
v
ali
d
atio
n
lo
s
s
cu
r
v
es.
Activ
atio
n
f
u
n
ctio
n
s
th
at
s
u
p
p
o
r
t
s
tab
le
g
r
ad
ien
ts
co
r
r
elate
with
h
ig
h
co
n
v
er
g
e
n
ce
r
ates,
th
er
eb
y
p
r
e
v
en
tin
g
g
r
ad
ien
t
v
an
is
h
in
g
o
r
b
lo
wu
p
.
T
h
ese
ass
ess
m
en
t
m
etr
ics
allo
w
f
o
r
c
r
o
s
s
-
co
m
p
a
r
is
o
n
b
etwe
en
ac
tiv
atio
n
f
u
n
ctio
n
s
ac
r
o
s
s
d
is
p
ar
ate
n
e
u
r
al
n
etwo
r
k
ar
c
h
itectu
r
es
an
d
d
ata
s
ets.
T
h
o
u
g
h
ac
cu
r
ac
y
an
d
lo
s
s
ex
is
t
as
m
ea
s
u
r
es
f
o
r
p
r
ed
ictiv
e
ac
c
u
r
ac
y
,
tr
ain
in
g
d
u
r
atio
n
e
x
is
ts
as
an
in
d
icato
r
f
o
r
co
m
p
u
t
atio
n
al
co
s
t,
with
g
r
ad
ien
t
s
tab
ilit
y
e
x
is
tin
g
as
an
i
n
d
icato
r
f
o
r
o
p
tim
izatio
n
r
o
b
u
s
tn
ess
.
T
o
g
et
h
er
,
th
ese
m
etr
ics
u
n
d
er
lie
d
r
awin
g
e
m
p
ir
ical
in
f
er
en
ce
s
as
well
as
ac
tiv
atio
n
f
u
n
ctio
n
s
elec
tio
n
ac
r
o
s
s
d
is
p
ar
ate
DL
m
eth
o
d
o
lo
g
ies.
I
n
co
m
b
in
atio
n
,
th
ese
m
et
r
ics
allo
w
f
o
r
in
d
ir
ec
t
co
m
p
ar
is
o
n
b
etwe
en
ea
ch
ac
tiv
atio
n
f
u
n
ct
io
n
with
r
esp
ec
t
t
o
p
r
ed
ictiv
e
ac
cu
r
ac
y
,
tr
ai
n
in
g
c
o
s
ts
,
an
d
o
p
tim
izatio
n
d
y
n
a
m
ics.
4
.
3
.
5
.
Rela
t
iv
e
r
o
o
t
m
ea
ns
s
qu
a
re
er
ro
r
R
elativ
e
r
o
o
t
m
ea
n
s
q
u
ar
e
er
r
o
r
(
R
R
MSE
)
r
ep
r
esen
ts
th
e
n
o
r
m
alize
d
er
r
o
r
m
etr
ic
th
at
m
ea
s
u
r
es
th
e
d
if
f
er
en
ce
s
b
etwix
t
th
e
p
r
ed
ict
ed
an
d
g
r
o
u
n
d
-
tr
u
th
v
al
u
es.
Her
e,
th
e
R
R
MSE
v
alu
es
r
ef
lect
th
e
co
n
s
is
ten
t
an
d
s
tab
le
m
o
d
el
p
r
ed
ictio
n
.
T
h
u
s
,
th
is
m
etr
ic
p
lay
s
a
v
ital
r
o
le
in
id
en
tify
in
g
th
e
ab
ilit
y
o
f
th
e
ac
tiv
atio
n
f
u
n
ctio
n
s
to
m
ain
tain
s
tab
le
er
r
o
r
b
e
h
av
io
r
ac
r
o
s
s
d
if
f
er
e
n
t e
p
o
ch
s
.
R
R
M
SE
=
√
(
1
/
)
∑
(
−
̂
)
2
=
1
/
(
1
6
)
4
.
4
.
M
et
ho
do
lo
g
ic
a
l
pip
eline
T
h
e
m
eth
o
d
o
lo
g
ical
p
ip
elin
e
o
f
th
e
p
r
o
p
o
s
ed
s
y
s
tem
is
illu
s
tr
ated
in
Fig
u
r
e
3
.
T
h
e
s
tep
-
by
-
s
tep
p
r
o
ce
s
s
f
o
llo
wed
in
t
h
is
s
tu
d
y
s
tar
ts
with
th
e
s
elec
tio
n
o
f
d
a
tasets
(
MN
I
ST,
C
I
F
AR
-
1
0
,
an
d
C
I
FAR
-
1
0
0
)
an
d
m
o
v
es
to
th
e
im
p
lem
en
tatio
n
o
f
m
o
d
el
ar
ch
itectu
r
es
(
ML
P
an
d
C
NN)
,
f
o
llo
wed
b
y
th
e
in
tr
o
d
u
cin
g
ac
tiv
atio
n
f
u
n
ctio
n
s
,
d
eter
m
in
in
g
ex
p
er
i
m
en
tal
p
ar
am
eter
s
(
i
n
clu
d
in
g
weig
h
t
in
itializatio
n
,
b
atc
h
s
ize,
an
d
o
p
tim
izer
)
,
an
d
u
ltima
tely
ar
r
iv
in
g
at
a
f
in
al
ass
ess
m
en
t
ce
n
ter
ed
o
n
m
etr
ics
lik
e
ac
cu
r
ac
y
,
l
o
s
s
,
tr
ain
in
g
tim
e,
a
n
d
g
r
ad
ien
t
co
n
v
er
g
e
n
ce
.
Her
e,
t
h
e
ep
o
ch
s
lik
e
1
0
,
3
0
,
an
d
5
0
ep
o
ch
s
ar
e
ch
o
s
en
f
o
r
f
u
r
th
er
p
r
o
ce
s
s
in
g
.
T
h
e
ML
P
co
n
s
is
t
s
o
f
an
in
p
u
t
lay
er
,
a
h
id
d
en
lay
e
r
,
an
d
a
n
o
u
tp
u
t
lay
er
,
wh
e
r
ea
s
th
e
C
NN
in
clu
d
es
a
co
n
v
o
l
u
tio
n
al
lay
er
,
a
p
o
o
lin
g
lay
er
,
a
f
latten
in
g
lay
er
,
an
d
a
n
o
u
tp
u
t
lay
er
.
Nin
e
ac
tiv
atio
n
f
u
n
ctio
n
s
,
n
am
ely
R
eL
U,
T
an
h
,
Sig
m
o
id
,
E
L
U,
SEL
U,
Swi
s
h
,
Mish
,
GE
L
U,
a
n
d
SMU,
ar
e
in
d
iv
id
u
ally
ap
p
l
ied
to
th
e
ML
P a
n
d
C
NN
m
o
d
els.
T
h
is
s
y
s
tem
atic
f
r
am
ewo
r
k
p
r
o
v
id
es
a
f
air
an
d
co
n
s
is
ten
t
ev
alu
atio
n
o
f
all
ac
tiv
atio
n
f
u
n
ctio
n
s
ac
r
o
s
s
b
o
th
d
atasets
an
d
ar
c
h
itectu
r
al
co
n
f
i
g
u
r
atio
n
s
,
t
h
er
eb
y
en
ab
lin
g
p
r
ec
is
e
co
m
p
a
r
is
o
n
.
Fig
u
r
e
3
.
Me
th
o
d
o
lo
g
ical
p
ip
e
lin
e
f
o
r
e
v
alu
atin
g
ac
tiv
atio
n
f
u
n
ctio
n
s
ac
r
o
s
s
n
eu
r
al
n
etwo
r
k
ar
ch
itectu
r
es
an
d
d
atasets
Fo
llo
win
g
th
e
m
eth
o
d
o
lo
g
ic
al
p
ip
elin
e
illu
s
tr
ated
in
Fig
u
r
e
3
,
a
n
o
r
m
aliza
tio
n
p
r
o
c
ed
u
r
e
was
ap
p
lied
p
r
io
r
to
r
ad
a
r
p
lo
t
v
is
u
aliza
tio
n
to
en
ab
le
f
air
c
o
m
p
ar
is
o
n
ac
r
o
s
s
h
eter
o
g
en
e
o
u
s
p
er
f
o
r
m
a
n
ce
m
etr
ics.
Sin
ce
th
e
co
n
s
id
er
ed
m
et
r
ics
d
if
f
er
in
s
ca
le
an
d
o
p
tim
izati
o
n
d
ir
ec
tio
n
,
a
m
in
–
m
ax
n
o
r
m
aliza
tio
n
s
tr
ateg
y
was e
m
p
lo
y
ed
.
T
h
e
n
o
r
m
aliza
tio
n
s
tep
s
u
s
ed
in
th
is
ev
alu
ati
o
n
p
ip
elin
e
ar
e
s
u
m
m
ar
ize
d
in
T
ab
le
4
.
T
ab
le
4
.
R
ad
ar
p
lo
t n
o
r
m
aliza
tio
n
s
u
m
m
ar
y
S
t
e
p
D
e
scri
p
t
i
o
n
F
o
r
mu
l
a
1
C
o
l
l
e
c
t
r
a
w
m
e
t
r
i
c
v
a
l
u
e
s
-
2
I
d
e
n
t
i
f
y
d
i
r
e
c
t
i
o
n
A
c
c
u
r
a
c
y
(
h
i
g
h
e
r
)
,
S
t
a
b
i
l
i
t
y
(
h
i
g
h
e
r
)
,
Lo
ss (
h
i
g
h
e
r
)
,
Ti
m
e
(
h
i
g
h
e
r
)
3
I
n
v
e
r
t
l
o
w
e
r
v
a
l
u
e
s
(
i
n
v
e
r
t
)
=
1
−
4
M
i
n
-
ma
x
n
o
r
mal
i
z
e
p
e
r
m
e
t
r
i
c
N
o
r
mal
i
z
a
t
i
o
n
=
−
(
mi
n
i
mu
m
)
(
max
i
mu
m
)
−
(
mi
n
i
mu
m
)
5
F
i
n
a
l
r
a
d
a
r
sc
o
r
e
r
a
n
g
e
0
=
w
o
r
s
t
,
1
=
b
e
s
t
Evaluation Warning : The document was created with Spire.PDF for Python.
I
SS
N
:
2
0
8
8
-
8
7
0
8
I
n
t J E
lec
&
C
o
m
p
E
n
g
,
Vo
l.
1
6
,
No
.
2
,
Ap
r
il
20
2
6
:
9
4
5
-
963
954
5.
P
E
RF
O
RM
A
NCE CO
M
P
A
RIS
O
N
T
h
is
s
ec
tio
n
co
n
d
u
cts
a
co
m
p
ar
ativ
e
s
tu
d
y
o
f
n
in
e
ac
ti
v
atio
n
f
u
n
ctio
n
s
u
s
in
g
ML
P
an
d
C
NN
ar
ch
itectu
r
es
o
n
th
e
C
I
FAR
-
1
0
,
MN
I
ST,
an
d
C
I
FAR
-
1
0
0
d
atasets
with
r
esp
ec
t
to
ac
c
u
r
ac
y
,
lo
s
s
,
tr
ain
in
g
tim
e,
an
d
g
r
a
d
ien
t
s
tab
ilit
y
.
T
h
e
v
alid
atio
n
ac
cu
r
ac
y
r
esu
lts
in
d
icate
d
th
at
ac
r
o
s
s
b
o
th
ar
ch
itectu
r
es
a
n
d
d
atasets
,
GE
L
U
an
d
Mish
p
r
o
v
id
ed
th
e
g
r
ea
test
ac
cu
r
ac
y
.
GE
L
U
ac
h
iev
ed
a
m
ax
im
u
m
ac
cu
r
ac
y
o
f
9
8
.
0
3
%
f
o
r
t
h
e
ML
P
m
o
d
el
o
n
MN
I
ST,
clo
s
ely
f
o
llo
wed
b
y
Swis
h
an
d
Mish
.
I
n
C
NNs
tr
ain
ed
o
n
C
I
FAR
-
1
0
an
d
C
I
FAR
-
1
0
0
,
GE
L
U,
Mish
,
a
n
d
Swis
h
r
eg
u
lar
ly
s
u
r
p
ass
ed
co
n
v
en
tio
n
al
ac
tiv
atio
n
f
u
n
c
tio
n
s
lik
e
T
an
h
an
d
Sig
m
o
id
.
As
d
ep
icted
in
Fig
u
r
e
4
,
r
a
d
ar
p
lo
ts
wer
e
u
tili
ze
d
t
o
g
iv
e
a
n
o
v
e
r
all,
s
id
e
-
by
-
s
id
e
c
o
m
p
ar
i
s
o
n
o
f
th
e
tr
ad
e
-
o
f
f
s
o
v
e
r
th
e
th
r
ee
b
asic
m
ea
s
u
r
es
o
f
ac
c
u
r
ac
y
,
lo
s
s
,
a
n
d
tr
ai
n
in
g
tim
e
f
o
r
ea
ch
ac
tiv
atio
n
f
u
n
ctio
n
.
T
o
en
s
u
r
e
th
e
v
is
u
al
co
m
p
ar
is
o
n
with
d
if
f
er
en
t
s
ca
les,
all
r
ad
ar
v
is
u
aliza
tio
n
s
wer
e
g
en
er
ated
u
s
in
g
m
in
-
m
ax
n
o
r
m
aliza
tio
n
.
T
h
is
n
o
r
m
aliz
atio
n
en
s
u
r
ed
t
h
at
all
th
e
ax
es
in
th
e
r
ad
ar
p
l
o
ts
r
ef
lecte
d
th
e
co
m
p
ar
ativ
e
p
er
f
o
r
m
an
ce
,
th
e
r
eb
y
p
r
e
v
en
t
in
g
b
iased
in
ter
p
r
etatio
n
.
Ne
v
er
th
eless
,
th
e
p
h
o
t
o
s
in
d
icat
ed
th
at
GE
L
U
an
d
Mish
h
ad
th
e
m
o
s
t
well
-
r
o
u
n
d
ed
p
r
o
f
iles
b
y
c
o
u
p
lin
g
b
etter
ac
cu
r
ac
y
,
less
lo
s
s
,
an
d
r
ea
s
o
n
ab
le
tr
ain
in
g
tim
e.
Swis
h
also
d
em
o
n
s
tr
ated
s
tr
o
n
g
p
e
r
f
o
r
m
an
ce
with
s
lig
h
tly
b
etter
co
m
p
u
tatio
n
al
ef
f
icien
c
y
,
wh
er
ea
s
Sig
m
o
i
d
d
is
p
lay
ed
s
ig
n
if
ican
tly
lim
ite
d
u
tili
ty
ac
r
o
s
s
all
m
etr
ics.
I
n
ter
m
s
o
f
v
alid
atio
n
l
o
s
s
,
E
L
U
an
d
Mish
d
ep
icted
th
e
lo
west
v
alu
es,
r
ef
lectin
g
ef
f
icien
t
co
n
v
er
g
en
ce
an
d
b
ette
r
g
en
er
aliza
tio
n
.
GE
L
U
an
d
S
wis
h
also
d
is
p
lay
ed
co
n
s
is
ten
tly
d
ec
lin
in
g
an
d
le
s
s
v
o
latile
lo
s
s
g
r
ap
h
s
,
wh
e
r
ea
s
Sig
m
o
id
an
d
T
an
h
wer
e
m
ar
k
e
d
b
y
g
r
ea
ter
f
lu
ctu
atio
n
s
,
p
a
r
ticu
lar
ly
in
C
NN
m
o
d
els.
Alth
o
u
g
h
R
eL
U
o
f
f
er
e
d
th
e
f
astes
t
tr
ain
in
g
tim
es
d
u
e
t
o
its
co
m
p
u
tatio
n
al
s
im
p
licity
,
GE
L
U
an
d
Mish
in
cu
r
r
ed
l
o
n
g
er
d
u
r
atio
n
s
,
w
h
ich
wer
e
ju
s
tifie
d
b
y
th
eir
en
h
a
n
ce
d
ac
cu
r
ac
y
an
d
g
r
ad
ien
t stab
ilit
y
.
Fig
u
r
e
4
.
R
ad
ar
c
h
ar
ts
p
er
ac
ti
v
atio
n
f
u
n
ctio
n
Gr
ad
ien
t
f
lo
w
s
tu
d
y
r
e
v
ea
led
th
at
Mish
,
GE
L
U,
an
d
Swis
h
allo
wed
m
o
r
e
c
o
n
s
is
ten
t
co
n
v
er
g
e
n
ce
,
th
u
s
r
ed
u
cin
g
o
v
e
r
f
itti
n
g
an
d
o
s
cillatio
n
s
.
C
o
n
tr
ar
ily
,
Sig
m
o
id
s
u
f
f
er
ed
f
r
o
m
th
e
v
an
is
h
in
g
g
r
ad
ie
n
t
is
s
u
e
o
win
g
to
th
e
r
ed
u
ce
d
g
r
a
d
ie
n
t
m
ag
n
itu
d
es.
An
ac
tiv
atio
n
f
u
n
ctio
n
'
s
g
en
er
al
ef
f
icac
y
was
also
r
elian
t
o
n
ar
ch
itectu
r
e.
R
eL
U
an
d
E
L
U
p
er
f
o
r
m
ed
b
etter
i
n
ML
Ps
,
wh
ile
GE
L
U
an
d
Mish
ex
ce
led
in
C
NNs.
Alth
o
u
g
h
GE
L
U
was
f
ir
s
t
in
g
en
er
al
p
er
f
o
r
m
a
n
ce
,
p
r
ac
titi
o
n
er
s
h
ad
to
tak
e
in
to
ac
co
u
n
t
m
o
d
el
d
ep
th
,
task
-
s
p
ec
if
i
c
lim
itatio
n
s
,
an
d
co
m
p
u
tin
g
c
o
s
t.
Evaluation Warning : The document was created with Spire.PDF for Python.