Au1CMA1lCn

nLLu Cl Au1CMA1lCn
W hyslcal Lask of performlng Lhe LesLs can Lake a
greaL deal of Llme and efforL
W ?ou could spend more Llme equlvalence
parLlLlonlng your LesL cases
W 8educlng Lhe number LhaL you run buL Lhen you
Lake on more rlsk because youre reduclng
coverage chooslng noL Lo LesL lmporLanL
feaLures
W 1he answer ls Lo do whaL people have done for
years ln every oLher fleld and lndusLry develop
and use Lools Lo make Lhe [ob easler and more
efflclenL
LnLll1S Cl Au1CMA1lCn Anu 1CCLS
W 8egresslon 1esLlng 1he codeLesLflx loop can repeaL
several Llmes before Lhe sofLware ls released Whlle
LesLlng a parLlcular feaLure we need Lo run LesLs noL once
buL repeaLed Llmes We check LhaL Lhe bugs found ln
prevlous LesL runs were lndeed flxed and LhaL no new bugs
were lnLroduced 1hls process of rerunnlng your LesLs ls
known as reqression testinq
W lf a small sofLware pro[ecL had several Lhousand LesL cases
Lo run Lhere mlghL be barely enough Llme Lo execuLe
Lhem [usL once 8unnlng Lhem numerous Llmes mlghL be
lmposslble
W SofLware LesL Lools and auLomaLlon can help solve Lhls
problem by provldlng a more efflclenL means Lo run your
LesLs Lhan by manual LesLlng
A118lu1LS Cl Au1CMA1lCn Anu 1CCLS
W Speed 1hlnk of Llme requlred Lo do few
Lhousand LesL cases for Lhe Wlndows CalculaLor
Average may be LesL case every flve seconds or
so AuLomaLlon mlghL be able Lo run 10 100
even 1000 Llmes LhaL fasL
W Lfflclency Whlle busy runnlng LesL cases we
canL be dolng anyLhlng else lf we have a LesL
Lool LhaL reduces Lhe Llme lL Lakes for you Lo run
your LesLs we can have more Llme for LesL
plannlng and Lhlnklng up new LesLs
W Accuracy and reclslon AfLer Lrylng a few
hundred cases Cur aLLenLlon span wlll wane and
well sLarL Lo make mlsLakes A LesL Lool wlll
perform Lhe same LesL and check Lhe resulLs
perfecLly each and every Llme
W 8esource 8educLlon SomeLlmes number of
people or Lhe amounL of equlpmenL requlred Lo
creaLe Lhe LesL condlLlon could be prohlblLlve A
LesL Lool can be used Lo slmulaLe Lhe real world
and greaLly reduce Lhe physlcal resources
necessary Lo perform Lhe LesLlng
W SlmulaLlon and LmulaLlon 1esL Lools are ofLen
used Lo replace hardware or sofLware LhaL would
normally lnLerface Lo your producL 1hls fake
devlce or appllcaLlon can Lhen be used Lo drlve or
respond Lo your sofLware ln ways LhaL you choose
and ways LhaL mlghL oLherwlse be dlfflculL Lo
achleve
W 8elenLlessness 1esL Lools and auLomaLlon never
Llre or glve up 1heyre llke baLLeryoperaLed bunny
of Lhe 1v commerclals Lhey can keep golng and
golng
W SofLware LesL Lools arenL a subsLlLuLe for sofLware
LesLers Lhey [usL help sofLware LesLers perform Lhelr
[obs beLLer
1LS1 1CCLS
W 1here ls a wlde range of sofLware LesLlng Lools
W SelecLlon of LesL we use ls based on Lhe Lype of
sofLware LhaL we are LesLlng and Lype of LesL llke
blackbox or whlLebox LesLs
VILWLkS AND MCNI1CkS 1CCLS
W Allows you Lo see deLalls of Lhe sofLwares operaLlon
LhaL you wouldnL normally be able Lo see
W Code coverage analyzers provlde a means for you Lo
see whaL llnes of code are execuLed whaL funcLlons
are run and whaL code paLhs are followed when you
run your LesLs
W vlewer a communlcaLlons analyzer (or comm
analyzer for shorL) 1hls Lool allows you Lo see Lhe
raw proLocol daLa movlng across a neLwork or oLher
communlcaLlons cable
W lL slmply Laps lnLo Lhe llne pulls off Lhe daLa as lL passes by
and dlsplays lL on anoLher compuLer
W We enLer a LesL case on CompuLer #1 conflrm LhaL Lhe
resulLlng communlcaLlons daLa ls correcL on CompuLer #3
and check LhaL Lhe approprlaLe resulLs occurred on CompuLer
#2
W We also use Lhls sysLem Lo lnvesLlgaLe why a bug occurs
W y looklng aL Lhe daLa pulled off Lhe wlre we can deLermlne
lf Lhe problem lles ln creaLlng Lhe daLa (CompuLer #1) or
lnLerpreLlng Lhe daLa (CompuLer #2)
W 1hls Lype of sysLem ls nondangerous Lo Lhe sofLware
W ln neLworklng such a monlLor ls known as a sniffer because
llke an elecLronlc nose lL snlffs Lhe daLa from Lhe alr
W Code debuggers LhaL come wlLh compllers are also vlewers
because Lhey allow programmers or whlLebox LesLers Lo
vlew lnLernal varlable values and program sLaLes
W AnyLhlng LhaL leLs you see lnLo Lhe sysLem and look aL daLa
LhaL Lhe average user wouldnL be able Lo see can be
classlfled as a vlewer LesL Lool
DkIVLkS
W urlvers are Lools used Lo conLrol and operaLe Lhe sofLware
belng LesLed
W Cne of Lhe slmplesL examples of a drlver ls a baLch flle a slmple
llsL of programs or commands LhaL are execuLed sequenLlally
W 1he Wlndows 1ask Scheduler can execuLe varlous LesL programs
aL cerLaln Llmes LhroughouL Lhe day
W Suppose LhaL Lhe sofLware were LesLlng requlres large amounLs
of daLa Lo be enLered for we LesL cases
W WlLh some hardware modlflcaLlons and a few sofLware Lools
we can replace Lhe keyboard and mouse of Lhe sysLem belng
LesLed wlLh an addlLlonal compuLer LhaL acLs as a drlver
W We can wrlLe slmple programs on Lhls drlver compuLer LhaL
auLomaLlcally generaLe Lhe approprlaLe keysLrokes and mouse
movemenLs Lo LesL Lhe sofLware
WP? SuCP SL1u lS 8LCul8Lu
W lLs posslble LhaL Lhe sofLware or operaLlng
sysLem lsnL mulLlLasklng maklng lL lmposslble Lo
run anoLher drlver program concurrenLly
W y sendlng keysLrokes and mouse movemenLs
from an exLernal compuLer Lhe LesL sysLem ls
nonlnvaslve
W lf a drlver program ls runnlng on Lhe same
sysLem as Lhe sofLware belng LesLed lLs lnvaslve
and may noL be consldered an accepLable LesL
scenarlo
S1uS
W SLubs llke drlvers are whlLebox LesLlng
Lechnlques
W SLubs are essenLlally Lhe opposlLe of drlvers ln
LhaL Lhey donL conLrol or operaLe Lhe
sofLware belng LesLed
W 1hey lnsLead recelve or respond Lo daLa LhaL
Lhe sofLware sends
W lor LesLlng sofLware LhaL sends daLa Lo a prlnLer one
way Lo LesL lL ls Lo enLer daLa prlnL lL and look aL Lhe
resulLlng paper prlnLouL
W lL would work buL lLs falrly slow lnefflclenL and
error prone Can you Lell lf Lhe ouLpuL had a slngle
mlsslng plxel or lf lL was [usL sllghLly off ln color?
W lnsLead we replaced Lhe prlnLer wlLh anoLher
compuLer LhaL was runnlng sLub sofLware LhaL could
read and lnLerpreL Lhe prlnLer daLa lL could much
more qulckly and accuraLely check Lhe LesL resulLs
W SLubs are frequenLly used when sofLware needs Lo
communlcaLe wlLh exLernal devlces CfLen durlng
developmenL Lhese devlces arenL avallable or are
scarce
W A sLub allows LesLlng Lo occur desplLe noL havlng Lhe
hardware and lL makes LesLlng more efflclenL
W 1he Lerm emulaLor used Lo descrlbe a devlce
LhaLs a plugln replacemenL for Lhe real
devlce
W 1he dlfference beLween an emulaLor and a
sLub ls LhaL a sLub also provldes Lhe means for
a LesLer Lo vlew and lnLerpreL Lhe daLa senL Lo
lL A sLub ls a superseL of an emulaLor
S1kLSS AND LCAD 1CCLS
W SLress and load Lools lnduce sLresses and loads Lo Lhe sofLware belng
LesLed
W A word processor runnlng as Lhe only appllcaLlon on Lhe sysLem
wlLh all avallable memory and dlsk space probably works [usL flne
W uL lf Lhe sysLem runs low on Lhese resources youd expecL a
greaLer poLenLlal for bugs
W ?ou could copy flles Lo flll up Lhe dlsk run loLs of programs Lo eaL up
memory and so on buL Lhese meLhods are lnefflclenL and non
exacL
W A sLress Lool speclflcally deslgned for Lhls would make LesLlng much
easler
W 1he MlcrosofL SLress uLlllLy LhaL comes wlLh lLs programmlng
language developmenL sofLware
W 1he SLress program allows you Lo lndlvldually seL Lhe amounLs of
memory dlsk space flles and oLher resources avallable Lo Lhe
sofLware runnlng on Lhe machlne
W Load Lools are slmllar Lo sLress Lools ln LhaL
Lhey creaLe slLuaLlons for your sofLware LhaL
mlghL oLherwlse be dlfflculL Lo creaLe
W Commerclally avallable programs can be run
on web servers Lo load Lhem down by
slmulaLlng a seL number of connecLlons and
hlLs
W We mlghL wanL Lo check LhaL 10000
slmulLaneous users and 1 mllllon hlLs a day
can be handled wlLhouL slowlng response
Llmes
W WlLh a load Lool you can slmply dlal ln LhaL
level run your LesLs and see whaL happens
IN1LkILkLNCL INILC1CkS AND NCISL GLNLkA1CkS
W Are slmllar Lo sLress and load Lools buL are more
random ln whaL Lhey do
W 1he SLress uLlllLy has an execuLor mode LhaL
randomly changes Lhe avallable resources
W A program mlghL run flne wlLh loLs of memory
and mlghL handle low memory slLuaLlons buL lL
could have problems lf Lhe amounL of avallable
memory ls consLanLly changlng
W 1he execuLor mode of Lhe SLress uLlllLy would
uncover Lhese Lypes of bugs
W ln Lhls scenarlo Lhe vlewer ls replaced wlLh
hardware and sofLware LhaL allows noL only
vlewlng Lhe daLa on Lhe communlcaLlons llne
buL also modlfylng lL
W lL becomes an lnLerference ln[ecLor Such a seLup
could slmulaLe all Lypes of communlcaLlons
errors caused by daLa dropouLs nolsy or bad
cables and so on
ANALSIS 1CCLS
W We call Lhls lasL caLegory of Lools analysls Lools a besLof
LheresL group
Word processlng sofLware
SpreadsheeL sofLware
uaLabase sofLware
llle comparlson sofLware
Screen capLure and comparlson sofLware
uebugger
lnaryhex calculaLor
SLopwaLch
vC8 or camera
W SofLware complexlLy and dlrecLlon change all Lhe Llme ?ou
need Lo look aL your lndlvldual slLuaLlon Lo declde whaL
Lhe mosL effecLlve Lools would be and how besL Lo apply
Lhem
SCl1WA8L 1LS1 Au1CMA1lCn
W 1he sofLware LesL Lools LhaL we learned abouL so
far are lndeed effecLlve buL Lhey sLlll musL be
operaLed or monlLored manually
W SofLware LesL auLomaLlon
1ools could be comblned sLarLed and run wlLh llLLle
or no lnLervenLlon from LesLer
1hey could run your LesL cases look for bugs analyze
whaL Lhey see and log Lhe resulLs
MACkC kLCCkDING AND LA8ACk
W 1he mosL baslc Lype of LesL auLomaLlon ls recordlng your
keyboard and mouse acLlons as you run your LesLs for Lhe
flrsL Llme and Lhen playlng Lhem back when you need Lo
run Lhem agaln
W Cn Lhe Mac you can use Culckeys on Wlndows Lhe
shareware program Macro Maglc ls a good cholce
W Macro recorders and players are a Lype of drlver Lool
W urlvers are Lools used Lo conLrol and operaLe Lhe sofLware
belng LesLed WlLh a macro program youre dolng [usL LhaL
Lhe macros you record are played back repeaLlng Lhe
acLlons LhaL you performed Lo LesL Lhe sofLware
W Macro Maglc SeLup Wlzard whlch walks you Lhrough Lhe
sLeps necessary Lo conflgure and capLure your macros
W 1he Macro Maglc SeLup Wlzard allows you Lo seL
Lhe followlng opLlons for macros
name Clvlng Lhe macro a name provldes a way Lo
ldenLlfy lL laLer Lven for a small sofLware pro[ecL you
could wrlLe hundreds of macros
8epeLlLlons 8epeLlLlon LesLlng ls a greaL way Lo flnd
bugs ?ou can seL Lhe number of Llmes Lhe macro wlll
repeaL or loop when lL runs
1rlggers ?ou can seL how Lhe macro ls sLarLed
W 1hls can be by a hoL key (for example CLrl+ShlfL+1)
W by a seL of Lypedln characLers (maybe run macro 1)
W by cllcklng a shorLcuL when a cerLaln wlndow ls dlsplayed
(whenever CalculaLor ls sLarLed for example) or
W when Lhe sysLem has ldled unused for a cerLaln lengLh of
Llme
WhaLs capLured ?ou can selecL Lo capLure (record) [usL
keysLrokes or boLh keysLrokes and mouse acLlons such as
movlng and cllcklng
layback speed 1he macro can play back from up Lo 20
slower Lo 300 fasLer Lhan how you orlglnally recorded lL
1hls ls lmporLanL lf your sofLwares performance can vary
WhaL would happen lf Lhe sofLware youre LesLlng became
a blL slower and Lhe buLLon Lhe macro was Lo cllck on
wasnL yeL onscreen?
layback poslLlon 1hls opLlon deLermlnes lf Lhe mouse
movemenLs and cllcks should be absoluLe or relaLlve Lo a
cerLaln wlndow onscreen lf youre LesLlng an appllcaLlon
LhaL mlghL change onscreen poslLlons maklng your
movemenLs relaLlve Lo LhaL appllcaLlon ls a good ldea
oLherwlse Lhe mouse may noL cllck where you would
expecL
u8AWACkS Cl uSlnC MAC8C
W We flnd ls LhaL alLhough macros can do some auLomaLed LesLlng for you
maklng lL much easler and fasLer Lo rerun your LesLs Lheyre noL perfecL
W 1he blggesL problem ls lack of verlflcaLlon
W 1he macros canL check LhaL Lhe sofLware does whaL lLs supposed Lo do
W 1he macro could Lype 10099 lnLo Lhe CalculaLor buL lL canL LesL LhaL Lhe
resulL ls 1you sLlll need Lo do LhaL
W 1hls ls an lssuebuL many LesLers are happy [usL ellmlnaLlng all Lhe repeLlLlve
Lyplng and mouse movlng
W lLs a much easler [ob Lo slmply waLch Lhe macros run and conflrm LhaL Lhe
resulLs are whaLs expecLed
W layback speed can be anoLher dlfflculLy wlLh macros
W We can ad[usL Lhe speed of playback lL may noL always be enough Lo keep
Lhe macros ln sync
W A web page may Lake 1 second or 10 seconds Lo load We can slow down
our macros Lo accounL for Lhe expecLed worsL case buL Lhen Lheyd run
slowly even lf Lhe sofLware was runnlng fasL And lf Lhe web page
unexpecLedly Look 13 seconds Lo load your macros would sLlll geL confused
cllcklng Lhe wrong Lhlngs aL Lhe wrong Llme
8CC8AMMLu MAC8CS
W rogrammed macros are a sLep up ln evoluLlon
from Lhe slmple record and playback varleLy
W We can creaLe programmed macros by recordlng
your acLlons as you run Lhe LesL for Lhe flrsL Llme
creaLe Lhem by programmlng slmple lnsLrucLlons
for Lhe playback sysLem Lo follow
W 1hls Lype of macro can be programmed by
selecLlng lndlvldual acLlons from a menu of
cholces you donL even need Lo Lype ln Lhe
commands
CalculaLor 1esL #2
2 LxLCu1LCWlnuCWSS?S1LM32Calcexe
3 LCCklC8CalculaLorSLCS3
4 123100
3 8CM11he answer should be 23
6 CLCSLCalculaLor
8AnuCM 1LS1lnC
MCnkL?S Anu CC8lLLAS
W SofLware LesLlng auLomaLlon Lools and Lechnlques
makes [ob of sofLware LesLer more easler and
efflclenL
W Pelps ln runnlng LesL cases or ldeally runnlng LesL
cases auLomaLlcally wlLhouL Lhe need for consLanL
aLLenLlon
W uslng Lools and auLomaLlon wlll help us Lo flnd
bugs whlle Lhe Lools are busy dolng regresslon
LesLlng we have Llme Lo plan new LesLs and
deslgn new and lnLeresLlng cases Lo Lry
1LS1 MCnkL?S
W noL deslgned Lo help run or auLomaLlcally run LesL cases
W Pave goal Lo slmulaLe whaL your users mlghL do 1hls Lype
of auLomaLlon Lool ls called a LesL monkey
W When sofLware ls released Lo Lhe publlc lL wlll have
Lhousands or posslbly mllllons of people uslng lL
W uesplLe besL efforLs aL deslgnlng LesL cases Lo flnd bugs
some bugs wlll sllp by and be found by Lhose users
W WhaL lf we can supplemenL our LesL case approach wlLh a
slmulaLlon of whaL all Lhose users would do before we
release our producL?
W We can poLenLlally flnd bugs LhaL would have oLherwlse
made lL pasL your LesLlng 1haLs whaL a LesL monkey can
do
uuM MCnkL?S
W 1he easlesL and mosL sLralghLforward Lype of LesL monkey
W A dumb monkey doesnL know anyLhlng abouL Lhe sofLware belng LesLed lL [usL
cllcks or Lypes randomly
W 1 8AnuCMlZL 1lML8
2 lC8 l1 1C 10000
3 LA? CLlCk +S18$(ln1(8nu*640))+ +S18$(ln1(8nu*480))+
4 LA? CP8$(8nu*236)
3 nLx1 l
W 1hls monkey ls dolng absoluLely no verlflcaLlon
W lL [usL cllcks and Lypes unLll one of Lwo Lhlngs happens elLher lL flnlshes lLs loop
or Lhe sofLware or Lhe operaLlng sysLem crashes
W lf Lhe sofLware under LesL crashes Lhe monkey wonL even know lL and wlll
conLlnue cllcklng and Lyplng away
Clven enough Llme and aLLempLs Lhe random lnpuLs wlll evenLually sLumble onLo a
maglc sequence LhaL Lhe programmers and LesLers dldnL Lhlnk of Maybe Lhe
monkey enLers some daLa and lmmedlaLely deleLes lL or Lypes ln a huge sLrlng where
a shorL one was expecLed Who knows? lL wlll flnd lL Lhough
A dumb monkey wlLh lLs conLlnuous repeLlLlon and use can expose bugs such as
memory leaks LhaL mlghL noL occur unLll many hours or days of normal use lf youve
ever used sofLware LhaL seemed Lo become less and less sLable Lhe longer you used
lL youve seen a problem LhaL could have been found wlLh a dumb monkey
SLMl SMA81 MCnkL?S
W uumb monkeys can be exLremely effecLlve
W 1heyre easy Lo wrlLe and can flnd serlous crashlng bugs 1hey
lack a few lmporLanL feaLures Lhough LhaL would make Lhem
even more effecLlve Addlng Lhese feaLures ralses your
monkeys lC a blL maklng hlm semlsmarL
W Say LhaL monkey ran for several hours logglng Lhousands of
random lnpuLs before Lhe sofLware crashed ?oud know Lhere
was a problem buL you couldnL show Lhe programmer exacLly
how Lo recreaLe lL
W ?ou could rerun your monkey wlLh Lhe same random seed buL
lf lL Look several hours agaln Lo fall youd be wasLlng a loL of
Llme
W 1he soluLlon ls Lo add logglng Lo your monkey so LhaL
everyLhlng lL does ls recorded Lo a flle When Lhe monkey flnds
a bug you need only Lo look aL Lhe log flle Lo see whaL lL was
dolng before Lhe fallure
W lLs also a good ldea Lo program your monkey Lo operaLe only
on Lhe sofLware youre LesLlng
W lf lLs randomly cllcklng all over Lhe screen lL could (and wlll
evenLually) cllck Lhe exlL command and sLop Lhe program
W Slnce Lhe monkey doesnL know LhaL Lhe program closed lLll
keep on golng
W WhaL would happen lf Lhe monkey was cllcklng all over your
compuLers screen Louch! MosL programmable auLomaLlon
Lools provlde a way Lo always LargeL a speclflc appllcaLlon or Lo
sLop worklng lf Lhe appllcaLlon ls no longer presenL
W AnoLher good feaLure Lo make your monkey semlsmarL ls crash
recognlLlon
W lf you sLarLed your monkey runnlng for Lhe nlghL and lL found a
bug as soon as you walked ouL Lhe door youd lose many hours
of valuable LesL Llme
W lf you can add programmlng Lo your monkey Lo recognlze LhaL
a crash has occurred resLarL Lhe compuLer and sLarL runnlng
agaln you could poLenLlally flnd several bugs each nlghL
SMA81 MCnkL?S
W 1akes Lhe effecLlveness of random LesLlng from hls less
lnLelllgenL broLhers and adds Lo LhaL an awareness of hls
surroundlngs Pe doesnL [usL pound on Lhe keyboard
randomly he pounds on lL wlLh a purpose
W A Lrue smarL monkey knows
Where he ls
WhaL he can do Lhere
Where he can go
Where hes been
lf whaL hes seelng ls correcL
W A smarL monkey lsnL llmlLed Lo [usL looklng for crashlng
bugs elLher lL can examlne daLa as lL goes checklng Lhe
resulLs of lLs acLlons and looklng for dlfferences from whaL
lL expecLs lf you programmed ln your LesL cases Lhe smarL
monkey could randomly execuLe Lhem look for bugs and
log Lhe resulLs
W SmarL monkey called koko named afLer Lhe gorllla LhaL
could speak ln slgn language 1o program koko you feed lL
Lhe sLaLe Lable LhaL descrlbes your sofLware by deflnlng
each sLaLe Lhe acLlons LhaL can be performed ln LhaL sLaLe
and clalms LhaL deLermlne wheLher Lhe resulL of execuLlng
an acLlon ls rlghL or wrong
W When koko runs lL drlves Lhe sofLware Lo a known sLaLe
randomly selecLs an acLlon based on a welghLlng LhaL
slmulaLes realworld llkellhood performs LhaL acLlon and
Lhen checks Lhe resulL lf Lhe acLlon resulLs ln Lhe sofLware
changlng sLaLes koko knows LhaL and uses a new seL of
acLlons LhaL apply Lo LhaL new sLaLe
W WlLh such a sysLem you could slmulaLe realworld usage
of your sofLware compresslng Lhousands of hours of use
lnLo [usL a few hours SmarL monkeys are Lruly bugflndlng
machlnes!
uC ASPLS Anu L1A 1LS1lnC
W AuLomaLed 1esLlng and 1esL 1ools can help ln
maklng our LesLlng more efflclenL
W uslng sofLware Lo LesL sofLware ls a greaL way Lo
speed up your work and Lo help flnd bugs LhaL
mlghL oLherwlse be mlssed
W AnoLher means Lo be a more effecLlve LesLer ls
Lo lnfluence oLher people
W lf we can geL more people even lf Lhey arenL
professlonal LesLers looklng aL sofLware before
lLs released Lhey may be able Lo flnd bugs LhaL
we and our fellow LesLers falled Lo see
W AfLer flnlsh looklng have several frlends do Lhe
same search and compare your llsLs
W WhaL we flnd ls LhaL everyone has very dlfferenL
resulLs
W 1he number of dlfferences found Lhe order LhaL
Lhey were found even whlch ones were found
wlll be dlfferenL
W llnally comblne all Lhe llsLs and Lhrow ouL Lhe
dupllcaLes one have a compleLe llsL of all Lhe
dlfferences buL even Lhen Lhere sLlll may be a
few LhaL were mlssed
W SofLware LesLlng works exacLly Lhe same way buL
Lhere are several reasons and soluLlons for lL Llke
Pavlng anoLher seL of eyes look aL Lhe sofLware helps
break Lhe pesLlclde paradox
eople donL [usL see dlfferenLly from each oLher
Lhey go abouL Lhelr LesLlng dlfferenLly Loo
Pavlng someone asslsL ln your LesLlng helps ellmlnaLe
boredom lL can geL preLLy monoLonous runnlng Lhe
same LesLs over and over uslng Lhe same sofLware
feaLures agaln and agaln 1he boredom wlll also
cause our aLLenLlon Lo wane and one mlghL sLarL
mlsslng obvlous bugs
WaLchlng how someone else approaches a problem ls
a greaL way Lo learn new LesLlng Lechnlques 1here
are always new and dlfferenL approaches Lo LesLlng
LhaL we can add Lo our bag of LesLlng Lrlcks
1LS1 SPA8lnC
W 1he odds are unless pro[ecL ls very small Lhere
wlll be aL leasL several LesLers LesLlng Lhe
sofLware Lven lf Lhere are [usL a few Lhere are
Lhlngs we can do Lo geL more Lhan your eyes
looklng for bugs
W Cne common approach ls Lo slmply swap LesL
responslblllLles wlLh anoLher LesLer for a few
hours or a few days
uC ASP
W A fun way Lo share Lhe LesLlng Lasks ls Lo schedule a q fsh
W A bug bash ls a perlod of Llme (usually a couple hours) ln whlch Lhe
enLlre LesL Leam sLops Lhelr normally asslgned LesLlng Lasks Lo
parLlclpaLe ln Lhe bash
W ln a bug bash a speclflc area of Lhe sofLware ls selecLed and all Lhe
LesLers concenLraLe Lhelr LesLlng on LhaL one area or seL of feaLures
W 1he selecLlon mlghL be an area LhaLs been especlally buggy Lo see lf
Lhere are sLlll more problems lurklng abouL
W A bug bash could deLermlne lf Lhe bugs have been mlssed by normal
LesLlng or lf lLs [usL well wrlLLen code 1here are loLs of poLenLlal
crlLerla for chooslng Lhe area buL ulLlmaLely a bug bash geLs many
dlfferenL people looklng aL one parLlcular area of Lhe sofLware for
bugs
W Cne of your greaLesL allles Lo flnd bugs ls producL supporL or
cusLomer servlce Leam Lhe people who wlll Lalk wlLh cusLomers when
Lhey call or emall wlLh quesLlons and problems
L1A 1LS1lnC
W 1he LesL sharlng ls a lnLernal meLhod Lhe people
LhaL would help share Lhe LesLlng are elLher from
LesL Leam or Lhe pro[ecL developmenL Leam
W eLa LesLlng ls an oLher common meLhod for havlng
oLhers verlfy and valldaLe Lhe sofLware
W eLa LesLlng ls Lhe Lerm used Lo descrlbe Lhe
exLernal LesLlng process ln whlch Lhe sofLware ls
senL ouL Lo a selecL group of poLenLlal cusLomers
who use lL ln a realworld envlronmenL
W eLa LesLlng usually occurs Loward Lhe end of Lhe
producL developmenL cycle and ldeally should [usL
be a valldaLlon LhaL Lhe sofLware ls ready Lo release
Lo real cusLomers
W lrom a LesL sLandpolnL Lhere are several Lhlngs Lo be
consldered whlle relylng on a beLa LesL Llke
Who are Lhe beLa LesLers? Llke lL can be beglnner user
lnLermedlaLe user advances user
Pow wlll you know lf Lhe beLa LesLers even use Lhe
sofLware? Make sure LhaL you or someone runnlng Lhe
beLa program follows up wlLh Lhe parLlclpanLs Lo make
sure LhaL Lheyre uslng Lhe sofLware and meeLlng Lhe
plans goals
eLa LesLs can be a good way Lo flnd compaLlblllLy and
conflguraLlon bugs
usablllLy LesLlng ls anoLher area LhaL beLa LesLlng can
conLrlbuLe Lo lf Lhe parLlclpanLs are well chosen a good
mlx of experlenced and lnexperlenced users 1heyll be
seelng Lhe sofLware for Lhe flrsL Llme and wlll readlly flnd
anyLhlng LhaLs confuslng or dlfflculL Lo use
W esldes conflguraLlon compaLlblllLy and usablllLy beLa
LesLs are surprlslngly poor ways Lo flnd bugs because
1he parLlclpanLs ofLen donL have a loL of Llme Lo use Lhe sofLware so
Lhey wonL flnd much more Lhan superflclal obvlous problems ones
LhaL you llkely already know abouL
eLa LesLlng usually occurs near Lhe end of Lhe developmenL cycle
Lheres noL much Llme Lo flx bugs LhaL are found
W eLa LesLlng can prove Lo be a valuable meLhod for
geLLlng solld lndependenL LesL daLa back on your
sofLware buL Lo be effecLlve lL musL be properly deflned
and managed we could almosL say lL needs Lo be LesLed
Cu1SCu8ClnC 1LS1lnC
W A common pracLlce ln many corporaLlons ls Lo
ouLsource or subconLracL a porLlon of Lhe LesL
work Lo oLher companles LhaL speclallze ln
varlous aspecLs of sofLware LesLlng
lL can cumbersome and more expenslve Lhan havlng
Lhe work done by LesLers on Lhe producL Leam
lL can be an effecLlve way Lo share Lhe LesLlng lf done
properly
W ConflguraLlon and compaLlblllLy LesLlng can be
done uslng ouLsourclng 1hey requlre a large LesL
lab conLalnlng many dlfferenL hardware and
sofLware comblnaLlons and a sLaff of several
people Lo manage lL
W LocallzaLlon LesLlng ls anoLher concern Lo
ouLsourclng
W unless we have a very large LesL Leam lL would
be lmposslble Lo sLaff LesLers LhaL speak all Lhe
dlfferenL languages LhaL your producL supporLs
W lL would be beneflclal Lo have a couple of
forelgn language speaklng LesLers ln Leam Lo
look for fundamenLal locallzaLlon problems buL
lLs probably more efflclenL Lo ouLsource Lhe
LesLlng of Lhe speclflc languages
W Peres a llsL of Lhlngs Lo conslder and Lo dlscuss for maklng
Lhe [ob of ouLsourclng run more smooLhly Llke
WhaL exacLly are Lhe LesLlng Lasks LhaL Lhe LesLlng company ls
Lo perform? Who wlll deflne Lhem? Who wlll approve Lhem?
WhaL schedule wlll Lhey follow? Who wlll seL Lhe schedule?
WhaL happens lf Lhe deadllne ls mlssed?
WhaL dellverables are you Lo provlde Lo Lhe LesLlng company?
1he sofLwares speclflcaLlon perlodlc sofLware updaLes and
LesL cases are some examples
WhaL dellverables are Lhey Lo provlde Lo you? A llsL of Lhe bugs
Lhey flnd would be Lhe mlnlmum
Pow wlll you communlcaLe wlLh Lhem? hone emall lnLerneL
cenLral daLabase dally vlslL? Who are Lhe polnLs of conLacL aL
boLh ends?
Pow wlll you know lf Lhe LesLlng company ls meeLlng your
expecLaLlons? Pow wlll Lhey know lf Lheyre meeLlng your
expecLaLlons?