L14: Software Testing

Question 1

What is software testing?

Answer 1

Testing is intended to
- show that a program does what it is intended to do
- discover program defects before it is put into use

Usually uses artificial data.
The results of the test run are checked for errors, anomalies or information about non-functional attributes.
Testing is part of a more general verification and validation process.
Testing can find errors but not prove their absence.
Formally specified tests replace extensive system specification documents in agile approaches.

Question 2

What is verification and validation?

Answer 2

They can be used to establish whether the system is fit for purpose

Verification
- “Are we building the product right?”
- The software should conform to its specification

Validation
- “Are we building the right product?”
- The software should do what the user requires

Question 3

What are some goals of testing?

Answer 3

• Demonstrate that software meets requirements
• System performs as expected given test cases
• Discover faults, undesirable behaviour
• Expose defects

Question 4

What are some verification and validation methods?

Answer 4

• Walk-through
• Code review
• Inspection
• Formal methods

Question 5

What is the difference between inspections and testing?

Answer 5

Inspections are concerned with the analysis of the static system representation to discover problems (static).

Testing is concerned with exercising and observing product behaviour (dynamic). System is executed with test data and operational behaviour is observed. (You actually run the code)

Testing looks at the program
Inspection looks at all the artefacts (requirements, architecture, UML, database schema, program)

Question 6

Compare inspections and testing (adv/disadv.)

Answer 6

Inspections
- During testing, some errors can mask other errors
- Testing is not suitable for determining certain quality attributes

Testing
- Inspections are not suitable for finding errors due to unexpected interactions and timing/performance issues
- Small companies may not be able to find an independent inspection team

Question 7

When do we test?

Answer 7

• Development
• Release
• Production (after deployment)

Question 8

Who does the testing?

Answer 8

• Developers
• Quality Assurance team (test engineers)
• Domain experts
• Users

Question 9

What are the types of testing?

Answer 9

Functional
- Unit, component, interface, integration, acceptance, system

Non-functional
- Performance, scalability, security, usability, accessibility

Question 10

How can testing be performed?

Answer 10

We want to automate testing as much as possible.

Knowledge of internal structure
- Opaque (PKA black box)
- Transparent (PKA white box)
- Partially transparent (PKA grey box)

Automation
- Manual vs automated

Question 11

What is development testing?

Answer 11

All testing carried out during development:
- Unit
- Component
- System
- Acceptance (for agile development)

Question 12

What is unit testing?

Answer 12

• Testing the functionality of individual program units
• Units may be functions, methods, class
• Defect testing process
• Tests that component acts as expected
• Or show errors with abnormal inputs

Question 13

What are some testing strategies?

Answer 13

Partition testing
- identify groups if inputs with common characteristics that should be processed in the same way, choose tests from each group

Path-based testing
- ensure path through the code under test is executed at least once

Guideline-based testing
- use testing guidelines to choose test cases
- guidelines can reflect commonly made/ previous errors

Question 14

What is component testing?

Answer 14

• Testing the functionality of components as defined in well-defined component interfaces
• Software components are composite (made up of interacting components)
• Shows that each component behaves as expected
• Assumes unit testing on individual objects/components within the component have been testing

Question 15

What is interface testing?

Answer 15

Aims to detect faults in interface

Interface types
- Parameter interfaces (data passed between methods)
- Shared-memory interfaces (memory shared between methods)
- Procedural interfaces
- Message passing interfaces

Question 16

What is system testing?

Answer 16

• Integrating components to create a version of the system
• Testing the integrated system as a whole
• Focuses on interactions between components
• Checks component compatibility, interaction and transfer of data

Question 17

Compare system and component testing

Answer 17

• There are individual components that have been developed and tested individually (potentially by different teams)
• System testing is a collective rather than an individual process

Question 18

What drivers can be used for system testing?

Answer 18

• Requirements/use cases (1 test per requirement)
• Structure (check elements of system works correctly)
• Statistics (assess trustworthiness using randomly sampled input data)
• Risk (check system is not vulnerable)
• Level of coverage achieved

Question 19

What are system testing policies?

Answer 19

Exhaustive system testing is impossible so testing policies which define the required system test coverage may be developed

Examples of testing policies
- All system functions that are accessed through menus should be
- Combinations of functions that are accessed through the same menu must be tested
- Where user input is provided, all functions must be tested with both correct and incorrect input

Question 20

What is acceptance testing?

Answer 20

Used throughout agile, and at the end of plan-driven

Give us a way of saying that something is ‘done’.

Question 21

What is test-driven development?

Answer 21

Test-driven development is a key component of most agile software development approaches.

• Tests are written before code, passing the tests is a critical driver of development
• Code developed incrementally, can only move onto next increment if current increment passes tests
• Provides coverage as every code segment is tested
• Debugging is simplified
• A regression test suite is developed incrementally

Question 22

What is regression testing?

Answer 22

• Testing to check that changes in code has not broken previously working code
• Can be expensive with a manual testing process
• More simple with automated testing (each time a change is made, tests are rerun)
• Tests must be successful before change is committed

Question 23

What is release testing?

Answer 23

• Testing a particular release of system outside development team
• Convince supplier that it meets functionality, performance, dependability goals
• While system testing focuses on bugs (defect testing), release testing focuses on requirements being met (validation testing)

Question 24

What is performance testing?

Answer 24

• Test emergent properties of system, e.g. performance, reliability
• Test while slowly increasing load and find limits of system to identify reasons for failure (stress testing)

Question 25

What is user testing?

Answer 25

- User or customer is testing - User's working environment can effect on reliability, performance, usability - Hard to test user environments and behaviour

Question 26

What are the types of user testing?

Answer 26

Alpha testing - Users work with development at development site Beta testing - A release of software is made available to users to experiment with and raise problems Acceptance testing - Customers test a system to decide whether or not it should be accepted and deployed

Question 27

Describe acceptance testing with agile methods.

Answer 27

- User is part of the development team and deciding acceptability of the system - Tests defined by user are integrated with other tests - No separate acceptance testing process - User representative must represent different behaviours, proficiencies of wide range of users

Question 28

What are the best practices in agile testing?

Answer 28

- Continuous testing and feedback - Involvement of whole team - Test-driven development - Customer satisfaction is the primary concern - Exploratory testing

Question 29

How do we test in production?

Answer 29

- Test in a live environment after software is deployed - Ideally used with smaller and frequent deployments - Features can be turned on/off to see how the rest of the system copes - Attention to customer experience is important

Question 30

What are the costs of testing?

Answer 30

- Cost of fixing errors rises through successive phases of development process - Continuous testing allows errors to be caught early, reducing costs of fixing them - A system should have a suite of tests at different granularities - Automate as much as possible **Test early, test often**