data structures and software dependability
computer science department
brandenburg university of technology cottbus - senftenberg
Software Testing
Exercise, Prof. Dr.-Ing. M. Heiner, WS 2021/2022
In case of demand, this course will be in English.
latest updated: January 20, 2022, at 08:28 PM
when and where:
- when: time to be decided, start: 2nd teaching week
- where: VG 1c, R 2.01;
Preliminaries
There are three criteria to fulfil for successfully passing the exercises for the course "Software Testing":
- continuous and active participation in all exercises;
- presentation of TWO tools for software testing;
- compiling a brief summary (data sheet) for the test tool presented;
Please follow this template for the summary in the style of a data sheet:
Date sheet test tool (MS Word, German)
Date sheet test tool (MS Word, english)
. . . you may also create your latex form.
Schedule
| Date | Tool | Student | presentation/slides, Data sheet |
| xx/yy/2021 | -- | -- | -- |
| 29/11/2021 | afl | Osama | slides - datasheet |
| 06/12/2021 | Jest | Christian | slides - datasheet |
| 13/12/2021 | RAPISE | Farhan | -- |
| 10/01/2022 | BurpSuite | Osama | slides - datasheet |
| 17/01/2022 | Selenium | Christian | slides - datasheet |
| 24/01/2022 | Backlogs | Farhan | -- |
| xx/yy/2022 | -- | -- | -- |
Tools
Category Index
- Static Program Checkers
- Code Coverage
- Systematic Software Testing
- Regression Test
- Mutation Test
- Determining Security Defects
- Memory Checks
- Fuzz Testing
- GUI Testing
- Profiling
- Distributed Testing
- Test Management and Bug Tracking
Tool Index
A - abbot - afl - autohotkey - AutoIt - B - Badboy - bugzilla - C - checkstyle - cobertura - CppUnit - csunit - D - Dogtail - E - emma - expect - F - findbugs - Fitnesse - flyspray - fxcop - G - gcov - gprof - GraphWalker - J - Jacareto - Jameleon - javapathfinder - javaprof - jDiffChaser - jester - jfcunit - jlint - junit - L - lint/splint - M - marathon - mbunit - memcheck - O - OProfile - P - pmd - pounder - pywinauto - Q - qat - S - Sahi - specexplorer - Sourcery Analyzer - staf - T - TestMaster - V - valgrind - Z - zanebug
STATIC PROGRAM CHECKERS
lint / splint
Splint is a tool for statically checking C programs for security vulnerabilities and coding mistakes. With minimal effort, Splint can be used as a better lint. If additional effort is invested adding annotations to programs, Splint can perform stronger checking than can be done by any standard lint.
checkstyle
Checkstyle can check many aspects of your java source code. Historically it's main functionality has been to check code layout issues, but since the internal architecture was changed in version 3, more and more checks for other purposes have been added. Now Checkstyle provides checks that find class design problems, duplicate code, or bug patterns like double checked locking.
jlint
A Java Programm checker. Jlint will check your Java code and find bugs, inconsistencies and synchronization problems by doing data flow analysis and building the lock graph.
findbugs
FindBugs is a program to find bugs in Java programs. It looks for instances of "bug patterns" --- code instances that are likely to be errors.
PMD
is a Java source code analyzer. It finds unused variables, empty catch blocks, unnecessary object creation, and so forth. PMD scans Java source code and looks for potential problems. It has plugins for JEdit, JBuilder, Netbeans/Forte/Sun ONE, IntelliJ IDEA, Maven, Ant, Eclipse, Gel, and Emacs.
FxCop
FxCop is a code analysis tool that checks .NET managed code assemblies for conformance to the Microsoft .NET Framework Design Guidelines. It uses reflection, MSIL parsing, and callgraph analysis to inspect assemblies for more than 200 defects in the following areas:: Library design, Localization, Naming conventions, Performance, Security. FxCop includes both GUI and command line versions of the tool.
CODE COVERAGE
gcov coverage testing tool
gcov is a test coverage program. Use it in concert with GNU CC to analyze your programs to help create more efficient, faster running code. You can use gcov as a profiling tool to help discover where your optimization efforts will best affect your code.
cobertura
Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage.
EMMA: a free Java code coverage tool
EMMA is an open-source toolkit for measuring and reporting Java code coverage.
SYSTEMATIC SOFTWARE TESTING
GraphWalker
GraphWalker is an implementation of Model-based testing built in Java. It allows you to generate test sequences from a finite-state machine (graph). The test sequences can be created statically, or run dynamically.
Java PathFinder
Java PathFinder is a system to verify executable Java bytecode programs. In its basic form, it is a Java Virtual Machine (JVM) that is used as an explicit state software model checker.
Spec Explorer
Spec Explorer is a software-development tool for advanced model-based specification and conformance testing. Spec Explorer can help software-development teams detect errors in the design, the specification, and the implementation of their systems. The tool is intended to be used by software testers, designers, and implementers.
REGRESSION TEST
Expect / Dejagnu
programmed dialogue with interactive programs
DejaGnu is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A Test Harness is the testing infrastructure that is created to support a specific program or tool.
Junit
JUnit is a regression testing framework written by Erich Gamma and Kent Beck. It is used by the developer who implements unit tests in Java.
CppUnit
CppUnit is the C++ port of the famous JUnit framework for unit testing. Test output is in XML or text format for automatic testing and GUI based for supervised tests.
Jameleon
Jameleon is a data-driven automated testing tool that is easily extensible via plug-ins. Features of applications are automated in Java and tied together independently in XML, creating self-documenting automated test cases.
csUnit
csUnit is a unit testing tool for the Microsoft .NET Framework. You can use it with all .NET languages including C#, Visual Basic .NET, J#, and managed C++.
MbUnit
- https://code.google.com/p/mb-unit/ (outdated) :
MbUnit is an extensible .Net test framework. As in NUnit, tests are created at runtime using Reflection and custom attributes. MbUnit differentiates itself from NUnit in it's extensibility model. It contains a number of tests that go beyond the simple unit testing, such as combinatorial testing, data oriented testing, etc...
Zanebug
Zanebug was developed to provide a unified platform for unit and integration testing. The current release addresses component-level unit testing, with a fair amount of development time focused on performance statistics and a flexible architecture. The long-term view for Zanebug is a tool that provides integrated component, form and web-based testing both locally and remotely with all the necessary performance information necessary for enterprise development.
MUTATION TEST
Jester
Jester - the JUnit test tester. Jester finds code that is not covered by tests. Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed. Jester includes a script for generating web pages that show the changes made that did not cause the tests to fail.
DETERMINING SECURITY DEFECTS
Material
Paper Linux journal http://www.linuxjournal.com//article.php?sid=5673
Tools
- FlawFinder
- http://www.dwheeler.com/flawfinder
- Rats
- http://www.securesw.com/download_rats.htm
- PSCAN
- http://deployingradius.com/pscan/ (outdated?)
- bfbtester
- http://bfbtester.sourceforge.net/
MEMORY CHECKS
Memory checker / leaktracer
LeakTracer is a small tool for checking a C++ program for memory leaks. To use LeakTracer, run your program using the provided LeakCheck script. It uses the LD_PRELOAD feature to "overlay" some functions on top of your functions (no recompile needed). LeakTracer uses gdb to print out the exact line where the memory was allocated and not freed - this of course means you have to free all dynamically allocated data. LeakTracer also overrides the global operator new and operator delete.
Valgrind
Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.
FUZZ TESTING
american fuzzy lop (afl)
- Article
- LWN 2015, Wikipedia, Wikipedia/afl
- Tool
- http://lcamtuf.coredump.cx/afl/
American fuzzy lop is a security-oriented fuzzer that employs a novel type of compile-time instrumentation and genetic algorithms to automatically discover clean, interesting test cases that trigger new internal states in the targeted binary. This substantially improves the functional coverage for the fuzzed code. The compact synthesized corpora produced by the tool are also useful for seeding other, more labor- or resource-intensive testing regimes down the road.
GUI TEST
JFCUnit
[-jfcUnit enables developers to write test cases for Java Swing based applications. It provides support for: 1.Obtaining handles on Windows/Dialogs opened by the Java code. 2.Locating components within a component hierarchy that occur within the containers found above. 3.Ra\\