data structures and software dependability
computer science department
brandenburg university of technology cottbus - senftenberg
Testen von Software
Übung, Prof. Dr.-Ing. M. Heiner, WS 2015/2016
latest updated: February 08, 2016, at 04:25 PM
Allgemeines zur Übung
Zum erfolgreichen Absolvieren der Übung "Testen von Software" sind folgende drei Kriterien zu erfüllen:
- kontinuierliche und aktive Teilnahme an den Übungsveranstaltungen
- Präsentation zweier Testwerkzeuge
- Schreiben einer Kurzzusammenfassung über das referierte Testwerkzeug
Für die Zusammenfassung bitte folgendes Template nutzen:
Zusammenfassung Testwerkzeug (MS Word Version)
Zusammenfassung Testwerkzeug (OpenOffice 2.0 Version)
. . . es darf aber auch in Latex sein.
Termine
| Datum | Toolbeschreibung | Student | Vortrag |
| 04.12.2015 | Junit | Christian Mevius | slides, docu |
| 11.12.2015 | lint/splint | Daniel Schulz | slides, docu |
| 08.01.2016 | GraphWalker | Jonas Ahlers | slides, docu |
| 15.01.2016 | Jacareto | Christian Mevius | slides, docu |
| 22.01.2016 | TestMaster | Daniel Schulz | slides, docu |
| 29.01.2016 | FitNesse | Jonas Ahlers | slides, docu |
Tools
Category Index
- Static Program Checkers
- Code Coverage
- Systematic Software Testing
- Regression Test
- Mutation Test
- Determining Security Defects
- Memory Checks
- Fuzz Testing
- GUI Testing
- Profiling
- Distributed Testing
- Test Management and Bug Tracking
- General Test Frameworks
- Licensed Software
Tool Index
A - abbot - afl - autohotkey - AutoIt - B - Badboy - bugzilla - C - checkstyle - cobertura - CppUnit - csunit - D - Dogtail - E - emma - expect - F - findbugs - Fitnesse - flyspray - fxcop - G - gcov - gprof - GraphWalker - J - Jacareto - Jameleon - javapathfinder - javaprof - jDiffChaser - jester - jfcunit - jlint - junit - L - lint - M - marathon - mbunit - memcheck - O - OProfile - P - pmd - pounder - pywinauto - Q - qat - qmtest - S - Sahi - specexplorer - staf - T - TestMaster - TPTP - V - valgrind - verisoft - Z - zanebug
STATIC PROGRAM CHECKERS
lint / splint
Splint is a tool for statically checking C programs for security vulnerabilities and coding mistakes. With minimal effort, Splint can be used as a better lint. If additional effort is invested adding annotations to programs, Splint can perform stronger checking than can be done by any standard lint.
checkstyle
Checkstyle can check many aspects of your java source code. Historically it's main functionality has been to check code layout issues, but since the internal architecture was changed in version 3, more and more checks for other purposes have been added. Now Checkstyle provides checks that find class design problems, duplicate code, or bug patterns like double checked locking.
jlint
A Java Programm checker. Jlint will check your Java code and find bugs, inconsistencies and synchronization problems by doing data flow analysis and building the lock graph.
findbugs
FindBugs is a program to find bugs in Java programs. It looks for instances of "bug patterns" --- code instances that are likely to be errors.
PMD
is a Java source code analyzer. It finds unused variables, empty catch blocks, unnecessary object creation, and so forth. PMD scans Java source code and looks for potential problems. It has plugins for JEdit, JBuilder, Netbeans/Forte/Sun ONE, IntelliJ IDEA, Maven, Ant, Eclipse, Gel, and Emacs.
FxCop
FxCop is a code analysis tool that checks .NET managed code assemblies for conformance to the Microsoft .NET Framework Design Guidelines. It uses reflection, MSIL parsing, and callgraph analysis to inspect assemblies for more than 200 defects in the following areas:: Library design, Localization, Naming conventions, Performance, Security. FxCop includes both GUI and command line versions of the tool.
CODE COVERAGE
gcov coverage testing tool
gcov is a test coverage program. Use it in concert with GNU CC to analyze your programs to help create more efficient, faster running code. You can use gcov as a profiling tool to help discover where your optimization efforts will best affect your code. Material: Buch: Zeller/Krinke, Programmierwekzeuge S.229 ? 235
cobertura
Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage.
EMMA: a free Java code coverage tool
EMMA is an open-source toolkit for measuring and reporting Java code coverage.
SYSTEMATIC SOFTWARE TESTING
GraphWalker
GraphWalker is an implementation of Model-based testing built in Java. It allows you to generate test sequences from a finite-state machine (graph). The test sequences can be created statically, or run dynamically.
Java PathFinder
Java PathFinder is a system to verify executable Java bytecode programs. In its basic form, it is a Java Virtual Machine (JVM) that is used as an explicit state software model checker.
Spec Explorer
Spec Explorer is a software-development tool for advanced model-based specification and conformance testing. Spec Explorer can help software-development teams detect errors in the design, the specification, and the implementation of their systems. The tool is intended to be used by software testers, designers, and implementers.
REGRESSION TEST
Expect / Dejagnu
programmed dialogue with interactive programs
DejaGnu is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A Test Harness is the testing infrastructure that is created to support a specific program or tool.
Junit
JUnit is a regression testing framework written by Erich Gamma and Kent Beck. It is used by the developer who implements unit tests in Java.
CppUnit
CppUnit is the C++ port of the famous JUnit framework for unit testing. Test output is in XML or text format for automatic testing and GUI based for supervised tests.
Jameleon
Jameleon is a data-driven automated testing tool that is easily extensible via plug-ins. Features of applications are automated in Java and tied together independently in XML, creating self-documenting automated test cases.
csUnit
csUnit is a unit testing tool for the Microsoft .NET Framework. You can use it with all .NET languages including C#, Visual Basic .NET, J#, and managed C++.
MbUnit
- https://code.google.com/p/mb-unit/ (outdated) :
MbUnit is an extensible .Net test framework. As in NUnit, tests are created at runtime using Reflection and custom attributes. MbUnit differentiates itself from NUnit in it's extensibility model. It contains a number of tests that go beyond the simple unit testing, such as combinatorial testing, data oriented testing, etc...
Zanebug
Zanebug was developed to provide a unified platform for unit and integration testing. The current release addresses component-level unit testing, with a fair amount of development time focused on performance statistics and a flexible architecture. The long-term view for Zanebug is a tool that provides integrated component, form and web-based testing both locally and remotely with all the necessary performance information necessary for enterprise development.
MUTATION TEST
Jester
Jester - the JUnit test tester. Jester finds code that is not covered by tests. Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed. Jester includes a script for generating web pages that show the changes made that did not cause the tests to fail.
DETERMINING SECURITY DEFECTS
Material
Paper Linux journal http://www.linuxjournal.com//article.php?sid=5673
Tools
- Flawfinder
- http://www.dwheeler.com/flawfinder
- Rats
- http://www.securesw.com/download_rats.htm
- PSCAN
- http://deployingradius.com/pscan/ (outdated?)
- bfbtester
- http://bfbtester.sourceforge.net/
MEMORY CHECKS
Memory checker / leaktracer
LeakTracer is a small tool for checking a C++ program for memory leaks. To use LeakTracer, run your program using the provided LeakCheck script. It uses the LD_PRELOAD feature to "overlay" some functions on top of your functions (no recompile needed). LeakTracer uses gdb to print out the exact line where the memory was allocated and not freed - this of course means you have to free all dynamically allocated data. LeakTracer also overrides the global operator new and operator delete.
Valgrind
Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.
FUZZ TESTING
american fuzzy lop (afs)
- Article
- LWN 2015, Wikipedia
- Tool
- http://lcamtuf.coredump.cx/afl/
American fuzzy lop is a security-oriented fuzzer that employs a novel type of compile-time instrumentation and genetic algorithms to automatically discover clean, interesting test cases that trigger new internal states in the targeted binary. This substantially improves the functional coverage for the fuzzed code. The compact synthesized corpora produced by the tool are also useful for seeding other, more labor- or resource-intensive testing regimes down the road.
GUI TEST
JFCUnit
jfcUnit enables developers to write test cases for Java Swing based applications. It provides support for: 1.Obtaining handles on Windows/Dialogs opened by the Java code. 2.Locating components within a component hierarchy that occur within the containers found above. 3.Raising events on the found components, e.g. clicking a button, typing text in a TextComponent. 4.Handling testing of components in a thread safe manner. Since version 2.0, jfcUnit provides XML Recording and playback. This allows users to quickly and automatically generate/edit� scripts to drive the testing. The XML API's are open and allow for developers to define there own XML tag handlers. See the example JFCUnit via XML.
Abbot Java GUI Test Framework
The Abbot framework provides automated event generation and validation of Java GUI components, improving upon the very rudimentary functions provided by the java.awt.Robot class (A Better 'Bot). The framework may be invoked directly from Java code or accessed without programming through the use of scripts. It is suitable for use both by developers for unit tests and QA for functional testing.
Jacareto
Java Capture & Replay Toolkit - Jacareto is a capture&replay framework for Java applications and applets. It allows you to easily create your own capture&replay tools.
X11::GUITest / Win32::GuiTest
X11::GUITest is a Perl package intended to facilitate the testing of GUI applications by means of user emulation. It can be used to test and interact with GUI applications which have been built (Xlib, GTK, etc.) upon the X toolkit. (GUItest of Ped?)
Pounder (no longer being actively developed)
Pounder is a utility for automating Java GUI tests. It allows developers to dynamically load GUI- Components , record scripts, and then use those scripts in a test harness. Pounder differentiates itself from other utilities by allowing you to examine the results of a test run in source, while maintaining a separate GUI script that can be re-recorded if necessary.
Marathon
Marathon is a testing framework for GUI applications developed using Java/Swing. Marathon composes of recorder, runner and editor. The testscripts are composed of python code. Marathon captures the semantic actions on components instead of the mouse & keyboard events. This results in scripts that are simple and at a higher granular level. Marathon suites well for applications that mostly depend on forms-paradigm.
Badboy
Badboy is a tool designed to aid in testing and development of dynamic web applications. It makes web testing and development easier with dozens of features including a simple yet comprehensive capture/replay interface, powerful load testing support, detailed reports, graphs and much more!
Sahi
Sahi is an automation and testing tool for web applications, with the facility to record and playback scripts. Developed in java and javascript, this tool uses simple javascript to execute events on the browser.
Dogtail
Dogtail is a GUI test tool and automation framework written in Python. It uses Accessibility (a11y) technologies to communicate with Linux desktop applications.
Python Win32 Automation
pywinauto is a python package that allows you to automate the windows GUI. Very easy to get started, and quite powerful.
jDiffChaser
jDiffChaser is a Java Open Source Software that automates visual GUI comparisons between two versions of a Swing application.
AutoIt
AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. It uses a combination of simulated keystrokes, mouse movement and window/control manipulation in order to automate tasks in a way not possible or reliable with other languages (e.g. VBScript and SendKeys). AutoIt is also very small, self-contained and will run on all versions of Windows out-of-the-box with no annoying "runtimes" required!
Autohotkey
AutoHotkey is a free, open-source utility for Windows. With it, you can for example automate almost anything by sending keystrokes and mouse clicks. You can write a mouse or keyboard macro by hand or use the macro recorder.
PROFILING
google performance tools
These tools are for use by developers so that they can create more robust applications. Especially of use to those developing multi-threaded applications in C++ with templates. Includes TCMalloc, heap-checker, heap-profiler and cpu-profiler.
OProfile
OProfile is a system-wide profiler for Linux systems, capable of profiling all running code at low overhead. OProfile is released under the GNU GPL. It consists of a kernel driver and a daemon for collecting sample data, and several post-profiling tools for turning data into information. OProfile leverages the hardware performance counters of the CPU to enable profiling of a wide variety of interesting statistics, which can also be used for basic time-spent profiling. All code is profiled: hardware and software interrupt handlers, kernel modules, the kernel, shared libraries, and applications.
gprof Gnu Profiler
Material
- Buch Programmierwerkzeuge S. 223 -- 228
- https://sourceware.org/binutils/docs/gprof/
Profiling Java Programs
- Jrat
- http://jrat.sourceforge.net
- Hpjmeter
- http://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER
- JunitPerf
- http://clarkware.com/software/JUnitPerf.html
DISTRIBUTED TESTING
QAT (Quality Assurance Tests )
QAT was developed to ease the issues encountered by having to perform Quality Assurance tests across a variety of hardware and software combinations. The QAT tool can be divided into two main sections, the Agent, responsible for actually running each test or group of tests, and the Harness, which is responsible for test selection, management, result and agent co-ordination.
QMTest
CodeSourcery's QMTest provides a cost-effective general purpose testing solution that allows an organization to implement a robust, easy-to-use testing program tailored to its needs. QMTest's extensible architecture allows it to handle a wide range of application domains: everything from compilers to graphical user interfaces to web-based applications.
STAF
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of reusable components, called services. STAF is a framework designed to improve the level of reuse and automation in test cases and test environments. The goal of STAF is to provide a complete end-to-end automation solution for testers.
TEST MANAGEMENT AND BUG TRACKING
TestMaster
A testcase management, logging, reporting and test automation tool, similar to the commercial product Test Director. Features: Progress stats, reports, test case import from CSV,doc,web or SQL, STAF plugin. (Remark: not supported anymore?)
Fitnesse
FitNesse is a web server, a wiki and an automated testing tool for software. It is a simple tool that allows non-technical users to specify and run acceptance tests for software systems.
Bugzilla
Bugzilla is a bug- or issue-tracking system. Bug-tracking systems allow individual or groups of developers effectively to keep track of outstanding problems with their product. Bugzilla was originally written by Terry Weissman in a programming language called TCL, to replace a rudimentary bug-tracking database used internally by Netscape Communications. Terry later ported Bugzilla to Perl from TCL, and in Perl it remains to this day. Most commercial defect-tracking software vendors at the time charged enormous licensing fees, and Bugzilla quickly became a favorite of the open-source crowd (with its genesis in the open-source browser project, Mozilla). It is now the de-facto standard defect-tracking system against which all others are measured.
Flyspray
Flyspray is an uncomplicated, web-based bug tracking system for assisting with software development.
GENERAL TEST FRAMEWORKS
Eclipse Test and Performance Tools Platform
The Eclipse Test and Performance Tools Platform (TPTP) Project is an open source Top Level Project of the Eclipse Foundation. TPTP is divided into four projects: TPTP Platform, Monitoring Tools, Testing Tools, Tracing and Profiling Tools.
LICENSED SOFTWARE
VeriSoft
is a tool for systematically exploring the state spaces of systems composed of several concurrent processes executing arbitrary code written in any language. The state space of a system is a directed graph that represents the combined behavior of all the components of the system. Paths in this graph correspond to sequences of operations (scenarios) that can be observed during executions of the system. VeriSoft systematically explores the state space of a system by controlling and observing the execution of all the components, and by reinitializing their executions. It searches for coordination problems (deadlocks, divergences, etc.) between concurrent components, and for violations of user-specified assertions. VeriSoft can always guarantee a complete coverage of the state space up to some depth; hence, all possible executions of the system up to that depth are guaranteed to be covered.