Property-Based Testing

Property-based testing is an approach that verifies that properties of a program hold true for a wide range of inputs. In Haskell, the primary tool for property-based testing is the QuickCheck library. This approach is especially useful for testing functions that operate on lists (Lists and List Comprehensions) or that involve algebraic laws (Equational Reasoning).

Core Concepts

What is Property-Based Testing?

Property-based testing:

• Tests that certain properties of your code hold across a range of inputs
• Generates random test data automatically
• Tries to find the simplest counterexample if a property fails
• Provides a systematic way to test code without writing individual test cases

Advantages over Unit Testing

Compared to traditional unit testing:

• Tests behavior rather than specific cases
• Explores edge cases you might not have considered
• Often requires less code to test more thoroughly
• Finds bugs that manual test cases might miss

QuickCheck Basics

Installation

QuickCheck can be installed using Cabal:

cabal update
cabal install --lib QuickCheck

Importing QuickCheck

import Test.QuickCheck

Defining Properties

A property in QuickCheck is a function that returns a Bool or a Property value:

prop_reverseReverse :: [Int] -> Bool
prop_reverseReverse xs = reverse (reverse xs) == xs

Running Tests

quickCheck prop_reverseReverse
-- +++ OK, passed 100 tests.

For more verbose output:

verboseCheck prop_reverseReverse
-- Passed:
-- [0]
-- Passed:
-- [-3,4]
-- ...

Common Properties to Test

Identity Properties

prop_reverseReverse :: [Int] -> Bool
prop_reverseReverse xs = reverse (reverse xs) == xs
 
prop_sortIdempotent :: [Int] -> Bool
prop_sortIdempotent xs = sort (sort xs) == sort xs

Invariant Properties

prop_lengthPreserved :: [Int] -> Bool
prop_lengthPreserved xs = length (sort xs) == length xs

Transformation Properties

prop_mapPreservesLength :: [Int] -> Bool
prop_mapPreservesLength xs = length (map (+1) xs) == length xs

Algebraic Properties

prop_appendAssociative :: [Int] -> [Int] -> [Int] -> Bool
prop_appendAssociative xs ys zs = (xs ++ ys) ++ zs == xs ++ (ys ++ zs)

Constraining Inputs

Sometimes we need to limit the property to certain kinds of inputs:

Using Conditional Properties

prop_divisionInverse :: Int -> Int -> Property
prop_divisionInverse x y = y /= 0 ==> (x `div` y) * y + (x `mod` y) == x

Using Generators

prop_divisionInverse2 :: Int -> NonZero Int -> Bool
prop_divisionInverse2 x (NonZero y) = (x `div` y) * y + (x `mod` y) == x

Custom Generators

For more control, we can define our own generators:

genEven :: Gen Int
genEven = do
  n <- arbitrary
  return (n * 2)
 
prop_evenNumberIsEven :: Property
prop_evenNumberIsEven = forAll genEven (\n -> even n)

Testing Example: A Sorting Function

Let’s verify that a sorting function behaves correctly:

-- Properties for a sorting function
prop_sortOrders :: [Int] -> Bool
prop_sortOrders xs = ordered (sort xs)
  where ordered [] = True
        ordered [_] = True
        ordered (x:y:zs) = x <= y && ordered (y:zs)
 
prop_sortPreservesLength :: [Int] -> Bool
prop_sortPreservesLength xs = length (sort xs) == length xs
 
prop_sortPreservesElements :: [Int] -> Bool
prop_sortPreservesElements xs = 
  all (`elem` xs) (sort xs) && all (`elem` sort xs) xs

Finding Bugs with QuickCheck

When a property fails, QuickCheck attempts to shrink the counterexample to a minimal case, which is useful for debugging recursive functions (Pattern Matching and Recursion):

-- Bug: doesn't handle negative numbers correctly
isAscending :: [Int] -> Bool
isAscending [] = True
isAscending [x] = True
isAscending (x:y:zs) = (x < y) && isAscending (y:zs)
 
-- Testing this buggy function
prop_sortIsAscending :: [Int] -> Bool
prop_sortIsAscending xs = isAscending (sort xs)
 
-- QuickCheck will find a counterexample like:
-- *** Failed! Falsified (after 15 tests and 10 shrinks):
-- [0,0]

Correcting the Bug

-- Fixed version
isAscending :: [Int] -> Bool
isAscending [] = True
isAscending [x] = True
isAscending (x:y:zs) = (x <= y) && isAscending (y:zs)
                      -- ^ Changed < to <=

Performance Considerations

• QuickCheck generates and tests many cases, which can be slow for complex properties
• Use quickCheckWith to configure test parameters:

quickCheckWith stdArgs {maxSuccess = 1000} prop_myProperty

Best Practices

1. Define simple, specific properties that can be checked independently
2. Think about edge cases and invariants in your functions
3. Test algebraic laws when appropriate (associativity, commutativity, etc.)
4. Use appropriate generators for your data types
5. Combine property-based testing with unit testing for critical code paths

Key Points to Remember

1. QuickCheck generates random test cases to verify properties of your code
2. Properties are expressed as functions that return Bool or Property values
3. When a property fails, QuickCheck tries to find the simplest counterexample
4. Custom generators can be defined for specific test data requirements
5. Property-based testing is especially powerful for functional code, where properties and invariants are well-defined

Common properties to test include identity properties (e.g., reverse (reverse xs) == xs), which relate to the concept of pure functions (Functions and Equations), and algebraic properties such as associativity of list append (++), which is a key law for monads (Monads Basics) and functors/applicatives (Functors and Applicatives).

When testing functions that manipulate lists, such as map, filter, or sort, you are often indirectly testing higher-order functions (Higher-Order Functions).

Best practices for property-based testing include thinking about invariants and algebraic laws, which are also central to Equational Reasoning and Monad Laws.