[Haskell-cafe] ANN: leancheck-v0.7.4 with providers for Tasty, Hspec and test-framework

[Rudy Matela]

Hello Haskell-Café,

A new version of LeanCheck is out: v0.7.4. LeanCheck is a property
testing library (like QuickCheck) that tests values enumeratively rather
than at random (unlike QuickCheck).

_Whats new?_ I have created providers to make it easy to incorporate
LeanCheck properties in Tasty, Hspec and test-framework test suites. This
is optional and provided in separate packages: LeanCheck can still be used
alone.

LeanCheck and its framework providers are available on Hackage.
You can install them with:

$ cabal install leancheck
$ cabal install tasty-leancheck
$ cabal install hspec-leancheck
$ cabal install test-framework-leancheck

Check out the README files for examples of use:

* https://github.com/rudymatela/leancheck
* https://github.com/rudymatela/tasty-leancheck
* https://github.com/rudymatela/hspec-leancheck
* https://github.com/rudymatela/test-framework-leancheck


Here's a sneak peek of LeanCheck+Tasty:


## Test program

import Test.Tasty
import Test.Tasty.LeanCheck as LC
import Data.List

main :: IO ()
main = defaultMain tests

tests :: TestTree
tests = testGroup "Test properties checked by LeanCheck"
[ LC.testProperty "sort == sort . reverse" $
\list -> sort (list :: [Int]) == sort (reverse list)
, LC.testProperty "Fermat's little theorem" $
\x -> ((x :: Integer)^7 - x) `mod` 7 == 0
-- the following property do not hold
, LC.testProperty "Fermat's last theorem" $
\x y z n ->
(n :: Integer) >= 3 LC.==> x^n + y^n /= (z^n :: Integer)
]


## Output for the test program

$ ./test
Test properties checked by LeanCheck
sort == sort . reverse: OK
+++ OK, passed 200 tests.
Fermat's little theorem: OK
+++ OK, passed 200 tests.
Fermat's last theorem: FAIL
*** Failed! Falsifiable (after 71 tests):
0 0 0 3

1 out of 3 tests failed (0.00s)

You can't see it here, but Tasty's output is highlighted in colours:
"FAIL" appears in red so you can quickly see which properties failed.


Best Regards,
Rudy

[waldmann]

Hi.

I do prefer the enumerative approach for test case generation
over the randomized one (quickcheck) because I have use cases
(testing student code) where I want to avoid IO,
and I just want a pure list of (small) counterexamples.

Among the enumerative testing libraries,
the "competition" is between leancheck and smallcheck.
For, me the important differences are

Enumeration of algebraic data types (ADTs), via built-in
serial combinators, e.g., cons0 Leaf \/ cons2 Branch

* leancheck enumerate by size (number of nodes in tree)
* smallcheck enumerates by depth (longest path in tree)

I (much!) prefer "by size"
because this delays combinatorial explosion somewhat.

Automated Listable/Serial instances for ADTs

* leancheck uses template-haskell
* smallcheck uses DeriveGeneric

Here I would actually prefer DeriveGeneric.
But this certainly could be added to leancheck?

Oh, and leancheck does not use fancy typeclassery
(smallcheck needs Multiparamtypeclasses and Monads
just to declare a Serial instance)
and it is also easier to get at the list of counterexamples.

- J.W.

[Rudy Matela]

Hi,

I'm glad you liked LeanCheck. :-)

Post by waldmann
Among the enumerative testing libraries,
the "competition" is between leancheck and smallcheck.
For, me the important differences are
Enumeration of algebraic data types (ADTs), via built-in
serial combinators, e.g., cons0 Leaf \/ cons2 Branch
* leancheck enumerate by size (number of nodes in tree)
* smallcheck enumerates by depth (longest path in tree)
I (much!) prefer "by size"
because this delays combinatorial explosion somewhat.

Yes, size bounded enumeration is more tractable. The combinatorial
explosion is still there, but a lot of the times much later in the
enumeration.

LeanCheck also avoids repeating tests. SmallCheck may/will repeat tests
when testing successive "depths".

Post by waldmann
Automated Listable/Serial instances for ADTs
* leancheck uses template-haskell
* smallcheck uses DeriveGeneric
Here I would actually prefer DeriveGeneric.

Might I ask why the preference?

Post by waldmann
But this certainly could be added to leancheck?

Yes it could. I guess LeanCheck could provide a genericTiers /
defaultTiers function (of type something like `Generic a => [[a]]`) on an
optional module so that one would write:

import Test.LeanCheck.Generic

instance Listable MyType where
tiers = genericTiers

Patches or GitHub pull requests are always welcome. :-) -- I may add
this myself if I ever get the time.

Another addition could be defining a "default" inside the Listable
typeclass itself so that one could write `data MyType = ... deriving
Listable`. But I'm not very keen on this second option, as it would
require an extension on LeanCheck's core which is compliant with
Haskell98+Namespaces. It would also hinder the beginner-friendliness of
reading that module. (Or maybe there is a way to define "orphan" default
declarations for typeclasses in separate modules?)

Post by waldmann
Oh, and leancheck does not use fancy typeclassery
(smallcheck needs Multiparamtypeclasses and Monads
just to declare a Serial instance)
and it is also easier to get at the list of counterexamples.

This was intentional, I'm a fan of simple types and interfaces. I'm glad
you liked this.

- Rudy