August 9, 2016
Related Material::
Additional Participants: Alex Shi, Ard Biesheuvel, Christian Borntraeger, Dan Williams, Dmitry Torokhov, Geert Uytterhoeven, Greg KH, Guenter Roeck, Hannes Reinecke, James Bottomley, Jan Kara, Jiri Kosina, Kevin Hilman, Mark Brown, NeilBrown, Olof Johansson, Rik van Riel, and Vlastimil Babka.
People tagged: Geert.
Trond Myklebust suggested splitting kernel unit testing off from the stable workflow discussion. Guenter Roeck noted that making tests easier to implement does not necessarily cause the tests to be executed frequently, and suggested the topics of formal testing of release candidates and improving test coverage. Mark Brown indicated interest, and added that ease of implementation and frequent use are both important, arguing that initial progress will require pushing on all fronts simultaneously, though David Woodhouse suspects that there isn't that much difference. Alex Shi also indicated interest.
Hannes Reinecke put forward a multipath testbed implemented in FCoE over virtio within qemu, which he would like to include in an automated kernel-ci testbed. Dan Williams put forward his libnvdimm unit-test framework. Kevin Hilman is working on a real-hardware test farm that includes about 30 unique SoC families, with members from ARM, ARM64, and x86, and is starting to run kselftests on these platforms. Guenter Roeck would like to improve qemu-based testing, and perhaps also merge kerneltests.org into kernelci.org. Christian Borntraeger would like to integrate x390 into regression testing based on some existing s390 server farms, a sentiment that Hannes Reinecke seconded. Greg KH put forward the LTSI Test Project.
Dan Williams put forward three questions:
Jan Kara said that while userspace breakage is always a judgement call, it seems clear that if the unit test is the only thing that breaks, that should be fine.
Jan Kara's experience with xfstests indicates that it is good for unit tests to be runnable by more than just the developers. The tests should therefore detect missing functionality and gracefully avoid spewing errors in such cases. Such grace makes testing of backports much easier. Dan Williams agrees that testing for missing functionality is good when there is some sort of feature flag that makes it easy to detect exactly what functionality is present, but notes that in many cases detailed knowledge of the kernel is needed. Dan suggests that this might have a bearing on whether or not the test suite is in-tree or out-of-tree. Jan Kara noted that xfstests just use syscalls to determine what should and should not be tested, and argued that tests that required too many details about the kernel should instead be in-tree.
Test Suites Laurent Pinchart asked if other test suites were of interest, including this one. Tim Bird expressed an interest in integrating such test suites into the Fuego project. Kevin Hilman is also interested in more test suites, but notes that parsing output and producing useful regression reports is the hard part. Laurent Pinchart noted that this situation calls for a common format for test invocation and test results. [ Ed. note: Obligatory xkcd cartoon. ] Mark Brown argued that getting the tests running was nevertheless the first step.