Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 25 additions & 12 deletions docs/COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,25 +14,38 @@ SmalltalkCISpec {
#testing : {
...
#coverage : {
#packages : [ 'Packages-To-Cover.*' ],
#classes : [ #ClassToCover, #'ClassToCover class' ],
#categories : [ 'Categories-To-Cover*' ],
#packages : [ 'SomePackage', 'SomePack*' ],
#classes : [ #ClassToCover ],
#categories : [ 'SomeClassCategory', 'SomeClassCat*' ],
#format : #coveralls
}
}
}
```

The `#coverage` dictionary can contain the following options:

- `#packages` (recommended)
- Measure coverage of all instance side methods in the provided packages
- Measure coverage of all methods in the provided packages (including extension methods)
- Items that end with a trailing `*` (or `.*`) match all packages that start with the given name
- `#classes`
- Measures all methods of all provided classes (instance side)
- Measures all methods of all provided classes (from both their instance and their class sides)
- `#categories`
- Measure coverage for all classes' methods as well as their meta classes' methods
- Measure coverage for all classes' and metaclasses' methods in the provided system categories (does NOT include extension methods)
- Items that end with a trailing `*` or `.*` match all packages that start with the given name
- `#format` (defaults to `#coveralls`)
- The output format of the Coverage data
- May be either `#coveralls` or `#lcov`

If multiple of the option `#packages`, `#classes`, and `#categories` are provided, the union of all matched methods is used for coverage testing.

*Traits* in the specified packages, classes, or categories are honored as well.
The coverage of traits is captured globally: Even if a specified trait method is used by another class that is not covered, the trait will be marked as covered.
This follows the behavior of SUnit in Squeak.

> **Warning**
> *Traits* are currently only supported in Squeak 6.1Alpha and newer. For other Smalltalk versions, methods that are defined in a trait wil incorrectly be displayed as uncovered in coverage reports until these versions receive support for the required Traits protocol. See [#572](https://github.com/hpi-swa/smalltalkCI/pull/572) for more details.

When running smalltalkCI on TravisCI or AppVeyor with the `#coveralls` coverage format, the results will be uploaded to [Coveralls][coveralls] automatically.
Make sure your repository is [added to Coveralls][coveralls_new].

Expand All @@ -51,13 +64,13 @@ Most coverage services already support uploading coverage in the LCOV format wit
For the most common usecases, see these instructions:
- [Inspecting coverage locally](#inspecting-coverage-locally)
- [Coveralls](#coveralls)
- [Travis CI](#coveralls-%26-travis-ci)
- [GitHub actions](#coveralls-%26-github-actions)
- [Travis CI](#coveralls--travis-ci)
- [GitHub Actions](#coveralls--github-actions)
- [CodeCov](#codecov)
- [Travis CI](#codecov-%26-travisci)
- [GitHub actions](#codecov-%26-github-actions)
- [Travis CI](#codecov--travisci)
- [GitHub Actions](#codecov--github-actions)
- [Cobertura](#cobertura)
- [GitLab CI](#cobertura-%26-gitlabci)
- [GitLab CI](#cobertura--gitlab-ci)

### Inspecting coverage locally
On Linux distributions, LCOV is available as a set of tools that can generate a coverage report as HTML/CSS files.
Expand Down Expand Up @@ -150,7 +163,7 @@ Generally it will be:
bash <(curl -s https://codecov.io/bash)
```

#### CodeCov & TravisCI
#### CodeCov & Travis CI
Add this to your `.travis.yml`
```yml
after_success:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
class organization
extensionMethodsForPackageNames: packageNames

^ packageNames gather: [:packageName |
self extensionMethodsInPackage: packageName]
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
class organization
extensionMethodsForPackages: unresolvedPackageNames

^ self extensionMethodsForPackageNames:
(self resolvedPackageNames: unresolvedPackageNames)
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
class organization
extensionMethodsInPackage: packageName

^ self platformClass extensionMethodsInPackage: packageName
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@
"ensureNoExistingBuildStatusFile" : "fn 11/20/2017 11:55",
"escape" : "fn 9/26/2016 16:12",
"explicitTestKeys" : "fn 11/13/2016 22:23",
"extensionMethodsForPackageNames:" : "ct 9/29/2022 16:57",
"extensionMethodsForPackages:" : "ct 9/29/2022 13:38",
"extensionMethodsInPackage:" : "ct 9/29/2022 13:39",
"failAndQuit:" : "EstebanLorenzano 5/27/2021 21:44",
"fileExists:" : "fn 11/22/2017 14:19",
"fold:block:" : "fn 10/7/2016 21:26",
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
(name 'SmalltalkCI-Core-EstebanLorenzano.296' message 'clean two remaining variables' id '04be3ce6-0280-0d00-bf8e-d02f0115559c' date '31 May 2021' time '4:37:18.269174 pm' author 'EstebanLorenzano' ancestors ((name 'SmalltalkCI-Core-mml.295' message 'The last couple of bytes of the UUID are fixed...Back to using Random with as little dependecy on things that might change as possible.' id 'c441f974-1f26-4555-8215-d648236d37a2' date '31 May 2021' time '1:27:22.435831 am' author 'mml' ancestors ((name 'SmalltalkCI-Core-mml.294' message 'Use the last 4 bytes of a UUID for the Travis ID.Using Random bears a great risk of breaking the code because Pharo and Squeak have (and still are) diverging strongly w.r.t. to the Random class.' id 'daaa04a2-34c1-4027-831a-05be17fa2b79' date '31 May 2021' time '1:10:22.395023 am' author 'mml' ancestors ((name 'SmalltalkCI-Core-mml.293' message 'Create Travis ID from RandomCopying 4 bytes from the beginning of a UUID can lead to clashes when the UUID is clock based and the clock hasn''t yet advanced far enough for the next generation (happens in tests).It is cleaner to generate 2 random bytes directly using Random and then produce a hexadecimal representation.' id '7b365d25-36c3-4a28-aa7e-bd03a6e2c54d' date '31 May 2021' time '12:49:43.910095 am' author 'mml' ancestors ((name 'SmalltalkCI-Core-EstebanLorenzano.292' message '- use Smalltalk dictionary to access classes that may not be in the system- remove shadowed variables' id '81a5a210-b77f-0d00-9442-20e900cb1d42' date '27 May 2021' time '10:08:52.076073 pm' author 'EstebanLorenzano' ancestors ((name 'SmalltalkCI-Core-fn.291' message 'Use `String cr` instead of `Character cr` (#521).' id '4ea49fea-a012-4d18-a006-a896368c182b' date '17 May 2021' time '10:12:25.663103 am' author 'fn' ancestors () stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())
(name 'SmalltalkCI-Core-ct.297' message 'Complements SmalltalkCI-Coverage-Core-ct.34 (extension methods for coverage testing).' id '40f925fc-21b7-2249-91ea-1d83b6b7a0a1' date '29 September 2022' time '4:58:30.835693 pm' author 'ct' ancestors ((name 'SmalltalkCI-Core-EstebanLorenzano.296' message 'clean two remaining variables' id '04be3ce6-0280-0d00-bf8e-d02f0115559c' date '31 May 2021' time '4:37:18.269174 pm' author 'EstebanLorenzano' ancestors ((name 'SmalltalkCI-Core-mml.295' message 'The last couple of bytes of the UUID are fixed...Back to using Random with as little dependecy on things that might change as possible.' id 'c441f974-1f26-4555-8215-d648236d37a2' date '31 May 2021' time '1:27:22.435831 am' author 'mml' ancestors ((name 'SmalltalkCI-Core-mml.294' message 'Use the last 4 bytes of a UUID for the Travis ID.Using Random bears a great risk of breaking the code because Pharo and Squeak have (and still are) diverging strongly w.r.t. to the Random class.' id 'daaa04a2-34c1-4027-831a-05be17fa2b79' date '31 May 2021' time '1:10:22.395023 am' author 'mml' ancestors ((name 'SmalltalkCI-Core-mml.293' message 'Create Travis ID from RandomCopying 4 bytes from the beginning of a UUID can lead to clashes when the UUID is clock based and the clock hasn''t yet advanced far enough for the next generation (happens in tests).It is cleaner to generate 2 random bytes directly using Random and then produce a hexadecimal representation.' id '7b365d25-36c3-4a28-aa7e-bd03a6e2c54d' date '31 May 2021' time '12:49:43.910095 am' author 'mml' ancestors ((name 'SmalltalkCI-Core-EstebanLorenzano.292' message '- use Smalltalk dictionary to access classes that may not be in the system- remove shadowed variables' id '81a5a210-b77f-0d00-9442-20e900cb1d42' date '27 May 2021' time '10:08:52.076073 pm' author 'EstebanLorenzano' ancestors ((name 'SmalltalkCI-Core-fn.291' message 'Use `String cr` instead of `Character cr` (#521).' id '4ea49fea-a012-4d18-a006-a896368c182b' date '17 May 2021' time '10:12:25.663103 am' author 'fn' ancestors () stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
helpers
cannotBeCovered: aMethodReference
"Answer true is aMethodReference is not suitable for covering, such as an abstract method or a method derived from a trait."

^ aMethodReference compiledMethod isAbstract
or: [self methodIsTraitMethod: aMethodReference compiledMethod "trait methods are derived from a method in a trait - only the original method is relevant for us"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
helpers
eventualOriginalTraitMethodFor: aCompiledMethod

^ (aCompiledMethod respondsTo: #eventualOriginalTraitMethod)
ifTrue: [aCompiledMethod eventualOriginalTraitMethod]
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
helpers - file name specials
fileNameForSelector: selector
"Copied from MCFileTreeStCypressWriter"
^ (selector includes: $:)
ifTrue: [
selector
collect: [ :each |
each = $:
ifTrue: [ $. ]
ifFalse: [ each ] ] ]
ifFalse: [
(self specials includes: selector first)
ifFalse: [ selector ]
ifTrue: [
| output specials |
specials := self specials.
output := String new writeStream.
output nextPut: $^.
selector
do: [ :each |
output
nextPutAll:
((specials includes: each)
ifTrue: [ specials at: each ]
ifFalse: [ each asString ]) ]
separatedBy: [ output nextPut: $. ].
output contents ] ]
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
helpers
filePathFor: aMethodReference in: aDirectoryName
"Generates the filename for the file where the method related to aCompiledMethod is specified in.
aDirectoryName (a String) states the subfolder in the projectDirectory where the method comes from."
| method class package instanceOrClass |
method := aMethodReference selector.
filePathFor: aMethodReference in: directoryName
"Generates the filename for the file where the method related to aMethodReference is specified in.
directoryName (a String) states the subfolder in the projectDirectory where the method comes from."
| selector class package behaviorType instanceOrClass |
selector := aMethodReference selector.
class := aMethodReference actualClass.
class isMeta
ifTrue: [ instanceOrClass := 'class' ]
ifFalse: [ instanceOrClass := 'instance' ].
package := self packageNameForClass: class.
^ aDirectoryName, SmalltalkCI pathNameDelimiter,
package := self packageNameForMethod: aMethodReference.
behaviorType := (self packageNameForClass: class) = package
ifFalse: ['extension']
ifTrue:
[class isTrait
ifTrue: ['trait']
ifFalse: ['class']].
instanceOrClass := class isMeta
ifTrue: ['class']
ifFalse: ['instance'].
^ directoryName, SmalltalkCI pathNameDelimiter,
package, '.package', SmalltalkCI pathNameDelimiter,
class name, '.class', SmalltalkCI pathNameDelimiter,
class instanceSide name, '.', behaviorType , SmalltalkCI pathNameDelimiter,
instanceOrClass, SmalltalkCI pathNameDelimiter,
(method asString replaceAll: $: with: $.), '.st'
(self fileNameForSelector: selector), '.st'
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
helpers - file name specials
initializeSpecials
"Copied from MCFileTreeStCypressWriter"
| map |
map := Dictionary new.
map
at: $+ put: 'plus';
at: $- put: 'minus';
at: $= put: 'equals';
at: $< put: 'less';
at: $> put: 'more';
at: $% put: 'percent';
at: $& put: 'and';
at: $| put: 'pipe';
at: $* put: 'star';
at: $/ put: 'slash';
at: $\ put: 'backslash';
at: $~ put: 'tilde';
at: $? put: 'wat';
at: $, put: 'comma';
at: $@ put: 'at'.
map keys do: [ :key | map at: (map at: key) put: key ].
^ map
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
helpers
isExcluded: aMethodReference
^ (self ignoredSelectors includes: aMethodReference selector)
or: [ aMethodReference compiledMethod isAbstract
or: [ aMethodReference compiledMethod hasLiteral: #ignoreForCoverage ]]
or: [aMethodReference compiledMethod hasLiteral: #ignoreForCoverage]
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
helpers
localSelectorsIn: aClassDescription

^ aClassDescription localSelectors
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
helpers
methodIsTraitMethod: aCompiledMethod
<ignoreForCoverage>

^ (aCompiledMethod respondsTo: #isTraitMethod)
and: [aCompiledMethod isTraitMethod]
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
helpers
methodReferencesFor: classes
^ (classes gather: [ :cls |
cls selectors collect: [ :selector |
(self localSelectorsIn: cls) collect: [ :selector |
self methodReferenceFor: cls selector: selector ]]) asOrderedCollection
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
compatibility
packageNameForMethod: aMethodReference
self subclassResponsibility
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
helpers - file name specials
specials
"Copied from MCFileTreeStCypressWriter"
^ Specials ifNil: [ Specials := self initializeSpecials ]
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
helpers
allMethodReferencesToCover
^ self class methodReferencesFor: self allClassesToCover

| methods |
methods := self class methodReferencesFor: self allClassesToCover.

methods := methods , self extensionMethodsForPackages asArray.

^ methods reject: [:method | self class cannotBeCovered: method]
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
helpers
classIsTrait: aClassOrMetaClass

^ (aClassOrMetaClass respondsTo: #isTrait)
and: [aClassOrMetaClass isTrait]
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
helpers
eventualUserReferencesOf: methodReferenceInATrait

^ (methodReferenceInATrait respondsTo: #eventualUserReferences)
ifTrue: [methodReferenceInATrait eventualUserReferences]
ifFalse: [#()]
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
helpers
eventualUsersForTraitsIn: methods

^ ((methods select: [:each | self classIsTrait: each actualClass])
gather: [:each | self eventualUserReferencesOf: each])
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
helpers
extensionMethodsForPackages

^ self coverageAt: #packages do: [:packageNames |
SmalltalkCI extensionMethodsForPackages: packageNames]
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@ setup
finishUp
coverageWrappers do: [ :wrapper | wrapper uninstall ].
coveredMethods := (coverageWrappers
select: [ :each | each hasRun ])
select: [ :each | each hasRun
and: [(self class methodIsTraitMethod: each) not "trait methods' wrappers update their original method wrapper automatically, see SCICoverageWrapper>>#mark"] ])
collect: [ :each | each reference ]
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
setup
startUp
| methods |
| methods methodsForWrappers |
methods := self allMethodReferencesToCover.
excludedMethods := methods select: [ :method | self class isExcluded: method ].
includedMethods := methods copyWithoutAll: excludedMethods.
coverageWrappers := includedMethods collect: [ :each | SCICoverageWrapper on: each ].

"Methods in traits are copied to their users. Wrap these copies as well to catch all evaluations of the original trait method."
methodsForWrappers := includedMethods , (self eventualUsersForTraitsIn: methods).
SCICoverageWrapper updateCodeCoverageClass.
coverageWrappers := methodsForWrappers collect: [ :each | SCICoverageWrapper on: each ].
coverageWrappers do: [ :each | each install ]
Original file line number Diff line number Diff line change
@@ -1,25 +1,34 @@
{
"class" : {
"cannotBeCovered:" : "ct 9/29/2022 19:59",
"classNamesNotUnderTest" : "fn 11/22/2017 18:05",
"coverallsDictFor:in:projectDirectory:coverageValue:" : "fn 8/6/2020 13:32",
"filePathFor:in:" : "fn 11/22/2017 15:02",
"eventualOriginalTraitMethodFor:" : "ct 9/29/2022 21:59",
"fileNameForSelector:" : "ct 9/9/2022 22:47",
"filePathFor:in:" : "ct 9/10/2022 02:03",
"ignoredPackages" : "fn 11/26/2017 18:26",
"ignoredSelectors" : "fn 11/22/2017 11:23",
"isExcluded:" : "fn 11/22/2017 12:51",
"initializeSpecials" : "ct 9/9/2022 22:48",
"isExcluded:" : "ct 9/29/2022 13:27",
"linesOf:" : "fn 11/22/2017 14:41",
"localSelectorsIn:" : "ct 9/30/2022 13:50",
"md5Of:" : "fn 11/22/2017 14:42",
"methodIsTraitMethod:" : "ct 9/29/2022 22:34",
"methodReferenceFor:selector:" : "fn 10/8/2016 15:10",
"methodReferencesFor:" : "fn 11/22/2017 13:46",
"methodReferencesFor:" : "ct 9/30/2022 13:41",
"packageNameForClass:" : "fn 10/8/2016 15:11",
"packageNameForMethod:" : "ct 9/9/2022 22:37",
"relativeUnixPathOf:to:" : "fn 8/6/2020 13:32",
"run:spec:in:" : "smalltalkCI 8/26/2020 14:08",
"specials" : "ct 9/9/2022 22:48",
"theNonMetaClassOf:" : "GabrielOmarCotelli 2/20/2019 18:02" },
"instance" : {
"allClassesFromSpec" : "fn 11/26/2017 23:23",
"allClassesNotUnderTestOf:" : "GabrielOmarCotelli 2/20/2019 18:00",
"allClassesToCover" : "fn 11/26/2017 23:28",
"allMethodReferencesToCover" : "fn 11/22/2017 13:29",
"allMethodReferencesToCover" : "ct 9/29/2022 13:43",
"allPackagesUnderTestOf:" : "GabrielOmarCotelli 2/20/2019 18:00",
"classIsTrait:" : "ct 9/29/2022 13:15",
"classesAndMetaclassesOf:" : "GabrielOmarCotelli 2/20/2019 18:00",
"classesToCover" : "fn 11/22/2017 13:33",
"classesToCoverForCategories" : "fn 11/22/2017 12:40",
Expand All @@ -30,12 +39,15 @@
"coverallsSourceFilesIn:" : "fn 11/22/2017 14:47",
"coveredMethods" : "fn 11/22/2017 13:58",
"directoryFor:in:" : "fn 11/22/2017 14:35",
"eventualUserReferencesOf:" : "ct 9/29/2022 21:33",
"eventualUsersForTraitsIn:" : "ct 9/29/2022 13:49",
"excludedMethods" : "fn 11/22/2017 13:58",
"exportResultsIn:" : "smalltalkCI 8/26/2020 14:07",
"finishUp" : "fn 11/22/2017 13:57",
"extensionMethodsForPackages" : "ct 9/29/2022 13:37",
"finishUp" : "ct 9/29/2022 13:57",
"includedMethods" : "fn 11/22/2017 13:58",
"run:" : "fn 11/30/2017 14:42",
"startUp" : "fn 3/26/2018 14:57",
"startUp" : "ct 9/30/2022 14:47",
"theMetaClassOf:" : "GabrielOmarCotelli 2/20/2019 17:58",
"theNonMetaClassOf:" : "GabrielOmarCotelli 2/20/2019 18:02",
"uncoveredMethods" : "fn 11/22/2017 14:30" } }
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"classinstvars" : [
],
"classvars" : [
],
"Specials" ],
"commentStamp" : "",
"instvars" : [
"includedMethods",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
accessing
updateCodeCoverageClass

CodeCoverageClass := SmalltalkCI codeCoverageClass.
Loading