[Bug]: 0 Mutations for 0 Files created

[schonhoff]

What Happened

Hello,

I tried out the mutation plugin for pest but every time I tried it, it tells me that 0 Mutations for 0 Files were created.

~\Herd\example-app git:[master]
./vendor/bin/pest --mutate

  ...

  Tests:    3 passed (4 assertions)
  Duration: 0.89s

  Mutating application files...
  0 Mutations for 0 Files created
  Mutations: 0 tested
  Score:     0.00%
  Duration:  0.00s

   INFO  No mutations created.

I tried it with the new PHPStorm version, too. But it looks like the same result:

C:\Users\schon\.config\herd\bin\php82\php.exe -dzend_extension=C:\Users\schon\.config\herd\bin\xdebug\xdebug-8.2.dll -dxdebug.mode=coverage C:\Users\schon\Herd\example-app\vendor\pestphp\pest\bin\pest --teamcity --mutate --configuration C:/Users/schon/Herd/example-app/phpunit.xml C:\Users\schon\Herd\example-app\tests\Feature\Http\Controllers\TestControllerTest.php

  Tests:    3 passed (4 assertions)
  Duration: 0.85s

  Mutating application files...
  0 Mutations for 0 Files created
  Mutations: 0 tested
  Score:     0.00%
  Duration:  0.00s

   INFO  No mutations created.


Process finished with exit code 0

Hopefully someone has a guess why it won't work for me and my colleges at work.

How to Reproduce

You can use my sample repo and just run ./vendor/bin/pest --mutate.
I added the example mentioned in the mutation site on pestphp to the example repo (https://pestphp.com/docs/mutation-testing) and tried it. It is the same result in the demo as in my main repo.

Sample Repository

https://github.com/schonhoff/mutate-fail-example

Pest Version

3.5.1

PHP Version

8.3.13

Operation System

Windows

Notes

I'm using Laravel Herd on Windows.

Here is my config:

php.ini

[PHP]
; php.ini Options  ;
;user_ini.filename = ".user.ini"
;user_ini.filename =
;user_ini.cache_ttl = 300

; Language Options ;

engine = On
short_open_tag = Off
precision = 14
output_buffering = 4096
;output_handler =
;url_rewriter.tags
;url_rewriter.hosts
zlib.output_compression = Off
;zlib.output_compression_level = -1
;zlib.output_handler =
implicit_flush = Off
unserialize_callback_func =
;unserialize_max_depth = 4096
serialize_precision = -1
;open_basedir =
disable_functions =
disable_classes =
;highlight.string  = #DD0000
;highlight.comment = #FF9900
;highlight.keyword = #007700
;highlight.default = #0000BB
;highlight.html    = #000000
;ignore_user_abort = On
;realpath_cache_size = 4096k
;realpath_cache_ttl = 120
zend.enable_gc = On
;zend.multibyte = Off
;zend.script_encoding =
zend.exception_ignore_args = Off
zend.exception_string_param_max_len = 15

; Miscellaneous ;
expose_php = On

; Resource Limits ;
max_execution_time = 30
max_input_time = 60
;max_input_nesting_level = 64
;max_input_vars = 1000
;max_multipart_body_parts = 1500
memory_limit=2048M

; Error handling and logging ;
error_reporting = E_ALL
display_errors = On
display_startup_errors = On
log_errors = On
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
;report_zend_debug = 0
;xmlrpc_errors = 0
;xmlrpc_error_number = 0
;html_errors = On
;docref_root = "/phpmanual/"
;docref_ext = .html
;error_prepend_string = "<span style='color: #ff0000'>"
;error_append_string = "</span>"

; Log errors to specified file. PHP's default behavior is to leave this value
; empty.
; https://php.net/error-log
; Example:
;error_log = php_errors.log
; Log errors to syslog (Event Log on Windows).
;error_log = syslog
;syslog.ident = php
;syslog.facility = user
;syslog.filter = ascii

;windows.show_crt_warning
; Default value: 0
; Development value: 0
; Production value: 0

; Data Handling ;

;arg_separator.output = "&amp;"
;arg_separator.input = ";&"
variables_order = "EGPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
;enable_post_data_reading = Off
post_max_size=20M
auto_prepend_file=C:\\Program Files\\Herd\\resources\\app.asar.unpacked\\resources\\valet\\dump-loader.php
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
;internal_encoding =
;input_encoding =
;output_encoding =

; Paths and Directories ;

; UNIX: "/path1:/path2"
;include_path = ".:/php/includes"
;
; Windows: "\path1;\path2"
;include_path = ".;c:\php\includes"

doc_root =
user_dir =

; Directory in which the loadable extensions (modules) reside.
; https://php.net/extension-dir
;extension_dir = "./"
; On windows:

extension_dir = "ext"
;sys_temp_dir = "/tmp"
enable_dl = Off
;cgi.force_redirect = 1
;cgi.nph = 1
;cgi.redirect_status_env =
;cgi.fix_pathinfo=1
;cgi.discard_path=1
;fastcgi.impersonate = 1
;fastcgi.logging = 0
;cgi.rfc2616_headers = 0
;cgi.check_shebang_line=1

; File Uploads ;

file_uploads = On
;upload_tmp_dir =
upload_max_filesize=20M
max_file_uploads = 20

; Fopen wrappers ;

allow_url_fopen = On
allow_url_include = Off
;from="[email protected]"
;user_agent="PHP"
default_socket_timeout = 60
;auto_detect_line_endings = Off

; Dynamic Extensions ;

extension=bz2
;extension=ldap
extension=curl
extension=ffi
;extension=ftp
extension=fileinfo
extension=gd
;extension=gettext
extension=gmp
extension=intl
;extension=imap
extension=mbstring
extension=exif      ; Must be after mbstring as it depends on it
extension=mysqli
;extension=oci8_12c  ; Use with Oracle Database 12c Instant Client
;extension=oci8_19  ; Use with Oracle Database 19 Instant Client
;extension=odbc
extension=openssl
;extension=pdo_firebird
extension=pdo_mysql
;extension=pdo_oci
;extension=pdo_odbc
extension=pdo_pgsql
extension=pdo_sqlite
extension=pgsql
extension=shmop
;extension=snmp
extension=soap
extension=sockets
extension=sodium
extension=sqlite3
;extension=tidy
;extension=xsl
extension=zip
zend_extension=opcache

;;;;;;;;;;;;;;;;;;;
; Module Settings ;
;;;;;;;;;;;;;;;;;;;

[CLI Server]
cli_server.color = On

[Date]
;date.timezone =
;date.default_latitude = 31.7667
;date.default_longitude = 35.2333
;date.sunrise_zenith = 90.833333
;date.sunset_zenith = 90.833333

[filter]
;filter.default = unsafe_raw
;filter.default_flags =

[iconv]
;iconv.input_encoding =
;iconv.internal_encoding =
;iconv.output_encoding =

[imap]
;imap.enable_insecure_rsh=0

[intl]
;intl.default_locale =
;intl.error_level = E_WARNING
;intl.use_exceptions = 0

[sqlite3]
;sqlite3.extension_dir =
;sqlite3.defensive = 1

[Pcre]
;pcre.backtrack_limit=100000
;pcre.recursion_limit=100000
;pcre.jit=1

[Pdo]
;pdo_odbc.connection_pooling=strict

[Pdo_mysql]
pdo_mysql.default_socket=

[Phar]
;phar.readonly = On
;phar.require_hash = On
;phar.cache_list =

[mail function]
; For Win32 only.
; https://php.net/smtp
SMTP = localhost
; https://php.net/smtp-port
smtp_port = 25
;sendmail_from = [email protected]
;sendmail_path =
;mail.force_extra_parameters =
mail.add_x_header = Off
mail.mixed_lf_and_crlf = Off
;mail.log = syslog

[ODBC]
;odbc.default_db    =  Not yet implemented
;odbc.default_user  =  Not yet implemented
;odbc.default_pw    =  Not yet implemented
;odbc.default_cursortype
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1

[MySQLi]
mysqli.max_persistent = -1
;mysqli.allow_local_infile = On
;mysqli.local_infile_directory =
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
;mysqli.rollback_on_cached_plink = Off

[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = On
;mysqlnd.debug =
;mysqlnd.log_mask = 0
;mysqlnd.mempool_default_size = 16000
;mysqlnd.net_cmd_buffer_size = 2048
;mysqlnd.net_read_buffer_size = 32768
;mysqlnd.net_read_timeout = 31536000
;mysqlnd.sha256_server_public_key =

[OCI8]
;oci8.privileged_connect = Off
;oci8.max_persistent = -1
;oci8.persistent_timeout = -1
;oci8.ping_interval = 60
;oci8.connection_class =
;oci8.events = Off
;oci8.statement_cache_size = 20
;oci8.default_prefetch = 100
; oci8.prefetch_lob_size = 0
;oci8.old_oci_close_semantics = Off

[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

[bcmath]
bcmath.scale = 0

[browscap]
;browscap = extra/browscap.ini

[Session]
session.save_handler = files
;session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
;session.cookie_secure =
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.cookie_samesite =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.sid_length = 26
session.trans_sid_tags = "a=href,area=href,frame=src,form="
;session.trans_sid_hosts=""
session.sid_bits_per_character = 5
;session.upload_progress.enabled = On
;session.upload_progress.cleanup = On
;session.upload_progress.prefix = "upload_progress_"
;session.upload_progress.name = "PHP_SESSION_UPLOAD_PROGRESS"
;session.upload_progress.freq =  "1%"
;session.upload_progress.min_freq = "1"
;session.lazy_write = On

[Assertion]
zend.assertions = 1

[COM]
;com.typelib_file =
;com.allow_dcom = true
;com.autoregister_typelib = true
;com.autoregister_casesensitive = false
;com.autoregister_verbose = true
;com.code_page=
;com.dotnet_version=

[mbstring]
;mbstring.language = Japanese
;mbstring.internal_encoding =
;mbstring.http_input =
;mbstring.http_output =
;mbstring.encoding_translation = Off
;mbstring.detect_order = auto
;mbstring.substitute_character = none
;mbstring.strict_detection = Off
;mbstring.http_output_conv_mimetypes=
;mbstring.regex_stack_limit=100000
;mbstring.regex_retry_limit=1000000

[gd]
;gd.jpeg_ignore_warning = 1

[exif]
;exif.encode_unicode = ISO-8859-15
;exif.decode_unicode_motorola = UCS-2BE
;exif.decode_unicode_intel    = UCS-2LE
;exif.encode_jis =
;exif.decode_jis_motorola = JIS
;exif.decode_jis_intel    = JIS

[Tidy]
;tidy.default_config = /usr/local/lib/php/default.t
tidy.clean_output = Off

[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5

[sysvshm]
;sysvshm.init_mem = 10000

[ldap]
ldap.max_links = -1

[dba]
;dba.default_handler=

[opcache]
opcache.enable=1
;opcache.enable_cli=0
;opcache.memory_consumption=128
;opcache.interned_strings_buffer=8
;opcache.max_accelerated_files=10000
;opcache.max_wasted_percentage=5
;opcache.use_cwd=1
opcache.validate_timestamps=1
;opcache.revalidate_freq=2
;opcache.revalidate_path=0
;opcache.save_comments=1
;opcache.record_warnings=0
;opcache.enable_file_override=0
;opcache.optimization_level=0x7FFFBFFF
;opcache.dups_fix=0
;opcache.blacklist_filename=
;opcache.max_file_size=0
;opcache.force_restart_timeout=180
;opcache.error_log=
;opcache.log_verbosity_level=1
;opcache.preferred_memory_model=
;opcache.protect_memory=0
;opcache.restrict_api=
;opcache.mmap_base=
;opcache.cache_id=
;opcache.file_cache=
;opcache.file_cache_only=0
;opcache.file_cache_consistency_checks=1
;opcache.file_cache_fallback=1
;opcache.huge_code_pages=0
;opcache.validate_permission=0
;opcache.validate_root=0
;opcache.opt_debug_level=0
;opcache.preload=
;opcache.preload_user=
;opcache.file_update_protection=2
;opcache.lockfile_path=/tmp

[curl]
;curl.cainfo =

[openssl]
openssl.cafile="C:\Users\schon\.config\herd\config\php\cacert.pem"
;openssl.capath=

[ffi]
;ffi.enable=preload
;ffi.preload=

[XDEBUG]
zend_extension=C:\Users\schon\.config\herd\bin\xdebug\xdebug-8.3.dll
xdebug.mode=coverage
xdebug.start_with_request=yes
xdebug.start_upon_error=yes

[Herd]
; Enable PHP Redis
extension="C:\Program Files\Herd\resources\app.asar.unpacked\resources\bin\phpredis\php_redis-8.3.dll"
extension = 'C:\Program Files\Herd\resources\app.asar.unpacked\resources\bin\phpherd\php_herd-8.3.dll'

[Herd]
; Enable MongoDB
extension="C:\Program Files\Herd\resources\app.asar.unpacked\resources\bin\phpmongodb\php_mongodb-8.3.dll"

[adevade]

Yeah, same deal for me! I've tried to run it on at least two different test suites and it always reports 0 mutations. I haven't dug deeper, but would like to know why it is like this?

Also running Windows with Herd.

[unisys12]

@schonhoff & @adevade I just started playing with Mutation Testing myself and ran into the exact same issue. Windows 10 & Herd. After some toying around though, here's what I found to actually get any results - You need to specify a path to the directory or file you want to test. Like so:

.\vendor\bin\pest --mutate --parallel --path=App\Models

You cannot run any of the other available options. There is a optional parameter to pass a class instead of a path, but it yields the same empty results.

Non-Working Example

C:\Users\unisy\Herd\paws-out>.\vendor\bin\pest --mutate --parallel --class=App\Models

  .....................................................................

  Tests:    69 passed (86 assertions)
  Duration: 4.86s
  Parallel: 8 processes

  Mutating application files...
  0 Mutations for 0 Files created


  Mutations: 0 tested
  Score:     0.00%
  Duration:  0.00s
  Parallel:  8 processes

   INFO  No mutations created.

Working Example

C:\Users\unisy\Herd\paws-out>.\vendor\bin\pest --mutate --parallel --path=App\Models

  .....................................................................

  Tests:    69 passed (86 assertions)
  Duration: 4.89s
  Parallel: 8 processes

  Mutating application files...
  35 Mutations for 3 Files created

  ..................-----............
  ------------------------------------------------------------------------------------------------------------------------------------------------
   UNCOVERED  C:\Users\unisy\Herd\paws-out\app\Models\User.php  > Line 25: RemoveArrayItem - ID: da586a3cffe63063

  ------------------------------------------------------------------------------------------------------------------------------------------------
   UNCOVERED  C:\Users\unisy\Herd\paws-out\app\Models\User.php  > Line 26: RemoveArrayItem - ID: 07695e7f735bb13d

  ------------------------------------------------------------------------------------------------------------------------------------------------
   UNCOVERED  C:\Users\unisy\Herd\paws-out\app\Models\User.php  > Line 27: RemoveArrayItem - ID: 9782a18c0a0f8762

  ------------------------------------------------------------------------------------------------------------------------------------------------
   UNCOVERED  C:\Users\unisy\Herd\paws-out\app\Models\User.php  > Line 36: RemoveArrayItem - ID: 13a8405aad2eea49

  ------------------------------------------------------------------------------------------------------------------------------------------------
   UNCOVERED  C:\Users\unisy\Herd\paws-out\app\Models\User.php  > Line 37: RemoveArrayItem - ID: 1f18c3373ffe0049

  Mutations: 5 uncovered, 30 tested
  Score:     85.71%
  Duration:  0.49s
  Parallel:  8 processes

To be honest, passing a path is not the worst option in the world. In the example I have above, it will perform mutations on any models that I have added the covers() method to within my test suite. At least I think it does. The report above says that it performed 35 mutations for 3 files created. But I have 12 models currently. Like I said before, I just started playing with this (yesterday actually) so I am learning and just wanted to share what I found that kinda works for me.