fastlane-community / trainer

Convert xcodebuild plist and xcresult files to JUnit reports

Home Page:https://krausefx.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support for Xcode 11

thibault-ml opened this issue · comments

Xcode 11 uses a new format for xctest (version 3), which doesn't have a TestSummary.plist file, and uses Zstandard-compressed "database-like" files.

Xcode 11 Release Notes state:

The format of result bundles changed in Xcode 11. A result bundle is an asset produced by Xcode 11 with the xcresult file extension that contains information about the build, tests, code coverage, and more. Any xcresult files produced with Xcode 10 or earlier cannot be read by Xcode 11. A result bundle can be produced by passing -resultBundlePath ./Example.xcresult to an xcodebuild invocation and the Example.xcresult can then be opened in Xcode. Xcode also creates result bundles in Derived Data. The current result bundle version number is 3, which can be specified by passing the xcodebuild flag -resultBundleVersion 3. Version 3 is the default in Xcode 11, but it is still recommended for automation to explicitly pass the flag, so that any potential future versions that become the default do not cause issues to existing tools. Result bundles can be inspected using xcresulttool. A JSON representation of the root object of the result bundle can be exported using xcrun xcresulttool get --format json --path ./Example.xcresult and any nested object, identified by its reference found in the JSON output, can be exported by adding the flag --id REF. xcresulttool also provides the description of its format using xcrun xcresulttool formatDescription. (41633595)

Basically, to obtain results from an Xcode 11 xcresult, one would have to do the following:

// The output of this command will be useful to automatically parse the output of the next commands
$ xcrun xcresulttool formatDescription get --format json
[...]
$ xcrun xcresulttool get --format json --path Project.xcresult
[...]
          "testsRef" : {
            "_type" : {
              "_name" : "Reference"
            },
            "id" : {
              "_type" : {
                "_name" : "String"
              },
              "_value" : "0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=="
            },
[...]
// We take the value of the `testsRef` key (if any) for each `ActionRecord` entry
$ xcrun xcresulttool get --format json --path Project.xcresult --id '0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=='
{
  "_type" : {
    "_name" : "ActionTestPlanRunSummaries"
  },
  "summaries" : {
    "_type" : {
      "_name" : "Array"
    },
    "_values" : [
[...]

hi @thibault-ml thanks for bringing this up. I will take a look at this once I get some of these JSON files.

@thibault-ml Looking into! Will hopefully have something for you soon 😊

@thibault-ml Hey! Do you happen to have a xcresult file you could send me? 😊 It would be good to have a legit file that I can test on 🙏

@joshdholtz I'll have a look if we can, not sure 100% I'm at liberty to just yet. Will have a look :-)

@thibault-ml If you don't want to send but are able to run my PR that would also be 💯 😊 The PR works for me on my simple test use cases (I don't have any huge test suites). I got a few fixes to do on it and then will merge.

@joshdholtz Here's a .xcresult directory of one of our PSPDFKit test suites. Should contain over 1000 test results:

PSPDFKit.xcresult.zip

Your PR fails with:

bundler: failed to load command: trainer (/Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer)
NoMethodError: [!] undefined method `[]' for nil:NilClass
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:308:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:337:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:337:in `block in initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:336:in `map'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:336:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:238:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:238:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:209:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:209:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:181:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:181:in `block in initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:180:in `map'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/xcresult.rb:180:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:141:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:141:in `parse_xcresult'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:61:in `initialize'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:47:in `new'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:47:in `block in auto_convert'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:33:in `each'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/test_parser.rb:33:in `auto_convert'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:34:in `block (2 levels) in run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in `call'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in `run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in `run_active_command'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/fastlane-2.130.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in `run!'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:40:in `run'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/lib/trainer/commands_generator.rb:10:in `start'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/trainer-e6907dbe9186/bin/trainer:6:in `<top (required)>'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer:23:in `load'
  /Volumes/CI/ci/.asdf/installs/ruby/2.4.1/lib/ruby/gems/2.4.0/bin/trainer:23:in `<top (required)>'

@juliangrosshauser Thanks for sending that over! I had a feeling my PR would need some tweaking for something a little more complex 🙃 I will make sure it works with you sent over ❤️

@juliangrosshauser Got it solved! Thanks sending this 🙌

@juliangrosshauser Alright, PR should be good to go again if you'd like to test when you get some time. I was able to get a successful output from the xcresults that you sent me 👌 I made sure all the data is getting pulled out safely now so that shouldn't happen anymore 😇

@joshdholtz Great work! Thank you so much, really appreciate it!

I tested your PR and it works flawlessly now. The only problematic .xcresult directory I could find it this one: PSPDFKit-1-test-failure.zip

It contains 1008 test results. 1007 tests succeed, but the testStyledStringToPDFAsFileURL test fails. The problem is that the resulting XML file contains 1008 successful tests, but no failures.

Oh boy 😱 I found the reason. The testCaseName name in the xcresulttool JSON output is something not as straight forward to match on 🤔 Will work on a solution though

"producingTarget" : {
    "_type" : {
        "_name" : "String"
    },
    "_value" : "PSPDFKitTests"
},
"testCaseName" : {
    "_type" : {
        "_name" : "String"
    },
    "_value" : "-[PSPDFProcessorTests testStyledStringToPDFAsFileURL]"
}

@juliangrosshauser Thanks for testing again! Here is a commit for fixing that 😊 3f88286

@joshdholtz I tested your newest changes with Objective-C and Swift failures. Looks like everything works now! 🎉 Thank you again!

@juliangrosshauser Thanks so much for feeding me files needed to make this change 😊 I will merge the PR and get a new release out today at some point 💪🚀

Thank you guys for working on this!

Is the correct way to use this to specify the -resultBundlePath to xcodebuild, like this, when invoking scan?

xcargs: "-resultBundlePath ./TestSummaries.xcresult"

@vrutberg I think that should work! But you should also be able to do...

scan(
  result_bundle: true,
  output_directory: "./" # optional 
)