Adding end to end tests for the guides and samples by nmetulev · Pull Request #351 · microsoft/winappCli

nmetulev · 2026-03-09T20:34:16Z

Description

This PR adds tests for all the guides and samples to run on each pr

Type of Change

🔧 Config/build

Checklist

New tests added for new functionality (if applicable)
Tested locally on Windows

- Create SampleTestHelpers.psm1 shared PowerShell module with assertion, logging, CLI invocation, and MSIX packaging helpers - Add self-contained test.ps1 for each sample: cpp-app, dotnet-app, electron, flutter-app, rust-app, tauri-app, wpf-app - Add test-samples.yml GitHub Actions workflow with matrix strategy running 7 samples in parallel, triggered by Build and Package workflow - Add scripts/test-samples.ps1 local orchestrator for running tests locally with pass/fail summary - Update AGENTS.md with sample testing conventions and instructions Each test validates: prerequisites -> build -> package MSIX -> verify output. Tests run without elevation. electron-winml skipped (requires ML models). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rewrite all 7 sample test.ps1 scripts to follow a guide-first approach: - Phase 1: From-scratch guide workflow in a temp directory (scaffold project, winapp init, build, cert generate, cert info, pack MSIX) - Phase 2: Quick build of existing sample code to verify freshness Add new packaging-cli guide test (samples/packaging-cli/test.ps1): - Tests winapp manifest generate, cert generate, cert info, pack, sign - Validates the docs/guides/packaging-cli.md workflow end-to-end Update shared module with new helpers: - New-TempTestDirectory / Remove-TempTestDirectory for temp dir lifecycle - Assert-WinappInitOutput for verifying winapp init creates expected files - Assert-CertInfo for verifying winapp cert info output Add packaging-cli to CI workflow matrix (8 parallel jobs total). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Fix [switch]$Verbose conflict with PowerShell common parameter in all test scripts and orchestrator (use [CmdletBinding()] instead) - Fix $PSScriptRoot in module functions pointing to wrong directory (pass -SampleDir $PSScriptRoot from callers) - Fix Assert-WinappInitOutput defaults (switches default to $false) - Remove -ExpectWinappYaml for .NET and --setup-sdks=none projects (.NET uses .csproj, Rust/Tauri with no SDKs skip winapp.yaml) - Fix dotnet/wpf tests to explicitly call winapp pack after build (auto-packaging MSBuild targets are optional, not added by init) - Find .exe output directory recursively (handles RID subdirectories) - Fix winapp sign syntax: positional args, not --cert flag - Simplify electron test to sample freshness check only (from-scratch Electron guide workflow is covered by E2E test in test-e2e-electron.ps1) - Rename workflow and headings to 'Sample & Guide' - Update CI matrix: packaging-cli needs Node.js, electron no longer needs .NET Validated locally: packaging-cli, dotnet-app, rust-app, electron, wpf-app Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Replace 8 raw PowerShell test.ps1 files with Pester test.Tests.ps1 files - Simplify SampleTestHelpers.psm1 from ~460 to ~180 lines (Pester handles assertions/reporting) - Use BeforeDiscovery + BeforeAll dual-phase pattern for prerequisite skip logic - Update orchestrator (scripts/test-samples.ps1) as thin Pester wrapper with comma-split support - Update CI workflow for Invoke-Pester with JUnit XML test result reporting - Update AGENTS.md with Pester conventions for sample tests - Fix winapp cert info positional arg syntax in wpf-app test - Resolve WinappPath to absolute in orchestrator before passing to containers All 8 sample tests validated locally: packaging-cli: 10/10, electron: 7/7, dotnet-app: 15/15, rust-app: 15/15, wpf-app: 10/10, tauri-app: 17/17, cpp-app: 8 skipped (no cmake), flutter-app: 7 skipped (no flutter) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…l sample tests - Replace all Invoke-Expression 'winapp ...' calls with Invoke-WinappCommand -Arguments in dotnet-app, packaging-cli, rust-app, and tauri-app tests. This ensures tests work in environments where winapp isn't on PATH by using the helper's fallback chain (npx -> dotnet run -> PATH). - Fix flutter-app to use -Arguments named parameter consistently. - Remove duplicate \ assignments in dotnet-app and rust-app. - Update AGENTS.md to document Context-level -Skip as acceptable when BeforeAll has prerequisite-dependent setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-09T20:46:03Z

Build Metrics Report

Binary Sizes

Artifact	Baseline	Current	Delta
CLI (ARM64)	25.92 MB	25.92 MB	✅ 0.0 KB (0.00%)
CLI (x64)	26.54 MB	26.54 MB	✅ 0.0 KB (0.00%)
MSIX (ARM64)	11.08 MB	11.08 MB	📉 -0.2 KB (-0.00%)
MSIX (x64)	11.79 MB	11.79 MB	📉 -0.1 KB (-0.00%)
NPM Package	23.01 MB	23.01 MB	📉 -0.1 KB (-0.00%)

Test Results

✅ 414 passed out of 414 tests in 397.6s (-162.4s vs. baseline)

Test Coverage

❌ 42.3% line coverage, 46.1% branch coverage · ✅ no change vs. baseline

CLI Startup Time

35ms median (x64, winapp --version) · ✅ no change vs. baseline

Updated 2026-03-30 19:10:40 UTC · commit cf29dda · workflow run

Migrate the full Electron guide workflow from scripts/test-e2e-electron.ps1 into samples/electron/test.Tests.ps1 as Phase 1, making electron consistent with all other sample tests (Phase 1: from-scratch guide + Phase 2: sample freshness check). Changes: - Rewrite electron test.Tests.ps1 with Phase 1 covering: Electron app creation, winapp init, C++/C# addon creation and build, debug identity, Electron packaging, certificate generation, and MSIX packaging. - Remove e2e-test job from build-package.yml (now covered by test-samples.yml). - Delete scripts/test-e2e-electron.ps1 (replaced by Pester test). - Add .NET SDK setup for electron in test-samples.yml (needed for C# addon). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The workflow_run trigger only works for workflow files on the default branch, so test-samples.yml won't trigger until merged. Adding pull_request trigger lets sample tests run as proper PR checks. For PR events, a build job produces the npm-package artifact first. The test-sample jobs then download it, same as workflow_run/dispatch. The build job is skipped for non-PR triggers since the artifact comes from the Build and Package workflow instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR introduces a new Pester-based end-to-end testing approach for the repository’s samples and corresponding guides, and wires it into CI so these workflows run on each PR.

Changes:

Added per-sample Pester 5.x tests (samples/*/test.Tests.ps1) plus shared helpers (samples/SampleTestHelpers.psm1) to validate guide workflows and sample build freshness.
Added CI workflow .github/workflows/test-samples.yml to run each sample test in a matrix on PRs / workflow runs.
Removed the legacy standalone Electron E2E script/job and updated internal docs (AGENTS + packaging-cli guide).

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/test-samples.ps1	Local orchestrator to discover and run sample Pester tests via containers.
scripts/test-e2e-electron.ps1	Removes legacy standalone Electron E2E script in favor of Pester suite.
samples/SampleTestHelpers.psm1	Adds shared helpers for resolving/installing winapp and running CLI commands in tests.
samples/cpp-app/test.Tests.ps1	Adds Pester coverage for C++/CMake guide workflow + sample build check.
samples/dotnet-app/test.Tests.ps1	Adds Pester coverage for .NET guide workflow + sample build check.
samples/electron/test.Tests.ps1	Adds Pester coverage for Electron guide workflow + sample freshness checks.
samples/flutter-app/test.Tests.ps1	Adds Pester coverage for Flutter guide workflow + sample build check.
samples/packaging-cli/test.Tests.ps1	Adds Pester coverage for packaging-cli guide workflow.
samples/rust-app/test.Tests.ps1	Adds Pester coverage for Rust guide workflow + sample build check.
samples/tauri-app/test.Tests.ps1	Adds Pester coverage for Tauri guide workflow + sample build check.
samples/wpf-app/test.Tests.ps1	Adds Pester coverage for WPF guide workflow + sample build check.
.github/workflows/test-samples.yml	New CI workflow to run sample tests in a Windows matrix and publish JUnit XML.
.github/workflows/build-package.yml	Removes the prior Electron E2E job from the build pipeline.
AGENTS.md	Documents the new sample/guide testing conventions and how to run/write tests.
docs/guides/packaging-cli.md	Improves packaging-cli guide steps and adds install/verify instructions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-07T20:45:30Z

samples/SampleTestHelpers.psm1

+function Invoke-WinappCommand {
+    <#
+    .SYNOPSIS
+    Invokes the winapp CLI with the given arguments and returns stdout lines.
+    Uses npx if in a Node project, falls back to dotnet run, then PATH.
+    Throws on non-zero exit code.
+    #>
+    param(
+        [Parameter(Mandatory)]
+        [string]$Arguments,
+        [string]$FailMessage = "winapp $Arguments failed"
+    )
+
+    $npxWinapp = Join-Path (Get-Location) "node_modules\.bin\winapp.cmd"
+    if (Test-Path $npxWinapp) {
+        $cmd = "npx winapp $Arguments"
+    } else {
+        $cliProject = Join-Path $PSScriptRoot "..\src\winapp-CLI\WinApp.Cli\WinApp.Cli.csproj"
+        if (Test-Path $cliProject) {
+            $cmd = "dotnet run --project `"$cliProject`" -- $Arguments"
+        } else {
+            $cmd = "winapp $Arguments"
+        }
+    }


Invoke-WinappCommand currently prefers dotnet run --project ...WinApp.Cli.csproj whenever the repo source is present, even when tests install a specific winapp npm package. This means the E2E sample tests are not actually validating the packaged CLI artifact (and also implicitly require .NET even for npm-only samples). Consider preferring an installed winapp on PATH (or local node_modules/.bin) first, and only falling back to dotnet run when explicitly requested (e.g., via a switch/env var) or when winapp is not available.

Copilot · 2026-04-07T20:45:30Z

samples/SampleTestHelpers.psm1

+        $WinappPath = Join-Path $repoRoot "artifacts\npm"
+        if (-not (Test-Path $WinappPath)) {
+            $WinappPath = Join-Path $repoRoot "src\winapp-npm"
+        }
+    }
+
+    if (-not (Test-Path $WinappPath)) {


Resolve-WinappCliPath defaults to artifacts\npm, but the repo's packaging script (scripts/package-npm.ps1) outputs the .tgz to artifacts\ (repo root), not artifacts\npm. This makes the default resolution fail for common local workflows unless callers always pass -WinappPath. Consider also checking artifacts\ for .tgz files before falling back to src\winapp-npm.

Suggested change

$WinappPath = Join-Path $repoRoot "artifacts\npm"

if (-not (Test-Path $WinappPath)) {

$WinappPath = Join-Path $repoRoot "src\winapp-npm"

}

}

if (-not (Test-Path $WinappPath)) {

$defaultCandidates = @(

(Join-Path $repoRoot "artifacts\npm"),

(Join-Path $repoRoot "artifacts"),

(Join-Path $repoRoot "src\winapp-npm")

)

$WinappPath = $defaultCandidates | Where-Object { Test-Path $_ } | Select-Object -First 1

}

if (-not $WinappPath -or -not (Test-Path $WinappPath)) {

Copilot · 2026-04-07T20:45:31Z

samples/packaging-cli/test.Tests.ps1

+BeforeDiscovery {
+    $script:skip = $null -eq (Get-Command npm -ErrorAction SilentlyContinue)
+}
+
+Describe "Packaging CLI Guide Workflow" {
+    BeforeAll {
+        Import-Module "$PSScriptRoot\..\SampleTestHelpers.psm1" -Force
+        $script:skip = $null -eq (Get-Command npm -ErrorAction SilentlyContinue)
+        $script:tempDir = $null
+
+        if ($script:skip) { return }
+
+        $resolvedPkg = Resolve-WinappCliPath -WinappPath $WinappPath
+        Install-WinappGlobal -PackagePath $resolvedPkg
+


BeforeDiscovery only checks for npm, but this test invokes Invoke-WinappCommand which (with the current helper implementation) can run dotnet run against the repo CLI project. That will fail on machines where .NET isn't installed while the test is not skipped. Either include dotnet in the prerequisite/skip check, or (preferably) adjust Invoke-WinappCommand to use the installed winapp CLI on PATH so this test truly only depends on npm + the packaged winapp artifact.

Copilot · 2026-04-07T20:45:31Z

.github/workflows/test-samples.yml

+    - name: Download npm package (workflow_dispatch)
+      if: >-
+        steps.check.outputs.skip != 'true' &&
+        github.event_name == 'workflow_dispatch'
+      uses: actions/download-artifact@v4
+      with:
+        name: npm-package
+        path: artifacts/npm
+      continue-on-error: true
+
+    # --- Toolchain setup (conditional per sample) ---
+
+    - name: Setup .NET
+      if: >-
+        steps.check.outputs.skip != 'true' &&
+        contains(fromJson('["dotnet-app", "wpf-app", "packaging-cli", "electron"]'), matrix.sample)
+      uses: actions/setup-dotnet@v5
+      with:
+        dotnet-version: '10.0.x'
+
+    - name: Setup Node.js
+      if: >-
+        steps.check.outputs.skip != 'true' &&
+        contains(fromJson('["electron", "tauri-app", "cpp-app", "dotnet-app", "wpf-app", "rust-app", "flutter-app", "packaging-cli"]'), matrix.sample)
+      uses: actions/setup-node@v5
+      with:
+        node-version: '24'
+
+    - name: Setup Flutter
+      if: >-
+        steps.check.outputs.skip != 'true' &&
+        matrix.sample == 'flutter-app'
+      uses: subosito/flutter-action@v2
+      with:
+        channel: stable
+
+    - name: Setup Rust
+      if: >-
+        steps.check.outputs.skip != 'true' &&
+        contains(fromJson('["rust-app", "tauri-app"]'), matrix.sample)
+      uses: dtolnay/rust-toolchain@stable
+
+    # --- Run the sample's self-contained Pester test ---
+
+    - name: Install Pester
+      if: steps.check.outputs.skip != 'true'
+      shell: pwsh
+      run: |
+        Install-Module -Name Pester -Force -SkipPublisherCheck -Scope CurrentUser -MinimumVersion 5.0
+
+    - name: Run ${{ matrix.sample }} test
+      if: steps.check.outputs.skip != 'true'
+      shell: pwsh
+      run: |
+        $winappPath = "artifacts/npm"
+        if (-not (Test-Path $winappPath)) {
+          $winappPath = "src/winapp-npm"
+        }
+        $container = New-PesterContainer -Path "samples/${{ matrix.sample }}/test.Tests.ps1" -Data @{
+          WinappPath = $winappPath
+        }
+        $config = New-PesterConfiguration
+        $config.Run.Container = $container
+        $config.Run.Exit = $true
+        $config.Output.Verbosity = 'Detailed'
+        $config.TestResult.Enabled = $true
+        $config.TestResult.OutputPath = "test-results-${{ matrix.sample }}.xml"
+        $config.TestResult.OutputFormat = 'JUnitXml'
+        Invoke-Pester -Configuration $config


For workflow_dispatch, the workflow attempts to download the npm-package artifact but continues on error; it then falls back to src/winapp-npm. That directory does not contain a built dist/ (it’s generated by npm run compile / packaging), so installing/running winapp from that path will fail. Consider either (1) running the build job on workflow_dispatch too, or (2) requiring a run-id input and using it to download artifacts, or (3) building the npm package in this job when the artifact is missing.

nmetulev and others added 5 commits March 6, 2026 23:07

nmetulev and others added 4 commits March 9, 2026 13:46

Merge branch 'main' into nm/sample-tests

1fc047a

Changes to the packaging cli guide

9c17a1a

Copilot AI review requested due to automatic review settings April 7, 2026 20:39

Copilot started reviewing on behalf of Jaylyn-Barbee April 7, 2026 20:42 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

dotnet tests improved and added winapp run tests for dotnet apps

c1be139

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding end to end tests for the guides and samples#351

Adding end to end tests for the guides and samples#351
nmetulev wants to merge 10 commits intomainfrom
nm/sample-tests

nmetulev commented Mar 9, 2026

Uh oh!

github-actions bot commented Mar 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-        $WinappPath = Join-Path $repoRoot "artifacts\npm"
-        if (-not (Test-Path $WinappPath)) {
-            $WinappPath = Join-Path $repoRoot "src\winapp-npm"
-        }
-    }
-    if (-not (Test-Path $WinappPath)) {
+        $defaultCandidates = @(
+            (Join-Path $repoRoot "artifacts\npm"),
+            (Join-Path $repoRoot "artifacts"),
+            (Join-Path $repoRoot "src\winapp-npm")
+        )
+        $WinappPath = $defaultCandidates | Where-Object { Test-Path $_ } | Select-Object -First 1
+    }
+    if (-not $WinappPath -or -not (Test-Path $WinappPath)) {

Conversation

nmetulev commented Mar 9, 2026

Description

Type of Change

Checklist

Uh oh!

github-actions bot commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Metrics Report

Binary Sizes

Test Results

Test Coverage

CLI Startup Time

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Mar 9, 2026 •

edited

Loading