Broken test in SignatureTrustAndValidityVerificationProviderTests(functional test)

[heng-liu]

In SignatureTrustAndValidityVerificationProviderTests(functional test),
there're 19 tests failed on Mac , 15 on Linux, 1 on Windows(netcore)
The list is :
On Mac:

1.GetTrustResultAsync_PrimarySignatureWithUntrustedRoot_NotInAllowedUntrustedRootList_AllowUntrustedFalse_ErrorAsync   
2.GetTrustResultAsync_WithUnavailableRevocationInformationAndAllowUnknownRevocation_WithOnlineRevocationMode_WarnsAsync   
3..GetTrustResultAsync_WithUnavailableRevocationInformationAndAllowUnknownRevocation_WithOfflineRevocationMode_WarnsAsync    
4.GetTrustResultAsync_WithTrustedButExpiredPrimaryAndTimestampCertificates_WithUnavailableRevocationInformationAndAllowUnknownRevocation_WarnsAsync    
5.GetTrustResultAsync_RepositoryCountersignatureWithUntrustedRoot_EmptyAllowedUntrustedRootList_AllowUntrustedFalse_ErrorAsync    
6.GetTrustResultAsync_PrimarySignatureWithUntrustedRoot_EmptyAllowedUntrustedRootList_AllowUntrustedFalse_ErrorAsync   
7.GetTrustResultAsync_RepositoryCountersignatureWithUntrustedRoot_NotInAllowedUntrustedRootList_AllowUntrustedFalse_ErrorAsync    
8.GetTrustResultAsync_WithUnavailableRevocationInformationInVerify_WarnsAsync   
9.GetTrustResultAsync_WithTimestampChainingToUntrustedRoot_NotAllowIgnoreTimestamp_FailAsync    
10.GetTrustResultAsync_WithUnavailableRevocationInformationInRequireMode_WarnsAsync     
11.AuthorPrimarySignatures.GetTrustResultAsync_WithRevokedTimestampCertificate_ReturnsStatusAsync(allowIgnoreTimestamp: False, expectedStatus: Disallowed)
12.AuthorPrimarySignatures.GetTrustResultAsync_WithRevokedPrimaryCertificate_ReturnsSuspectAsync(allowEverything: True)
13.AuthorPrimarySignatures.GetTrustResultAsync_WithRevokedPrimaryCertificate_ReturnsSuspectAsync(allowEverything: False)
14.RepositoryPrimarySignatures.GetTrustResultAsync_WithRevokedTimestampCertificate_ReturnsStatusAsync(allowIgnoreTimestamp: False, expectedStatus: Disallowed)
15.RepositoryPrimarySignatures.GetTrustResultAsync_WithRevokedPrimaryCertificate_ReturnsSuspectAsync(allowEverything: False) 
16.RepositoryPrimarySignatures.GetTrustResultAsync_WithRevokedPrimaryCertificate_ReturnsSuspectAsync(allowEverything: True)
17.RepositoryCountersignatures.VerifyAsync_WithRevokedTimestampCertificate_ReturnsStatusAsync(allowIgnoreTimestamp: False, expectedStatus: Disallowed)
18.RepositoryCountersignatures.VerifyAsync_WithRevokedCountersignatureCertificate_ReturnsSuspectAsync(allowEverything: True)   
19.RepositoryCountersignatures.VerifyAsync_WithRevokedCountersignatureCertificate_ReturnsSuspectAsync(allowEverything: False)

On Linux:

1.GetTrustResultAsync_PrimarySignatureWithUntrustedRoot_NotInAllowedUntrustedRootList_AllowUntrustedFalse_ErrorAsync
2.GetTrustResultAsync_WithUnavailableRevocationInformationAndAllowUnknownRevocation_WithOnlineRevocationMode_WarnsAsync 
3.GetTrustResultAsync_WithUnavailableRevocationInformationAndAllowIllegal_WarnsAsync
4.GetTrustResultAsync_WithUnavailableRevocationInformationInAcceptMode_DoesNotWarnAsync
5.GetTrustResultAsync_WithUnavailableRevocationInformationAndAllowUnknownRevocation_WithOfflineRevocationMode_WarnsAsync 
6.GetTrustResultAsync_RepositoryCountersignatureWithUntrustedRoot_InAllowedUntrustedRootList_AllowUntrustedFalse_SucceedsAsync
7.GetTrustResultAsync_WithTrustedButExpiredPrimaryAndTimestampCertificates_WithUnavailableRevocationInformationAndAllowUnknownRevocation_WarnsAsync
8.GetTrustResultAsync_RepositoryCountersignatureWithUntrustedRoot_EmptyAllowedUntrustedRootList_AllowUntrustedFalse_ErrorAsync 
9.GetTrustResultAsync_WithSignedAndCountersignedPackage_SucceedsAsync
10.GetTrustResultAsync_PrimarySignatureWithUntrustedRoot_EmptyAllowedUntrustedRootList_AllowUntrustedFalse_ErrorAsync
11.GetTrustResultAsync_RepositoryCountersignatureWithUntrustedRoot_NotInAllowedUntrustedRootList_AllowUntrustedFalse_ErrorAsync
12.GetTrustResultAsync_WithUnavailableRevocationInformationInVerify_WarnsAsync
13.GetTrustResultAsync_WithSignedTimestampedCountersignedAndCountersignatureTimestampedPackage_SucceedsAsync
14.GetTrustResultAsync_WithTimestampChainingToUntrustedRoot_NotAllowIgnoreTimestamp_FailAsync
15.GetTrustResultAsync_WithUnavailableRevocationInformationInRequireMode_WarnsAsync

On Windows(netcore):
RepositoryCountersignatures.VerifyAsync_WithRevokedCountersignatureCertificate_ReturnsSuspectAsync

[heng-liu]

The 9 out of 19 tests are fixed or confirmed as exception message issues.
The remaining 10 tests needs to be checked.

[heng-liu]

Status:
On Linux: 14 out of 15 tests are fixed.
On Mac: 10 out of 19 tests are fixed.
On Windows: 1 test is fixed.

[heng-liu]

Disabled the following tests on Mac. Those tests pass on Windows and Linux, but not for macOS.
GetTrustResultAsync_WithRevokedPrimaryCertificate_ReturnsSuspectAsync
GetTrustResultAsync_WithRevokedTimestampCertificate_ReturnsStatusAsync
VerifyAsync_WithRevokedCountersignatureCertificate_ReturnsSuspectAsync
VerifyAsync_WithRevokedTimestampCertificate_ReturnsStatusAsync
Those tests dynamically revoke a certificate. During debugging, the OCSP responder sends the response, but the chain.build results has the status of "RevocationStatusUnknown" on Mac.
It may be the problem of simulated OCSP responder( but Windows and Linux works fine with the OCSP responder)
How does it work with OCSP responder is in Apple’s SecTrust* functions from Security.framework used in chain.build. Have no idea how should we change the tests for macOS specifically.
So we disabled those tests for Mac only (tests are kept on Windows and Linux) for now.
This issue looks the same with dotnet/runtime#31249

[heng-liu]

Closed as fixed.