Deep Dive into NuGet v2 & v3 Repository Behaviour

This blog post details a comprehensive investigation into the behavior of NuGet v2 and v3 repositories, specifically focusing on the discrepancies and inconsistencies observed across different repository managers. The aim is to provide clarity on the observed issues and offer practical recommendations for users encountering similar problems.

Our investigation revealed inconsistencies in the data provided by various NuGet repository managers (Nexus, Artifactory, and ProGet) when queried using both v2 and v3 NuGet API endpoints. While both Chocolatey CLI (Chocolatey CLI) and NuGet CLI (NuGet CLI) consistently presented the data received, the data itself varied between repository implementations. Below we have outlined our methodology, findings, and recommendations for mitigating these issues.

Terms and Definitions

For consistency, we have used some wording that we should define before we go on:

• Group(ed) Repository: A repository that aggregates queries from multiple other repositories. (Nexus calls these Group Repositories, and Artifactory calls them Virtual Repositories).
• Proxy Repository: An intermediary repository between a client and an upstream repository. (Nexus calls these Proxy Repositories, and Artifactory calls them Remote Repositories).
• Correct/Incorrect Endpoint: Using the appropriate NuGet API endpoint version (v2 or v3) for the corresponding repository version.

Methodology

To ensure reproducibility and thoroughness, our methodology consisted of several key phases:

1. Environment Setup

We used the following versions during our investigation:

Software Name	Version Number
Chocolatey CLI	2.3.0
NuGet CLI	6.11.1.2
Fiddler Classic	5.0.20244.10953
SQL Server (Docker Container)	2019-CU28-ubuntu-20.04
Artifactory (Docker Container)	7.77.8
Nexus (Docker Container)	3.71.0
ProGet (Docker Container)	24.0.13

Repository Manager Setup

We used Docker Compose (provided in the Git repository) to provision isolated environments for each repository manager (the version used is shown in the table above) using docker compose up --wait. This ensured consistency and prevented interference between tests. The Nexus repository’s default administrator password was retrieved using docker compose exec nexus cat /nexus-data/password.admin.

SQL Server was used as the database backend for those repository managers that require it.

Each repository manager was configured with the following settings:

• Consistent administrator username and password across all systems.
• Anonymous access enabled to simplify testing.
• Nexus: NuGet API-Key Realm was explicitly enabled.
• Port mappings were carefully configured to allow access to each repository manager’s web interface and API endpoints:
  • Nexus: Port 9000
  • Artifactory: Ports 8081 and 8082
  • ProGet: Port 9003

2. Test Repository Configuration

Within each repository manager, we created a standardized set of repositories:

• nuget-hosted: A local NuGet v3 repository.
• nuget-v2-hosted: A local NuGet v2 repository.
• nuget.org-proxy: A proxy NuGet v3 repository pointing to https://api.nuget.org/v3/index.json.
• nuget.org-v2-proxy: A proxy NuGet v2 repository pointing to https://nuget.org/api/v2/.
• nuget-group: A group repository that aggregated nuget-hosted and nuget.org-proxy.
• nuget-v2-group: A group repository that aggregated nuget-v2-hosted and nuget.org-v2-proxy.

ProGet’s configuration differed slightly as it does not have an explicit group and proxy repository type in the same way that Nexus and Artifactory do. Its nuget-group repository included a connector to https://api.nuget.org/v3/index.json.

3. Package Creation and Publication

Using the provided PowerShell scripts, New-Dependencies.ps1 and New-Packages.ps1, we generated many test packages with varying dependencies. A Vagrantfile is provided to further simplify the package creation process.

We used Push-Repo.ps1, along with the appropriate API keys and source URLs for each repository, to upload all the generated test packages.

4. Automated Testing

The testRepos.ps1 PowerShell script automated the testing process. This script:

• Took the Docker host as a parameter.
• Created a timestamped output directory for each test run.
• Iterated through all the defined repositories and endpoint combinations (NuGet v2 and NuGet v3).

For each combination:

1. Created an output file (<RepositoryBeingTested>-<RepositoryType>-<RepositoryQueryType>-queriedas-<TypeOfQuery>.txt).
2. Added the source URL to the output file.
3. Executed choco search --source <SourceUrl> --ignore-http-cache and captured the results.
4. Recorded the Chocolatey CLI exit code.
5. Executed nuget search -Source <SourceUrl> -Take 1000 and captured both standard error and standard output.
6. Recorded the NuGet CLI exit code.

For example, if your Docker Host is corbob-docker, you would run ./testRepos.ps1 -dockerHost corbob-docker.

5. Network Traffic Analysis

Fiddler Classic was used to capture and analyze the network traffic during testing with the captured traffic being saved as a .saz file, and stored along with the corresponding test output files. This allowed us to inspect the raw HTTP requests and responses, and verify the data being exchanged.

6. Expanded Testing Scenarios

Based on initial observations and user reports, we designed additional test scenarios:

• Configuring a NuGet v2 upstream repository as a NuGet v3 proxy.
• Testing dependency resolution when downloading/installing packages.
• Investigating incomplete results when querying NuGet v2 repositories.

7. Data Analysis

The captured output files and Fiddler traces were then analyzed to identify inconsistencies and discrepancies in the behavior of the different repository managers.

Initial Results

The initial results were mostly as we’d expected, and as we’d seen in previous testing.

Nexus Results

When queried with the incorrect endpoint, the group repository returned a 502 result indicating that there was a mismatch between the query and the repositories.

When queried with the correct endpoint, the results were that although both Chocolatey CLI and NuGet CLI were wanting 1000 results, they only received and subsequently outputted less than 1000 results. While both NuGet v2 and NuGet v3 endpoints returned less than 1000 results, the NuGet v3 endpoint returned a totalHits property that was populated with the number of results reported by Chocolatey CLI and NuGet CLI. And both requested results until that number.

Artifactory Results

When queried with the incorrect endpoint, the group repository returned just the hosted repository packages.

When queried with the correct endpoint, the NuGet v3 results were that both Chocolatey CLI and NuGet CLI were requesting 1000 results and that is how many they output. There were some discrepancies in the NuGet v2 endpoints where sometimes Chocolatey CLI would output more, and other times NuGet CLI would. By our understanding, the repository providers have deprecated the NuGet v2 endpoints, so we have focused our analysis on the NuGet v3 endpoints.

ProGet Results

ProGet was by far the most consistent in our testing. It responded with 1000 packages on every endpoint tested.

Expanding the Testing

Based on reported issues, we expanded our testing to include the following scenarios:

1. NuGet v2 Upstream Repository as a NuGet v3 Proxy

Configuring a NuGet v2 upstream repository as a NuGet v3 proxy in Nexus resulted in the group repository reporting everything as fine, but only returning hosted repository packages. Direct NuGet v3 queries to the proxy repository resulted in a 502 error. Artifactory exhibited similar behavior, while ProGet remained consistent.

2. Dependency Detection

Testing the download/install of the dep100 package revealed that Nexus consistently downloaded only dep100, while Artifactory and ProGet downloaded the full dependency chain (dep1 to dep100). Further investigation showed that Nexus required version numbers to be present in the dependencyGroups element of the .nuspec file to return dependencies correctly, while Artifactory and ProGet did not.

3. NuGet v2 Repository Package Retrieval

To address issues with incomplete results from NuGet v2 repositories, the usePackageRepositoryOptimizations feature in Chocolatey CLI was disabled. This, along with an increase in the number of test packages to 100, resolved the issue in Nexus and ProGet. However, Artifactory still showed inconsistencies, returning varying numbers of results to Chocolatey CLI and NuGet CLI.

Conclusion

Throughout this investigation, both Chocolatey CLI and NuGet CLI consistently reported the data they received from the repository providers. Discrepancies arose from the differing implementations of the NuGet API by the repository managers. Chocolatey CLI’s default query optimization can also contribute to some of these discrepancies, but can be disabled using choco feature disable --name usePackageRepositoryOptimizations.

Recommendations

Based on our findings, we make the following recommendations when using repository managers:

• If you encounter incomplete search results from a group repository, consider splitting it into multiple Chocolatey source repositories.
• If you encounter incomplete results from a NuGet v2 repository, try querying it as a NuGet v3 repository.
• When using Nexus, ensure that your .nuspec files include version numbers for dependencies to ensure correct dependency resolution.
• If encountering inconsistencies with Chocolatey CLI, consider temporarily disabling the usePackageRepositoryOptimizations feature using choco feature disable --name usePackageRepositoryOptimizations.

Summary

This study provides a snapshot of the current behavior of several NuGet repository managers. Ongoing changes and updates to these systems may affect these findings. Continued testing and monitoring are recommended to ensure compatibility and identify any new issues.

We hope this blog post provides insight into the complexities of NuGet repository management. By understanding the nuances of each repository manager’s implementation, you can make informed decisions and avoid potential pitfalls.