.Net: Bump Microsoft.ML.Tokenizers and System.Text.Json in /dotnet (#…

…9702)

Bumps
[Microsoft.ML.Tokenizers](https://github.com/dotnet/machinelearning) and
[System.Text.Json](https://github.com/dotnet/runtime). These
dependencies needed to be updated together.
Updates `Microsoft.ML.Tokenizers` from 0.22.0-preview.24378.1 to 1.0.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dotnet/machinelearning/releases">Microsoft.ML.Tokenizers's
releases</a>.</em></p>
<blockquote>
<h2>ML.NET v1.0.0</h2>
<p>ML.NET is now <code>1.0.0</code>. 🍰</p>
<p>This is our stable API. In this final sprint we have worked mainly on
improving the documentation. Please let us know what you like about
ML.NET and what we can improve to make your use of machine learning
easier in .NET. With this release we are committed to staying backward
compatible.</p>
<p><a
href="https://github.com/dotnet/machinelearning/blob/master/docs/release-notes/1.0.0/release-1.0.0.md">Release
Notes</a>
<a
href="https://github.com/dotnet/machinelearning/blob/master/docs/release-notes/1.0.0/release-1.0.0.md#installation">Download
and Install</a></p>
<h2>ML.NET v1.0.0-preview</h2>
<p>This is the <code>RC1</code> release for <a
href="https://aka.ms/mlnet">ML.NET</a> version <code>1.0.0</code>. The
work on the API project has been concluded. The focus before releasing
version <code>1.0.0</code> would be to enhance documentation and samples
as well as addressing any critical issues. Please note that NuGets have
now 1.0.0-preview as well as 0.12.0-preview versions depending on which
one will become stable release. Also <code>IDataView</code> is now in
<code>Microsoft.ML</code> namespace. As always thank you so much for
being an awesome community of Machine Learning enthusiasts.</p>
<p><a
href="https://github.com/dotnet/machinelearning/blob/master/docs/release-notes/1.0.0-preview/release-1.0.0-preview.md">Release
Notes</a>
<a
href="https://github.com/dotnet/machinelearning/blob/master/docs/release-notes/1.0.0-preview/release-1.0.0-preview.md#installation">Download
and Install</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/dotnet/machinelearning/commits/v1.0.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `System.Text.Json` from 8.0.5 to 8.0.5
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/dotnet/runtime/compare/v8.0.5...v8.0.5">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Roger Barreto <[email protected]>

dotnet/Directory.Packages.props

@@ -14,6 +14,7 @@
     <PackageVersion Include="Dapr.AspNetCore" Version="1.14.0" />
     <PackageVersion Include="Microsoft.AspNetCore.Mvc.Testing" Version="8.0.0" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="8.0.0" />
+    <PackageVersion Include="Microsoft.ML.Tokenizers.Data.Cl100kBase" Version="1.0.0" />
     <PackageVersion Include="Microsoft.IdentityModel.JsonWebTokens" Version="6.34.0" />
     <PackageVersion Include="Microsoft.VisualStudio.Threading" Version="17.11.20" />
     <PackageVersion Include="MSTest.TestFramework" Version="3.6.1" />
@@ -57,7 +58,7 @@
     <PackageVersion Include="System.Text.Json" Version="8.0.5" />
     <PackageVersion Include="OllamaSharp" Version="4.0.6" />
     <!-- Tokenizers -->
-    <PackageVersion Include="Microsoft.ML.Tokenizers" Version="0.22.0-preview.24378.1" />
+    <PackageVersion Include="Microsoft.ML.Tokenizers" Version="1.0.0" />
     <PackageVersion Include="Microsoft.DeepDev.TokenizerLib" Version="1.3.3" />
     <PackageVersion Include="SharpToken" Version="2.0.3" />
     <!-- Microsoft.Extensions.* -->

dotnet/src/SemanticKernel.UnitTests/SemanticKernel.UnitTests.csproj

@@ -10,6 +10,7 @@
   </PropertyGroup>
 
   <ItemGroup>
+    <PackageReference Include="Microsoft.ML.Tokenizers.Data.Cl100kBase" />
     <PackageReference Include="Microsoft.NET.Test.Sdk" />
     <PackageReference Include="Moq" />
     <PackageReference Include="Verify.Xunit" />