Skip to content

Commit

Permalink
feat: huge changes to facilitate the bad adult classification we have
Browse files Browse the repository at this point in the history
- Adult classification is killing so many titles.
Introduced a new adult (xxx) category into the database.
- Wired up torznab capabilities for the adult category.
- Introduced category filtering to limit what gets through in the postrges search function
- Build a full dashboard allowing you to add / remove / edit entries in your db with ease. You have to toggle this on with `Zilean__EnableDashboard`: true, or set it in your settings file. Its off by default because I dont know who exposes their zilean, so its safer to have it opt in.
- Fixed tvSeries searches when searching by IMDB Id
- Introduced better category handling for movies, tvSearch and the new XXX search.
- Allow overriding the year, imdb id, category and adult + trash states from the dashboard to override anything RTN parsed the files as during ingestion.
- Reworked the python service to ensure all PyObjects are disposed efficiently after the bridge to c#
Same goes for any imported PyModules, and enforce a call to gc in python.
- Moved the python support from the scraper to the shared library so the Api can use it. When you insert a new torrent, or edit one - RTN reruns on the fly for the edited record.
- Removed the Adult and trash options - we just ingest everything now, but anything that would have been removed under the adult flag previously, gets stored in the new xxx category, and because the endpoints are filtered, you wont get these results in regular movie and tvSearch results. If something is in an incorrect category - you can override on the dashboard.
- Implement Index Vaccuming after imports to clean up database, and stop bloat.
  • Loading branch information
iPromKnight committed Nov 21, 2024
1 parent 970f14d commit 5149cd1
Show file tree
Hide file tree
Showing 66 changed files with 1,627 additions and 149 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,6 @@ riderModule.iml
.idea/

src/Zilean.Scraper/python/
src/Zilean.ApiService/python/

Zilean.sln.DotSettings.user
4 changes: 4 additions & 0 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
<PackageVersion Include="Dapper" Version="2.1.35" />
<PackageVersion Include="EFCore.BulkExtensions.PostgreSql" Version="8.1.1" />
<PackageVersion Include="FluentAssertions" Version="6.12.2" />
<PackageVersion Include="Microsoft.AspNetCore.Components.WebAssembly.Server" Version="9.0.0" />
<PackageVersion Include="Microsoft.EntityFrameworkCore" Version="9.0.0" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="9.0.0">
<PrivateAssets>all</PrivateAssets>
Expand All @@ -26,7 +27,10 @@
<PackageVersion Include="Scalar.AspNetCore" Version="1.2.39" />
<PackageVersion Include="SimCube.Aspire" Version="1.0.5" />
<PackageVersion Include="Spectre.Console.Cli" Version="0.49.2-preview.0.50" />
<PackageVersion Include="Syncfusion.Blazor.Grid" Version="27.2.2" />
<PackageVersion Include="Syncfusion.Blazor.Themes" Version="27.2.2" />
<PackageVersion Include="System.Linq.Async" Version="6.0.1" />
<PackageVersion Include="System.Private.Uri" Version="4.3.2" />
<PackageVersion Include="TestableIO.System.IO.Abstractions.TestingHelpers" Version="21.1.3" />
<PackageVersion Include="TestableIO.System.IO.Abstractions.Wrappers" Version="21.1.3" />
<PackageVersion Include="Testcontainers" Version="4.0.0" />
Expand Down
1 change: 1 addition & 0 deletions Zilean.sln
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "scripts", "scripts", "{DAAA
eng\install-python-reqs-benchmarks.sh = eng\install-python-reqs-benchmarks.sh
eng\install-python-reqs-dmmscraper.sh = eng\install-python-reqs-dmmscraper.sh
eng\create-new-migration.sh = eng\create-new-migration.sh
eng\install-python-reqs-dmmscraper.ps1 = eng\install-python-reqs-dmmscraper.ps1
EndProjectSection
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Internal Apps", "Internal Apps", "{2042A18C-6933-4FE9-B4F6-E164BE2D514A}"
Expand Down
3 changes: 1 addition & 2 deletions docs/Writerside/snippets/default-settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
"Zilean": {
"ApiKey": "da3a4ee25d3749ad87301d701a924eb9154c4a95c9b740c5a179469ab0f10578",
"FirstRun": true,
"EnableDashboard": false,
"Dmm": {
"EnableScraping": true,
"EnableEndpoint": true,
Expand Down Expand Up @@ -40,8 +41,6 @@
"RequestTimeout": 10000
},
"Parsing": {
"IncludeAdult": false,
"IncludeTrash": true,
"BatchSize": 5000
}
}
Expand Down
3 changes: 1 addition & 2 deletions docs/Writerside/snippets/settings-with-ingestion.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
"Zilean": {
"ApiKey": "da3a4ee25d3749ad87301d701a924eb9154c4a95c9b740c5a179469ab0f10578",
"FirstRun": true,
"EnableDashboard": false,
"Dmm": {
"EnableScraping": true,
"EnableEndpoint": true,
Expand Down Expand Up @@ -46,8 +47,6 @@
"RequestTimeout": 10000
},
"Parsing": {
"IncludeAdult": false,
"IncludeTrash": true,
"BatchSize": 5000
}
}
Expand Down
12 changes: 4 additions & 8 deletions docs/Writerside/topics/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ _Default: `Generated on first run`_
_Indicates whether this is the first run of the application._
_Default: `true`_

**EnableDashboard**
_Indicates whether the Zilean dashboard should be enabled allowing management of ingested data._
_Default: `false`_

### DMM Configuration
**Dmm.EnableScraping**
_Indicates whether the DMM indexer should scrape from Dmm Hashlists._
Expand Down Expand Up @@ -139,14 +143,6 @@ _The timeout in milliseconds for requests to Zurg and Zilean instances._
_Default: `10000`_

### Parsing Configuration
**Parsing.IncludeAdult**
_Indicates whether adult content should be included in the indexer._
_Default: `false`_

**Parsing.IncludeTrash**
_Indicates whether trash content should be included in the indexer._
_Default: `true`_

**Parsing.BatchSize**
_The batch size for parsing content._
_Default: `5000`_
Expand Down
5 changes: 4 additions & 1 deletion eng/install-python-reqs-dmmscraper.ps1
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
Remove-Item -Path ../src/Zilean.Scraper/python -Recurse -Force
New-Item -Path ../src/Zilean.Scraper/python -ItemType Directory
python -m pip install -r ../requirements.txt -t ../src/Zilean.Scraper/python/ --no-user
Remove-Item -Path ../src/Zilean.ApiService/python -Recurse -Force
New-Item -Path ../src/Zilean.ApiService/python -ItemType Directory
python -m pip install -r ../requirements.txt -t ../src/Zilean.Scraper/python/ --no-user
python -m pip install -r ../requirements.txt -t ../src/Zilean.ApiService/python/ --no-user
5 changes: 4 additions & 1 deletion eng/install-python-reqs-dmmscraper.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
#!/bin/bash
rm -rf ../src/Zilean.Scraper/python
mkdir -p ../src/Zilean.Scraper/python
python3.11 -m pip install -r ../requirements.txt -t ../src/Zilean.Scraper/python/
rm -rf ../src/Zilean.ApiService/python
mkdir -p ../src/Zilean.ApiService/python
python3.11 -m pip install -r ../requirements.txt -t ../src/Zilean.Scraper/python/
python3.11 -m pip install -r ../requirements.txt -t ../src/Zilean.ApiService/python/
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
using Zilean.ApiService.Features.Dashboard.Components.Pages.Dashboard;

namespace Zilean.ApiService.Features.Bootstrapping;

[ExcludeFromCodeCoverage]
Expand Down Expand Up @@ -72,4 +74,23 @@ public static IServiceCollection AddApiKeyAuthentication(this IServiceCollection

return services;
}

public static IServiceCollection AddDashboardSupport(this IServiceCollection services, ZileanConfiguration configuration)
{
if (!configuration.EnableDashboard)
{
return services;
}

services.AddRazorComponents()
.AddInteractiveServerComponents()
.AddInteractiveWebAssemblyComponents();

services.AddSyncfusionBlazor();

services.AddScoped<DashboardDataAdapter>();
services.AddSingleton<ParseTorrentNameService>();

return services;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,55 @@ namespace Zilean.ApiService.Features.Bootstrapping;

public static class WebApplicationExtensions
{
public static WebApplication EnableSwagger(this WebApplication app)
public static WebApplication UseZileanRequired(this WebApplication app, ZileanConfiguration configuration)
{
app.MapOpenApi();
app.MapScalarApiReference();
if (configuration.EnableDashboard)
{
if (app.Environment.IsDevelopment())
{
app.UseWebAssemblyDebugging();
}
else
{
app.UseExceptionHandler("/Error", createScopeForErrors: true);
app.UseHsts();
}

app.UseStaticFiles();
app.UseAntiforgery();
}

app.UseAuthentication();
app.UseAuthorization();

return app;
}

public static WebApplication MapZileanEndpoints(this WebApplication app, ZileanConfiguration configuration) =>
app
.MapDmmEndpoints(configuration)
public static WebApplication MapZileanEndpoints(this WebApplication app, ZileanConfiguration configuration)
{
app.MapDefaultEndpoints();

app.MapDmmEndpoints(configuration)
.MapImdbEndpoints(configuration)
.MapTorznabEndpoints(configuration)
.MapTorrentsEndpoints(configuration)
.MapBlacklistEndpoints()
.MapHealthCheckEndpoints();

if (configuration.EnableDashboard)
{
app.MapStaticAssets();

app.MapRazorComponents<Dashboard.Components.ZileanWebApp>()
.AddInteractiveServerRenderMode()
.AddInteractiveWebAssemblyRenderMode();

Syncfusion.Licensing.SyncfusionLicenseProvider.RegisterLicense("MzU4OTgxNkAzMjM3MmUzMDJlMzBuY2dtNzRCZjAzRmtPTDdGcmFRNXVXTDhTOHdjaU9sNDZPUjBWMEsxSmlNPQ==");
}

app.MapOpenApi();
app.MapScalarApiReference();

return app;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
@inherits LayoutComponentBase

<div class="page">
<main>
<article class="content px-4">
@Body
</article>
</main>
</div>
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
.page {
position: relative;
display: flex;
flex-direction: column;
}

main {
flex: 1;
}

.sidebar {
background-image: linear-gradient(180deg, rgb(5, 39, 103) 0%, #3a0647 70%);
}

.top-row {
background-color: #f7f7f7;
border-bottom: 1px solid #d6d5d5;
justify-content: flex-end;
height: 3.5rem;
display: flex;
align-items: center;
}

.top-row ::deep a, .top-row ::deep .btn-link {
white-space: nowrap;
margin-left: 1.5rem;
text-decoration: none;
}

.top-row ::deep a:hover, .top-row ::deep .btn-link:hover {
text-decoration: underline;
}

.top-row ::deep a:first-child {
overflow: hidden;
text-overflow: ellipsis;
}

@media (max-width: 640.98px) {
.top-row {
justify-content: space-between;
}

.top-row ::deep a, .top-row ::deep .btn-link {
margin-left: 0;
}
}

@media (min-width: 641px) {
.page {
flex-direction: row;
}

.sidebar {
width: 250px;
height: 100vh;
position: sticky;
top: 0;
}

.top-row {
position: sticky;
top: 0;
z-index: 1;
}

.top-row.auth ::deep a:first-child {
flex: 1;
text-align: right;
width: 0;
}

.top-row, article {
padding-left: 2rem !important;
padding-right: 1.5rem !important;
}
}

#blazor-error-ui {
background: lightyellow;
bottom: 0;
box-shadow: 0 -1px 2px rgba(0, 0, 0, 0.2);
display: none;
left: 0;
padding: 0.6rem 1.25rem 0.7rem 1.25rem;
position: fixed;
width: 100%;
z-index: 1000;
}

#blazor-error-ui .dismiss {
cursor: pointer;
position: absolute;
right: 0.75rem;
top: 0.5rem;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
@page "/"
@rendermode InteractiveServer

<div class="container-fluid">
<div class="row">
<div class="col-12">
<img src="images/zilean-logo.png" alt="Zilean" class="imt-fluid mb-5" />
</div>
</div>
<div class="row">
<div class="col-12">
<DashboardTorrentsGrid />
</div>
</div>
</div>
Loading

0 comments on commit 5149cd1

Please sign in to comment.