新增M3U8启发式广告检测算法并支持动态关键词配置

This commit is contained in:
Troray
2026-01-19 21:52:38 +08:00
parent 7dd3ceea17
commit c3d01e95a4
11 changed files with 947 additions and 114 deletions
+118
View File
@@ -0,0 +1,118 @@
# 🛡️ 广告过滤功能使用指南
KVideo 新增了强大的 M3U8 广告过滤系统,支持多种过滤模式,可有效去除视频中插入的切片广告。
## ✨ 功能特性
- **多模式选择**:支持关闭、关键词过滤、智能启发式过滤(Beta)和激进模式。
- **UI 集成**:在播放器设置菜单中直接切换模式,实时生效。
- **自定义关键词**:支持通过环境变量扩展过滤关键词。
- **高性能**:基于流式处理,对播放加载速度几乎无影响。
## 🎮 使用说明
1. 打开任意视频播放。
2. 点击播放器左上角的 **(···)** 按钮。
3.**广告过滤** 下拉菜单中选择模式:
- **关闭**:不过滤任何内容(默认)。
- **关键词**:仅过滤 URL 中包含特定广告关键词(如 `adjump`, `pre-roll`)的片段。安全稳定,误杀率极低。
- **智能(Beta)**:推荐使用。结合关键词、文件名特征、时间戳跳变标记 (DISCONTINUITY) 和 HLS 标签进行综合评分。能识别大多数无明显关键词的广告。
- **激进**:降低判定阈值,过滤更多疑似广告,但可能会误伤正片。仅在智能模式无效时尝试。
> ⚠️重要提示:添加关键词时,请勿使用类似`ad` `ads` `video` `20260118` 这类简短相对通用的词汇,会造成误杀率极高。
---
## 🛠️ 部署配置指南
你可以通过环境变量 `NEXT_PUBLIC_AD_KEYWORDS` 自定义额外的广告关键词。
### 1. 本地开发 (Local Development)
在项目根目录的 `.env.local` 文件中添加。支持多行格式(推荐使用引号包裹):
```env
# 单行模式
NEXT_PUBLIC_AD_KEYWORDS=spam,promo,intro-ad
# 多行模式 (更清晰)
NEXT_PUBLIC_AD_KEYWORDS="
spam
promo
intro-ad
/unwanted-segment/
"
```
### 2. Docker 部署 (运行时热加载)
我们支持三种方式注入关键词,优先级从高到低:
1. **文件加载 (推荐)**:通过挂载文件并在运行时读取。
2. **运行时环境变量**:直接通过 `-e` 传入 `AD_KEYWORDS`
3. **构建时环境变量**:构建镜像时打入的 `NEXT_PUBLIC_AD_KEYWORDS`
#### 方案 A:挂载配置文件 (支持运行时修改)
创建一个关键词文件 `ad_keywords.txt`
```text
spam
promo
/ad-segment/
intro
```
运行 Docker 时挂载该文件并设置 `AD_KEYWORDS_FILE` 环境变量:
```bash
docker run -d -p 3000:3000 \
-v $(pwd)/ad_keywords.txt:/app/ad_keywords.txt \
-e AD_KEYWORDS_FILE=/app/ad_keywords.txt \
--name kvideo kuekhaoyang/kvideo:latest
```
*注:修改宿主机文件后重启容器即可生效,无需重新构建镜像。*
#### 方案 B:直接注入环境变量
```bash
docker run -d -p 3000:3000 \
-e AD_KEYWORDS="spam,promo,intro-ad" \
--name kvideo kuekhaoyang/kvideo:latest
```
### 3. Vercel 部署
1. 进入 Vercel 项目 Dashboard。
2. 点击 **Settings** -> **Environment Variables**
3. 添加新变量:
- **Key**: `NEXT_PUBLIC_AD_KEYWORDS`
- **Value**: `spam,promo,intro-ad`
4. 重新部署项目以生效。
### 4. Cloudflare Pages 部署
1. 进入 Cloudflare Pages 项目 Dashboard。
2. 点击 **Settings** -> **Environment variables**
3. 添加变量:
- **Variable name**: `NEXT_PUBLIC_AD_KEYWORDS`
- **Value**: `spam,promo,intro-ad`
4. 重新部署项目 (Retry deployment) 以生效。
---
## 🔍 技术实现细节
广告检测逻辑位于 `lib/utils/m3u8-ad-detector.ts`,采用分块评分机制:
1. **Block 解析**:将 M3U8 播放列表按 `#EXT-X-DISCONTINUITY` 分割成多个块。
2. **特征学习**:分析最长的块(通常是正片)提取特征,如文件命名模式、路径前缀等。
3. **评分系统**
- **HLS 标签**`#EXT-X-CUE-OUT`/`IN` (+10分,确认为广告)
- **路径前缀**:与正片路径不一致 (+5分,高度疑似)
- **关键词**URL 包含已知广告词 (+2.5分)
- **文件名模式**:与正片命名规则不符 (+1.5分)
4. **决策过滤**
- **智能模式**:移除评分 >= 5.0 的块。
- **激进模式**:移除评分 >= 3.0 的块。
+5
View File
@@ -83,6 +83,11 @@
- **内容隔离**:高级内容与普通内容完全物理隔离,互不干扰
- **专属设置**:拥有独立的内容源管理和功能设置
### 🛡️ 广告过滤
- **多模式选择**:支持关闭、关键词过滤、智能启发式过滤(Beta)和激进模式。
- **UI 集成**:在播放器设置菜单中直接切换模式,实时生效。
- **自定义关键词**:支持通过环境变量扩展过滤关键词。
- **高性能**:基于流式处理,对播放加载速度几乎无影响。
## 🔐 隐私保护
+43
View File
@@ -1,3 +1,4 @@
import React from 'react';
import type { Metadata } from "next";
import { Geist, Geist_Mono } from "next/font/google";
import "./globals.css";
@@ -6,7 +7,48 @@ import { Analytics } from "@vercel/analytics/react";
import { ServiceWorkerRegister } from "@/components/ServiceWorkerRegister";
import { PasswordGate } from "@/components/PasswordGate";
import { siteConfig } from "@/lib/config/site-config";
import { AdKeywordsInjector } from "@/components/AdKeywordsInjector";
import fs from 'fs';
import path from 'path';
// Server Component specifically for reading env/file (async for best practices)
async function AdKeywordsWrapper() {
let keywords: string[] = [];
try {
// 1. Try reading from file (Docker runtime support)
const keywordsFile = process.env.AD_KEYWORDS_FILE;
if (keywordsFile) {
// Resolve absolute path or relative to CWD
const filePath = path.isAbsolute(keywordsFile)
? keywordsFile
: path.join(process.cwd(), keywordsFile);
try {
const content = await fs.promises.readFile(filePath, 'utf-8');
keywords = content.split(/[\n,]/).map((k: string) => k.trim()).filter((k: string) => k);
console.log(`[AdFilter] Loaded ${keywords.length} keywords from file: ${filePath}`);
} catch (fileError: unknown) {
// Handle file not found (ENOENT) gracefully
if ((fileError as NodeJS.ErrnoException).code !== 'ENOENT') {
console.warn('[AdFilter] Error reading keywords file:', fileError);
}
}
}
// 2. Fallback to Env var (Runtime or Build time)
if (keywords.length === 0) {
const envKeywords = process.env.AD_KEYWORDS || process.env.NEXT_PUBLIC_AD_KEYWORDS;
if (envKeywords) {
keywords = envKeywords.split(/[\n,]/).map((k: string) => k.trim()).filter((k: string) => k);
}
}
} catch (error) {
console.warn('[AdFilter] Failed to load keywords:', error);
}
return <AdKeywordsInjector keywords={keywords} />;
}
const geistSans = Geist({
variable: "--font-geist-sans",
@@ -39,6 +81,7 @@ export default function RootLayout({
>
<ThemeProvider>
<PasswordGate hasEnvPassword={!!process.env.ACCESS_PASSWORD}>
<AdKeywordsWrapper />
{children}
</PasswordGate>
<Analytics />
+22
View File
@@ -0,0 +1,22 @@
'use client';
import { useEffect, useRef } from 'react';
import { usePlayerSettings } from '@/components/player/hooks/usePlayerSettings';
interface AdKeywordsInjectorProps {
keywords: string[];
}
export function AdKeywordsInjector({ keywords }: AdKeywordsInjectorProps) {
const { setAdKeywords } = usePlayerSettings();
const initialized = useRef(false);
useEffect(() => {
if (!initialized.current && keywords.length > 0) {
setAdKeywords(keywords);
initialized.current = true;
}
}, [keywords, setAdKeywords]);
return null;
}
+54 -1
View File
@@ -3,6 +3,7 @@
import React from 'react';
import { Icons } from '@/components/ui/Icon';
import { usePlayerSettings } from '../hooks/usePlayerSettings';
import { settingsStore, AdFilterMode } from '@/lib/store/settings-store';
import { createPortal } from 'react-dom';
@@ -32,16 +33,28 @@ export function DesktopMoreMenu({
autoSkipOutro,
skipOutroSeconds,
showModeIndicator,
adFilter,
setAutoNextEpisode,
setAutoSkipIntro,
setSkipIntroSeconds,
setAutoSkipOutro,
setSkipOutroSeconds,
setShowModeIndicator,
setAdFilter,
adFilterMode,
setAdFilterMode,
} = usePlayerSettings();
const buttonRef = React.useRef<HTMLButtonElement>(null);
const [menuPosition, setMenuPosition] = React.useState({ top: 0, left: 0 });
const [isAdFilterOpen, setAdFilterOpen] = React.useState(false);
const AD_FILTER_LABELS: Record<string, string> = {
off: '关闭',
keyword: '关键词',
heuristic: '智能(Beta)',
aggressive: '激进'
};
React.useEffect(() => {
if (showMoreMenu && buttonRef.current && containerRef.current) {
@@ -130,6 +143,46 @@ export function DesktopMoreMenu({
</button>
</div>
{/* Ad Filter Mode Selector */}
<div className="px-4 py-2.5 flex items-center justify-between">
<div className="flex items-center gap-3 text-sm text-[var(--text-color)]">
<Icons.ShieldAlert size={18} />
<span>广</span>
</div>
{/* Custom Ad Filter Mode Selector */}
<div className="relative">
<button
onClick={() => setAdFilterOpen(!isAdFilterOpen)}
className="flex items-center gap-1.5 bg-[var(--glass-bg)] border border-[var(--glass-border)] text-[var(--text-color)] text-xs rounded-md px-2.5 py-1.5 outline-none hover:border-[var(--accent-color)] hover:bg-[color-mix(in_srgb,var(--accent-color)_5%,transparent)] transition-all cursor-pointer"
>
<span>{AD_FILTER_LABELS[adFilterMode] || '关闭'}</span>
<Icons.ChevronDown size={14} className={`text-[var(--text-color-secondary)] transition-transform duration-200 ${isAdFilterOpen ? 'rotate-180' : ''}`} />
</button>
{isAdFilterOpen && (
<>
<div className="fixed inset-0 z-10 cursor-default" onClick={() => setAdFilterOpen(false)} />
<div className="absolute right-0 top-full mt-1 w-28 bg-[var(--glass-bg)] backdrop-blur-xl border border-[var(--glass-border)] rounded-lg shadow-xl overflow-hidden z-20 flex flex-col animate-in fade-in zoom-in-95 duration-150">
{Object.entries(AD_FILTER_LABELS).map(([mode, label]) => (
<button
key={mode}
onClick={() => {
setAdFilterMode(mode as AdFilterMode);
setAdFilterOpen(false);
}}
className={`text-left text-xs px-3 py-2.5 hover:bg-[color-mix(in_srgb,var(--accent-color)_15%,transparent)] transition-colors w-full flex items-center justify-between group ${adFilterMode === mode ? 'text-[var(--accent-color)] font-medium bg-[color-mix(in_srgb,var(--accent-color)_5%,transparent)]' : 'text-[var(--text-color)]'
}`}
>
<span>{label}</span>
{adFilterMode === mode && <Icons.Check size={12} className="text-[var(--accent-color)]" />}
</button>
))}
</div>
</>
)}
</div>
</div>
{/* Auto Next Episode Switch */}
<div className="px-4 py-2.5 flex items-center justify-between">
<div className="flex items-center gap-3 text-sm text-[var(--text-color)]">
@@ -225,7 +278,7 @@ export function DesktopMoreMenu({
</div>
)}
</div>
</div>
</div >
);
return (
+210 -53
View File
@@ -1,5 +1,7 @@
import { useEffect, useRef } from 'react';
import Hls from 'hls.js';
import { usePlayerSettings } from './usePlayerSettings';
import { filterM3u8Ad } from '@/lib/utils/m3u8-utils';
interface UseHlsPlayerProps {
videoRef: React.RefObject<HTMLVideoElement | null>;
@@ -17,6 +19,8 @@ export function useHlsPlayer({
onError
}: UseHlsPlayerProps) {
const hlsRef = useRef<Hls | null>(null);
const { adFilterMode, adKeywords } = usePlayerSettings();
const isAdFilterEnabled = adFilterMode !== 'off';
useEffect(() => {
const video = videoRef.current;
@@ -29,45 +33,66 @@ export function useHlsPlayer({
}
let hls: Hls | null = null;
let extraBlobs: string[] = [];
// Check if HLS is supported natively (Safari, Mobile Chrome)
// We prefer native playback if available as it's usually more battery efficient
const isNativeHlsSupported = video.canPlayType('application/vnd.apple.mpegurl');
if (Hls.isSupported()) {
// Use hls.js for browsers without native support (Desktop Chrome, Firefox, Edge)
// OR if we want to force hls.js for better control (optional, but sticking to native first is safer)
// Note: Some desktop browsers (like Safari) support native HLS.
// We usually prefer native, BUT sometimes native implementation is buggy or lacks features.
// For now, we follow the standard pattern: Native first, then HLS.js.
// EXCEPT for Chrome on Desktop which reports canPlayType as '' (false).
// Define custom loader class to intercept manifest loading
// We use 'any' cast because default loader type might not be strictly exposed in all typings
const DefaultLoader = (Hls as any).DefaultConfig.loader;
if (!isNativeHlsSupported) {
hls = new Hls({
class AdFilterLoader extends DefaultLoader {
load(context: any, config: any, callbacks: any) {
if (isAdFilterEnabled && (context.type === 'manifest' || context.type === 'level')) {
const originalOnSuccess = callbacks.onSuccess;
callbacks.onSuccess = (response: any, stats: any, context: any, networkDetails: any) => {
if (typeof response.data === 'string') {
try {
// Filter the content
response.data = filterM3u8Ad(response.data, context.url, adFilterMode, adKeywords);
} catch (e) {
console.warn('[HLS] Ad filter error:', e);
}
}
originalOnSuccess(response, stats, context, networkDetails);
};
}
super.load(context, config, callbacks);
}
}
if (!isNativeHlsSupported || isAdFilterEnabled) {
// If ad filtering is on, we force Hls.js even on native-supported desktop browsers
// Exceptions might exist for iOS where MSE is strictly not available, check Hls.isSupported() result carefully.
// Hls.isSupported() is false on iOS Safari usually, so this block won't run there.
const config: any = {
// Worker & Performance
enableWorker: true,
lowLatencyMode: false, // Disable low latency for more stable playback
lowLatencyMode: false,
// Buffer Settings - More aggressive buffering for smoother playback
maxBufferLength: 60, // Buffer up to 60 seconds ahead
maxMaxBufferLength: 120, // Allow up to 2 minutes of buffer
maxBufferSize: 60 * 1000 * 1000, // 60MB buffer size
maxBufferHole: 0.5, // Allow small gaps in buffer
// Buffer Settings
maxBufferLength: 60,
maxMaxBufferLength: 120,
maxBufferSize: 60 * 1000 * 1000,
maxBufferHole: 0.5,
// Start with more buffer before playing
startFragPrefetch: true, // Enable prefetching next fragment
// Start with more buffer
startFragPrefetch: true,
// ABR (Adaptive Bitrate) Settings - Be more conservative
abrEwmaDefaultEstimate: 500000, // Start with conservative bandwidth estimate (500kbps)
abrEwmaFastLive: 3, // Fast adaptation for live
abrEwmaSlowLive: 9, // Slow adaptation for live
abrEwmaFastVoD: 3, // Fast adaptation for VoD
abrEwmaSlowVoD: 9, // Slow adaptation for VoD
abrBandWidthFactor: 0.8, // Use 80% of estimated bandwidth (conservative)
abrBandWidthUpFactor: 0.7, // Even more conservative when switching up
// ABR Settings
abrEwmaDefaultEstimate: 500000,
abrEwmaFastLive: 3,
abrEwmaSlowLive: 9,
abrEwmaFastVoD: 3,
abrEwmaSlowVoD: 9,
abrBandWidthFactor: 0.8,
abrBandWidthUpFactor: 0.7,
// Loading Settings - More retries and longer timeouts
// Loading Settings
fragLoadingMaxRetry: 6,
fragLoadingRetryDelay: 1000,
fragLoadingMaxRetryTimeout: 64000,
@@ -79,29 +104,35 @@ export function useHlsPlayer({
levelLoadingMaxRetryTimeout: 64000,
// Timeouts
fragLoadingTimeOut: 20000, // 20 seconds for fragment loading
manifestLoadingTimeOut: 10000, // 10 seconds for manifest
levelLoadingTimeOut: 10000, // 10 seconds for level
fragLoadingTimeOut: 20000,
manifestLoadingTimeOut: 10000,
levelLoadingTimeOut: 10000,
// Backbuffer - Keep some played content for seeking back
backBufferLength: 30, // Keep 30 seconds of played content
});
// Backbuffer
backBufferLength: 30,
};
// Use custom loader if ad filtering is enabled
if (isAdFilterEnabled) {
config.loader = AdFilterLoader;
}
hls = new Hls(config);
hlsRef.current = hls;
hls.loadSource(src);
hls.attachMedia(video);
// Auto Play Handler
hls.on(Hls.Events.FRAG_LOADED, (event, data) => {
// Force play if we have the first segment and it's not playing yet
// detailed: data.frag.sn is the sequence number
if (autoPlay && video.paused && data.frag.start === 0) {
video.play().catch(console.warn);
}
});
// Manifest Parsed Handler
hls.on(Hls.Events.MANIFEST_PARSED, () => {
// Check for HEVC/H.265 codec (limited browser support)
// Check for HEVC
if (hls) {
const levels = hls.levels;
if (levels && levels.length > 0) {
@@ -110,10 +141,7 @@ export function useHlsPlayer({
level.videoCodec?.toLowerCase().includes('h265')
);
if (hasHEVC) {
console.warn('[HLS] ⚠️ HEVC/H.265 codec detected - may not play in all browsers');
console.warn('[HLS] Supported: Safari with hardware acceleration, some Edge versions');
console.warn('[HLS] Not supported: Most Chrome/Firefox versions');
// Notify parent about potential codec issues
console.warn('[HLS] ⚠️ HEVC detected');
onError?.('检测到 HEVC/H.265 编码,当前浏览器可能不支持');
}
}
@@ -121,12 +149,13 @@ export function useHlsPlayer({
if (autoPlay) {
video.play().catch((err) => {
console.warn('[HLS] Autoplay prevented:', err);
// console.warn('[HLS] Autoplay prevented:', err);
onAutoPlayPrevented?.(err);
});
}
});
// Error Handling
let networkErrorRetries = 0;
let mediaErrorRetries = 0;
const MAX_RETRIES = 3;
@@ -136,22 +165,18 @@ export function useHlsPlayer({
switch (data.type) {
case Hls.ErrorTypes.NETWORK_ERROR:
networkErrorRetries++;
console.error(`[HLS] Network error (${networkErrorRetries}/${MAX_RETRIES}), trying to recover...`, data);
if (networkErrorRetries <= MAX_RETRIES) {
hls?.startLoad();
} else {
console.error('[HLS] Too many network errors, giving up');
onError?.('网络错误:无法加载视频流');
hls?.destroy();
}
break;
case Hls.ErrorTypes.MEDIA_ERROR:
mediaErrorRetries++;
console.error(`[HLS] Media error (${mediaErrorRetries}/${MAX_RETRIES}), trying to recover...`, data);
if (mediaErrorRetries <= MAX_RETRIES) {
hls?.recoverMediaError();
} else {
console.error('[HLS] Too many media errors, giving up');
onError?.('媒体错误:视频格式不支持或已损坏');
hls?.destroy();
}
@@ -162,20 +187,151 @@ export function useHlsPlayer({
hls?.destroy();
break;
}
} else {
// Non-fatal errors
console.warn('[HLS] Non-fatal error:', data.type, data.details);
}
});
} else {
// Native HLS support
// Native HLS (Desktop Safari, no Filter)
video.src = src;
}
} else if (isNativeHlsSupported) {
// Fallback for environments where Hls.js is not supported but native is (e.g. iOS without MSE?)
video.src = src;
// Native HLS (iOS, Mobile Safari)
// Limitations: Native HLS cannot easily intercept sub-playlist requests.
// We use fetch+blob for the master playlist as a best 'first-level' filter.
// If the ad discontinuity is in the master playlist (rare for ads, common for periods), it works.
// If it's in sub-playlists, it might fail unless we parse and blob those too (complex).
if (isAdFilterEnabled) {
const processMasterPlaylist = async (masterSrc: string) => {
// Move blob tracking outside try to ensure cleanup on error
const createdBlobs: string[] = [];
// Safely resolve relative URLs to absolute (handles iOS Safari scenarios)
let absoluteMasterSrc: string;
try {
absoluteMasterSrc = new URL(masterSrc, window.location.href).toString();
} catch {
absoluteMasterSrc = masterSrc; // Fallback if URL parsing fails
}
try {
const response = await fetch(absoluteMasterSrc);
const masterContent = await response.text();
// If it's a simple playlist (no variants), just filter and play
if (!masterContent.includes('#EXT-X-STREAM-INF')) {
const filtered = filterM3u8Ad(masterContent, absoluteMasterSrc, adFilterMode, adKeywords);
const blob = new Blob([filtered], { type: 'application/vnd.apple.mpegurl' });
const blobUrl = URL.createObjectURL(blob);
createdBlobs.push(blobUrl);
return { masterBlobUrl: blobUrl, allBlobs: createdBlobs };
}
// It IS a master playlist. Use map + Promise.all for clean concurrent processing.
const lines = masterContent.split(/\r?\n/);
// Helper to safely compare origins
const isSameOrigin = (url: string): boolean => {
try {
return new URL(url).origin === new URL(absoluteMasterSrc).origin;
} catch {
return false;
}
};
// Process each line, looking back at previous line to determine context
const lineProcessingPromises = lines.map(async (line, index) => {
const trimmedLine = line.trim();
// Handle #EXT-X-MEDIA:URI="..."
if (trimmedLine.startsWith('#EXT-X-MEDIA') && trimmedLine.includes('URI="')) {
const uriMatch = trimmedLine.match(/URI="([^"]+)"/);
const uri = uriMatch?.[1];
if (uri) {
// Process if relative URL or same-origin absolute URL
const isRelative = !uri.startsWith('http');
const sameOrigin = uri.startsWith('http') && isSameOrigin(uri);
if (isRelative || sameOrigin) {
try {
const absoluteUrl = isRelative ? new URL(uri, absoluteMasterSrc).toString() : uri;
const subRes = await fetch(absoluteUrl);
const subContent = await subRes.text();
const filteredSub = filterM3u8Ad(subContent, absoluteUrl, adFilterMode, adKeywords);
const subBlob = new Blob([filteredSub], { type: 'application/vnd.apple.mpegurl' });
const subBlobUrl = URL.createObjectURL(subBlob);
createdBlobs.push(subBlobUrl);
return line.replace(`URI="${uri}"`, `URI="${subBlobUrl}"`);
} catch (e) {
console.warn('[HLS Native] Failed to process EXT-X-MEDIA URI:', e);
return line;
}
}
}
}
// Handle playlist URL (line after #EXT-X-STREAM-INF)
const prevLine = index > 0 ? lines[index - 1].trim() : '';
if (prevLine.startsWith('#EXT-X-STREAM-INF') && trimmedLine && !trimmedLine.startsWith('#')) {
// Process if relative URL or same-origin absolute URL
const isRelative = !trimmedLine.startsWith('http');
const sameOrigin = trimmedLine.startsWith('http') && isSameOrigin(trimmedLine);
if (isRelative || sameOrigin) {
try {
const absoluteUrl = isRelative ? new URL(trimmedLine, absoluteMasterSrc).toString() : trimmedLine;
const subRes = await fetch(absoluteUrl);
const subContent = await subRes.text();
const filteredSub = filterM3u8Ad(subContent, absoluteUrl, adFilterMode, adKeywords);
const subBlob = new Blob([filteredSub], { type: 'application/vnd.apple.mpegurl' });
const subBlobUrl = URL.createObjectURL(subBlob);
createdBlobs.push(subBlobUrl);
return subBlobUrl;
} catch (e) {
console.warn('[HLS Native] Failed to process variant playlist:', e);
return line;
}
}
}
// All other lines pass through unchanged
return line;
});
const processedLines = await Promise.all(lineProcessingPromises);
// Join back
const finalMasterContent = processedLines.join('\n');
const masterBlob = new Blob([finalMasterContent], { type: 'application/vnd.apple.mpegurl' });
const masterBlobUrl = URL.createObjectURL(masterBlob);
createdBlobs.push(masterBlobUrl);
return { masterBlobUrl, allBlobs: createdBlobs };
} catch (e) {
// Critical: Clean up any blobs created before the error
for (const blobUrl of createdBlobs) {
try {
URL.revokeObjectURL(blobUrl);
} catch { /* ignore cleanup errors */ }
}
console.error('[HLS Native] Recursive fetch failed', e);
throw e;
}
};
processMasterPlaylist(src).then((result) => {
video.src = result.masterBlobUrl;
extraBlobs = result.allBlobs;
}).catch((e) => {
console.warn('[HLS Native] Ad filtering failed, falling back to original source.', e);
onError?.('广告过滤失败,已回退到原始视频流');
video.src = src;
});
} else {
video.src = src;
}
} else {
console.error('[HLS] HLS not supported in this browser');
console.error('[HLS] HLS not supported');
onError?.('当前浏览器不支持 HLS 视频播放');
}
@@ -183,6 +339,7 @@ export function useHlsPlayer({
if (hls) {
hls.destroy();
}
extraBlobs.forEach(url => URL.revokeObjectURL(url));
};
}, [src, videoRef, autoPlay, onAutoPlayPrevented, onError]);
}, [src, videoRef, autoPlay, onAutoPlayPrevented, onError, isAdFilterEnabled, adFilterMode, adKeywords]);
}
+22 -1
View File
@@ -1,7 +1,7 @@
'use client';
import { useState, useEffect, useCallback } from 'react';
import { settingsStore } from '@/lib/store/settings-store';
import { settingsStore, AdFilterMode } from '@/lib/store/settings-store';
/**
* Hook to access and update player settings from the settings store
@@ -17,6 +17,9 @@ export function usePlayerSettings() {
autoSkipOutro: stored.autoSkipOutro,
skipOutroSeconds: stored.skipOutroSeconds,
showModeIndicator: stored.showModeIndicator,
adFilter: stored.adFilter,
adFilterMode: stored.adFilterMode,
adKeywords: stored.adKeywords,
};
});
@@ -31,6 +34,9 @@ export function usePlayerSettings() {
autoSkipOutro: stored.autoSkipOutro,
skipOutroSeconds: stored.skipOutroSeconds,
showModeIndicator: stored.showModeIndicator,
adFilter: stored.adFilter,
adFilterMode: stored.adFilterMode,
adKeywords: stored.adKeywords,
});
});
return unsubscribe;
@@ -71,6 +77,18 @@ export function usePlayerSettings() {
updateSetting('showModeIndicator', value);
}, [updateSetting]);
const setAdFilter = useCallback((value: boolean) => {
updateSetting('adFilter', value);
}, [updateSetting]);
const setAdFilterMode = useCallback((value: AdFilterMode) => {
updateSetting('adFilterMode', value);
}, [updateSetting]);
const setAdKeywords = useCallback((value: string[]) => {
updateSetting('adKeywords', value);
}, [updateSetting]);
return {
...settings,
setAutoNextEpisode,
@@ -79,5 +97,8 @@ export function usePlayerSettings() {
setAutoSkipOutro,
setSkipOutroSeconds,
setShowModeIndicator,
setAdFilter,
setAdFilterMode,
setAdKeywords,
};
}
+8
View File
@@ -15,6 +15,14 @@ export const UtilityIcons = {
</svg>
),
ShieldAlert: ({ className = "", size = 24 }: IconProps) => (
<svg width={size} height={size} viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
<path d="M12 22s8-4 8-10V5l-8-3-8 3v7c0 6 8 10 8 10z" />
<line x1="12" y1="8" x2="12" y2="12" />
<line x1="12" y1="16" x2="12.01" y2="16" />
</svg>
),
Sparkles: ({ className = "", size = 24 }: IconProps) => (
<svg width={size} height={size} viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round" className={className}>
<path d="M12 3v18M5.2 8.2l13.6 7.6M18.8 8.2L5.2 15.8" />
+38 -59
View File
@@ -18,6 +18,7 @@ export type SortOption =
| 'name-desc';
export type SearchDisplayMode = 'normal' | 'grouped';
export type AdFilterMode = 'off' | 'keyword' | 'heuristic' | 'aggressive';
export interface AppSettings {
sources: VideoSource[];
@@ -35,6 +36,9 @@ export interface AppSettings {
autoSkipOutro: boolean;
skipOutroSeconds: number;
showModeIndicator: boolean; // Show '直连模式'/'代理模式' badge on player
adFilter: boolean; // Filter ad tags from m3u8 (legacy, kept for compatibility)
adFilterMode: AdFilterMode; // 'off' | 'keyword' | 'heuristic' | 'aggressive'
adKeywords: string[]; // Dynamically loaded ad keywords
// Search & Display settings
realtimeLatency: boolean; // Enable real-time latency ping updates
searchDisplayMode: SearchDisplayMode; // 'normal' = individual cards, 'grouped' = group same-name videos
@@ -87,51 +91,43 @@ function getEnvSubscriptions(customValue?: string): SourceSubscription[] {
// Debugging helper
// console.log("Environment Subscriptions:", getEnvSubscriptions());
// Shared default settings factory to avoid code duplication
function getDefaultAppSettings(): AppSettings {
return {
sources: getDefaultSources(),
premiumSources: getDefaultPremiumSources(),
subscriptions: getEnvSubscriptions(),
sortBy: 'default',
searchHistory: true,
watchHistory: true,
passwordAccess: false,
accessPasswords: [],
autoNextEpisode: true,
autoSkipIntro: false,
skipIntroSeconds: 0,
autoSkipOutro: false,
skipOutroSeconds: 0,
showModeIndicator: false,
adFilter: false,
adFilterMode: 'heuristic',
adKeywords: [],
realtimeLatency: false,
searchDisplayMode: 'normal',
episodeReverseOrder: false,
};
}
export const settingsStore = {
getSettings(): AppSettings {
// SSR: Return defaults
if (typeof window === 'undefined') {
return {
sources: getDefaultSources(),
premiumSources: getDefaultPremiumSources(),
subscriptions: getEnvSubscriptions(),
sortBy: 'default',
searchHistory: true,
watchHistory: true,
passwordAccess: false,
accessPasswords: [],
autoNextEpisode: true,
autoSkipIntro: false,
skipIntroSeconds: 0,
autoSkipOutro: false,
skipOutroSeconds: 0,
showModeIndicator: false,
realtimeLatency: false,
searchDisplayMode: 'normal',
episodeReverseOrder: false,
};
return getDefaultAppSettings();
}
// Client: No stored settings, return defaults
const stored = localStorage.getItem(SETTINGS_KEY);
if (!stored) {
return {
sources: getDefaultSources(),
premiumSources: getDefaultPremiumSources(),
subscriptions: getEnvSubscriptions(),
sortBy: 'default',
searchHistory: true,
watchHistory: true,
passwordAccess: false,
accessPasswords: [],
autoNextEpisode: true,
autoSkipIntro: false,
skipIntroSeconds: 0,
autoSkipOutro: false,
skipOutroSeconds: 0,
showModeIndicator: false,
realtimeLatency: false,
searchDisplayMode: 'normal',
episodeReverseOrder: false,
};
return getDefaultAppSettings();
}
try {
@@ -187,33 +183,16 @@ export const settingsStore = {
autoSkipOutro: parsed.autoSkipOutro !== undefined ? parsed.autoSkipOutro : false,
skipOutroSeconds: typeof parsed.skipOutroSeconds === 'number' ? parsed.skipOutroSeconds : 0,
showModeIndicator: parsed.showModeIndicator !== undefined ? parsed.showModeIndicator : false,
adFilter: parsed.adFilter !== undefined ? parsed.adFilter : false,
adFilterMode: parsed.adFilterMode || 'heuristic',
adKeywords: Array.isArray(parsed.adKeywords) ? parsed.adKeywords : [],
realtimeLatency: parsed.realtimeLatency !== undefined ? parsed.realtimeLatency : false,
searchDisplayMode: parsed.searchDisplayMode === 'grouped' ? 'grouped' : 'normal',
episodeReverseOrder: parsed.episodeReverseOrder !== undefined ? parsed.episodeReverseOrder : false,
};
} catch {
// Even if localStorage fails, we should return defaults + ENV subscriptions
const envSubscriptions = getEnvSubscriptions();
return {
sources: getDefaultSources(),
premiumSources: getDefaultPremiumSources(),
subscriptions: envSubscriptions,
sortBy: 'default',
searchHistory: true,
watchHistory: true,
passwordAccess: false,
accessPasswords: [],
autoNextEpisode: true,
autoSkipIntro: false,
skipIntroSeconds: 0,
autoSkipOutro: false,
skipOutroSeconds: 0,
showModeIndicator: false,
realtimeLatency: false,
searchDisplayMode: 'normal',
episodeReverseOrder: false,
};
return getDefaultAppSettings();
}
},
+265
View File
@@ -0,0 +1,265 @@
/**
* Heuristic Ad Detection Module
*
* Provides block-based analysis for detecting ads in M3U8 playlists
* using filename pattern matching and other heuristics.
*/
// Ad-related path keywords for scoring
export const AD_PATH_KEYWORDS = [
'advert', 'preroll', 'midroll', 'postroll',
'dai', 'vast', 'ima', 'adjump', 'commercial', 'sponsor'
];
/**
* Represents a segment in the playlist
*/
interface Segment {
url: string;
duration: number;
lineIndex: number;
}
/**
* Represents a block of segments between DISCONTINUITY markers
*/
interface Block {
segments: Segment[];
startLineIndex: number;
endLineIndex: number;
hasCueTag: boolean;
}
/**
* Pattern extracted from main content for comparison
*/
interface MainPattern {
filenameRegex: RegExp | null;
avgDuration: number;
commonPrefix: string;
pathPrefix: string; // Directory path prefix (e.g., "/20230907/73PWifvT/1392kb/hls/")
}
/**
* Parse M3U8 content into blocks separated by DISCONTINUITY markers
*/
export function parseBlocks(lines: string[]): Block[] {
const blocks: Block[] = [];
let currentBlock: Block = {
segments: [],
startLineIndex: 0,
endLineIndex: 0,
hasCueTag: false
};
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
// Check for CUE tags
if (line.startsWith('#EXT-X-CUE-OUT') || line.startsWith('#EXT-X-CUE-IN')) {
currentBlock.hasCueTag = true;
}
// DISCONTINUITY marks block boundary
if (line === '#EXT-X-DISCONTINUITY') {
if (currentBlock.segments.length > 0) {
currentBlock.endLineIndex = i - 1;
blocks.push(currentBlock);
}
currentBlock = {
segments: [],
startLineIndex: i + 1,
endLineIndex: 0,
hasCueTag: false
};
continue;
}
// Parse EXTINF and the following URL
if (line.startsWith('#EXTINF:')) {
const durationMatch = line.match(/#EXTINF:([\d.]+)/);
const duration = durationMatch ? parseFloat(durationMatch[1]) : 0;
// Next line should be the URL
if (i + 1 < lines.length) {
const url = lines[i + 1].trim();
if (url && !url.startsWith('#')) {
currentBlock.segments.push({
url,
duration,
lineIndex: i + 1
});
}
}
}
}
// Don't forget the last block
if (currentBlock.segments.length > 0) {
currentBlock.endLineIndex = lines.length - 1;
blocks.push(currentBlock);
}
return blocks;
}
/**
* Extract filename from URL (handles both relative and absolute URLs)
*/
function extractFilename(url: string): string {
try {
const path = url.includes('://') ? new URL(url).pathname : url;
const parts = path.split('/');
return parts[parts.length - 1] || '';
} catch {
return url.split('/').pop() || '';
}
}
/**
* Find common prefix among an array of strings
*/
function findCommonPrefix(strings: string[]): string {
if (!strings || strings.length < 2) return '';
let prefix = '';
const first = strings[0];
for (let i = 0; i < first.length; i++) {
const char = first[i];
if (strings.every(s => s[i] === char)) {
prefix += char;
} else {
break;
}
}
return prefix;
}
/**
* Extract path prefix (directory) from URL
* e.g., "/20230907/73PWifvT/1392kb/hls/" from "/20230907/73PWifvT/1392kb/hls/gFE6lwIk.ts"
*/
function extractPathPrefix(url: string): string {
try {
const path = url.includes('://') ? new URL(url).pathname : url;
const lastSlash = path.lastIndexOf('/');
return lastSlash >= 0 ? path.substring(0, lastSlash + 1) : '';
} catch {
const lastSlash = url.lastIndexOf('/');
return lastSlash >= 0 ? url.substring(0, lastSlash + 1) : '';
}
}
/**
* Learn pattern from the largest block (assumed to be main content)
*/
export function learnMainPattern(blocks: Block[]): MainPattern {
// Find the largest block by segment count (likely main content)
const mainBlock = blocks.length > 0 ? blocks.reduce((largest, block) =>
block.segments.length > largest.segments.length ? block : largest
) : null;
if (!mainBlock || mainBlock.segments.length === 0) {
return { filenameRegex: null, avgDuration: 0, commonPrefix: '', pathPrefix: '' };
}
// Extract filenames
const filenames = mainBlock.segments.map(s => extractFilename(s.url));
// Find common prefix
const commonPrefix = findCommonPrefix(filenames);
// Calculate average duration
const totalDuration = mainBlock.segments.reduce((sum, s) => sum + s.duration, 0);
const avgDuration = totalDuration / mainBlock.segments.length;
// Try to build a regex pattern from the filenames
// Common patterns: "0000001.ts", "seg-1.ts", "segment_001.ts"
let filenameRegex: RegExp | null = null;
if (commonPrefix.length >= 2) {
// Escape special regex characters in prefix
const escapedPrefix = commonPrefix.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
filenameRegex = new RegExp(`^${escapedPrefix}`);
}
// Extract path prefix (directory path without filename)
// e.g., "/20230907/73PWifvT/1392kb/hls/" from "/20230907/73PWifvT/1392kb/hls/gFE6lwIk.ts"
const firstUrl = mainBlock.segments[0].url;
const pathPrefix = extractPathPrefix(firstUrl);
return { filenameRegex, avgDuration, commonPrefix, pathPrefix };
}
/**
* Score a block for ad likelihood based on heuristics
* Returns a score where higher = more likely to be an ad
*/
export function scoreBlock(block: Block, mainPattern: MainPattern, extraKeywords: string[] = []): number {
let score = 0;
// If block has CUE tag, it's definitely an ad
if (block.hasCueTag) {
return 10; // Max score
}
// Check path keywords (Built-in + Custom)
// We filter out very short custom keywords to avoid false positives in scoring
const safeExtraKeywords = extraKeywords.filter(k => k.length > 2);
const allKeywords = [...AD_PATH_KEYWORDS, ...safeExtraKeywords];
for (const segment of block.segments) {
const urlLower = segment.url.toLowerCase();
for (const keyword of allKeywords) {
if (urlLower.includes(keyword.toLowerCase())) {
score += 2.5;
break; // Only count once per segment
}
}
}
// Check filename pattern mismatch
if (mainPattern.filenameRegex) {
const mismatchCount = block.segments.filter(s => {
if (!mainPattern.filenameRegex) return false; // No pattern to compare against
const filename = extractFilename(s.url);
return !mainPattern.filenameRegex.test(filename);
}).length;
if (mismatchCount === block.segments.length && block.segments.length > 0) {
score += 1.5; // All filenames differ from main pattern
}
}
// **KEY FEATURE**: Check path prefix mismatch (e.g., different date/folder/bitrate)
// This is the most reliable indicator for ads that come from different CDN paths
if (mainPattern.pathPrefix && block.segments.length > 0) {
const pathMismatchCount = block.segments.filter(s => {
const segmentPathPrefix = extractPathPrefix(s.url);
return segmentPathPrefix !== mainPattern.pathPrefix;
}).length;
if (pathMismatchCount === block.segments.length) {
// ALL segments have different path prefix - strong ad indicator
score += 5.0;
}
}
return score;
}
/**
* Threshold configuration
*/
export const THRESHOLDS = {
HIGH: 5.0, // Definitely an ad
LOW: 3.0 // Possibly an ad (for future "fuzzy" mode)
};
/**
* Determine if a block should be filtered based on its score
*/
export function shouldFilterBlock(score: number, threshold: number = THRESHOLDS.HIGH): boolean {
return score >= threshold;
}
+162
View File
@@ -0,0 +1,162 @@
/**
* Utility functions for M3U8 playlist manipulation
*/
import { parseBlocks, learnMainPattern, scoreBlock, shouldFilterBlock } from './m3u8-ad-detector';
/**
* Filters ads from specific M3U8 content using multiple detection strategies:
* 1. Keyword matching (configurable via env)
* 2. CUE-OUT/CUE-IN standard tags
* 3. Heuristic block analysis (filename patterns, ad path keywords)
*
* Also converts relative URLs to absolute URLs for Blob playback.
*
* @param content The raw M3U8 content string
* @param baseUrl The base URL of the M3U8 file (to resolve relative paths)
* @returns The filtered M3U8 content
*/
export type AdFilterMode = 'off' | 'keyword' | 'heuristic' | 'aggressive';
export function filterM3u8Ad(content: string, baseUrl: string, mode: AdFilterMode = 'heuristic', customKeywords: string[] = []): string {
if (!content) return '';
// 1. Performance: Parse Keywords & Base URL ONLY ONCE
// Merge env keywords (build time) with custom keywords (runtime injected)
const envKeywordsStr = process.env.NEXT_PUBLIC_AD_KEYWORDS || '';
const envKeywords = envKeywordsStr.split(/[\n,]/).map(k => k.trim()).filter(k => k);
// Combine and deduplicate
const keywords = Array.from(new Set([...envKeywords, ...customKeywords]));
const basePath = baseUrl.substring(0, baseUrl.lastIndexOf('/') + 1);
let origin = '';
try {
origin = new URL(baseUrl).origin;
} catch (e) { /* ignore */ }
// 2. Global Scan: Check if any ad keywords exist in the content
const hasKeywordMatch = mode !== 'off' && keywords.some(k => content.includes(k));
const hasCueTag = mode !== 'off' && (content.includes('#EXT-X-CUE-OUT') || content.includes('#EXT-X-CUE-IN'));
// 3. Heuristic Analysis: If no explicit ad signals, use block-based detection
const lines = content.split(/\r?\n/);
let adLineIndices = new Set<number>();
if (!hasCueTag && (mode === 'heuristic' || mode === 'aggressive')) {
// No obvious ad signals - run heuristic analysis
const blocks = parseBlocks(lines);
if (blocks.length > 1) {
const mainPattern = learnMainPattern(blocks);
for (const block of blocks) {
// Pass all keywords (including custom ones) to heuristic scorer
const score = scoreBlock(block, mainPattern, keywords);
const threshold = mode === 'aggressive' ? 3.0 : 5.0;
if (shouldFilterBlock(score, threshold)) {
// Mark all lines in this block for removal
for (const segment of block.segments) {
adLineIndices.add(segment.lineIndex);
adLineIndices.add(segment.lineIndex - 1); // EXTINF line
}
}
}
}
}
const processedLines: string[] = [];
// State machine for CUE-OUT/CUE-IN tracking
let insideCueAdBlock = false;
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
const trimmedLine = line.trim();
// Skip lines marked by heuristic analysis
if (adLineIndices.has(i)) {
continue;
}
// 3. CUE Tag Detection (SCTE-35 Standard)
// EXT-X-CUE-OUT marks start of ad, EXT-X-CUE-IN marks end
if (mode !== 'off' && trimmedLine.startsWith('#EXT-X-CUE-OUT')) {
insideCueAdBlock = true;
// Remove preceding DISCONTINUITY if present
if (processedLines.length > 0 && processedLines[processedLines.length - 1].trim() === '#EXT-X-DISCONTINUITY') {
processedLines.pop();
}
continue; // Skip the CUE-OUT tag itself
}
if (trimmedLine.startsWith('#EXT-X-CUE-IN')) {
insideCueAdBlock = false;
// Also skip the next line if it's a DISCONTINUITY (ad block ending marker)
if (i + 1 < lines.length && lines[i + 1].trim() === '#EXT-X-DISCONTINUITY') {
i++; // Skip the following DISCONTINUITY
}
continue; // Skip the CUE-IN tag itself
}
// Skip all content inside CUE ad block
if (insideCueAdBlock) {
continue;
}
// 4. Keyword-based Ad Detection & Backtrack (skip if no keywords configured)
if (keywords.length > 0 && hasKeywordMatch && keywords.some(keyword => trimmedLine.includes(keyword))) {
// Found Ad: Remove it and backtrack to remove associated metadata
while (processedLines.length > 0) {
const lastIndex = processedLines.length - 1;
const lastLine = processedLines[lastIndex].trim();
if (lastLine.startsWith('#EXTINF:') || lastLine === '#EXT-X-DISCONTINUITY') {
processedLines.pop();
} else {
break;
}
}
continue; // Skip the ad line itself
}
// 5. Discontinuity Handling (Conservative Mode)
// Keep all Discontinuity tags by default.
// They will ONLY be removed via backtracking when a confirmed ad segment is found.
// This prevents false positives on legitimate concatenated streams.
if (trimmedLine === '#EXT-X-DISCONTINUITY') {
processedLines.push(line);
continue;
}
// 6. General Cleanup & URL Normalization
if (!trimmedLine || trimmedLine.startsWith('http') || trimmedLine.startsWith('blob:')) {
processedLines.push(line);
continue;
}
if (trimmedLine.startsWith('#')) {
// Handle URI="..." in attributes (e.g. #EXT-X-KEY)
if (trimmedLine.includes('URI="')) {
processedLines.push(line.replace(/URI="([^"]+)"/g, (match, uri) => {
if (uri.startsWith('http')) return match; // Already absolute
if (uri.startsWith('/')) {
return `URI="${origin}${uri}"`; // Root-relative
}
return `URI="${basePath}${uri}"`; // Path-relative
}));
} else {
processedLines.push(line);
}
continue;
}
// 7. Resolve Relative URLs (for Blob support)
if (trimmedLine.startsWith('/')) {
processedLines.push(origin ? `${origin}${trimmedLine}` : trimmedLine);
} else {
processedLines.push(`${basePath}${trimmedLine}`);
}
}
return processedLines.join('\n');
}