From 6bbadb69c73e3b9694fb341754e79d708182ef49 Mon Sep 17 00:00:00 2001 From: Marek Date: Sun, 5 Apr 2026 14:54:10 +0200 Subject: [PATCH] update --- CLAUDE.md | 50 +++-- architecture.md | 2 +- backend/models.py | 2 +- .../routes/__pycache__/videos.cpython-312.pyc | Bin 4849 -> 5552 bytes backend/routes/videos.py | 14 +- .../__pycache__/video_service.cpython-312.pyc | Bin 2284 -> 2750 bytes backend/services/video_service.py | 9 +- browser_extension/background.js | 4 +- browser_extension/content.js | 185 +++++++----------- commication.md | 2 +- features.md | 8 +- 11 files changed, 129 insertions(+), 147 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index dfde07a..360d800 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,36 +17,43 @@ Drei Komponenten: ### Browser Extension (`browser_extension/`) - **Manifest V2**, Firefox-kompatibel (`browser.*` API) - `content.js` — extrahiert Videodaten direkt aus dem YouTube-DOM: - - `yt-lockup-view-model` (Homepage, Abos, Kanalseiten) + - `ytd-rich-item-renderer` (Homepage, Abos, Kanalseiten) - `ytd-video-renderer` (Suchergebnisse) - - Debounced MutationObserver (250ms) fuer dynamisch geladene Cards + - IntersectionObserver (threshold 50%) — nur sichtbare Cards erfassen + - MutationObserver registriert neue Cards beim IntersectionObserver - `yt-navigate-finish` Event-Listener fuer SPA-Navigation - - Deduplizierung ueber `sentUrls` Set -- `background.js` — empfaengt Nachrichten vom Content Script, sendet POST an Server + - Deduplizierung ueber `sentUrls` Set, wird bei Navigation geleert + - Batch-Versand: sammelt sichtbare Videos, sendet als Array + - Selektoren ohne Klassen: nur Tags (`h3`, `img`) und Attribute (`href`, `src`) +- `background.js` — empfaengt Batch vom Content Script, sendet POST an Server - Laden via `about:debugging#/runtime/this-firefox` → "Temporaeres Add-on laden" → `manifest.json` ### Server (`backend/`) - **Python, FastAPI, SQLAlchemy, SQLite** (`videos/youtubeapp.db`) -- **yt-dlp + ffmpeg** fuer Video-Download +- **yt-dlp + ffmpeg** fuer Video-Download und Live-Streaming - Dockerisiert: `docker compose up --build -d` im `backend/` Verzeichnis - Laeuft auf `http://localhost:8000` - Download-Service speichert Videos unter `/videos/{id}.mp4` -- Beim Streamen wird automatisch heruntergeladen falls noetig +- Streaming: heruntergeladene Videos von Datei, sonst Live-Stream via yt-dlp + ffmpeg (fragmentiertes MP4) +- Dedup: beim Batch-Import wird bestehender Eintrag mit gleicher Video-ID geloescht und neu eingefuegt +- Sortierung: nach ID absteigend (erstes Video im Batch bekommt hoechste ID) ### App (`app/`) - **Kotlin, Jetpack Compose**, Android/Android TV - Gradle-Projekt, Modul `frontend` -- Aktueller Stand: Skeleton (nur `MainActivity` mit Platzhalter-Text) -- Geplante Screens: AllVideos, Downloaded, VideoDetail, VideoPlayer +- Screens: AllVideos (Grid), Downloaded, VideoDetail, VideoPlayer +- Retrofit fuer API-Calls, Coil fuer Thumbnails, ExoPlayer fuer Streaming +- Navigation mit Bottom Bar, Dark Theme +- Server-IP konfigurierbar in `ApiClient.kt` (aktuell `192.168.178.92`) - Emulator: Android Studio → Device Manager → Pixel 6a, API 35 ## API Endpoints -- `POST /videos` — Videodaten von Extension empfangen -- `GET /videos` — alle Videos abrufen +- `POST /videos` — Video-Batch von Extension empfangen (Liste von Videos, Dedup + Reverse-Insert) +- `GET /videos` — alle Videos abrufen (sortiert nach ID absteigend) - `GET /videos/downloaded` — heruntergeladene Videos abrufen - `POST /videos/{id}/download` — Download auf Server triggern -- `GET /videos/{id}/stream` — Video streamen +- `GET /videos/{id}/stream` — Video streamen (von Datei oder Live via yt-dlp/ffmpeg) - `GET /videos/{id}/file` — Video-Datei zum Download auf Client ausliefern ## Projektstruktur @@ -59,19 +66,28 @@ backend/ schemas.py — Pydantic Schemas (VideoCreate, VideoResponse) routes/videos.py — Alle API-Routen services/ - video_service.py — CRUD-Operationen - download_service.py — yt-dlp Download-Logik + video_service.py — CRUD-Operationen, Dedup + download_service.py — yt-dlp Download + Live-Streaming Dockerfile — Python 3.12 + ffmpeg docker-compose.yml — Service-Definition, Port 8000, Volume /videos + .dockerignore — videos/, __pycache__/ + .gitignore — videos/, __pycache__/ browser_extension/ manifest.json — Manifest V2, Permissions fuer youtube.com + localhost - content.js — DOM-basierte Video-Extraktion + MutationObserver - background.js — POST an Server + content.js — DOM-Extraktion + IntersectionObserver + Batch-Versand + background.js — Batch-POST an Server app/ + .gitignore — .gradle/, build/, .idea/, local.properties frontend/src/main/java/com/youtubeapp/ - MainActivity.kt — Einstiegspunkt (noch Skeleton) + MainActivity.kt — Einstiegspunkt + data/ — Video, ApiClient, VideoApi, VideoRepository + ui/screens/ — AllVideos, Downloaded, VideoDetail, VideoPlayer + ui/components/ — VideoCard + ui/viewmodel/ — VideoViewModel + ui/navigation/ — AppNavigation, Routes + ui/theme/ — Theme (Dark) ``` ## Entscheidungen @@ -81,4 +97,6 @@ app/ - Keine Benutzerprofile im Prototyp - Videos werden auf dem Server gespeichert, Client speichert nur bei explizitem Download - DOM-Extraktion statt ytInitialData-Parsing — funktioniert auch bei SPA-Navigation und Scrollen +- IntersectionObserver statt blindem Scan — nur sichtbare Videos erfassen +- Live-Streaming via yt-dlp/ffmpeg statt synchronem Download vor dem Streamen - Sprache der Dokumentation: Deutsch diff --git a/architecture.md b/architecture.md index 4a3aade..e681501 100644 --- a/architecture.md +++ b/architecture.md @@ -1,6 +1,6 @@ # Browser Extension - Content Script — YouTube-DOM auslesen, Videodaten extrahieren -- Background Script — Daten an Server senden (POST /videos) +- Background Script — Daten gruppiert an Server senden (POST /videos) # Server ## API - POST /videos — Videodaten von Extension empfangen diff --git a/backend/models.py b/backend/models.py index f647058..48f29a4 100644 --- a/backend/models.py +++ b/backend/models.py @@ -12,6 +12,6 @@ class Video(Base): title = Column(String, nullable=False) youtuber = Column(String, nullable=False) thumbnail_url = Column(String, nullable=False) - youtube_url = Column(String, nullable=False, unique=True) + youtube_url = Column(String, nullable=False) file_path = Column(String, nullable=True) created_at = Column(DateTime, default=datetime.utcnow) diff --git a/backend/routes/__pycache__/videos.cpython-312.pyc b/backend/routes/__pycache__/videos.cpython-312.pyc index aa80bbebd1bd1fa5fdf884d893b4ff3865a9a7d4..5f7519f3fcdbdfedd110d0e10f0b821fda97a7d0 100644 GIT binary patch delta 1674 zcmZWpO-vkB9Di?iW_FhSq`<;ZP-yvDploSbY^_j~FCzi0CLT=aW?=?c+-1p|0fg=f zF&;ROSn`ZwjFFRpU=y0X)Lv}RLlZ9@5x1RisD~c(1m#pO`hPpDl=da_d;j^p|L_0t z-kUk7-fHmt>UQTL7~c&37EQZWJSBK|rFP3%qWWZDfADN_^d2s*^r|JQmobV@b&m2% z{%67_lfn2b?Ji=L-vS}$2o6> zW4&2yx$S?JXKaesSNDE1c8 zTOO$jqvabsN^)K3b8&hndM=5(sOxi&u9jQt|B_$;d zFN=CXF4aU~&N`lxQm!kARK`Qbl=MCzW|lacv!q!R5C-PCnYO?Fg|T4b8p3)FI8BMD zGb+(k#dOXkruFGzH9SpXrl3v5qB>#ynuUrQQ*|{wJQvQnL>1FDLevPb*;6&e$kdcN zshAwBIL1`nqKS*?^F~P`9*!pB^Q&KQv05-Jmaf3eC&OG$M^b@@K7qqNTr5Ux*GcK($|-ErK)`gDsDW; zLsy1-@Q%CZ$A+zo%khk-+VE7b4X>Zw^)%gSSnPS=?s?Sw+K#jBwtxN2yUok}8CR9z zs#?3Y>#E)I8_u=|eA~09HstBW&rD~Vb_zP#_LO%P@Y~Jy&Juq6axoyi$$x3vzyq{F zC_i@wTsCakJ5UF4ervcy^=@Fs_cyfVJ?9fRK<3)WOZjYkz@WRfJ1Sod+g-&>_ zqh*d6?hvKFaKy2*g-JpTTGK9PBgO%G*%idyH0nx?vy4P!(9S?&;AOya6V{(8svk!+ zT?+^=R6yQkE`tD?0KjVK$nJ#c(sU>-esqD~gT)XXaIfI5^kaA5iGpaWx2E827N-Yb zo^OgzwFEfQOB3Frp%-dn#gJ!*KuKb5+`3RIG-^dsL-gC-w?KC>6xw88gh9Vj~_(-xi73ENr1n#l&(K!mBShNU;) z)0~c}?L-BR{b;p$I+x$k%C8L>6&Dvz@8Fh49liI(pS=f2!tJa2TA%Ui`F&<@jXh=4 Up$&1NeGaUBfBH$jgn1VJKM4Pnz5oCK delta 1001 zcmZvaO-vI(6o5P1ZP|9WwEUN!YNgOpAkY+{iBstsg zKUbNp) zTO-$_BJ+USRHhkC88m5yU<#_-Gc1=)W^*ddMD45&RIDEL4$DW!XnfGw)Xy5xgz&-C z+W||q?@HsA5-us>0$aVXsq`)leU|-Oa;PMS3PYQ6=LfNK4;*cLS$ZP+QP{2sbTxsn zX1A_k7|bN|);+;skhQ>`9X~)lKO_Ahejap10(goA0Y`n?P~FDkQVm~l#Yk)Mh3hqu zO;?&4)yr#SAD6zNDN>GcU+>EtrAnGnkf6Q@x(R9FKPzKofFJYTUyP&Pi8zIj5MD$t z0;_5|=^afs^r+)-3TPcdTm|S308}M7Xw}73bOZXA-EZFr7-Q(Q4EVj@j8I7l6)vyx zSJe$X?wclo;v?VakzDz9Rd}%dty&^?sQaOeXud587)I+Q&}37tPjKfc?Raq zh7lF$?l3eXLyy)FjaKd=NSJ#$bQAKdyct1we|PU=5Wc8PKah&=KKLMp3dtGaB>#dw1mYjyZetgAW|{fsn{Q|L&&cO-Wmi$$ z99{_JA6N&m+wOK!?7rK#)=chE%#rEQe%piCQP$P4kCh!0Y7QjkI*NVcT8L+(nkTq>`add|r5 zO#1&NQZLZm|4N#oPEJ8DzYfS{7#uagd;T35zyP<-ttq4qeYXHt;WXjQNey(ztV$uf zii?HZO;fi_%gPlCnhTH98Vs44q^67ZJC||3D`$&^(sHg~btS{hn!{kqBu&QJ=xv_{xSu1#bxwM)UNR{`Jzfzix?X1fO!X)p~ zMD|dAvgL|^VbXFf&IDP3V3qKm63o(={f9lKfpYoSwW3kZo5yjKCfFFQ201Ms4ac9{ z=y3qVpGO*EJ>F->o(zBvHm|a{4jREm`Avu=bn;f78smXy^FoiKZYR)ZuIDB#dGa6m CQ@0HO delta 283 zcmdld`bLoVG%qg~0}!a!T+H;D$ScV>ZK8UIFjp#93U4Y;8e0lq3riGlCATL3#!C~K zxVe&xQWHy3Q{od#CZ{o*Ga65x$1EXtUB%?0ipg~q%ZnLWjH;7ove{3b#!(Jr@o`Eqs!i79)MV8Fx>b5|IHw6C-{c-nZ*El}{}yv` zNl}r;3yhBM=wQn9RqeukXw_q5J~_Bcn6p0+SW23#~tc*&jF<8PgeOq=RJA X8CR&RC|#)yl1ZPO$)!4ZA(t!w=J{2X diff --git a/backend/services/video_service.py b/backend/services/video_service.py index cc421b9..9b54eec 100644 --- a/backend/services/video_service.py +++ b/backend/services/video_service.py @@ -13,17 +13,22 @@ def create_video(db: Session, video_data: VideoCreate) -> Video: def get_all_videos(db: Session) -> list[Video]: - return db.query(Video).order_by(Video.created_at.desc()).all() + return db.query(Video).order_by(Video.id.desc()).all() def get_downloaded_videos(db: Session) -> list[Video]: - return db.query(Video).filter(Video.file_path.isnot(None)).order_by(Video.created_at.desc()).all() + return db.query(Video).filter(Video.file_path.isnot(None)).order_by(Video.id.desc()).all() def get_video(db: Session, video_id: int) -> Video | None: return db.query(Video).filter(Video.id == video_id).first() +def delete_by_youtube_id(db: Session, youtube_id: str): + db.query(Video).filter(Video.youtube_url.contains(youtube_id)).delete(synchronize_session=False) + db.commit() + + def update_file_path(db: Session, video_id: int, path: str): video = get_video(db, video_id) if video: diff --git a/browser_extension/background.js b/browser_extension/background.js index 3e6d21b..c35da08 100644 --- a/browser_extension/background.js +++ b/browser_extension/background.js @@ -1,9 +1,9 @@ const SERVER_URL = "http://localhost:8000/videos"; -browser.runtime.onMessage.addListener((video) => { +browser.runtime.onMessage.addListener((videos) => { fetch(SERVER_URL, { method: "POST", headers: { "Content-Type": "application/json" }, - body: JSON.stringify(video), + body: JSON.stringify(videos), }).catch(() => {}); }); diff --git a/browser_extension/content.js b/browser_extension/content.js index f744a2b..7ae5737 100644 --- a/browser_extension/content.js +++ b/browser_extension/content.js @@ -2,143 +2,96 @@ console.log("[YT-Erfasser] Content Script geladen"); const sentUrls = new Set(); -function getVideoId(url) { - const match = url.match(/[?&]v=([^&]+)/); - return match ? match[1] : null; -} - -function getThumbnailUrl(videoId) { - return `https://img.youtube.com/vi/${videoId}/hqdefault.jpg`; -} - -// --- DOM Extractors --- - -function extractFromLockupViewModel(element) { - const div = element.querySelector(".yt-lockup-view-model"); - const classStr = div?.className || element.className || ""; - const idMatch = classStr.match(/content-id-([\w-]+)/); - let videoId = idMatch ? idMatch[1] : null; - - if (!videoId) { - const link = element.querySelector('a[href*="/watch?v="]'); - if (link) videoId = getVideoId(link.href); - } - if (!videoId) return null; - - const title = - element.querySelector("a.yt-lockup-metadata-view-model__title > span") - ?.textContent?.trim() || ""; - if (!title) return null; - - const youtuber = - element.querySelector('a.yt-core-attributed-string__link[href^="/@"]') - ?.textContent?.trim() || "Unbekannt"; - - return { - title, - youtuber, - thumbnail_url: getThumbnailUrl(videoId), - youtube_url: `https://www.youtube.com/watch?v=${videoId}`, - }; -} - -function extractFromVideoRenderer(element) { - const titleLink = element.querySelector("a#video-title"); - if (!titleLink) return null; - - const videoId = getVideoId(titleLink.href); - if (!videoId) return null; - - const title = - titleLink.textContent?.trim() || - element.querySelector("#video-title yt-formatted-string") - ?.textContent?.trim() || ""; - if (!title) return null; - - const youtuber = - element.querySelector("ytd-channel-name a")?.textContent?.trim() || - element.querySelector(".ytd-channel-name a")?.textContent?.trim() || - "Unbekannt"; - - return { - title, - youtuber, - thumbnail_url: getThumbnailUrl(videoId), - youtube_url: `https://www.youtube.com/watch?v=${videoId}`, - }; -} - function extractVideoFromCard(element) { - const lockup = element.matches?.("yt-lockup-view-model") - ? element - : element.querySelector("yt-lockup-view-model"); - if (lockup) return extractFromLockupViewModel(lockup); + const link = element.querySelector('a[href*="/watch?v="]'); + if (!link) return null; - const renderer = element.matches?.("ytd-video-renderer") - ? element - : element.querySelector("ytd-video-renderer"); - if (renderer) return extractFromVideoRenderer(renderer); + const match = link.href.match(/[?&]v=([^&]+)/); + if (!match) return null; - return null; + const title = element.querySelector("h3 a")?.textContent?.trim(); + if (!title) return null; + + const thumbnail = element.querySelector('a[href*="/watch?v="] img')?.src; + const youtuber = element.querySelector('a[href^="/@"]')?.textContent?.trim() || "Unbekannt"; + + return { + title, + youtuber, + thumbnail_url: thumbnail || `https://img.youtube.com/vi/${match[1]}/hqdefault.jpg`, + youtube_url: `https://www.youtube.com/watch?v=${match[1]}`, + }; } -// --- Processing --- - -function processCard(element) { - const video = extractVideoFromCard(element); - if (!video) return; - if (sentUrls.has(video.youtube_url)) return; - sentUrls.add(video.youtube_url); - console.log("[YT-Erfasser]", video.title, "-", video.youtuber); - browser.runtime.sendMessage(video); -} - -function scanExistingCards() { - document - .querySelectorAll("yt-lockup-view-model, ytd-video-renderer") - .forEach((el) => processCard(el)); -} - -// --- MutationObserver (debounced) --- - -let pendingElements = []; -let debounceTimer = null; - -function processPendingElements() { - const elements = pendingElements; - pendingElements = []; - debounceTimer = null; - +function collectVideos(elements) { + const videos = []; for (const el of elements) { - if (el.matches?.("yt-lockup-view-model, ytd-video-renderer")) { - processCard(el); - } - el.querySelectorAll?.("yt-lockup-view-model, ytd-video-renderer").forEach( - (card) => processCard(card) - ); + const video = extractVideoFromCard(el); + if (!video) continue; + if (sentUrls.has(video.youtube_url)) continue; + sentUrls.add(video.youtube_url); + videos.push(video); + } + return videos; +} + +// --- Debounced Batch-Versand --- + +let pendingVideos = []; +let sendTimer = null; + +function queueVideos(videos) { + pendingVideos.push(...videos); + if (!sendTimer) { + sendTimer = setTimeout(() => { + if (pendingVideos.length > 0) { + console.log(`[YT-Erfasser] ${pendingVideos.length} Videos senden`); + browser.runtime.sendMessage(pendingVideos); + } + pendingVideos = []; + sendTimer = null; + }, 250); } } -const observer = new MutationObserver((mutations) => { +// --- IntersectionObserver: nur sichtbare Cards erfassen --- + +const visibilityObserver = new IntersectionObserver((entries) => { + const cards = entries.filter((e) => e.isIntersecting).map((e) => e.target); + if (cards.length > 0) { + queueVideos(collectVideos(cards)); + } +}, { threshold: 0.5 }); + +function observeCard(el) { + visibilityObserver.observe(el); +} + +// --- MutationObserver: neue Cards registrieren --- + +const mutationObserver = new MutationObserver((mutations) => { for (const mutation of mutations) { for (const node of mutation.addedNodes) { if (node.nodeType !== Node.ELEMENT_NODE) continue; - pendingElements.push(node); + if (node.matches?.("ytd-rich-item-renderer, ytd-video-renderer")) { + observeCard(node); + } + node.querySelectorAll?.("ytd-rich-item-renderer, ytd-video-renderer").forEach(observeCard); } } - if (pendingElements.length > 0 && !debounceTimer) { - debounceTimer = setTimeout(processPendingElements, 250); - } }); -observer.observe(document.body, { childList: true, subtree: true }); +mutationObserver.observe(document.body, { childList: true, subtree: true }); // --- SPA Navigation --- document.addEventListener("yt-navigate-finish", () => { - setTimeout(scanExistingCards, 500); + sentUrls.clear(); + setTimeout(() => { + document.querySelectorAll("ytd-rich-item-renderer, ytd-video-renderer").forEach(observeCard); + }, 500); }); // --- Init --- -scanExistingCards(); +document.querySelectorAll("ytd-rich-item-renderer, ytd-video-renderer").forEach(observeCard); diff --git a/commication.md b/commication.md index d3f652d..11273e6 100644 --- a/commication.md +++ b/commication.md @@ -9,7 +9,7 @@ - Video streamen - Videodatei senden # App -> Server -- Videos abrufen +- Videos abrufen - Video download anfragen - Video stream anfragen # App intern diff --git a/features.md b/features.md index 85a6a66..920b748 100644 --- a/features.md +++ b/features.md @@ -1,13 +1,13 @@ # Aufgaben ## Browser -- Youtube Videos werden erfasst +- Sichtbare Youtube Videos werden erfasst - Videodaten (Titel, Youtuber, Bild, Url) werden nach dem erfassen an den Server gesendet +- Bereits erfasste Videos werden nicht erneut gesendet - Extension hat keine Einstellung ## App - Ansicht: Navigation mit Alle Videos, Heruntergeladen - Alle Videos: Videos als Cards auflisten (Untereinander: Bild, Youtuber, Titel) - Heruntergeladen: Heruntergeladene Videos als Cards auflisten (Untereinander: Bild, Youtuber, Titel) - Klick auf Card zeigt die Videoübersicht (Starten, Download, Zurück) -- Klick auf Startet startet dem Stream mit den Standard Video-Controls -- Klick auf Download lädt das Video herunter und übertragt es auf den Client -- Beim streamen wird das Video heruntergeladen, wenn noch nicht geschehen \ No newline at end of file +- Klick auf Startet startet den Stream über den Server mit den Standard Video-Controls +- Klick auf Download lädt das Video herunter und übertragt es auf den Client \ No newline at end of file