diff --git a/CLAUDE.md b/CLAUDE.md index dfde07a..360d800 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,36 +17,43 @@ Drei Komponenten: ### Browser Extension (`browser_extension/`) - **Manifest V2**, Firefox-kompatibel (`browser.*` API) - `content.js` — extrahiert Videodaten direkt aus dem YouTube-DOM: - - `yt-lockup-view-model` (Homepage, Abos, Kanalseiten) + - `ytd-rich-item-renderer` (Homepage, Abos, Kanalseiten) - `ytd-video-renderer` (Suchergebnisse) - - Debounced MutationObserver (250ms) fuer dynamisch geladene Cards + - IntersectionObserver (threshold 50%) — nur sichtbare Cards erfassen + - MutationObserver registriert neue Cards beim IntersectionObserver - `yt-navigate-finish` Event-Listener fuer SPA-Navigation - - Deduplizierung ueber `sentUrls` Set -- `background.js` — empfaengt Nachrichten vom Content Script, sendet POST an Server + - Deduplizierung ueber `sentUrls` Set, wird bei Navigation geleert + - Batch-Versand: sammelt sichtbare Videos, sendet als Array + - Selektoren ohne Klassen: nur Tags (`h3`, `img`) und Attribute (`href`, `src`) +- `background.js` — empfaengt Batch vom Content Script, sendet POST an Server - Laden via `about:debugging#/runtime/this-firefox` → "Temporaeres Add-on laden" → `manifest.json` ### Server (`backend/`) - **Python, FastAPI, SQLAlchemy, SQLite** (`videos/youtubeapp.db`) -- **yt-dlp + ffmpeg** fuer Video-Download +- **yt-dlp + ffmpeg** fuer Video-Download und Live-Streaming - Dockerisiert: `docker compose up --build -d` im `backend/` Verzeichnis - Laeuft auf `http://localhost:8000` - Download-Service speichert Videos unter `/videos/{id}.mp4` -- Beim Streamen wird automatisch heruntergeladen falls noetig +- Streaming: heruntergeladene Videos von Datei, sonst Live-Stream via yt-dlp + ffmpeg (fragmentiertes MP4) +- Dedup: beim Batch-Import wird bestehender Eintrag mit gleicher Video-ID geloescht und neu eingefuegt +- Sortierung: nach ID absteigend (erstes Video im Batch bekommt hoechste ID) ### App (`app/`) - **Kotlin, Jetpack Compose**, Android/Android TV - Gradle-Projekt, Modul `frontend` -- Aktueller Stand: Skeleton (nur `MainActivity` mit Platzhalter-Text) -- Geplante Screens: AllVideos, Downloaded, VideoDetail, VideoPlayer +- Screens: AllVideos (Grid), Downloaded, VideoDetail, VideoPlayer +- Retrofit fuer API-Calls, Coil fuer Thumbnails, ExoPlayer fuer Streaming +- Navigation mit Bottom Bar, Dark Theme +- Server-IP konfigurierbar in `ApiClient.kt` (aktuell `192.168.178.92`) - Emulator: Android Studio → Device Manager → Pixel 6a, API 35 ## API Endpoints -- `POST /videos` — Videodaten von Extension empfangen -- `GET /videos` — alle Videos abrufen +- `POST /videos` — Video-Batch von Extension empfangen (Liste von Videos, Dedup + Reverse-Insert) +- `GET /videos` — alle Videos abrufen (sortiert nach ID absteigend) - `GET /videos/downloaded` — heruntergeladene Videos abrufen - `POST /videos/{id}/download` — Download auf Server triggern -- `GET /videos/{id}/stream` — Video streamen +- `GET /videos/{id}/stream` — Video streamen (von Datei oder Live via yt-dlp/ffmpeg) - `GET /videos/{id}/file` — Video-Datei zum Download auf Client ausliefern ## Projektstruktur @@ -59,19 +66,28 @@ backend/ schemas.py — Pydantic Schemas (VideoCreate, VideoResponse) routes/videos.py — Alle API-Routen services/ - video_service.py — CRUD-Operationen - download_service.py — yt-dlp Download-Logik + video_service.py — CRUD-Operationen, Dedup + download_service.py — yt-dlp Download + Live-Streaming Dockerfile — Python 3.12 + ffmpeg docker-compose.yml — Service-Definition, Port 8000, Volume /videos + .dockerignore — videos/, __pycache__/ + .gitignore — videos/, __pycache__/ browser_extension/ manifest.json — Manifest V2, Permissions fuer youtube.com + localhost - content.js — DOM-basierte Video-Extraktion + MutationObserver - background.js — POST an Server + content.js — DOM-Extraktion + IntersectionObserver + Batch-Versand + background.js — Batch-POST an Server app/ + .gitignore — .gradle/, build/, .idea/, local.properties frontend/src/main/java/com/youtubeapp/ - MainActivity.kt — Einstiegspunkt (noch Skeleton) + MainActivity.kt — Einstiegspunkt + data/ — Video, ApiClient, VideoApi, VideoRepository + ui/screens/ — AllVideos, Downloaded, VideoDetail, VideoPlayer + ui/components/ — VideoCard + ui/viewmodel/ — VideoViewModel + ui/navigation/ — AppNavigation, Routes + ui/theme/ — Theme (Dark) ``` ## Entscheidungen @@ -81,4 +97,6 @@ app/ - Keine Benutzerprofile im Prototyp - Videos werden auf dem Server gespeichert, Client speichert nur bei explizitem Download - DOM-Extraktion statt ytInitialData-Parsing — funktioniert auch bei SPA-Navigation und Scrollen +- IntersectionObserver statt blindem Scan — nur sichtbare Videos erfassen +- Live-Streaming via yt-dlp/ffmpeg statt synchronem Download vor dem Streamen - Sprache der Dokumentation: Deutsch diff --git a/architecture.md b/architecture.md index 4a3aade..e681501 100644 --- a/architecture.md +++ b/architecture.md @@ -1,6 +1,6 @@ # Browser Extension - Content Script — YouTube-DOM auslesen, Videodaten extrahieren -- Background Script — Daten an Server senden (POST /videos) +- Background Script — Daten gruppiert an Server senden (POST /videos) # Server ## API - POST /videos — Videodaten von Extension empfangen diff --git a/backend/models.py b/backend/models.py index f647058..48f29a4 100644 --- a/backend/models.py +++ b/backend/models.py @@ -12,6 +12,6 @@ class Video(Base): title = Column(String, nullable=False) youtuber = Column(String, nullable=False) thumbnail_url = Column(String, nullable=False) - youtube_url = Column(String, nullable=False, unique=True) + youtube_url = Column(String, nullable=False) file_path = Column(String, nullable=True) created_at = Column(DateTime, default=datetime.utcnow) diff --git a/backend/routes/__pycache__/videos.cpython-312.pyc b/backend/routes/__pycache__/videos.cpython-312.pyc index aa80bbe..5f7519f 100644 Binary files a/backend/routes/__pycache__/videos.cpython-312.pyc and b/backend/routes/__pycache__/videos.cpython-312.pyc differ diff --git a/backend/routes/videos.py b/backend/routes/videos.py index 132575f..c1d998d 100644 --- a/backend/routes/videos.py +++ b/backend/routes/videos.py @@ -13,10 +13,16 @@ from services.download_service import download_video, stream_video_live router = APIRouter(prefix="/videos", tags=["videos"]) -@router.post("", response_model=VideoResponse) -def create_video(video_data: VideoCreate, db: Session = Depends(get_db)): - video = video_service.create_video(db, video_data) - return VideoResponse.from_model(video) +@router.post("", response_model=list[VideoResponse]) +def create_videos(videos_data: list[VideoCreate], db: Session = Depends(get_db)): + created_ids = [] + for video_data in reversed(videos_data): + video_id_match = video_data.youtube_url.split("v=")[-1].split("&")[0] + video_service.delete_by_youtube_id(db, video_id_match) + video = video_service.create_video(db, video_data) + created_ids.append(video.id) + videos = [video_service.get_video(db, vid) for vid in created_ids] + return [VideoResponse.from_model(v) for v in videos if v] @router.get("", response_model=list[VideoResponse]) diff --git a/backend/services/__pycache__/video_service.cpython-312.pyc b/backend/services/__pycache__/video_service.cpython-312.pyc index 65b9e0c..8a59c0a 100644 Binary files a/backend/services/__pycache__/video_service.cpython-312.pyc and b/backend/services/__pycache__/video_service.cpython-312.pyc differ diff --git a/backend/services/video_service.py b/backend/services/video_service.py index cc421b9..9b54eec 100644 --- a/backend/services/video_service.py +++ b/backend/services/video_service.py @@ -13,17 +13,22 @@ def create_video(db: Session, video_data: VideoCreate) -> Video: def get_all_videos(db: Session) -> list[Video]: - return db.query(Video).order_by(Video.created_at.desc()).all() + return db.query(Video).order_by(Video.id.desc()).all() def get_downloaded_videos(db: Session) -> list[Video]: - return db.query(Video).filter(Video.file_path.isnot(None)).order_by(Video.created_at.desc()).all() + return db.query(Video).filter(Video.file_path.isnot(None)).order_by(Video.id.desc()).all() def get_video(db: Session, video_id: int) -> Video | None: return db.query(Video).filter(Video.id == video_id).first() +def delete_by_youtube_id(db: Session, youtube_id: str): + db.query(Video).filter(Video.youtube_url.contains(youtube_id)).delete(synchronize_session=False) + db.commit() + + def update_file_path(db: Session, video_id: int, path: str): video = get_video(db, video_id) if video: diff --git a/browser_extension/background.js b/browser_extension/background.js index 3e6d21b..c35da08 100644 --- a/browser_extension/background.js +++ b/browser_extension/background.js @@ -1,9 +1,9 @@ const SERVER_URL = "http://localhost:8000/videos"; -browser.runtime.onMessage.addListener((video) => { +browser.runtime.onMessage.addListener((videos) => { fetch(SERVER_URL, { method: "POST", headers: { "Content-Type": "application/json" }, - body: JSON.stringify(video), + body: JSON.stringify(videos), }).catch(() => {}); }); diff --git a/browser_extension/content.js b/browser_extension/content.js index f744a2b..7ae5737 100644 --- a/browser_extension/content.js +++ b/browser_extension/content.js @@ -2,143 +2,96 @@ console.log("[YT-Erfasser] Content Script geladen"); const sentUrls = new Set(); -function getVideoId(url) { - const match = url.match(/[?&]v=([^&]+)/); - return match ? match[1] : null; -} - -function getThumbnailUrl(videoId) { - return `https://img.youtube.com/vi/${videoId}/hqdefault.jpg`; -} - -// --- DOM Extractors --- - -function extractFromLockupViewModel(element) { - const div = element.querySelector(".yt-lockup-view-model"); - const classStr = div?.className || element.className || ""; - const idMatch = classStr.match(/content-id-([\w-]+)/); - let videoId = idMatch ? idMatch[1] : null; - - if (!videoId) { - const link = element.querySelector('a[href*="/watch?v="]'); - if (link) videoId = getVideoId(link.href); - } - if (!videoId) return null; - - const title = - element.querySelector("a.yt-lockup-metadata-view-model__title > span") - ?.textContent?.trim() || ""; - if (!title) return null; - - const youtuber = - element.querySelector('a.yt-core-attributed-string__link[href^="/@"]') - ?.textContent?.trim() || "Unbekannt"; - - return { - title, - youtuber, - thumbnail_url: getThumbnailUrl(videoId), - youtube_url: `https://www.youtube.com/watch?v=${videoId}`, - }; -} - -function extractFromVideoRenderer(element) { - const titleLink = element.querySelector("a#video-title"); - if (!titleLink) return null; - - const videoId = getVideoId(titleLink.href); - if (!videoId) return null; - - const title = - titleLink.textContent?.trim() || - element.querySelector("#video-title yt-formatted-string") - ?.textContent?.trim() || ""; - if (!title) return null; - - const youtuber = - element.querySelector("ytd-channel-name a")?.textContent?.trim() || - element.querySelector(".ytd-channel-name a")?.textContent?.trim() || - "Unbekannt"; - - return { - title, - youtuber, - thumbnail_url: getThumbnailUrl(videoId), - youtube_url: `https://www.youtube.com/watch?v=${videoId}`, - }; -} - function extractVideoFromCard(element) { - const lockup = element.matches?.("yt-lockup-view-model") - ? element - : element.querySelector("yt-lockup-view-model"); - if (lockup) return extractFromLockupViewModel(lockup); + const link = element.querySelector('a[href*="/watch?v="]'); + if (!link) return null; - const renderer = element.matches?.("ytd-video-renderer") - ? element - : element.querySelector("ytd-video-renderer"); - if (renderer) return extractFromVideoRenderer(renderer); + const match = link.href.match(/[?&]v=([^&]+)/); + if (!match) return null; - return null; + const title = element.querySelector("h3 a")?.textContent?.trim(); + if (!title) return null; + + const thumbnail = element.querySelector('a[href*="/watch?v="] img')?.src; + const youtuber = element.querySelector('a[href^="/@"]')?.textContent?.trim() || "Unbekannt"; + + return { + title, + youtuber, + thumbnail_url: thumbnail || `https://img.youtube.com/vi/${match[1]}/hqdefault.jpg`, + youtube_url: `https://www.youtube.com/watch?v=${match[1]}`, + }; } -// --- Processing --- - -function processCard(element) { - const video = extractVideoFromCard(element); - if (!video) return; - if (sentUrls.has(video.youtube_url)) return; - sentUrls.add(video.youtube_url); - console.log("[YT-Erfasser]", video.title, "-", video.youtuber); - browser.runtime.sendMessage(video); -} - -function scanExistingCards() { - document - .querySelectorAll("yt-lockup-view-model, ytd-video-renderer") - .forEach((el) => processCard(el)); -} - -// --- MutationObserver (debounced) --- - -let pendingElements = []; -let debounceTimer = null; - -function processPendingElements() { - const elements = pendingElements; - pendingElements = []; - debounceTimer = null; - +function collectVideos(elements) { + const videos = []; for (const el of elements) { - if (el.matches?.("yt-lockup-view-model, ytd-video-renderer")) { - processCard(el); - } - el.querySelectorAll?.("yt-lockup-view-model, ytd-video-renderer").forEach( - (card) => processCard(card) - ); + const video = extractVideoFromCard(el); + if (!video) continue; + if (sentUrls.has(video.youtube_url)) continue; + sentUrls.add(video.youtube_url); + videos.push(video); + } + return videos; +} + +// --- Debounced Batch-Versand --- + +let pendingVideos = []; +let sendTimer = null; + +function queueVideos(videos) { + pendingVideos.push(...videos); + if (!sendTimer) { + sendTimer = setTimeout(() => { + if (pendingVideos.length > 0) { + console.log(`[YT-Erfasser] ${pendingVideos.length} Videos senden`); + browser.runtime.sendMessage(pendingVideos); + } + pendingVideos = []; + sendTimer = null; + }, 250); } } -const observer = new MutationObserver((mutations) => { +// --- IntersectionObserver: nur sichtbare Cards erfassen --- + +const visibilityObserver = new IntersectionObserver((entries) => { + const cards = entries.filter((e) => e.isIntersecting).map((e) => e.target); + if (cards.length > 0) { + queueVideos(collectVideos(cards)); + } +}, { threshold: 0.5 }); + +function observeCard(el) { + visibilityObserver.observe(el); +} + +// --- MutationObserver: neue Cards registrieren --- + +const mutationObserver = new MutationObserver((mutations) => { for (const mutation of mutations) { for (const node of mutation.addedNodes) { if (node.nodeType !== Node.ELEMENT_NODE) continue; - pendingElements.push(node); + if (node.matches?.("ytd-rich-item-renderer, ytd-video-renderer")) { + observeCard(node); + } + node.querySelectorAll?.("ytd-rich-item-renderer, ytd-video-renderer").forEach(observeCard); } } - if (pendingElements.length > 0 && !debounceTimer) { - debounceTimer = setTimeout(processPendingElements, 250); - } }); -observer.observe(document.body, { childList: true, subtree: true }); +mutationObserver.observe(document.body, { childList: true, subtree: true }); // --- SPA Navigation --- document.addEventListener("yt-navigate-finish", () => { - setTimeout(scanExistingCards, 500); + sentUrls.clear(); + setTimeout(() => { + document.querySelectorAll("ytd-rich-item-renderer, ytd-video-renderer").forEach(observeCard); + }, 500); }); // --- Init --- -scanExistingCards(); +document.querySelectorAll("ytd-rich-item-renderer, ytd-video-renderer").forEach(observeCard); diff --git a/commication.md b/commication.md index d3f652d..11273e6 100644 --- a/commication.md +++ b/commication.md @@ -9,7 +9,7 @@ - Video streamen - Videodatei senden # App -> Server -- Videos abrufen +- Videos abrufen - Video download anfragen - Video stream anfragen # App intern diff --git a/features.md b/features.md index 85a6a66..920b748 100644 --- a/features.md +++ b/features.md @@ -1,13 +1,13 @@ # Aufgaben ## Browser -- Youtube Videos werden erfasst +- Sichtbare Youtube Videos werden erfasst - Videodaten (Titel, Youtuber, Bild, Url) werden nach dem erfassen an den Server gesendet +- Bereits erfasste Videos werden nicht erneut gesendet - Extension hat keine Einstellung ## App - Ansicht: Navigation mit Alle Videos, Heruntergeladen - Alle Videos: Videos als Cards auflisten (Untereinander: Bild, Youtuber, Titel) - Heruntergeladen: Heruntergeladene Videos als Cards auflisten (Untereinander: Bild, Youtuber, Titel) - Klick auf Card zeigt die Videoübersicht (Starten, Download, Zurück) -- Klick auf Startet startet dem Stream mit den Standard Video-Controls -- Klick auf Download lädt das Video herunter und übertragt es auf den Client -- Beim streamen wird das Video heruntergeladen, wenn noch nicht geschehen \ No newline at end of file +- Klick auf Startet startet den Stream über den Server mit den Standard Video-Controls +- Klick auf Download lädt das Video herunter und übertragt es auf den Client \ No newline at end of file