Compare commits

...

40 Commits

Author SHA1 Message Date
Samantaz Fox
bad92093bf
Channels: Add sort options to streams (#4224) 2024-07-10 22:28:22 +02:00
Samantaz Fox
436a61e3bb
API: Fix error code for disabled popular endpoint (#4296)
When visiting /api/v1/popular and popular endpoint is disabled
Before:

500 {"error":"Closed stream"}

After

403 {"error":"Administrator has disabled this endpoint."}
2024-07-10 22:25:31 +02:00
Samantaz Fox
5e0f55333a
Allow embedding videos in local HTML files (#4450)
The current Content Security Policy does not allow to embed videos
inside local HTML files which are viewed in the browser via the file
protocol. This commit adds the file protocol to the allowed frame
ancestors, so that the embedded videos load correctly in local HTML
files.

This behaviour is consistent which how the official YouTube website
allows to embed videos from itself.

Closes issue 4448
2024-07-10 22:24:18 +02:00
Samantaz Fox
de61b163a3
CI: Bump Crystal version matrix (#4654) 2024-07-10 22:21:17 +02:00
Samantaz Fox
99c7e9e800
YtAPI: Remove API keys like official clients (#4655)
This PR removes API keys from innertube requests, as the official clients
did it too.
2024-07-10 22:19:51 +02:00
Samantaz Fox
e9bab06e90
HTML: Use full URL in the og:image property (#4675)
Some opengraph implementations don't support a URL without the domain
therefore failing to fetch the video thumbnail and channel image.
This pull request basically fixes that.
2024-07-10 22:17:45 +02:00
Samantaz Fox
a56a724a55
Rewrite transcript logic to be more generic (#4747)
The transcript logic in Invidious was written specifically as a workaround for
captions, and not transcripts as a feature.

This PR genericises the logic as so it can be used to implement transcripts
within Invidious.

The most notable change is the added parsing of section headings when it was
previously skipped over in favor of regular lines.
2024-07-10 22:14:56 +02:00
Samantaz Fox
0a54e26536
CI: Run Ameba (#4753)
This PR simply adds Ameba to the CI but doesn't actually fix any of the
detected issues.
2024-07-10 22:13:45 +02:00
Samantaz Fox
d135e5b7f7
CI: Add release based containers (#4763)
This PR changes the current master based container to use "master" tag instead
of "latest" tag and adds a new workflow to build a container on each new
release which has the "latest" tag, and a tag based on the current released
version.
2024-07-10 22:11:01 +02:00
syeopite
220cc9bd2f
Typo
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
2024-07-04 10:14:19 -07:00
syeopite
aace30b2b4
Bump nightly container build workflow crystal ver 2024-07-04 10:11:36 -07:00
syeopite
64d1f26ece
Fix trigger for stable container build 2024-07-01 21:39:14 -07:00
syeopite
8f5c6a602b
Rename container workflows 2024-07-01 21:35:08 -07:00
syeopite
dd38eef41a
Add workflow to build container on release 2024-06-24 11:45:00 -07:00
syeopite
848ab1e9c8
Specify which workflow builds from master 2024-06-24 11:36:11 -07:00
syeopite
933802b897
Use "master" label for master container build 2024-06-24 11:34:55 -07:00
syeopite
6b429575bf
Update ameba version 2024-06-16 16:22:01 -07:00
syeopite
e0ed094cc4
Cache ameba binary 2024-06-16 13:29:06 -07:00
syeopite
a644d76497
Update ameba config 2024-06-16 13:21:55 -07:00
syeopite
45fd4a1968
Add job to lint code through Ameba in CI 2024-06-16 13:21:55 -07:00
syeopite
f466116cd7
Extract label for transcript in YouTube response 2024-06-13 09:07:20 -07:00
syeopite
5b519123a7
Raise error when transcript does not exist 2024-06-11 18:46:34 -07:00
syeopite
0224162ad2
Rewrite transcript logic to be more generic
The transcript logic in Invidious was written specifically
as a workaround for captions, and not transcripts as a feature.

This commit genericises the logic a bit as so it can be used for
implementing transcripts within Invidious' API and UI as well.

The most notable change is the added parsing of section headings
when it was previously skipped over in favor of regular lines.
2024-06-11 18:23:01 -07:00
Fijxu
9d66676f2d
Use full URL in the og:image property. 2024-05-01 22:21:18 -04:00
Samantaz Fox
2fdb6dd644
CI: Bump Crystal version in docker too 2024-04-27 21:02:37 +02:00
Samantaz Fox
470245de54
YtAPI: Remove API keys like official clients 2024-04-27 20:48:42 +02:00
Samantaz Fox
b0ec359028
CI: Bump Crystal version matrix 2024-04-27 20:01:19 +02:00
Brahim Hadriche
a9e8aabe1f Merge commit '08390acd0c17875fddb84cabba54197a5b5740e4' into fix/popular-disabled-error 2024-04-01 10:03:37 -04:00
Brahim Hadriche
b0c6bdf44c use 403 code 2024-04-01 10:03:29 -04:00
Brahim Hadriche
c5eb10b21f Revert "Fix error code for disabled popular endpoint"
This reverts commit 1363fb809436464de57b90113864ff50867a9dae.
2024-04-01 10:02:49 -04:00
src-tinkerer
72fe8af850
Merge branch 'master' into stream-sort 2024-03-26 12:19:45 +00:00
Tomasz Wilczyński
4adb4c00d2
routes: Allow embedding videos in local HTML files (fixes #4448)
The current Content Security Policy does not allow to embed videos
inside local HTML files which are viewed in the browser via the file
protocol. This commit adds the file protocol to the allowed frame
ancestors, so that the embedded videos load correctly in local HTML
files.

This behaviour is consistent which how the official YouTube website
allows to embed videos from itself.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2024-02-24 20:01:16 +01:00
src-tinkerer
cf61af67ab Update src/invidious/routes/channels.cr sort_by for consistency 2023-11-30 14:34:01 +03:30
Brahim Hadriche
1363fb8094 Fix error code for disabled popular endpoint 2023-11-28 21:34:17 -05:00
src-tinkerer
5f2b43d653 Remove unecessary if condition in videos.cr 2023-11-25 00:48:27 +03:30
src-tinkerer
6251d8d43f Rename a variable in videos.cr 2023-11-25 00:46:11 +03:30
src-tinkerer
162b89d942 Fix format in videos.cr 2023-11-23 14:44:37 +03:30
src-tinkerer
0d63ad5a7f Use a single function for fetching channel contents 2023-11-22 14:52:17 +03:30
src-tinkerer
63e5d72466 Remove unused function produce_channel_livestream_url 2023-11-20 15:50:59 +03:30
src-tinkerer
b0df3774db Add sort options to streams 2023-11-01 21:56:25 +03:30
18 changed files with 259 additions and 144 deletions

View File

@ -20,6 +20,9 @@ Lint/ShadowingOuterLocalVar:
Excluded: Excluded:
- src/invidious/helpers/tokens.cr - src/invidious/helpers/tokens.cr
Lint/NotNil:
Enabled: false
# #
# Style # Style
@ -31,6 +34,13 @@ Style/RedundantBegin:
Style/RedundantReturn: Style/RedundantReturn:
Enabled: false Enabled: false
Style/ParenthesesAroundCondition:
Enabled: false
# This requires a rewrite of most data structs (and their usage) in Invidious.
Style/QueryBoolMethods:
Enabled: false
# #
# Metrics # Metrics
@ -39,50 +49,4 @@ Style/RedundantReturn:
# Ignore function complexity (number of if/else & case/when branches) # Ignore function complexity (number of if/else & case/when branches)
# For some functions that can hardly be simplified for now # For some functions that can hardly be simplified for now
Metrics/CyclomaticComplexity: Metrics/CyclomaticComplexity:
Excluded: Enabled: false
# get_about_info(ucid, locale) => [17/10]
- src/invidious/channels/about.cr
# fetch_channel_community(ucid, continuation, ...) => [34/10]
- src/invidious/channels/community.cr
# create_notification_stream(env, topics, connection_channel) => [14/10]
- src/invidious/helpers/helpers.cr:84:5
# get_index(plural_form, count) => [25/10]
- src/invidious/helpers/i18next.cr
# call(context) => [18/10]
- src/invidious/helpers/static_file_handler.cr
# show(env) => [38/10]
- src/invidious/routes/embed.cr
# get_video_playback(env) => [45/10]
- src/invidious/routes/video_playback.cr
# handle(env) => [40/10]
- src/invidious/routes/watch.cr
# playlist_ajax(env) => [24/10]
- src/invidious/routes/playlists.cr
# fetch_youtube_comments(id, cursor, ....) => [40/10]
# template_youtube_comments(comments, locale, ...) => [16/10]
# content_to_comment_html(content) => [14/10]
- src/invidious/comments.cr
# to_json(locale, json) => [21/10]
# extract_video_info(video_id, ...) => [44/10]
# process_video_params(query, preferences) => [20/10]
- src/invidious/videos.cr
#src/invidious/playlists.cr:327:5
#[C] Metrics/CyclomaticComplexity: Cyclomatic complexity too high [19/10]
# fetch_playlist(plid : String)
#src/invidious/playlists.cr:436:5
#[C] Metrics/CyclomaticComplexity: Cyclomatic complexity too high [11/10]
# extract_playlist_videos(initial_data : Hash(String, JSON::Any))

View File

@ -1,4 +1,4 @@
name: Build and release container name: Build and release container directly from master
on: on:
push: push:
@ -24,9 +24,9 @@ jobs:
uses: actions/checkout@v4 uses: actions/checkout@v4
- name: Install Crystal - name: Install Crystal
uses: crystal-lang/install-crystal@v1.8.0 uses: crystal-lang/install-crystal@v1.8.2
with: with:
crystal: 1.9.2 crystal: 1.12.2
- name: Run lint - name: Run lint
run: | run: |
@ -58,7 +58,7 @@ jobs:
images: quay.io/invidious/invidious images: quay.io/invidious/invidious
tags: | tags: |
type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }} type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }} type=raw,value=master,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
labels: | labels: |
quay.expires-after=12w quay.expires-after=12w
@ -83,7 +83,7 @@ jobs:
suffix=-arm64 suffix=-arm64
tags: | tags: |
type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }} type=sha,format=short,prefix={{date 'YYYY.MM.DD'}}-,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }} type=raw,value=master,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
labels: | labels: |
quay.expires-after=12w quay.expires-after=12w

View File

@ -0,0 +1,90 @@
name: Build and release container
on:
push:
tags:
- "v*"
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Crystal
uses: crystal-lang/install-crystal@v1.8.2
with:
crystal: 1.12.2
- name: Run lint
run: |
if ! crystal tool format --check; then
crystal tool format
git diff
exit 1
fi
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
with:
platforms: arm64
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to registry
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: quay.io/invidious/invidious
tags: |
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
labels: |
quay.expires-after=12w
- name: Build and push Docker AMD64 image for Push Event
uses: docker/build-push-action@v5
with:
context: .
file: docker/Dockerfile
platforms: linux/amd64
labels: ${{ steps.meta.outputs.labels }}
push: true
tags: ${{ steps.meta.outputs.tags }}
build-args: |
"release=1"
- name: Docker meta
id: meta-arm64
uses: docker/metadata-action@v5
with:
images: quay.io/invidious/invidious
flavor: |
suffix=-arm64
tags: |
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'master') }}
labels: |
quay.expires-after=12w
- name: Build and push Docker ARM64 image for Push Event
uses: docker/build-push-action@v5
with:
context: .
file: docker/Dockerfile.arm64
platforms: linux/arm64/v8
labels: ${{ steps.meta-arm64.outputs.labels }}
push: true
tags: ${{ steps.meta-arm64.outputs.tags }}
build-args: |
"release=1"

View File

@ -38,10 +38,10 @@ jobs:
matrix: matrix:
stable: [true] stable: [true]
crystal: crystal:
- 1.7.3
- 1.8.2
- 1.9.2 - 1.9.2
- 1.10.1 - 1.10.1
- 1.11.2
- 1.12.1
include: include:
- crystal: nightly - crystal: nightly
stable: false stable: false
@ -124,4 +124,28 @@ jobs:
- name: Test Docker - name: Test Docker
run: while curl -Isf http://localhost:3000; do sleep 1; done run: while curl -Isf http://localhost:3000; do sleep 1; done
ameba_lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Install Crystal
uses: crystal-lang/install-crystal@v1.8.0
with:
crystal: latest
- name: Cache Shards
uses: actions/cache@v3
with:
path: |
./lib
./bin
key: shards-${{ hashFiles('shard.lock') }}
- name: Install Shards
run: shards install
- name: Run Ameba linter
run: bin/ameba

View File

@ -1,4 +1,4 @@
FROM crystallang/crystal:1.8.2-alpine AS builder FROM crystallang/crystal:1.12.1-alpine AS builder
RUN apk add --no-cache sqlite-static yaml-static RUN apk add --no-cache sqlite-static yaml-static

View File

@ -1,5 +1,5 @@
FROM alpine:3.18 AS builder FROM alpine:3.19 AS builder
RUN apk add --no-cache 'crystal=1.8.2-r0' shards sqlite-static yaml-static yaml-dev libxml2-static zlib-static openssl-libs-static openssl-dev musl-dev xz-static RUN apk add --no-cache 'crystal=1.10.1-r0' shards sqlite-static yaml-static yaml-dev libxml2-static zlib-static openssl-libs-static openssl-dev musl-dev xz-static
ARG release ARG release

View File

@ -2,7 +2,7 @@ version: 2.0
shards: shards:
ameba: ameba:
git: https://github.com/crystal-ameba/ameba.git git: https://github.com/crystal-ameba/ameba.git
version: 1.5.0 version: 1.6.1
athena-negotiation: athena-negotiation:
git: https://github.com/athena-framework/negotiation.git git: https://github.com/athena-framework/negotiation.git

View File

@ -35,7 +35,7 @@ development_dependencies:
version: ~> 0.10.4 version: ~> 0.10.4
ameba: ameba:
github: crystal-ameba/ameba github: crystal-ameba/ameba
version: ~> 1.5.0 version: ~> 1.6.1
crystal: ">= 1.0.0, < 2.0.0" crystal: ">= 1.0.0, < 2.0.0"

View File

@ -1,4 +1,4 @@
def produce_channel_videos_continuation(ucid, page = 1, auto_generated = nil, sort_by = "newest", v2 = false) def produce_channel_content_continuation(ucid, content_type, page = 1, auto_generated = nil, sort_by = "newest", v2 = false)
object_inner_2 = { object_inner_2 = {
"2:0:embedded" => { "2:0:embedded" => {
"1:0:varint" => 0_i64, "1:0:varint" => 0_i64,
@ -16,6 +16,13 @@ def produce_channel_videos_continuation(ucid, page = 1, auto_generated = nil, so
.try { |i| Base64.urlsafe_encode(i) } .try { |i| Base64.urlsafe_encode(i) }
.try { |i| URI.encode_www_form(i) } .try { |i| URI.encode_www_form(i) }
content_type_numerical =
case content_type
when "videos" then 15
when "livestreams" then 14
else 15 # Fallback to "videos"
end
sort_by_numerical = sort_by_numerical =
case sort_by case sort_by
when "newest" then 1_i64 when "newest" then 1_i64
@ -27,7 +34,7 @@ def produce_channel_videos_continuation(ucid, page = 1, auto_generated = nil, so
object_inner_1 = { object_inner_1 = {
"110:embedded" => { "110:embedded" => {
"3:embedded" => { "3:embedded" => {
"15:embedded" => { "#{content_type_numerical}:embedded" => {
"1:embedded" => { "1:embedded" => {
"1:string" => object_inner_2_encoded, "1:string" => object_inner_2_encoded,
}, },
@ -62,6 +69,10 @@ def produce_channel_videos_continuation(ucid, page = 1, auto_generated = nil, so
return continuation return continuation
end end
def make_initial_content_ctoken(ucid, content_type, sort_by) : String
return produce_channel_content_continuation(ucid, content_type, sort_by: sort_by)
end
module Invidious::Channel::Tabs module Invidious::Channel::Tabs
extend self extend self
@ -69,10 +80,6 @@ module Invidious::Channel::Tabs
# Regular videos # Regular videos
# ------------------- # -------------------
def make_initial_video_ctoken(ucid, sort_by) : String
return produce_channel_videos_continuation(ucid, sort_by: sort_by)
end
# Wrapper for AboutChannel, as we still need to call get_videos with # Wrapper for AboutChannel, as we still need to call get_videos with
# an author name and ucid directly (e.g in RSS feeds). # an author name and ucid directly (e.g in RSS feeds).
# TODO: figure out how to get rid of that # TODO: figure out how to get rid of that
@ -94,7 +101,7 @@ module Invidious::Channel::Tabs
end end
def get_videos(author : String, ucid : String, *, continuation : String? = nil, sort_by = "newest") def get_videos(author : String, ucid : String, *, continuation : String? = nil, sort_by = "newest")
continuation ||= make_initial_video_ctoken(ucid, sort_by) continuation ||= make_initial_content_ctoken(ucid, "videos", sort_by)
initial_data = YoutubeAPI.browse(continuation: continuation) initial_data = YoutubeAPI.browse(continuation: continuation)
return extract_items(initial_data, author, ucid) return extract_items(initial_data, author, ucid)
@ -138,21 +145,18 @@ module Invidious::Channel::Tabs
# Livestreams # Livestreams
# ------------------- # -------------------
def get_livestreams(channel : AboutChannel, continuation : String? = nil) def get_livestreams(channel : AboutChannel, continuation : String? = nil, sort_by = "newest")
if continuation.nil? continuation ||= make_initial_content_ctoken(channel.ucid, "livestreams", sort_by)
# EgdzdHJlYW1z8gYECgJ6AA%3D%3D is the protobuf object to load "streams"
initial_data = YoutubeAPI.browse(channel.ucid, params: "EgdzdHJlYW1z8gYECgJ6AA%3D%3D")
else
initial_data = YoutubeAPI.browse(continuation: continuation) initial_data = YoutubeAPI.browse(continuation: continuation)
end
return extract_items(initial_data, channel.author, channel.ucid) return extract_items(initial_data, channel.author, channel.ucid)
end end
def get_60_livestreams(channel : AboutChannel, continuation : String? = nil) def get_60_livestreams(channel : AboutChannel, *, continuation : String? = nil, sort_by = "newest")
if continuation.nil? if continuation.nil?
# Fetch the first "page" of streams # Fetch the first "page" of stream
items, next_continuation = get_livestreams(channel) items, next_continuation = get_livestreams(channel, sort_by: sort_by)
else else
# Fetch a "page" of streams using the given continuation token # Fetch a "page" of streams using the given continuation token
items, next_continuation = get_livestreams(channel, continuation: continuation) items, next_continuation = get_livestreams(channel, continuation: continuation)

View File

@ -208,11 +208,12 @@ module Invidious::Routes::API::V1::Channels
get_channel() get_channel()
# Retrieve continuation from URL parameters # Retrieve continuation from URL parameters
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
continuation = env.params.query["continuation"]? continuation = env.params.query["continuation"]?
begin begin
videos, next_continuation = Channel::Tabs.get_60_livestreams( videos, next_continuation = Channel::Tabs.get_60_livestreams(
channel, continuation: continuation channel, continuation: continuation, sort_by: sort_by
) )
rescue ex rescue ex
return error_json(500, ex) return error_json(500, ex)

View File

@ -31,7 +31,7 @@ module Invidious::Routes::API::V1::Feeds
if !CONFIG.popular_enabled if !CONFIG.popular_enabled
error_message = {"error" => "Administrator has disabled this endpoint."}.to_json error_message = {"error" => "Administrator has disabled this endpoint."}.to_json
haltf env, 400, error_message haltf env, 403, error_message
end end
JSON.build do |json| JSON.build do |json|

View File

@ -89,9 +89,14 @@ module Invidious::Routes::API::V1::Videos
if CONFIG.use_innertube_for_captions if CONFIG.use_innertube_for_captions
params = Invidious::Videos::Transcript.generate_param(id, caption.language_code, caption.auto_generated) params = Invidious::Videos::Transcript.generate_param(id, caption.language_code, caption.auto_generated)
initial_data = YoutubeAPI.get_transcript(params)
webvtt = Invidious::Videos::Transcript.convert_transcripts_to_vtt(initial_data, caption.language_code) transcript = Invidious::Videos::Transcript.from_raw(
YoutubeAPI.get_transcript(params),
caption.language_code,
caption.auto_generated
)
webvtt = transcript.to_vtt
else else
# Timedtext API handling # Timedtext API handling
url = URI.parse("#{caption.base_url}&tlang=#{tlang}").request_target url = URI.parse("#{caption.base_url}&tlang=#{tlang}").request_target

View File

@ -30,7 +30,7 @@ module Invidious::Routes::BeforeAll
# Only allow the pages at /embed/* to be embedded # Only allow the pages at /embed/* to be embedded
if env.request.resource.starts_with?("/embed") if env.request.resource.starts_with?("/embed")
frame_ancestors = "'self' http: https:" frame_ancestors = "'self' file: http: https:"
else else
frame_ancestors = "'none'" frame_ancestors = "'none'"
end end

View File

@ -81,13 +81,12 @@ module Invidious::Routes::Channels
return env.redirect "/channel/#{channel.ucid}" return env.redirect "/channel/#{channel.ucid}"
end end
# TODO: support sort option for livestreams sort_by = env.params.query["sort_by"]?.try &.downcase || "newest"
sort_by = "" sort_options = {"newest", "oldest", "popular"}
sort_options = [] of String
# Fetch items and continuation token # Fetch items and continuation token
items, next_continuation = Channel::Tabs.get_60_livestreams( items, next_continuation = Channel::Tabs.get_60_livestreams(
channel, continuation: continuation channel, continuation: continuation, sort_by: sort_by
) )
selected_tab = Frontend::ChannelPage::TabsAvailable::Streams selected_tab = Frontend::ChannelPage::TabsAvailable::Streams

View File

@ -1,8 +1,26 @@
module Invidious::Videos module Invidious::Videos
# Namespace for methods primarily relating to Transcripts # A `Transcripts` struct encapsulates a sequence of lines that together forms the whole transcript for a given YouTube video.
module Transcript # These lines can be categorized into two types: section headings and regular lines representing content from the video.
record TranscriptLine, start_ms : Time::Span, end_ms : Time::Span, line : String struct Transcript
# Types
record HeadingLine, start_ms : Time::Span, end_ms : Time::Span, line : String
record RegularLine, start_ms : Time::Span, end_ms : Time::Span, line : String
alias TranscriptLine = HeadingLine | RegularLine
property lines : Array(TranscriptLine)
property language_code : String
property auto_generated : Bool
# User friendly label for the current transcript.
# Example: "English (auto-generated)"
property label : String
# Initializes a new Transcript struct with the contents and associated metadata describing it
def initialize(@lines : Array(TranscriptLine), @language_code : String, @auto_generated : Bool, @label : String)
end
# Generates a protobuf string to fetch the requested transcript from YouTube
def self.generate_param(video_id : String, language_code : String, auto_generated : Bool) : String def self.generate_param(video_id : String, language_code : String, auto_generated : Bool) : String
kind = auto_generated ? "asr" : "" kind = auto_generated ? "asr" : ""
@ -30,48 +48,79 @@ module Invidious::Videos
return params return params
end end
def self.convert_transcripts_to_vtt(initial_data : Hash(String, JSON::Any), target_language : String) : String # Constructs a Transcripts struct from the initial YouTube response
# Convert into array of TranscriptLine def self.from_raw(initial_data : Hash(String, JSON::Any), language_code : String, auto_generated : Bool)
lines = self.parse(initial_data) transcript_panel = initial_data.dig("actions", 0, "updateEngagementPanelAction", "content", "transcriptRenderer",
"content", "transcriptSearchPanelRenderer")
segment_list = transcript_panel.dig("body", "transcriptSegmentListRenderer")
if !segment_list["initialSegments"]?
raise NotFoundException.new("Requested transcript does not exist")
end
# Extract user-friendly label for the current transcript
footer_language_menu = transcript_panel.dig?(
"footer", "transcriptFooterRenderer", "languageMenu", "sortFilterSubMenuRenderer", "subMenuItems"
)
if footer_language_menu
label = footer_language_menu.as_a.select(&.["selected"].as_bool)[0]["title"].as_s
else
label = language_code
end
# Extract transcript lines
initial_segments = segment_list["initialSegments"].as_a
lines = [] of TranscriptLine
initial_segments.each do |line|
if unpacked_line = line["transcriptSectionHeaderRenderer"]?
line_type = HeadingLine
else
unpacked_line = line["transcriptSegmentRenderer"]
line_type = RegularLine
end
start_ms = unpacked_line["startMs"].as_s.to_i.millisecond
end_ms = unpacked_line["endMs"].as_s.to_i.millisecond
text = extract_text(unpacked_line["snippet"]) || ""
lines << line_type.new(start_ms, end_ms, text)
end
return Transcript.new(
lines: lines,
language_code: language_code,
auto_generated: auto_generated,
label: label
)
end
# Converts transcript lines to a WebVTT file
#
# This is used within Invidious to replace subtitles
# as to workaround YouTube's rate-limited timedtext endpoint.
def to_vtt
settings_field = { settings_field = {
"Kind" => "captions", "Kind" => "captions",
"Language" => target_language, "Language" => @language_code,
} }
# Taken from Invidious::Videos::Captions::Metadata.timedtext_to_vtt()
vtt = WebVTT.build(settings_field) do |vtt| vtt = WebVTT.build(settings_field) do |vtt|
lines.each do |line| @lines.each do |line|
# Section headers are excluded from the VTT conversion as to
# match the regular captions returned from YouTube as much as possible
next if line.is_a? HeadingLine
vtt.cue(line.start_ms, line.end_ms, line.line) vtt.cue(line.start_ms, line.end_ms, line.line)
end end
end end
return vtt return vtt
end end
private def self.parse(initial_data : Hash(String, JSON::Any))
body = initial_data.dig("actions", 0, "updateEngagementPanelAction", "content", "transcriptRenderer",
"content", "transcriptSearchPanelRenderer", "body", "transcriptSegmentListRenderer",
"initialSegments").as_a
lines = [] of TranscriptLine
body.each do |line|
# Transcript section headers. They are not apart of the captions and as such we can safely skip them.
if line.as_h.has_key?("transcriptSectionHeaderRenderer")
next
end
line = line["transcriptSegmentRenderer"]
start_ms = line["startMs"].as_s.to_i.millisecond
end_ms = line["endMs"].as_s.to_i.millisecond
text = extract_text(line["snippet"]) || ""
lines << TranscriptLine.new(start_ms, end_ms, text)
end
return lines
end
end end
end end

View File

@ -30,13 +30,13 @@
<meta property="og:site_name" content="Invidious"> <meta property="og:site_name" content="Invidious">
<meta property="og:url" content="<%= HOST_URL %>/channel/<%= ucid %>"> <meta property="og:url" content="<%= HOST_URL %>/channel/<%= ucid %>">
<meta property="og:title" content="<%= author %>"> <meta property="og:title" content="<%= author %>">
<meta property="og:image" content="/ggpht<%= channel_profile_pic %>"> <meta property="og:image" content="<%= HOST_URL %>/ggpht<%= channel_profile_pic %>">
<meta property="og:description" content="<%= channel.description %>"> <meta property="og:description" content="<%= channel.description %>">
<meta name="twitter:card" content="summary"> <meta name="twitter:card" content="summary">
<meta name="twitter:url" content="<%= HOST_URL %>/channel/<%= ucid %>"> <meta name="twitter:url" content="<%= HOST_URL %>/channel/<%= ucid %>">
<meta name="twitter:title" content="<%= author %>"> <meta name="twitter:title" content="<%= author %>">
<meta name="twitter:description" content="<%= channel.description %>"> <meta name="twitter:description" content="<%= channel.description %>">
<meta name="twitter:image" content="/ggpht<%= channel_profile_pic %>"> <meta name="twitter:image" content="<%= HOST_URL %>/ggpht<%= channel_profile_pic %>">
<link rel="alternate" type="application/rss+xml" title="RSS" href="/feed/channel/<%= ucid %>" /> <link rel="alternate" type="application/rss+xml" title="RSS" href="/feed/channel/<%= ucid %>" />
<%- end -%> <%- end -%>

View File

@ -10,7 +10,7 @@
<meta property="og:site_name" content="<%= author %> | Invidious"> <meta property="og:site_name" content="<%= author %> | Invidious">
<meta property="og:url" content="<%= HOST_URL %>/watch?v=<%= video.id %>"> <meta property="og:url" content="<%= HOST_URL %>/watch?v=<%= video.id %>">
<meta property="og:title" content="<%= title %>"> <meta property="og:title" content="<%= title %>">
<meta property="og:image" content="/vi/<%= video.id %>/maxres.jpg"> <meta property="og:image" content="<%= HOST_URL %>/vi/<%= video.id %>/maxres.jpg">
<meta property="og:description" content="<%= HTML.escape(video.short_description) %>"> <meta property="og:description" content="<%= HTML.escape(video.short_description) %>">
<meta property="og:type" content="video.other"> <meta property="og:type" content="video.other">
<meta property="og:video:url" content="<%= HOST_URL %>/embed/<%= video.id %>"> <meta property="og:video:url" content="<%= HOST_URL %>/embed/<%= video.id %>">

View File

@ -5,9 +5,6 @@
module YoutubeAPI module YoutubeAPI
extend self extend self
private DEFAULT_API_KEY = "AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"
private ANDROID_API_KEY = "AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w"
# For Android versions, see https://en.wikipedia.org/wiki/Android_version_history # For Android versions, see https://en.wikipedia.org/wiki/Android_version_history
private ANDROID_APP_VERSION = "19.14.42" private ANDROID_APP_VERSION = "19.14.42"
private ANDROID_USER_AGENT = "com.google.android.youtube/19.14.42 (Linux; U; Android 12; US) gzip" private ANDROID_USER_AGENT = "com.google.android.youtube/19.14.42 (Linux; U; Android 12; US) gzip"
@ -52,7 +49,6 @@ module YoutubeAPI
name: "WEB", name: "WEB",
name_proto: "1", name_proto: "1",
version: "2.20240304.00.00", version: "2.20240304.00.00",
api_key: DEFAULT_API_KEY,
screen: "WATCH_FULL_SCREEN", screen: "WATCH_FULL_SCREEN",
os_name: "Windows", os_name: "Windows",
os_version: WINDOWS_VERSION, os_version: WINDOWS_VERSION,
@ -62,7 +58,6 @@ module YoutubeAPI
name: "WEB_EMBEDDED_PLAYER", name: "WEB_EMBEDDED_PLAYER",
name_proto: "56", name_proto: "56",
version: "1.20240303.00.00", version: "1.20240303.00.00",
api_key: DEFAULT_API_KEY,
screen: "EMBED", screen: "EMBED",
os_name: "Windows", os_name: "Windows",
os_version: WINDOWS_VERSION, os_version: WINDOWS_VERSION,
@ -72,7 +67,6 @@ module YoutubeAPI
name: "MWEB", name: "MWEB",
name_proto: "2", name_proto: "2",
version: "2.20240304.08.00", version: "2.20240304.08.00",
api_key: DEFAULT_API_KEY,
os_name: "Android", os_name: "Android",
os_version: ANDROID_VERSION, os_version: ANDROID_VERSION,
platform: "MOBILE", platform: "MOBILE",
@ -81,7 +75,6 @@ module YoutubeAPI
name: "WEB", name: "WEB",
name_proto: "1", name_proto: "1",
version: "2.20240304.00.00", version: "2.20240304.00.00",
api_key: DEFAULT_API_KEY,
screen: "EMBED", screen: "EMBED",
os_name: "Windows", os_name: "Windows",
os_version: WINDOWS_VERSION, os_version: WINDOWS_VERSION,
@ -94,7 +87,6 @@ module YoutubeAPI
name: "ANDROID", name: "ANDROID",
name_proto: "3", name_proto: "3",
version: ANDROID_APP_VERSION, version: ANDROID_APP_VERSION,
api_key: ANDROID_API_KEY,
android_sdk_version: ANDROID_SDK_VERSION, android_sdk_version: ANDROID_SDK_VERSION,
user_agent: ANDROID_USER_AGENT, user_agent: ANDROID_USER_AGENT,
os_name: "Android", os_name: "Android",
@ -105,13 +97,11 @@ module YoutubeAPI
name: "ANDROID_EMBEDDED_PLAYER", name: "ANDROID_EMBEDDED_PLAYER",
name_proto: "55", name_proto: "55",
version: ANDROID_APP_VERSION, version: ANDROID_APP_VERSION,
api_key: "AIzaSyCjc_pVEDi4qsv5MtC2dMXzpIaDoRFLsxw",
}, },
ClientType::AndroidScreenEmbed => { ClientType::AndroidScreenEmbed => {
name: "ANDROID", name: "ANDROID",
name_proto: "3", name_proto: "3",
version: ANDROID_APP_VERSION, version: ANDROID_APP_VERSION,
api_key: DEFAULT_API_KEY,
screen: "EMBED", screen: "EMBED",
android_sdk_version: ANDROID_SDK_VERSION, android_sdk_version: ANDROID_SDK_VERSION,
user_agent: ANDROID_USER_AGENT, user_agent: ANDROID_USER_AGENT,
@ -123,7 +113,6 @@ module YoutubeAPI
name: "ANDROID_TESTSUITE", name: "ANDROID_TESTSUITE",
name_proto: "30", name_proto: "30",
version: ANDROID_TS_APP_VERSION, version: ANDROID_TS_APP_VERSION,
api_key: ANDROID_API_KEY,
android_sdk_version: ANDROID_SDK_VERSION, android_sdk_version: ANDROID_SDK_VERSION,
user_agent: ANDROID_TS_USER_AGENT, user_agent: ANDROID_TS_USER_AGENT,
os_name: "Android", os_name: "Android",
@ -137,7 +126,6 @@ module YoutubeAPI
name: "IOS", name: "IOS",
name_proto: "5", name_proto: "5",
version: IOS_APP_VERSION, version: IOS_APP_VERSION,
api_key: "AIzaSyB-63vPrdThhKuerbB2N_l7Kwwcxj6yUAc",
user_agent: IOS_USER_AGENT, user_agent: IOS_USER_AGENT,
device_make: "Apple", device_make: "Apple",
device_model: "iPhone14,5", device_model: "iPhone14,5",
@ -149,7 +137,6 @@ module YoutubeAPI
name: "IOS_MESSAGES_EXTENSION", name: "IOS_MESSAGES_EXTENSION",
name_proto: "66", name_proto: "66",
version: IOS_APP_VERSION, version: IOS_APP_VERSION,
api_key: DEFAULT_API_KEY,
user_agent: IOS_USER_AGENT, user_agent: IOS_USER_AGENT,
device_make: "Apple", device_make: "Apple",
device_model: "iPhone14,5", device_model: "iPhone14,5",
@ -161,7 +148,6 @@ module YoutubeAPI
name: "IOS_MUSIC", name: "IOS_MUSIC",
name_proto: "26", name_proto: "26",
version: "6.42", version: "6.42",
api_key: "AIzaSyBAETezhkwP0ZWA02RsqT1zu78Fpt0bC_s",
user_agent: "com.google.ios.youtubemusic/6.42 (iPhone14,5; U; CPU iOS 17_4 like Mac OS X;)", user_agent: "com.google.ios.youtubemusic/6.42 (iPhone14,5; U; CPU iOS 17_4 like Mac OS X;)",
device_make: "Apple", device_make: "Apple",
device_model: "iPhone14,5", device_model: "iPhone14,5",
@ -176,13 +162,11 @@ module YoutubeAPI
name: "TVHTML5", name: "TVHTML5",
name_proto: "7", name_proto: "7",
version: "7.20240304.10.00", version: "7.20240304.10.00",
api_key: DEFAULT_API_KEY,
}, },
ClientType::TvHtml5ScreenEmbed => { ClientType::TvHtml5ScreenEmbed => {
name: "TVHTML5_SIMPLY_EMBEDDED_PLAYER", name: "TVHTML5_SIMPLY_EMBEDDED_PLAYER",
name_proto: "85", name_proto: "85",
version: "2.0", version: "2.0",
api_key: DEFAULT_API_KEY,
screen: "EMBED", screen: "EMBED",
}, },
} }
@ -237,11 +221,6 @@ module YoutubeAPI
HARDCODED_CLIENTS[@client_type][:version] HARDCODED_CLIENTS[@client_type][:version]
end end
# :ditto:
def api_key : String
HARDCODED_CLIENTS[@client_type][:api_key]
end
# :ditto: # :ditto:
def screen : String def screen : String
HARDCODED_CLIENTS[@client_type][:screen]? || "" HARDCODED_CLIENTS[@client_type][:screen]? || ""
@ -606,7 +585,7 @@ module YoutubeAPI
client_config ||= DEFAULT_CLIENT_CONFIG client_config ||= DEFAULT_CLIENT_CONFIG
# Query parameters # Query parameters
url = "#{endpoint}?key=#{client_config.api_key}&prettyPrint=false" url = "#{endpoint}?prettyPrint=false"
headers = HTTP::Headers{ headers = HTTP::Headers{
"Content-Type" => "application/json; charset=UTF-8", "Content-Type" => "application/json; charset=UTF-8",