3265 Commits

Author SHA1 Message Date
syeopite
abaaeb7f4a
Use single global SSL context across HTTP Clients
The standard Crystal `HTTP::Client` creates a new SSL context for every
client/reconnection.  This is not ideal because creating a new SSL
context is an incredibly slow operation.

The standard practice instead is to cache or use a single SSL CTX
throughout the lifetime of the application. The OpenSSL context has been
designed with this in mind, and even explicitly recommends it. It even
works concurrently across many threads.

Reusing a single SSL context significantly speeds up the process of
starting and restarting clients. Which is incredibly important in the
case of the Invidious connection pool where new clients are often spun
up.

The only caveat is that a cached SSL context may become out of sync with
the system ca store. But for an Invidious instance following the best
practices of frequent restarts this is an non-issue.

See https://github.com/crystal-lang/crystal/issues/15419
2025-05-13 06:23:51 -07:00
syeopite
0270917756
Disable auto reconnect for companion pool clients 2025-05-13 06:23:51 -07:00
syeopite
5bc6d11d5e
Rework HTTP::Client logic
Implement force resolve via IO constructor

Use `DB::Pool#retry` in connection pool

`HTTP::Clients` created through its IO constructor cannot reconnect to
closed connections, and instead will result in an error on usage.

The `DB::Pool#retry` allows us to cycle until we find a client that
that still has an open connection, or to create a new one
if necessary. The poisoned clients are then removed once identified
by `#retry`'s internal logic

The fact that clients with closed connections will now be slowly removed
also means that Invidious will no longer use the same pattern of companion
connections within the pool; distributing routes more evenly.
2025-05-13 06:23:51 -07:00
syeopite
6ff616d9a4
Ensure http-proxy is not used for companion 2025-05-13 05:18:52 -07:00
syeopite
bd2f00f6df
Formatting 2025-05-13 05:18:52 -07:00
syeopite
d70b05e5c3
Remove idle_pool_size config
Clients created when idle_capacity is reached but not max_capacity
are discarded as soon as the client is checked back into the pool,
not when the connection is closed.

This means that allowing idle_capacity to be lower than max_capacity
essentially just makes the remaining clients a checkout timeout
deterrent that gets thrown away as soon as it is used. Not useful
for reusing connections whatsoever during peak load times
2025-05-13 05:18:52 -07:00
syeopite
924ae11721
Remove extraneous space 2025-05-13 05:18:52 -07:00
syeopite
4628f98eeb
Merge companion and standard pool into one 2025-05-13 05:18:52 -07:00
syeopite
60b45290bd
Pool: raise custom error when DB::PoolTimeout 2025-05-13 05:18:52 -07:00
syeopite
e5795cab3d
Pool: Make Pool#client method public 2025-05-13 05:18:52 -07:00
syeopite
7fe8f6f24e
Use non-streaming api when not invoked with block
Defaulting to the streaming api of `HTTP::Client` causes some issues
since the streaming respone content needs to be accessed
through #body_io rather than #body
2025-05-13 05:18:52 -07:00
syeopite
936c0ff61b
Update comment on reiniting proxy of pooled client 2025-05-13 05:18:52 -07:00
syeopite
802790e468
Simplify namespace calls to ConnectionPool 2025-05-13 05:18:52 -07:00
syeopite
3dd68872f9
Pool: remove redundant properties 2025-05-13 05:18:52 -07:00
syeopite
be9749aefd
Pool: Refactor logic for request methods
Make non-block request method internally call
the block based request method.
2025-05-13 05:18:51 -07:00
syeopite
7cd974dc24
Release client only when it still exists
@pool.release should not be called when the client has already been
deleted from the pool.
2025-05-13 05:18:51 -07:00
syeopite
bf2eccbcaf
Connection pool: ensure response is fully read
The streaming API of HTTP::Client has an internal buffer
that will continue to persist onto the next request unless
the response is fully read.

This commit privatizes the #client method of Pool and instead
expose various HTTP request methods that will call and yield
the underlying request and response.

This way, we can ensure that the resposne is fully read before
the client is passed back into the pool for another request.
2025-05-13 05:18:51 -07:00
syeopite
4e4a084f22
Delete broken clients from the pool explicitly 2025-05-13 05:18:51 -07:00
syeopite
65e6929037
Remove redundant pool.release
pool.checkout(&block) already ensures that the checked out item
will be released back into the pool
2025-05-13 05:18:51 -07:00
syeopite
97b13c042e
Add config to set connection pool checkout timeout 2025-05-13 05:18:51 -07:00
syeopite
c86846e7b7
Move ytimg pool logic to Invidious::ConnectionPool 2025-05-13 05:18:51 -07:00
syeopite
95a1f632d8
Move client logic file to connection subfolder 2025-05-13 05:18:51 -07:00
syeopite
220adf53f4
Refactor connection pooling logic
- Remove duplication between standard and companion pool
- Raises a wrapped exception on any DB:Error
- Don't use a non-pool client when client fails
- Ensure that client is always released
- Add documentation to various pool methods
2025-05-13 05:18:51 -07:00
syeopite
c2ede1d2a5
Add support for setting max idle http pool size 2025-05-13 05:18:51 -07:00
Émilien (perso)
9e3c0dfd85
fix: fallback first with TVHTML then MWEB
fixes #5273
2025-05-08 19:55:22 +02:00
absidue
2c1400c41e
Fix proxying live DASH streams (#4589) 2025-05-03 20:28:19 +00:00
Alex Maras
8fd0b82c38
feat: route to invidious companion on downloads (#5224) 2025-05-03 01:28:18 +02:00
Émilien (perso)
7579adc3a3
fix: fallback other yt clients no url found for adaptive formats (#5262) 2025-05-02 16:57:02 +02:00
Émilien (perso)
0c07e9d27a
chore: set dash by default (#5216) 2025-04-04 14:00:29 +02:00
Émilien (perso)
23ff6135bb
chore: enforce 16 characters for invidious_companion_key (#5220) 2025-03-26 15:27:59 +01:00
Émilien (perso)
70ff463cc6
Add invidious companion support (#4985)
* add support for invidious companion

* redirect latest_version and dash manifest to invidious companion

* fix Shadowing outer local variable `response`

* fixing condition for Content-Security-Policy

* throw error if inv_sig_helper and invidious_companion used same time

* Use sample instead of Random.rand

Co-authored-by: syeopite <70992037+syeopite@users.noreply.github.com>

* Remove debug puts functions

Co-authored-by: syeopite <70992037+syeopite@users.noreply.github.com>

* modify the description for config.example.yaml about invidious companion

* move config checks for invidious companion

* separate invidious_companion logic + better config.yaml config

* fixing "end" misplacement

* fix linting + use .empty?

* crystal handle decompression already by itself

* fix download function when invidious companion used

* fix linting

* invidious companion always used so always add CSP and redirect latest_version

* apply all the suggestions + rework invidious_companion parameter

* format watch.cr

* fix ameba Redundant use of `Object#to_s` in interpolation

* add ability for invidious companion to check request from invidious

* Better document private_url and public_url

* Better doc for invidious_companion_key

* !empty? to present?

* skip proxy for invidious companion

* fixing format

* missing ,

* add companion pooling http

* fix: don't use http proxy when sending requests to companion

* fix: logic where we want to have the invidious logic if companion is not used

* chore: remove baseurl usage from invidious companion

* chore: change from inv-sig-helper to companion for required playback

* fix: use puts + add warning for inv-sig-helper deprecated

---------

Co-authored-by: syeopite <70992037+syeopite@users.noreply.github.com>
2025-03-13 16:44:00 +01:00
syeopite
5c8b4eb379
Warn when po_token, visitor_data and/or inv-sig-helper is not configured (#5202)
* Warn when required configs for playback is missing

* Add link to documentation in warnings

* Direct users to /installation instead
2025-03-12 10:11:17 +01:00
syeopite
adcdb8cb92
Fix lint and formatting 2025-02-26 14:18:50 -08:00
syeopite
fe4fa0480a
Fix HLS being used for non-livestream videos (#5189)
Invidious does not currently support non-livestream hls playback

Originally, the HLS manifest check was essentially a boolean:
if the HLS manifest field was present, it was assumed to be a
livestream. Some videos include the HLS Manifest but aren't
livestreams.

In the case where they are livestreams, the video contains a videoType
field with the value "Livestream". In the case that they're normal
videos, the videoType is "Video". This is exposed via the
`video.live_now` property.

This commit just checks that `video.live_now` is true before treating
it as a livestream
2025-02-26 14:14:29 -08:00
syeopite
dbbcacc955
Images: fix typo in thumbnail logic 2025-02-26 14:13:58 -08:00
syeopite
58ad848d56
Channels: Support YouTube's change to from /community to /posts (#5183) 2025-02-26 14:13:22 -08:00
syeopite
6ac74f4362
Videos: Fix empty response when rv published field is nonexistent (#5162)
Fixes #5161 by checking recommended videos published field for presence
before attempting to parse it in api
2025-02-26 14:09:28 -08:00
syeopite
9fbe3944b0
Channels: Add Courses to channel page and channel API (#5158)
Closes #5144
2025-02-26 14:08:44 -08:00
syeopite
c5e9447f41
Pick a different instance upon redirect (#5154)
The automatic instance redirection has the potential to pick
the same instance the user is currently on. This is especially
prevalent when the instance list is limited in number like how it is
today.

This PR checks the domain of the instance and ensures that it is not
the same as the current instane before redirecting the user to it.
Otherwise, it just sends the user to rediret.invidious.io
2025-02-26 14:05:21 -08:00
syeopite
3e329410d1
Add the ability to listen on UNIX sockets (#5112) 2025-02-26 14:04:29 -08:00
syeopite
74dfda150e
i18n: Enable Tamil 2025-02-26 14:02:57 -08:00
syeopite
3d77635a5c
Add API endpoint for fetching transcripts from YouTube (#4788) 2025-02-26 13:56:39 -08:00
syeopite
4ea4878d1a
User: Batch notifications together 2025-02-26 13:55:25 -08:00
syeopite
1f0a89fb5f
RSS: Channel + Playlist improvements (#4298) 2025-02-26 13:55:01 -08:00
syeopite
f95f87e448
Frontend: Add a first page and previous page buttons for channel navigation (#4123) 2025-02-26 13:54:25 -08:00
Alex Maras
49afbf2a14 Fix an issue with the HLS manifest check for livestream videos
Originally, the HLS manifest check was essentially a boolean: if the HLS
manifest field was present, it was assumed to be a livestream. Some
videos include the HLS Manifest but aren't livestreams.

In the case where they are livestreams, the video contains a videoType
field with the value "Livestream". In the case that they're normal
videos, the videoType is "Video". This is exposed via the video.live_now
method.

This commit just checks that video.live_now is true before treating it
as a livestream
2025-02-21 16:30:39 +08:00
syeopite
d853b9f6dc
Typo
Co-authored-by: Samantaz Fox <coding@samantaz.fr>
2025-02-18 14:46:18 -08:00
Fijxu
d70681538a
Channels: Fix community tab 2025-02-18 19:20:55 -03:00
Drikanis
29219c46a1 fix 5161 by checking recommended videos published field for presence instead of just not nil 2025-01-28 19:40:15 -07:00
epicsam123
a77f083a0a
remove ! on reject 2025-01-26 16:42:59 -05:00