Reduce latency by 1 frame #646
Conversation
Add an option to set a custom window title. Signed-off-by: Romain Vimont <rom@rom1v.com>
add option window-title to set the title
rom1v
force-pushed
the
lowlatency
branch
2 times, most recently
from
0f0ccf4
to
0aa9828
Compare
A drag & drop always pushed the file to /sdcard/. Add an option to customize the target directory. Fixes <#659>
To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
|
Tried with scrcpy --bit-rate 1M --max-size 240 --max-fps 30 got 2 frame latency at 136x240, It's my device dependent, if it was faster than rom1v's unknown device, it would be 0 to 1 frame latency, without scrcpy, I would struggle with compressed image, inconsistent frame rate and image freeze spikes, thanks that scrcpy existed!!!!!! 😇 |
Wow, that's a lot of latency for such a small definition.
The device I used in this test was a Nexus 5. The results are better with newer devices (like a OnePlus 7 Pro). |
|
Actually x265 outperforms in everything x264: x265: |
|
I'm not sure to understand the screenshots, but it depends a lot on the device and hardware encoder. For example, on a One Plus 7 Pro, the h264 encoder has a lower latency than the h265 encoder. |
Basically i wrote a c++ program to display "i+1" number every 1ms, so i can take a photo and see how different are numbers on my phone and on my pc, basically latency. It would be so much better, if h265 was the default one. UPD: i just made a super slow motion video with both screens, it appears, that the latency is around 8-9ms, while using h265 |
This PR contains unrelated commit after I pushed force it (I cannot change the base branch). The relevant commit is 63af7fb.
To packetize the H.264 raw stream, av_parser_parse2() (called by
av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder!On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording).
On the client side, replace
av_read_frame()by manual packetizing and parsing.https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay
https://trac.ffmpeg.org/ticket/3354
I'm opening a pull request to get more tests before merging (the changes are quite sensitive), to avoid regressions.
To confirm that it actually reduces the end-to-end latency by 1 frame, I created a video which increments a number at 30fps (every 33.3ms):
I run this video in VLC on Android, and start scrcpy.
I compared the current master with this PR for both 1080x1920 (just running
scrcpy) and 448x800 (runningscrcpy -m800).1080x1920
on
masterEnd-to-end latency is about 3 frames (~100ms).
on
lowlatencyEnd-to-end latency is about 2 frames (~67ms).
Once a packet is received by the client, on average, the frame is rendered 12ms later.
(with "skip frames" disabled, i.e.
--render-expired-frames)448x800
on
masterEnd-to-end latency is about 2 frames (~67ms).
on
lowlatencyEnd-to-end latency is about 1 frame (~33ms).
Once a packet is received by the client, on average, the frame is rendered 8ms later.
(with "skip frames" disabled, i.e.
-m800 --render-expired-frames)For now, I preserved the compatibility with prebuilt server v1.9, so you can just:
git fetch origin && git checkout lowlatencythen build as usual.
For windows users, I built a binary (replace the one from v1.9):
scrcpy.exe.SHA-256: 6b8eae4eb4df811b6aa194320f00b51102cf3a254e432f353963863c82f362ea
Thank you for your feedbacks 😉