Android/scrcpy
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity
scrcpy/app/src/decoder.c

133 lines
3.7 KiB
C
Raw Normal View History

Initial commit Start a new clean history from here.
2017-12-12 22:12:07 +08:00
#include "decoder.h"
Add missing includes Refs #2616 <https://github.com/Genymobile/scrcpy/issues/2616>
2021-09-21 00:27:37 +08:00
#include <libavcodec/avcodec.h>
Initial commit Start a new clean history from here.
2017-12-12 22:12:07 +08:00
#include <libavformat/avformat.h>
#include "events.h"
Make decoder push frames to sinks Now that screen implements the packet sink trait, make decoder push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
#include "trait/frame_sink.h"
Move utilities to util/
2019-11-24 18:53:00 +08:00
#include "util/log.h"
Initial commit Start a new clean history from here.
2017-12-12 22:12:07 +08:00
Expose decoder as packet sink Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.
2021-04-11 21:01:05 +08:00
/** Downcast packet_sink to decoder */
Use sc_ prefix for decoder
2022-02-03 02:27:41 +08:00
#define DOWNCAST(SINK) container_of(SINK, struct sc_decoder, packet_sink)
Expose decoder as packet sink Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.
2021-04-11 21:01:05 +08:00
Make decoder push frames to sinks Now that screen implements the packet sink trait, make decoder push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
static bool
Use sc_ prefix for decoder
2022-02-03 02:27:41 +08:00
sc_decoder_open(struct sc_decoder *decoder, const AVCodec *codec) {
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
decoder->codec_ctx = avcodec_alloc_context3(codec);
if (!decoder->codec_ctx) {
Add OutOfMemory log helper Add a special LOG_OOM() function to log all OutOfMemory errors (i.e. allocations returning NULL).
2021-11-25 05:06:11 +08:00
LOG_OOM();
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
return false;
Initial commit Start a new clean history from here.
2017-12-12 22:12:07 +08:00
}
Set "low delay" decoder flag I don't really know the concrete benefits, but scrcpy definitely wants low delay decoding. Suggested-by: François Cartegnie <fcvlcdev@free.fr>
2021-12-07 20:59:20 +08:00
decoder->codec_ctx->flags |= AV_CODEC_FLAG_LOW_DELAY;
Pass AVCodecContext to frame sinks Frame consumers may need details about the frame format. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-25 23:19:58 +08:00
if (codec->type == AVMEDIA_TYPE_VIDEO) {
// Hardcoded video properties
decoder->codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;
}
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
if (avcodec_open2(decoder->codec_ctx, codec, NULL) < 0) {
Give a name to decoder instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-25 04:22:35 +08:00
LOGE("Decoder '%s': could not open codec", decoder->name);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
avcodec_free_context(&decoder->codec_ctx);
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
return false;
Initial commit Start a new clean history from here.
2017-12-12 22:12:07 +08:00
}
Make video_buffer more generic The video buffer took ownership of the producer frame (so that it could swap frames quickly). In order to support multiple sinks plugged to the decoder, the decoded frame must not be consumed by the display video buffer. Therefore, move the producer and consumer frames out of the video buffer, and use FFmpeg AVFrame refcounting to share ownership while avoiding copies.
2021-04-11 21:01:05 +08:00
decoder->frame = av_frame_alloc();
if (!decoder->frame) {
Add OutOfMemory log helper Add a special LOG_OOM() function to log all OutOfMemory errors (i.e. allocations returning NULL).
2021-11-25 05:06:11 +08:00
LOG_OOM();
Make video_buffer more generic The video buffer took ownership of the producer frame (so that it could swap frames quickly). In order to support multiple sinks plugged to the decoder, the decoded frame must not be consumed by the display video buffer. Therefore, move the producer and consumer frames out of the video buffer, and use FFmpeg AVFrame refcounting to share ownership while avoiding copies.
2021-04-11 21:01:05 +08:00
avcodec_close(decoder->codec_ctx);
avcodec_free_context(&decoder->codec_ctx);
return false;
}
Use frame source trait in decoder
2023-03-02 16:37:36 +08:00
if (!sc_frame_source_sinks_open(&decoder->frame_source,
decoder->codec_ctx)) {
Make decoder push frames to sinks Now that screen implements the packet sink trait, make decoder push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
av_frame_free(&decoder->frame);
avcodec_close(decoder->codec_ctx);
avcodec_free_context(&decoder->codec_ctx);
return false;
}
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
return true;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Enable video output file, with pts set by server
2018-10-11 13:12:36 +08:00
Make decoder push frames to sinks Now that screen implements the packet sink trait, make decoder push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
static void
Use sc_ prefix for decoder
2022-02-03 02:27:41 +08:00
sc_decoder_close(struct sc_decoder *decoder) {
Use frame source trait in decoder
2023-03-02 16:37:36 +08:00
sc_frame_source_sinks_close(&decoder->frame_source);
Make video_buffer more generic The video buffer took ownership of the producer frame (so that it could swap frames quickly). In order to support multiple sinks plugged to the decoder, the decoded frame must not be consumed by the display video buffer. Therefore, move the producer and consumer frames out of the video buffer, and use FFmpeg AVFrame refcounting to share ownership while avoiding copies.
2021-04-11 21:01:05 +08:00
av_frame_free(&decoder->frame);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
avcodec_close(decoder->codec_ctx);
avcodec_free_context(&decoder->codec_ctx);
}
Initial commit Start a new clean history from here.
2017-12-12 22:12:07 +08:00
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
static bool
Use sc_ prefix for decoder
2022-02-03 02:27:41 +08:00
sc_decoder_push(struct sc_decoder *decoder, const AVPacket *packet) {
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
bool is_config = packet->pts == AV_NOPTS_VALUE;
if (is_config) {
// nothing to do
return true;
}
Handle EAGAIN on send_packet in decoder EAGAIN was only handled on receive_frame. In practice, it should not be necessary, since one packet always contains one frame. But just in case.
2021-04-18 23:13:58 +08:00
int ret = avcodec_send_packet(decoder->codec_ctx, packet);
if (ret < 0 && ret != AVERROR(EAGAIN)) {
Give a name to decoder instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-25 04:22:35 +08:00
LOGE("Decoder '%s': could not send video packet: %d",
decoder->name, ret);
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
return false;
Initial commit Start a new clean history from here.
2017-12-12 22:12:07 +08:00
}
Call avcodec_receive_frame() in a loop Since in scrcpy a video packet passed to avcodec_send_packet() is always a complete video frame, it is sufficient to call avcodec_receive_frame() exactly once. In practice, it also works for audio packets: the decoder produces exactly 1 frame for 1 input packet. In theory, it is an implementation detail though, so avcodec_receive_frame() should be called in a loop.
2023-03-01 04:20:28 +08:00
for (;;) {
ret = avcodec_receive_frame(decoder->codec_ctx, decoder->frame);
if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {
break;
}
if (ret) {
LOGE("Decoder '%s', could not receive video frame: %d",
decoder->name, ret);
return false;
}
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
// a frame was received
Use frame source trait in decoder
2023-03-02 16:37:36 +08:00
bool ok = sc_frame_source_sinks_push(&decoder->frame_source,
decoder->frame);
Make video_buffer more generic The video buffer took ownership of the producer frame (so that it could swap frames quickly). In order to support multiple sinks plugged to the decoder, the decoded frame must not be consumed by the display video buffer. Therefore, move the producer and consumer frames out of the video buffer, and use FFmpeg AVFrame refcounting to share ownership while avoiding copies.
2021-04-11 21:01:05 +08:00
// A frame lost should not make the whole pipeline fail. The error, if
// any, is already logged.
(void) ok;
Unref decoder AVFrame immediately The frame can be unref immediately after it is pushed to the frame sinks. It was not really a memory leak because the frame was unref every time by avcodec_receive_frame() (and freed on close), but a reference was unnecessarily kept for too long.
2021-04-27 00:05:43 +08:00
av_frame_unref(decoder->frame);
Enable video output file, with pts set by server
2018-10-11 13:12:36 +08:00
}
Call avcodec_receive_frame() in a loop Since in scrcpy a video packet passed to avcodec_send_packet() is always a complete video frame, it is sufficient to call avcodec_receive_frame() exactly once. In practice, it also works for audio packets: the decoder produces exactly 1 frame for 1 input packet. In theory, it is an implementation detail though, so avcodec_receive_frame() should be called in a loop.
2023-03-01 04:20:28 +08:00
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
return true;
Initial commit Start a new clean history from here.
2017-12-12 22:12:07 +08:00
}
Reorder decoder functions This will make further commits more readable.
2021-04-11 21:01:05 +08:00
Expose decoder as packet sink Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.
2021-04-11 21:01:05 +08:00
static bool
Use sc_ prefix for decoder
2022-02-03 02:27:41 +08:00
sc_decoder_packet_sink_open(struct sc_packet_sink *sink, const AVCodec *codec) {
struct sc_decoder *decoder = DOWNCAST(sink);
return sc_decoder_open(decoder, codec);
Expose decoder as packet sink Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.
2021-04-11 21:01:05 +08:00
}
static void
Use sc_ prefix for decoder
2022-02-03 02:27:41 +08:00
sc_decoder_packet_sink_close(struct sc_packet_sink *sink) {
struct sc_decoder *decoder = DOWNCAST(sink);
sc_decoder_close(decoder);
Expose decoder as packet sink Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.
2021-04-11 21:01:05 +08:00
}
static bool
Use sc_ prefix for decoder
2022-02-03 02:27:41 +08:00
sc_decoder_packet_sink_push(struct sc_packet_sink *sink,
const AVPacket *packet) {
struct sc_decoder *decoder = DOWNCAST(sink);
return sc_decoder_push(decoder, packet);
Expose decoder as packet sink Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.
2021-04-11 21:01:05 +08:00
}
Reorder decoder functions This will make further commits more readable.
2021-04-11 21:01:05 +08:00
void
Give a name to decoder instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-25 04:22:35 +08:00
sc_decoder_init(struct sc_decoder *decoder, const char *name) {
decoder->name = name; // statically allocated
Use frame source trait in decoder
2023-03-02 16:37:36 +08:00
sc_frame_source_init(&decoder->frame_source);
Explicitly initialize decoder sink_count The zero-initialization relied on the fact that the decoder instance is static.
2021-05-29 03:23:10 +08:00
Expose decoder as packet sink Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.
2021-04-11 21:01:05 +08:00
static const struct sc_packet_sink_ops ops = {
Use sc_ prefix for decoder
2022-02-03 02:27:41 +08:00
.open = sc_decoder_packet_sink_open,
.close = sc_decoder_packet_sink_close,
.push = sc_decoder_packet_sink_push,
Expose decoder as packet sink Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.
2021-04-11 21:01:05 +08:00
};
decoder->packet_sink.ops = &ops;
Reorder decoder functions This will make further commits more readable.
2021-04-11 21:01:05 +08:00
}
Reference in a new issue Copy permalink