/** * Copyright 2012-2024 JogAmp Community. All rights reserved. * * Redistribution and use in source and binary forms, with or without modification, are * permitted provided that the following conditions are met: * * 1. Redistributions of source code must retain the above copyright notice, this list of * conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright notice, this list * of conditions and the following disclaimer in the documentation and/or other materials * provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY JogAmp Community ``AS IS'' AND ANY EXPRESS OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL JogAmp Community OR * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON * ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * The views and conclusions contained in the software and documentation are those of the * authors and should not be interpreted as representing official policies, either expressed * or implied, of JogAmp Community. */ package com.jogamp.opengl.util.av; import com.jogamp.opengl.GL; import com.jogamp.opengl.GLEventListener; import com.jogamp.opengl.GLException; import jogamp.opengl.Debug; import java.io.PrintStream; import java.util.List; import com.jogamp.common.av.AudioSink; import com.jogamp.common.av.PTS; import com.jogamp.common.av.TimeFrameI; import com.jogamp.common.net.Uri; import com.jogamp.math.Vec4f; import com.jogamp.opengl.util.texture.Texture; import com.jogamp.opengl.util.texture.TextureSequence; /** * GLMediaPlayer interface specifies a {@link TextureSequence} state machine * using a multiplexed audio/video stream as it's source. *

* Audio maybe supported and played back internally or via an {@link AudioSink} implementation. *

*

* Audio and video streams can be selected or muted via {@link #playStream(Uri, int, int, int, int)} * or {@link #playStream(Uri, int, String, int, String, int, int)} * using the appropriate stream id's. *

*

* Camera input can be selected using the {@link #CameraInputScheme} Uri. *

* *
StreamWorker Decoding Thread
*

* Most of the stream processing is performed on the decoding thread, a.k.a. StreamWorker: *

* StreamWorker generates it's own {@link GLContext}, shared with the one passed to {@link #initGL(GL)}. * The shared {@link GLContext} allows the decoding thread to push the video frame data directly into * the designated {@link TextureFrame}, later returned via {@link #getNextTexture(GL)} and used by the user. *

* StreamWorker Error Handling *

* Caught exceptions on StreamWorker are delivered as {@link StreamException}s, * which either degrades the {@link State} to {@link State#Uninitialized} or {@link State#Paused}. *

*

* An occurring {@link StreamException} triggers a {@link GLMediaEventListener#EVENT_CHANGE_ERR EVENT_CHANGE_ERR} event, * which can be listened to via {@link GLMediaEventListener#attributesChanged(GLMediaPlayer, int, long)}. *

*

* An occurred {@link StreamException} can be read via {@link #getStreamException()}. *

* *

*
GLMediaPlayer Lifecycle
*

* * * * * * * * * * * * * *
Action {@link State} Before {@link State} After {@link EventMask#Bit Event}
{@link #playStream(Uri, int, int, int, int)} {@link State#Uninitialized Uninitialized} {@link State#Initialized Initialized}1, {@link State#Uninitialized Uninitialized} {@link EventMask.Bit#Init Init} or ( {@link EventMask.Bit#Error Error} + {@link EventMask.Bit#Uninit Uninit} )
{@link #initGL(GL)} {@link State#Initialized Initialized}, {@link State#Uninitialized Uninitialized} {@link State#Playing Playing}, {@link State#Uninitialized Uninitialized} {@link EventMask.Bit#Play Play} or ( {@link EventMask.Bit#Error Error} + {@link EventMask.Bit#Uninit Uninit} )
{@link #pause(boolean)} {@link State#Playing Playing} {@link State#Paused Paused} {@link EventMask.Bit#Pause Pause}
{@link #resume()} {@link State#Paused Paused} {@link State#Playing Playing} {@link EventMask.Bit#Play Play}
{@link #stop()} {@link State#Playing Playing}, {@link State#Paused Paused} {@link State#Uninitialized Uninitialized} {@link EventMask.Bit#Pause Pause}
{@link #seek(int)} {@link State#Paused Paused}, {@link State#Playing Playing} {@link State#Paused Paused}, {@link State#Playing Playing} none
{@link #getNextTexture(GL)} any same none
{@link #getLastTexture()} any same none
{@link TextureFrame#END_OF_STREAM_PTS END_OF_STREAM} {@link State#Playing Playing} {@link State#Paused Paused} {@link EventMask.Bit#EOS EOS} + {@link EventMask.Bit#Pause Pause}
{@link StreamException} any {@link State#Paused Paused}, {@link State#Uninitialized Uninitialized} {@link EventMask.Bit#Error Error} + ( {@link EventMask.Bit#Pause Pause} or {@link EventMask.Bit#Uninit Uninit} )
{@link #destroy(GL)} any {@link State#Uninitialized Uninitialized} {@link EventMask.Bit#Uninit Uninit}
*

* *
Audio and video Stream IDs
*

* * * * * *
value request get
{@link #STREAM_ID_NONE} mute not available
{@link #STREAM_ID_AUTO} auto unspecified
≥0 specific stream specific stream
*

*

* Current implementations (check each API doc link for details): *

*

*

* Implementations of this interface must implement: *

 *    public static final boolean isAvailable();
 * 
* to be properly considered by {@link GLMediaPlayerFactory#create(ClassLoader, String)} * and {@link GLMediaPlayerFactory#createDefault()}. *

*
Timestamp Accuracy
*

*

* Timestamp type and value range has been chosen to suit embedded CPUs * and characteristics of audio and video streaming. See {@link TimeFrameI}. *

* *
Audio and video synchronization
*

* The class follows a passive A/V synchronization pattern. * Audio is being untouched, while {@link #getNextTexture(GL)} delivers a new video frame * only, if its timestamp is less than {@link #MAX_VIDEO_ASYNC} ahead of time. * If its timestamp is more than {@link #MAX_VIDEO_ASYNC} ahead of time, * the previous frame is returned. * If its timestamp is more than {@link #MAX_VIDEO_ASYNC} after time, * the frame is dropped and the next frame is being fetched. *

*

* https://en.wikipedia.org/wiki/Audio_to_video_synchronization *

 *   d_av = v_pts - a_pts;
 * 
*

*

* Recommendation of audio/video pts time lead/lag at production: *

*

*

* Recommendation of av pts time lead/lag at presentation: *

*

* *
Test Streams
*

* * * * * * * * * * * * * * * * * * *
Big Buck Bunny 24f 16:9
Big Buck Bunny320ph264aac 48000Hz 2 chanhttp://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4
Big Buck Bunny240ph264aac 48000Hz 2 chanhttp://archive.org/download/BigBuckBunny_328/BigBuckBunny_512kb.mp4
Big Buck Bunny720pmpeg4ac3 48000Hz 5.1 chanhttp://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_surround.avi
Big Buck Bunny720pmsmpeg4v2mp3 48000Hz 2 chanhttp://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_stereo.avi
Big Buck Bunny720ptheoravorbis 48000Hz 2 chanhttp://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_stereo.ogg
Big Buck Bunny1080pmpeg4ac3 48000Hz 5.1 chanhttp://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_surround.avi
WebM/Matroska (vp8/vorbis)
Big Buck Bunny Trailer640pvp8vorbis 44100Hz 1 chanhttp://video.webmfiles.org/big-buck-bunny_trailer.webm
Elephants Dream540pvp8vorbis 44100Hz 1 chanhttp://video.webmfiles.org/elephants-dream.webm
You Tube http/rtsp
Sintelhttp://www.youtube.com/watch?v=eRsGyueVLvQrtsp://v3.cache1.c.youtube.com/CiILENy73wIaGQn0LpXnygYbeRMYDSANFEgGUgZ2aWRlb3MM/0/0/0/video.3gp
Audio/Video Sync
Five-minute-sync-test1080phttps://www.youtube.com/watch?v=szoOsG9137Urtsp://v7.cache8.c.youtube.com/CiILENy73wIaGQm133VvsA46sxMYDSANFEgGUgZ2aWRlb3MM/0/0/0/video.3gp
Audio-Video-Sync-Test-Calibration-23.98fps-24fpshttps://www.youtube.com/watch?v=cGgf_dbDMsw
sound_in_sync_testhttps://www.youtube.com/watch?v=O-zIZkhXNLE