Using Android MediaPlayer framework to plug my player extension

Emboldened by using ISurface in the native code, I set out to follow Android MediaPlayer architecture and write my own media player (does nothing, renders same as the earlier one, visit this to understand how rendering works). Android MediaPlayer architecture is plugable architecture but still one needs to recompile Media libraries, I wonder why Google did not make this runtime plugable like Netscape plug-in.

There are 2 major tasks to plug our Media player into Android architecture,  Mediaplayer service creates appropriate instance of a media player based on the data source, in our case the extension “avi”. Since avi is not supported, I choose to add this as myPlayer support format.

1. We need to extend MediaPlayerInterface, defined in  media/MediaPlayerInterface.h. 
2. Insert our player instance at the appropriate place in  MediaPlayerService.cpp

Below is the interface  myPlayer I derived from MediaPlayerInterface. I have used Vorbis implementation as base to get this up and running in a day ( well .. spent a week on thinking how to..). I have added my understanding as comments in the code below

class myPlayer : public MediaPlayerInterface {
public:
    myPlayer();

    virtual             ~myPlayer();
    virtual void        onFirstRef();
    virtual status_t    initCheck(); //return player status 
    virtual status_t    setAudioStreamType(int type);
    virtual status_t    setDataSource(const char *url);
    virtual status_t    setDataSource(int fd, int64_t offset, int64_t length);
    virtual status_t    setVideoSurface(const sp& surface);
    virtual status_t    prepare();
    virtual status_t    prepareAsync();
    virtual status_t    start();
    virtual status_t    stop();
    virtual status_t    pause();
    virtual bool           isPlaying();
    virtual status_t    getVideoWidth(int *w);
    virtual status_t    getVideoHeight(int *h);
    virtual status_t    seekTo(int msec);
    virtual status_t    getCurrentPosition(int *msec);
    virtual status_t    getDuration(int *msec);
    virtual status_t    reset();
    virtual status_t    setLooping(int loop);
    virtual player_type playerType() { return MY_PLAYER; }


//myPlayer methods //---------------------------------------------------------------------------------------

    static  int         renderThread(void*);
    int render();

//-----------------------------------------------------------------------------------------------------------

private:

    bool                    mIsDataSourceSet;
    sp                       mSurface;
    status_t             mStatus;
    int                       mDuration;
    int                       mStreamType;
    Mutex                 mMutex;
    Condition           mCondition;
    pid_t                   mRenderTid;
    volatile bool       mExit;
    bool                     mPaused;
    volatile bool       mRender;


//myPlayer stuff for rendering //------------------------------------------------------------------------------------------//
    sp mFrameheap;
    int mHorstride;
    int mVerstride;
    int mWidth;
    int mHeight;
    int mFrameoffset[2];
    int mBufindex;
    short *mFramebuf;
    PixelFormat mPixformat;
//myPlayer stuff //------------------------------------------------------------------------------------------//

}; //class myPlayer

 In the  MediaPlayerService.cpp add .avi into FILE_EXTS. This will be used in getPlayerType (...) method

extmap FILE_EXTS [] =  {
        {".mid", SONIVOX_PLAYER},
        {".midi", SONIVOX_PLAYER},
        {".smf", SONIVOX_PLAYER},
        {".xmf", SONIVOX_PLAYER},
        {".imy", SONIVOX_PLAYER},
        {".rtttl", SONIVOX_PLAYER},
        {".rtx", SONIVOX_PLAYER},
        {".ota", SONIVOX_PLAYER},
        {".ogg", VORBIS_PLAYER},
        {".oga", VORBIS_PLAYER},
        {".avi", MY_PLAYER},
};

Now add myPlayer instance in createPlayer. We are done. Not to forget to add and libmyplayer to MediaService’s Android.mk file.


static sp createPlayer(player_type playerType, void* cookie, notify_callback_f notifyFunc){
 .
 .
 .
         case MY_PLAYER:
            LOGE(" create myPlayer");
            p = new myPlayer();
            break;
 .
 .
 .
}

Now, lets get back to myPlayer.cpp implementation.  Lets pay attention only to the following methods, others are just dummy returns No_ERROR. mStatus plays key role here as you can see it changes from method to method.

myPlayer, constructor initializes class variables, here esp, I have set screen size to 320X430. In reality this has to handled at runtime.
    myPlayer();
Destructor clears up Surface buffers and frees up thread
    virtual             ~myPlayer();
Create the render thread and wait for signal, the signal is sent out from start() method.
    virtual void        onFirstRef();
InitCheck does nothing just pass the mStatus
    virtual status_t    initCheck();
Set the file URL, here we do nothing but this is the place one can open the file and check it’s valid, do parsing and return OK
    virtual status_t    setDataSource(const char *url);
    virtual status_t    setDataSource(int fd, int64_t offset, int64_t length);
Important method, we store the Surface handle and use it for rendering. Also, register our buffers
    virtual status_t    setVideoSurface(const sp& surface);
Called from Java application class MediaPlayer, may be we can prepare CODECs here or initialize e double buffering etc. Now does nothing
    virtual status_t    prepare();
    virtual status_t    prepareAsync();
Called from Java application class MediaPlayer, this is where we send single() to render thread to start
    virtual status_t    start();

Rendering is done through double buffer,

//myPlayer methods  //---------------------------------------------------------------------------------------

    static  int         renderThread(void*);
    int render();

//-----------------------------------------------------------------------------------------------------------



#include "myPlayer.h"

#ifdef HAVE_GETTID
static pid_t myTid() { return gettid(); }
#else
static pid_t myTid() { return getpid(); }
#endif


/* max frame buffers */
#define MAX_FRAME_BUFFERS     2

#define WIDTH  320
#define HEIGHT  430

namespace android {


static status_t ERROR_NOT_OPEN = -1;
static status_t ERROR_OPEN_FAILED = -2;
static status_t ERROR_ALLOCATE_FAILED = -4;
static status_t ERROR_NOT_SUPPORTED = -8;
static status_t ERROR_NOT_READY = -16;
static status_t STATE_INIT = 0;
static status_t STATE_ERROR = 1;
static status_t STATE_OPEN = 2;

// ----------------------------------------------------------------------------------------------------------------
myPlayer::myPlayer(){
    LOGI ("myPlayer Constructor");
    mStatus = STATE_ERROR;
    mDuration = -1;
    mSurface = NULL;
    mRenderTid = 0;
   mExit = false;
   mPaused = false;
   mRender = false;


//----------------------------------------------------
//myPlayer stuff initialize

    mWidth = (WIDTH + 1) & -2;;
    mHeight = (HEIGHT + 1) & -2;;

    mHorstride = mWidth;
    mVerstride = mHeight;

   int frameSize = mWidth*mHeight*2;
   /* create frame buffer heap base */
 mFrameheap = new MemoryHeapBase(frameSize * MAX_FRAME_BUFFERS);
    if (mFrameheap->heapID() < 0) {
        LOGI("Error creating frame buffer heap!");
    }

    for ( int i = 0; i<MAX_FRAME_BUFFERS; i++){
     mFrameoffset[i] = i*frameSize;
    }

 mBufindex = 0;

 mFramebuf = new short[mWidth*mHeight];

    mPixformat = PIXEL_FORMAT_RGB_565;

}

// ----------------------------------------------------------------------------------------------------------------
void myPlayer::onFirstRef()
{
    LOGV("onFirstRef");
    // create playback thread
    Mutex::Autolock l(mMutex);
    createThreadEtc(renderThread, this, "vorbis decoder");
    mCondition.wait(mMutex);
    if (mRenderTid > 0) {
        LOGV("render thread(%d) started", mRenderTid);
        mStatus = STATE_INIT;
    }
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::initCheck(){
    LOGI ("mStatus = %s\n", (mStatus==NO_ERROR)?"NO_ERROR":"ERROR");
    if (mStatus != STATE_ERROR) return NO_ERROR;
    return ERROR_NOT_READY;
}

// ----------------------------------------------------------------------------------------------------------------

myPlayer::~myPlayer(){

    LOGI ("myPlayer destruct");

 mExit=true;
 mRender=false;

    if (mSurface != NULL){
  mSurface->unregisterBuffers();
        mSurface.clear();
    }

 if(mFramebuf)
  free(mFramebuf);

 mFrameheap.clear();

 mFramebuf = NULL;

}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::setDataSource(const char *url){
  
 //now nothing done, here you may add file stream code and paring the file
    LOGI("setDataSource: url = %s\n", url);

 if (url) {
  mStatus = STATE_OPEN;
  mExit=false;
 }
 else {
  mStatus = ERROR_NOT_OPEN;
 }
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::setDataSource(int fd, int64_t offset, int64_t length){

    LOGI("fd: %i, offset: %ld, len: %ld\n", fd, (long)offset, (long)length);

    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::setVideoSurface(const sp& surface){
    LOGI("ISurface: %p\n", surface.get());

    mSurface = surface;
    /* create frame buffer heap and register with surfaceflinger */
    ISurface::BufferHeap buffers( mWidth, mHeight, mHorstride, mVerstride, mPixformat, mFrameheap);

    if (mSurface->registerBuffers(buffers) < 0 ){
        LOGI("Cannot register frame buffer!");
        mFrameheap.clear();
        return STATE_ERROR;
    }

 mRender = true;
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::prepare(){
 LOGI("prepare");
 mStatus = STATE_OPEN;
 mRender = false;
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::prepareAsync(){

    LOGI("prepareAsync\n");

 if (mStatus != STATE_OPEN ) {
        sendEvent(MEDIA_ERROR);
        return NO_ERROR;
    }
    sendEvent(MEDIA_PREPARED);

    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::start(){
 LOGI("start");
    Mutex::Autolock l(mMutex);
    if (mStatus != STATE_OPEN) {
        return ERROR_NOT_OPEN;
    }
 mRender = true;
    mCondition.signal();
    return NO_ERROR;
}
// ----------------------------------------------------------------------------------------------------------------
int myPlayer::renderThread(void* p) {
 LOGI("renderThread");
    return ((myPlayer*)p)->render();
}

// ----------------------------------------------------------------------------------------------------------------

int myPlayer::render(){
 LOGI("render Enter");
 {
  Mutex::Autolock l(mMutex);
  mRenderTid = myTid();
  mCondition.signal();
 }

 int frame = 0;

  // nothing to render, wait for client thread to wake us up
 while (1) {
 
  if (!mExit && !mRender) {
   Mutex::Autolock l(mMutex);
   LOGI("render - signal wait\n");
   mCondition.wait(mMutex);
   LOGI("render - signal rx'd\n");
  }


/// ------//  myPlayer stuff -- replace with decoders //--------//
  if (++mBufindex == MAX_FRAME_BUFFERS)
   mBufindex = 0;
 
  for (int i=0; i< mWidth*mHeight; i++){
   mFramebuf[i] = (short)(i+frame*16) ;
  }
  memcpy (static_cast<unsigned char *>(mFrameheap->base()) + mFrameoffset[mBufindex],  mFramebuf, mWidth*mHeight*sizeof(short));
  mSurface->postBuffer(mFrameoffset[mBufindex]);
  if(++frame == 430)
   frame=0;
 
/// ------// myPlayer stuff end-----------------------------------//

  if (mExit) break;
 }

 {
  Mutex::Autolock l(mMutex);
  mRenderTid = -1;
  mCondition.signal();
 }

 LOGI("render Exit");
 return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------

status_t myPlayer::setAudioStreamType(int type){
    mStreamType = type;
    LOGI ("mStreamType = %d\n", mStreamType);
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::stop(){
 LOGI("stop");
 mRender = false;
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::pause(){
 LOGI("stop");
 mRender = false;
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
bool myPlayer::isPlaying(){
 LOGI("isPlaying");

 return mRender;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::getVideoWidth(int *w){
 LOGI("getVideoWidth");
 *w = 320;
    return NO_ERROR;

}
// ----------------------------------------------------------------------------------------------------------------

status_t myPlayer::getVideoHeight(int *h){
 LOGI("getVideoHeight");
 *h = 430;
 return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::getCurrentPosition(int *msec){
 LOGI("getCurrentPosition");
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::getDuration(int *msec){
 LOGI("getDuration");
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::seekTo(int msec){
 LOGI("seekTo");
    return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::reset(){
 LOGI("reset");  
 return NO_ERROR;
}

// ----------------------------------------------------------------------------------------------------------------
status_t myPlayer::setLooping(int loop){
  LOGI("setLooping");
 return NO_ERROR;
}

}; // namespace android


This class can be used as template for new player development. I created a project under ./external folder and linked this as libmyplayer to libmediaservice library. I used the ApiDemos/medial player to test this myPlayer engine

Using Surface in Android native code

Well, here I set out deeper to use Surface in the native code. In my search for more information on Surfaceflinger, again I dived deeper into MediaPlayer code only to be drowned once gain, nevertheless, these posts in the Android groups helped me in undertading and narrowing down to ISurface interface.  Thanks to Dave Sparks

http://groups.google.co.in/group/android-framework/browse_thread/thread/d3f242356375effa/44d88801c76fcfd3?hl=en&q=codec&pli=1

All we need is, to make call to three methods, defined in the ISurface interface.

1. registerBuffers

2. postBuffer

3. unregisterBuffer

It sounded easy in the beginning but it to was a quite a task to figure out the usage of many methods, as there were security restrictions to call registerBuffers 

Here, the idea is, Java application will pass the Surface object, created with PUSH_BUFFER  and in the native code, from the Surface object, we get the ISurface handle. Using this handle, we call register buffers and post RGB 565 format buffer, using double buffering.

Again we hack SimpleJNI sample, first the Native code, the required header files

#include "SkBlurMaskFilter.h"

#include "ui/Surface.h"

#include "ui/ISurface.h"

#include "utils/MemoryHeapBase.h"

#include "utils/MemoryBase.h"


Now add required libraries to Android.mk

libsgl  \

libui \

libsurfaceflinger

In JNI, we need to use Fields to retrieve the class objects and arguments passed in the JNI calls. Casting directly does not work, so here we declare a fieldID to get the surface instance  

static jfieldID gSurface_SurfaceFieldID;

Global variables for surface and frame buffer rendering

int frame_offset[2];

int buf_index;

short *frame_buf;

int width, height;

sp isurface = 0;

sp frame_heap;

Add new JNI method, this method is called from the Java application in the setSurfacChanged. 

static jint

setSurfaceChanged(JNIEnv *env, jobject thiz, jobject surf, jint w, jint h) {

Surface* const p = (Surface*) env->GetIntField(surf, gSurface_SurfaceFieldID);

const sp& native_Surface = sp(p);

sp native_iSurface = MediaPlayer::getISurface( native_Surface);

registersurface(native_iSurface,w,h);

}

Now we have the ISurface handle, we can go ahead and call those three methods. Here, I want highlight the  ISurface::getISurface  which is actually a private method but accessible through MediaPlayer as it is declared friend in the ISurface class. So, we declare a dummy MediaPlayer class and call ISurface::getISurface

Things did not work as expected until I stumbled upon http://groups.google.com/group/prajnashi, who is maintaining gst-plugin-android. There is an implementation of wrapper in folder sink that I used here to understand the Android Surface architecture, esp borrowed code in postBufers

//------------------------------------------------------------------------------------------------------------------

/* store the ISurface handle in a global variable, then we create MemoryHeapBase heap, use it create Frame buffer, then call registerBuffers

*/

void registerBuffer(sp& isurf, int w, int h){

  buf_index = 0;

width = w;

height = h;

   for ( int i = 0; i&MAX_FRAME_BUFFERS; i++){

frame_offset[i] = 0;

    }

    

isurface = isurf;

    /* use double buffer in post */

    int frameSize = width * height * 2;

    /* create frame buffer heap base */

    frame_heap = new MemoryHeapBase(frameSize * MAX_FRAME_BUFFERS);

    

if (frame_heap->heapID() < 0) {

      LOGI("Error creating memory base  heap!");

return;

    }

    /* create frame buffer heap and register with surfaceflinger */

    ISurface::BufferHeap buffers( (width + 1)& -2,(height + 1)  & -2 ,width,height, 

PIXEL_FORMAT_RGB_565, frame_heap);

    if (isurface->registerBuffers(buffers) < 0 ){

        LOGI("Cannot register frame buffer!");

        frame_heap.clear();

return;

     }

LOGI("Registered surface! %d, %d",width, height);

   for ( int i = 0; ii&MAX_FRAME_BUFFERS; i++){

        frame_offset[i] = i*frameSize;

    }

    buf_index = 0;

frame_buf = new short[width*height];

}

//--------------------------------------------------------------------------------------------------

/*

This is the JNI method called from SurfaceView::OnDraw. Here I use frame_buf to set some random values, then call postBuffer

*/

static jint

draw(JNIEnv *env, jobject thiz, jint n) {

if (++buf_index == MAX_FRAME_BUFFERS) 

buf_index = 0;

   

for (int i=0; i< width*height; i++){

frame_buf[i] = (short)(i+n*16) ;

}

memcpy (static_cast<unsigned char *>(frame_heap->base()) + frame_offset[buf_index],  frame_buf, 

width*height*sizeof(short));

    

isurface->postBuffer(frame_offset[buf_index]);

return 1;

}

//--------------------------------------------------------------------------------------------------

/* JNI method for final cleanup called from surfaceDestroyed */

static jint 

destroySurface(JNIEnv *env, jobject thiz, jint i){

isurface->unregisterBuffers();

LOGI("Unregistered surface!");

if(frame_buf)

free(frame_buf);

frame_buf = NULL;

isurface.clear();

LOGI("destroySurface done!");

return 1;

}

//--------------------------------------------------------------------------------------------------

Here is Java code changes in SimpleJNI.java  

public void surfaceChanged(SurfaceHolder holder, int format, int width,

         int height) {

    thread.setSurfaceSize(width, height);

Native.setSurfaceChanged(holder.getSurface(), width,height);

}

//--------------------------------------------------------------------------------------------------

There is one more hurdle to overcome, that is security, I did not want to bypass security, I tried to sign with platform key but still I got “access denied”, then I added   “dangerous” to frameworks/base/core/res/AndroidManifest.xml. still got  Permission failure: android.permission.ACCESS_SURFACE_FLINGER from uid=10012 pid=774 then I had no choice but to comment out the checkCallingPermission. Finally everything worked to my glee!