Friday, October 16, 2009

Using Surface in Android native code




Well, here I set out deeper to use Surface in the native code. In my search for more information on Surfaceflinger, again I dived deeper into MediaPlayer code only to be drowned once gain, nevertheless, these posts in the Android groups helped me in undertading and narrowing down to ISurface interface.  Thanks to Dave Sparks



All we need is, to make call to three methods, defined in the ISurface interface.


1. registerBuffers
2. postBuffer
3. unregisterBuffer

It sounded easy in the beginning but it to was a quite a task to figure out the usage of many methods, as there were security restrictions to call registerBuffers 

Here, the idea is, Java application will pass the Surface object, created with PUSH_BUFFER  and in the native code, from the Surface object, we get the ISurface handle. Using this handle, we call register buffers and post RGB 565 format buffer, using double buffering.

Again we hack SimpleJNI sample, first the Native code, the required header files

#include "SkBlurMaskFilter.h"
#include "ui/Surface.h"
#include "ui/ISurface.h"
#include "utils/MemoryHeapBase.h"
#include "utils/MemoryBase.h"


Now add required libraries to Android.mk





libsgl  \
libui \
libsurfaceflinger



In JNI, we need to use Fields to retrieve the class objects and arguments passed in the JNI calls. Casting directly does not work, so here we declare a fieldID to get the surface instance  

static jfieldID gSurface_SurfaceFieldID;

Global variables for surface and frame buffer rendering



int frame_offset[2];
int buf_index;
short *frame_buf;
int width, height;

sp isurface = 0;
sp frame_heap;


Add new JNI method, this method is called from the Java application in the setSurfacChanged. 


static jint
setSurfaceChanged(JNIEnv *env, jobject thiz, jobject surf, jint w, jint h) {

Surface* const p = (Surface*) env->GetIntField(surf, gSurface_SurfaceFieldID);
const sp& native_Surface = sp(p);
sp native_iSurface = MediaPlayer::getISurface( native_Surface);

registersurface(native_iSurface,w,h);

}

Now we have the ISurface handle, we can go ahead and call those three methods. Here, I want highlight the  ISurface::getISurface  which is actually a private method but accessible through MediaPlayer as it is declared friend in the ISurface class. So, we declare a dummy MediaPlayer class and call ISurface::getISurface

Things did not work as expected until I stumbled upon http://groups.google.com/group/prajnashi, who is maintaining gst-plugin-android. There is an implementation of wrapper in folder sink that I used here to understand the Android Surface architecture, esp borrowed code in postBufers

//------------------------------------------------------------------------------------------------------------------
/* store the ISurface handle in a global variable, then we create MemoryHeapBase heap, use it create Frame buffer, then call registerBuffers
*/
void registerBuffer(sp& isurf, int w, int h){

  buf_index = 0;
width = w;
height = h;

   for ( int i = 0; i&MAX_FRAME_BUFFERS; i++){
frame_offset[i] = 0;
    }
    
isurface = isurf;

    /* use double buffer in post */
    int frameSize = width * height * 2;

    /* create frame buffer heap base */
    frame_heap = new MemoryHeapBase(frameSize * MAX_FRAME_BUFFERS);
    
if (frame_heap->heapID() < 0) {
      LOGI("Error creating memory base  heap!");
return;
    }

    /* create frame buffer heap and register with surfaceflinger */
    ISurface::BufferHeap buffers( (width + 1)& -2,(height + 1)  & -2 ,width,height, 
PIXEL_FORMAT_RGB_565, frame_heap);

    if (isurface->registerBuffers(buffers) < 0 ){
        LOGI("Cannot register frame buffer!");
        frame_heap.clear();
return;
     }


LOGI("Registered surface! %d, %d",width, height);

   for ( int i = 0; ii&MAX_FRAME_BUFFERS; i++){
        frame_offset[i] = i*frameSize;
    }

    buf_index = 0;

frame_buf = new short[width*height];

}

//--------------------------------------------------------------------------------------------------
/*
This is the JNI method called from SurfaceView::OnDraw. Here I use frame_buf to set some random values, then call postBuffer
*/
static jint
draw(JNIEnv *env, jobject thiz, jint n) {

if (++buf_index == MAX_FRAME_BUFFERS) 
buf_index = 0;
   
for (int i=0; i< width*height; i++){
frame_buf[i] = (short)(i+n*16) ;
}

memcpy (static_cast<unsigned char *>(frame_heap->base()) + frame_offset[buf_index],  frame_buf, 
width*height*sizeof(short));
    
isurface->postBuffer(frame_offset[buf_index]);

return 1;
}

//--------------------------------------------------------------------------------------------------
/* JNI method for final cleanup called from surfaceDestroyed */

static jint 
destroySurface(JNIEnv *env, jobject thiz, jint i){

isurface->unregisterBuffers();

LOGI("Unregistered surface!");

if(frame_buf)
free(frame_buf);

frame_buf = NULL;
isurface.clear();

LOGI("destroySurface done!");

return 1;
}

//--------------------------------------------------------------------------------------------------

Here is Java code changes in SimpleJNI.java  

public void surfaceChanged(SurfaceHolder holder, int format, int width,
         int height) {
    thread.setSurfaceSize(width, height);
Native.setSurfaceChanged(holder.getSurface(), width,height);
}

//--------------------------------------------------------------------------------------------------

There is one more hurdle to overcome, that is security, I did not want to bypass security, I tried to sign with platform key but still I got “access denied”, then I added   “dangerous” to frameworks/base/core/res/AndroidManifest.xml. still got  Permission failure: android.permission.ACCESS_SURFACE_FLINGER from uid=10012 pid=774 then I had no choice but to comment out the checkCallingPermission. Finally everything worked to my glee! 



8 comments:

surya said...

am also working in similar lines i actually tried to make my application similar to Bootanimation ...I dont know where the surfaceflinger is Giving permission to Bootanimation but for me it says permission denied any idea ?

Siva a.k.a nondi said...

well, it's by design, only way is get the platform key and sign, probably the device manufacturer can supply this to you

Ste said...

great article. I am trying to implement. But where can I get libsgl.so ? it is not into android platform

Ste said...

Also seem that the five includes files that you specified are not enough. I've a lot of errors of miss include files. Could you post the source project ?

ganesh said...

Hi,
great article.For me, MediaPlayer::getSurface(videodev->surface.get());

returns NULL value.

Could you help me...??

siva said...

Hi, It was a very useful information. Thanks a lot. I have included some skia header file like SkCanvas.h , SkPaint.h in my native code. On compiling (that is running script (/ndk-build) at the project location , It is giving errors like :
SkCanvas.h : file or directory not found
SkPaint.h : file or directory not found

Could you help me.. how to solve this issue ?

siva said...

Hi, It was a very useful information. Thanks a lot. I have included some skia header file like SkCanvas.h , SkPaint.h in my native code. On compiling (that is running script (/ndk-build) at the project location , It is giving errors like :
SkCanvas.h : file or directory not found
SkPaint.h : file or directory not found

Could you help me.. how to solve this issue ?

Chetan said...

Hi, Can you please give some pointers on how to disable checkCallingPermission.
Thanks.