Need help with DEP in windows XP

[nakedostrich]

Hi,

I really need some help here!

I am using the latest version of ffmpeg compiled as shared DLL files. My application is being compiled under Visual Studio 6 and written in C++.

The problem is that whenever I call "av_find_stream_info" with DEP (Data execution prevention) turn on in Windows XP, windows stops the program from running, and then the program crashes. So basically it looks like there is a problem with FFMpeg and DEP in windows.

Ok, so the obvious answer is just to tell people to add my application as an exception in the DEP list (turn off DEP for that program), however that is not an option in this case - it must be able to function with DEP turned on.

I have since tried compiling ffmpeg as a static library into the application, but it still does exactly the same thing.

The strange thing is that the stone-alone ffplay.exe will play videos under windows XP with DEP turned on with no problem.

Any ideas?

Thanks.

[angustia]

Please figure out in which function the exception happens...

[nakedostrich]

Ok so we traced the problem into:

libavcodec\utils.c
avcodec_open()

then, the line:

ret = avctx->codec->init(avctx);

causes the problem. We cant go any further as this is a callback function and I guess it depends on which codec is trying to decode the video. We looked in the AVCodec struct and "init" seems to be a function pointer of some kind.

I dont know if the DEP problem is being caused something in the init function (or whatever it is pointing to), or perhaps it's the way in which the function is being called which is causing the issue. Maybe DEP doesnt like callbacks or function pointers?

I have tried playing videos in different formats and they all fail, and I would imagine that different formats would have different codecs and therefore different init functions?

Any help is appreciated.

Thanks.

[angustia]

99.9% chance the error is in your program... Is it reproducible with MinGW? Could you try adding printfs to the ffmpeg code to see how far it goes? Are you using the right calling convention?

[nakedostrich]

Well, here's the code my program is using to implement ffmpeg:

-------------------------------------------------------------------------------------------------------------
av_register_all();
if( av_open_input_file( &m_pFormatCtx, szPathName.Buffer(), NULL, 0, NULL)!=0)
return FALSE;

// Retrieve stream information
if( av_find_stream_info( m_pFormatCtx ) < 0 ) // < DEP EXCEPTION HAPPENS HERE
return FALSE;
-------------------------------------------------------------------------------------------------------------

And it's crashing on "av_find_stream_info"... so it's not as if I am doing a lot before it crashes. FFMpeg is being linked via lib files which call a set of external DLLs. I have also tried to link statically using .a files and the exact same thing happens. This happens on any video format which I have tested as well.

I have added printfs in the ffmpeg code, that's what I was referring to in the last post, I traced the problem into avcodec_open() in libavcodec\utils.c (see previous post).

Could you please be more specific about the calling conventions needed?

Thanks

[jesusdario]

Hi

I had the same problem in Windows Vista.

I have built a DLL using the FFMPEG static libraries. I read in :
http://technet.microsoft.com/en-us/libr ... 8483(WS.10).aspx
that Data Execution Prevention can arise isues to the applications, and to solve them,
/SECTION must be added to mark the segments causing problems with the EXECUTABLE attribute.

I used DUMPBIN with my libraries and determined that there were the segment .text.un.

I appened the post build event to the properties of VS2008:

editbin /section:.text.un,cer "$(OutDir)\libmultimedia.dll"

That (editbin) assigned the attributes "CODE,EXECUTABLE,READ" to the segment .text.un of my library and solved my problem.

I hope this helps.

Dario

[jjsafe]

Sorry for stirring things up.
I'm running ffmpeg 0.6 and is seeing similar exception when decoding h264 avi files. The program runs smoothly on mpeg4 or vid encoded files.
I only managed to trace to the point of ret = avctx->codec->decode(avctx, picture, got_picture_ptr, avpkt) called by avcodec_decode_video2 in utils.c of avcodec.
Spot this post due to its similarity of symptoms. But after I inspected the dll with dumpbin, the .text segment is already with code, execute, read attribute.
Actually I wonder why it is not so in your case.
trying my luck to see if anyone has solution for this.
Appreciate any feedback!

[Bonta]

I am using ffmpeg-r22941 and do not have any problem with DEP under Windows 7 Professional 64bit (with DEP turned on for all).

[jjsafe]

thanks for the hint. I'll give it a try.