quindo

D3DXCreateEffectFromFile failed (8007000e) - The bane of our server

Recommended Posts

First thing first: I'm not sure if I've chosen proper sub-forum for this, so if it isn't right place then please move it.

I'm writing this because we have pretty game-breaking problem on our server that we can't find a way to fix on our side. The shaders randomly fail to create with warning specified in the title, it's happening more and more as our server gets bigger. We do track how often this error/warning happens for our players, and it's in thousands per day.
It doesn't happen to all players, but when it happens we can't find any correlation to PC hardware. It happened on new high end pc's and on older hardware. The amount of memory used by MTA as shown in /showmemstat varies too, it's usually not even close to 3.5 gb limit, pretty often around 2gb. We did track, used/available VRAM by collecting data from dxGetStatus when the error happens, and usual there's lots of VRAM left too.
Some example numbers here (free video memory for MTA / Total VRAM): 

  • 602/1024
  • 2968/4084

It shows that there's enough RAM/VRAM available for MTA that it shouldn't be problem.
The full error message also includes the path to shader and file/line.

I looked over MTA sourcecode, and found that it fails at call to winapi, the returned HRESULT (8007000e) is E_OUTOFMEMORY which may happen if the memory couldn't be allocated, based on that we think it may be problem with MTA memory management.

Some more thoughts:

  • Once a single shader fails to create, all shaders fail to create after that (from what we observed)
  • Our server is pretty big, the issue affects a lot of players but happens seemingly randomly. Some players aren't affected by this at all.
  • I can't reproduce this on local server, neither by playing nor by forcing MTA to create a hundreds of thousands of shaders/textures and deleting them.
  • We're also getting errors with HRESULT code 80070008, but it happens way less than the title error, it looks like that (happens with other shaders too): "[D3DXCreateEffectFromFile failed (80070008)[Loaded 'ngui-r\r.fx' (6737 bytes)]] @ 'dxCreateShader' [r.fx]"
  • In last 16 hours this error happened more than 5300 times on our server, we limit the number of reported errors to 15 per player per session, based on that we can see it happened to a lot of players.
  • Only way to fix this issue is to RESTART MTA, reconnect may help for few minutes but it fails quickly afterwards again.

We would love any help with handling this issue, it's limiting our ability to work on server updates as our server relies heavily on shaders.

Share this post


Link to post

Hey, i wanted to try and use MTADiag on my own pc before sending it out to players and it returned
"https://pastebin.mtasa.com/
<head><title>413 Request Entity Too Large</title></head>
<body bgcolor="white">
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>"

I don't want to start asking players to run this while it's not certain that they will get the results, could you please check that out?

I'm gonna wait with MTADiag until the reply, for now i have one more idea what may be causing that, but for that i need to explain a bit what we use those shaders for.
Basically on our server we have cars paint jobs system, which plays a huge role in game play, we allow players to customize the cars paint by giving them ability to create their own paint jobs using layered images. The first time a vehicle is streamed, it's paint job must be generated into texture to use and cache it on player HDD, we do it by using rendertarget, drawing the entire paint job on this, and after that using dxGetTexturePixels and dxCreateTexture to save and use it. It may not seem at all related to shaders but lately we released standalone application that can act as rendertarget separate from mta, and we communicate with it using lua and files in our resources. (Released this because dxGetTexturePixels can take up to 1/4 second, which is pretty bad when it can happen at every vehicle stream in). It outputs ready dxt texture that we have just to load from file, we found out that players that use this application don't seem to be affected by the error in the post title at all. It's still only small percent of overall player base that uses this, but we started recommending this as a way to fight this error.

Share this post


Link to post

MTADiag should work now

Share this post


Link to post

I'm gonna send you the logs from MTADiag through private message because some players don't want it to be shown publicly here.

Share this post


Link to post
On 2017. 08. 27. at 02:28, quindo said:

First thing first: I'm not sure if I've chosen proper sub-forum for this, so if it isn't right place then please move it.

I'm writing this because we have pretty game-breaking problem on our server that we can't find a way to fix on our side. The shaders randomly fail to create with warning specified in the title, it's happening more and more as our server gets bigger. We do track how often this error/warning happens for our players, and it's in thousands per day.
It doesn't happen to all players, but when it happens we can't find any correlation to PC hardware. It happened on new high end pc's and on older hardware. The amount of memory used by MTA as shown in /showmemstat varies too, it's usually not even close to 3.5 gb limit, pretty often around 2gb. We did track, used/available VRAM by collecting data from dxGetStatus when the error happens, and usual there's lots of VRAM left too.
Some example numbers here (free video memory for MTA / Total VRAM): 

  • 602/1024
  • 2968/4084

It shows that there's enough RAM/VRAM available for MTA that it shouldn't be problem.
The full error message also includes the path to shader and file/line.

I looked over MTA sourcecode, and found that it fails at call to winapi, the returned HRESULT (8007000e) is E_OUTOFMEMORY which may happen if the memory couldn't be allocated, based on that we think it may be problem with MTA memory management.

Some more thoughts:

  • Once a single shader fails to create, all shaders fail to create after that (from what we observed)
  • Our server is pretty big, the issue affects a lot of players but happens seemingly randomly. Some players aren't affected by this at all.
  • I can't reproduce this on local server, neither by playing nor by forcing MTA to create a hundreds of thousands of shaders/textures and deleting them.
  • We're also getting errors with HRESULT code 80070008, but it happens way less than the title error, it looks like that (happens with other shaders too): "[D3DXCreateEffectFromFile failed (80070008)[Loaded 'ngui-r\r.fx' (6737 bytes)]] @ 'dxCreateShader' [r.fx]"
  • In last 16 hours this error happened more than 5300 times on our server, we limit the number of reported errors to 15 per player per session, based on that we can see it happened to a lot of players.
  • Only way to fix this issue is to RESTART MTA, reconnect may help for few minutes but it fails quickly afterwards again.

We would love any help with handling this issue, it's limiting our ability to work on server updates as our server relies heavily on shaders.

Sometimes it happens for us too, was there a solution for this?

Share this post


Link to post

Nope, we couldn't find what is causing it, neither did we get any reply from this thread.

Share this post


Link to post
3 hours ago, Dretax said:

Sometimes it happens for us too, was there a solution for this?

I can't reproduce the problem here.
What is your server IP/Port?

 

Share this post


Link to post
6 hours ago, ccw said:

I can't reproduce the problem here.
What is your server IP/Port?

 

I found It. It happens when you constantly keep creating shaders at a render, and then destroy them.

I made a boolean to only create the shader ONCE, so the render just changes the settings for It, and destroys the shader only, when the panel is closed.

Seems like the issue that refers that was already said here: 

 

I made resource restarts first, they didn't help, but once I did fix that render :~ (Was quite stupid also) It was fixed.

 

However the weird thing is that the actual method also checks if the dxShader is an element or the variable has a value atleast, before It would create It.

local elmosasShader
local kepernyoSource = dxCreateScreenSource(kepS[1], kepS[2])
local sotetsegRender = false
function elmosasBekapcsolasa(sotetseg)
	if not elmosasShader then
		if not isElement(elmosasShader) then
			elmosasShader, blurTec = dxCreateShader("shaderek/elmosasShader.fx")
			blurErosseg = 6
			sotetsegRender = sotetseg or false
		end
	end
end

function elmosasKikapcsolasa( )
	if isElement(elmosasShader) then
		destroyElement(elmosasShader)
		elmosasShader = nil
	end
end

addEventHandler("onClientPreRender", root,
function()
    if (elmosasShader and isElement(elmosasShader)) then
        dxUpdateScreenSource(kepernyoSource)
        
        dxSetShaderValue(elmosasShader, "ScreenSource", kepernyoSource)
        dxSetShaderValue(elmosasShader, "BlurStrength", blurErosseg)
		dxSetShaderValue(elmosasShader, "UVSize", kepS[1]+5, kepS[2]+5)

        dxDrawImage(0, 0, kepS[1], kepS[2], elmosasShader)
		if(sotetsegRender) then
			dxDrawRectangle(-5, -5, kepS[1]+10, kepS[2]+10, tocolor ( 0, 0, 0, 100 ))
		end
	end
end)

The only way for the error to disappear for me was to take out 

elmosasBekapcsolasa(true)

from the other render method, and only call It once when the panel was enabled/disabled.

 

I honestly think this isn't logical, but If It is I honestly believe that It didn't like getting called from a render and then being handled in a render. No idea.

It would seem logical in UnityEngine really, but not here. I will report back If we ever got the error again.

Share this post


Link to post

The server ip is 178.33.54.30:20256, it doesn't happen to all players. We are creating single shader and changing it's settings as needed in places that we can, but we need to use a lot of per entity shaders, all with different textures, at least 1 per player. The problem still is happening 1k+ times a day, we just apply a texture telling players to restart mta once shaders fail to create.

Share this post


Link to post
21 hours ago, quindo said:

The server ip is 178.33.54.30:20256, it doesn't happen to all players. We are creating single shader and changing it's settings as needed in places that we can, but we need to use a lot of per entity shaders, all with different textures, at least 1 per player. The problem still is happening 1k+ times a day, we just apply a texture telling players to restart mta once shaders fail to create.

I think thats the point. It seems there is a limit or somekind of. I don't know exactly.

Share this post


Link to post

I guess there's some kind of memory leak, and while the stats are reset the actual shader aren't destroyed.

 

Share this post


Link to post

dxCreateShader issue should have been fixed. Is it still happening?

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.