Jump to content

[TuT] Lua tables as a efficient data system


Recommended Posts

Hey.

After around a 2 years of scripting i've decided to share a bit of my knowledge which i've learnt - thanks to very special person which is @IIYAMA, without him i couldn't make this tutorial, greets!

When it comes to synchronising data you have two ways:

a) Element data

b) Tables

Which of course have own pros and cons. This tutorial is focused on tables. And covers such topics as:

- Basic details about tables & element data
- Data validation
- Usage of helper functions to create data system based on tables
- Usage of buffer to reduce trigger calls
- Data handlers
- Examples
- Efficiency comparison (vs element data)

Basic informations

Element data is accessible in every resource. It is synced by default for everyone. Stored data remains after resource restart. It's cleaned when element gets destroyed.

Table isn't accessible in every resource (unless you use exports), and it isn't synced by default. Stored data is lost after resource start. Needs to be cleaned manually. Requires more code to sync.

At first look you might say that table loses duel vs element data, but for sure? You will find out.
Whatever you choose - it doesn't really matter - it should be well maintained.

Validating data before triggering/applying changes

Do you even saw or heard about common issue in DayZ styled servers? Maybe yes? Maybe no?

If you thought about DUP items then it's a point for you.
The problem lies in blindly trusting client data, or even worse - applying changes via clientside.

And here comes... Serverside data validation.

Client(s) are listeners, and server is overseer which is giving orders to everyone.

But that doesn't mean that server doesn't listen at all... In some cases you need to trust client - to the some point. Like collision checks.

However... Just for sake, i recommend to verify data (and type of this data) which is coming from client. What's more, I recommend to do pre-validations on client including delay for sending triggers, or a boolean which will check if callback from server was received - do not give opportunity for player to flood server with triggers. And last point, but not least - do not forget to use client variable instead of source in your custom events! Quick example:

--[[Valid Client <-> Server interaction, as for example item synced on both sides]]--

-- Client

local itemToDrop = "First Aid Kit" -- item selected by player to drop

triggerServerEvent("onServerItemDrop", localPlayer, itemToDrop)

-- Use localPlayer as a lowest element in element tree to save server CPU
-- See: https://wiki.multitheftauto.com/wiki/TriggerServerEvent (Note under syntax)
-- We ask server to do some interaction, we need to do it server-side, because client data might be incorrect/faked

-- Server

function onServerItemDrop(pItem)
	if client then -- let's check if it's valid player - remember, do not use 'source'!
		local isValidString = type(pItem) == "string" -- let's verify if it's actual string

		if isValidString then -- if so
			local doesItemExist = getCustomData(client, pItem, false) -- retrieve synced data under pItem key, server-side would always have actual value, it can't be bugged unless function is working incorrectly

			if doesItemExist then -- we are 100% sure that player have this item in his inventory
				setCustomData(client, pItem, doesItemExist - 1, false, "onServerDropItem", "Loot interaction", 50) -- update player data: item count (doesItemExist) - 1
				-- data will be synced to all clients with defined onServerDropItem event (which could be used for refreshing for example - inventory, using data handler), within buffer (and timeout 50 ms), if two players would trigger this buffer at same time, trigger will be processed once - which means we save bandwith
			end
		end
	end
end
addEvent("onServerItemDrop", true)
addEventHandler("onServerItemDrop", root, onServerItemDrop)

Implementing custom functions for our data system & using buffer for bandwidth reducement

Probably, someone of you already saw a code which was using tables instead of element data, that's nothing new. But, what if we could make it more efficient than before? Sounds too beatiful to be real? I have good news, it is possible.

How? By implementing buffer solution (which i've had a chance to test more than twice). How does it help? By reducing trigger(Client/Server)Event calls by using a simple timer. Without it, server would die of endless count of triggers going forth and back. In my whole experience, i've used it just for onClientPlayerDamage & onClientVehicleDamage. But, there's nothing against to use it in any other case. There's just one con, data is being send with extra delay (which you set by yourself). Buffer could be implemented for element data aswell, however it will require usage of both element data & tables - in order to work, you would need disable sync, and track all changes with tables. While pure tables doesn't require such actions.

Keep in mind that best performance you would achieve by using it directly in your resource. This data system is taken from my gamemode, which consists of just two resources (and it's used just in 1st). Whole mode is separated on certain parts, of course with that i lose flexibility (can't simply restart desired resource), but i prefer it over using exports.

Client:

local localData = {} -- store our local (non-synced data)
local syncedData = {} -- store our synced data
local dataHandlers = {} -- store our data handlers
local playerElements = getElementByID("playerElements") -- we would need that aswell for binding handlers.
local otherElements = getElementByID("otherElements") -- we would need that aswell for binding handlers.

--[[
/***************************************************

***************************************************\
]]

function getCustomData(pElement, pKey, pIsLocal)
	local cachedTable = false

	if pIsLocal then -- reference to local or synced data
		cachedTable = localData[pElement]
	else
		cachedTable = syncedData[pElement]
	end

	if cachedTable then -- check if such index exists?
		local wholeData = pKey == nil -- do we need whole data or certain key?

		return wholeData and cachedTable or cachedTable[pKey] -- return requested data
	end
end

--[[
/***************************************************

***************************************************\
]]

function setCustomData(pElement, pKey, pValue, pIsLocal, pOnServerEvent)
	local cachedTable = pIsLocal and localData[pElement] or syncedData[pElement] -- do we need data from local or synced table?

	if not cachedTable then -- if sub-table under certain index doesn't exist...

		if pIsLocal then -- whether is local or not
			localData[pElement] = {} -- create sub-table for certain element
		else
			syncedData[pElement] = {} -- create sub-table for certain element
		end
	end

	cachedTable = pIsLocal and localData[pElement] or syncedData[pElement] -- let's reference this once again

	local oldValue = cachedTable[pKey] -- get old value for data handlers

	cachedTable[pKey] = pValue -- set our value

	handleDataChange(pElement, pKey, oldValue, pValue, pOnServerEvent) -- handle our functions (if there's any)

	return pElement, pKey, pValue -- perhaps, you would need those values afterwards, so let's return them.
end

--[[
/***************************************************

***************************************************\
]]

function addDataHandler(pElementTypes, pKeys, pFunction, pOnServerEvent)
	local validTypes = type(pElementTypes) == "string" or type(pElementTypes) == "table" -- check if it's valid type
	local validKeys = type(pKeys) == "string" or type(pKeys) == "table" -- check if it's valid type
	local validFunction = type(pFunction) == "function" -- check if it's valid type
	local validEvent = type(pOnServerEvent) == "string" or type(pOnServerEvent) == "nil" -- check if it's valid type

	if validTypes and validKeys and validFunction and validEvent then -- if all correct
		local cachedData = false -- remember, reuse it's always faster rather than recreating variable each time :)
		local currentSize = false -- remember, reuse it's always faster rather than recreating variable each time :)
		local currentHandlers = dataHandlers -- reference to main table

		pKeys = {pKeys, pOnServerEvent, pFunction} -- we need pack this into table, because it will be processed by loop

		if type(pElementTypes) == "string" then -- if element type was passed as a string

			if not currentHandlers[pElementTypes] then -- if sub table doesn't exist
				currentHandlers[pElementTypes] = {} -- create it
			end

			currentHandlers = currentHandlers[pElementTypes] -- update reference
			currentSize = #currentHandlers + 1 -- get new index for data handler
			currentHandlers[currentSize] = pKeys -- insert packed data
		else -- otherwise
			for i = 1, #pElementTypes do
				cachedData = pElementTypes[i]

				if not currentHandlers[cachedData] then -- if sub table doesn't exist
					currentHandlers[cachedData] = {} -- create it
				end

				currentHandlers = currentHandlers[cachedData] -- update reference
				currentSize = #currentHandlers + 1 -- get new index for data handler
				currentHandlers[currentSize] = pKeys -- insert packed data
			end
		end
	end
end

--[[
/***************************************************

***************************************************\
]]

function handleDataChange(pElement, pKey, pOldValue, pNewValue, pOnServerEvent)
	local isValidElement = isElement(pElement) -- we want element to exist at the time when handler was processed

	if isValidElement then
		local elementType = getElementType(pElement) -- get our element type
		local elementHandlers = dataHandlers[elementType] -- check if there's any handler for this type of element

		if elementHandlers then -- yup, apparently there is something
			local handlerData = false -- remember, reuse it's always faster rather than recreating variable each time :)
			local handlerKeys = false -- remember, reuse it's always faster rather than recreating variable each time :)
			local handlerKey = false -- remember, reuse it's always faster rather than recreating variable each time :)
			local handlerServerEvent = false -- remember, reuse it's always faster rather than recreating variable each time :)
			local handlerFunction = false -- remember, reuse it's always faster rather than recreating variable each time :)

			for i = 1, #elementHandlers do -- process our handlers by loop
				handlerData = elementHandlers[i]
				handlerKeys = handlerData[1]
				handlerServerEvent = handlerData[2]
				handlerFunction = handlerData[3]

				if handlerServerEvent == pOnServerEvent then -- if called event matches data handler event

					if type(handlerKeys) == "string" then -- if key is a string

						if handlerKeys == pKey then -- and it's equal to called key
							handlerFunction(pElement, pKey, pOldValue, pNewValue, pOnServerEvent)
						end
					else -- otherwise
						for i = 1, #handlerKeys do
							handlerKey = handlerKeys[i]

							if handlerKey == pKey then -- it's equal to called key
								handlerFunction(pElement, pKey, pOldValue, pNewValue, pOnServerEvent)
							end
						end
					end
				end
			end
		end
	end
end

--[[
/***************************************************

***************************************************\
]]

function onClientDataHandler(pElement, pKey, pOldValue, pNewValue, pOnServerEvent)
	print("onClientDataHandler got triggered :)")
end
addDataHandler("player", "Key", onClientDataHandler, nil)

--[[
/***************************************************

***************************************************\
]]

function onClientDataSync(pData)
	syncedData = pData -- update data
end
addEvent("onClientDataSync", true)
addEventHandler("onClientDataSync", localPlayer, onClientDataSync)

--[[
/***************************************************

***************************************************\
]]

function onClientReceiveData(...)
	local dataFromServer = {...} -- use vararg, because data coming from server might be packed in table or not
	local isBuffer = dataFromServer[1] -- verify if it's buffered
	local elementToSet = false -- declare it once for better readability, and later reuse it
	local keyToSet = false -- declare it once for better readability, and later reuse it
	local valueToSet = false -- declare it once for better readability, and later reuse it
	local serverEventToSet = false -- declare it once for better readability, and later reuse it

	if isBuffer then -- if yes, then use loop to iterate over table
		local dataPackage = dataFromServer[2]
		local cachedIndex = false

		for i = 1, #dataPackage do
			cachedIndex = dataPackage[i]
			elementToSet = cachedIndex[1]
			keyToSet = cachedIndex[2]
			valueToSet = cachedIndex[3]
			serverEventToSet = cachedIndex[4]

			setCustomData(elementToSet, keyToSet, valueToSet, false, serverEventToSet)
		end
	else -- otherwise process normally
		elementToSet = dataFromServer[2]
		keyToSet = dataFromServer[3]
		valueToSet = dataFromServer[4]
		serverEventToSet = dataFromServer[5]

		setCustomData(elementToSet, keyToSet, valueToSet, false, serverEventToSet)
	end
end
addEvent("onClientReceiveData", true)
addEventHandler("onClientReceiveData", root, onClientReceiveData)

--[[
/***************************************************

***************************************************\
]]

function onClientResourceStart()
	triggerServerEvent("onServerPlayerReady", localPlayer) -- let's tell server that client part is ready :)
end
addEventHandler("onClientResourceStart", resourceRoot, onClientResourceStart)

--[[
/***************************************************

***************************************************\
]]

function onClientPlayerQuit()
	localData[source] = nil -- clear any local data stored under player index
	syncedData[source] = nil -- clear any synced data stored under player index
end
addEventHandler("onClientPlayerQuit", playerElements, onClientPlayerQuit) -- let's bind handler just for players which are stored in our 'playerElements' parent

--[[
/***************************************************

***************************************************\
]]

function onClientElementDestroy()
	localData[source] = nil -- clear any local data stored under element index
	syncedData[source] = nil -- clear any synced data stored under element index
end
addEventHandler("onClientElementDestroy", otherElements, onClientElementDestroy) -- let's bind handler just for elements which are stored in our 'otherElements' parent

Server:

local localData = {} -- store our local (non-synced data)
local syncedData = {} -- store our synced data
local queuedData = {} -- store our data which will be processed in one trigger
local playerElements = createElement("playerElement", "playerElements") -- this element will hold our players which are ready to accept events, it's solution for "Server triggered client-side event onClientDoSomeMagic, but event is not added client-side.". We would need that aswell for binding handlers.
local otherElements = createElement("otherElement", "otherElements") -- this element will do the same, but it's desired for non-player elements

--[[
/***************************************************

***************************************************\
]]

function getCustomData(pElement, pKey, pIsLocal)
	local cachedTable = false

	if pIsLocal then -- reference to local or synced data
		cachedTable = localData[pElement]
	else
		cachedTable = syncedData[pElement]
	end

	if cachedTable then -- check if such index exists?
		local wholeData = pKey == nil -- do we need whole data or certain key?

		return wholeData and cachedTable or cachedTable[pKey] -- return requested data
	end
end

--[[
/***************************************************

***************************************************\
]]

function setCustomData(pElement, pKey, pValue, pIsLocal, pOnServerEvent, pBuffer, pTimeout)
	local cachedTable = pIsLocal and localData[pElement] or syncedData[pElement] -- do we need data from local or synced table?

	if not cachedTable then -- if sub-table under certain index doesn't exist...

		if pIsLocal then -- whether is local or not
			localData[pElement] = {} -- create sub-table for certain element
		else
			syncedData[pElement] = {} -- create sub-table for certain element
		end
	end

	cachedTable = pIsLocal and localData[pElement] or syncedData[pElement] -- let's reference this once again
	cachedTable[pKey] = pValue -- set our value

	if not pIsLocal then -- if our data isn't local, we want to it sync with client

		if pBuffer then -- if we want to send it in one big trigger :)
			local existingBuffer = queuedData[pBuffer] -- let's check if there's buffer under such name

			if not existingBuffer then -- if doesn't exist
				queuedData[pBuffer] = {} -- create a sub table using it's name
				existingBuffer = queuedData[pBuffer] -- update reference
				existingBuffer[1] = {pElement, pKey, pValue, pOnServerEvent} -- insert data to queue on 1st index, because table it's empty, so there's no need for getting it's length

				setTimer(function() -- use timer to pass data with delay
					triggerClientEvent(playerElements, "onClientReceiveData", getRandomPlayer() or playerElements, true, queuedData[pBuffer]) -- send as buffered data
					queuedData[pBuffer] = nil -- after data was sent, clear our queue
				end, pTimeout, 1)
			else -- otherwise, let's simply add it to queue
				local bufferSize = #existingBuffer + 1 -- get length of table

				existingBuffer[bufferSize] = {pElement, pKey, pValue, pOnServerEvent} -- add data to queue
			end
		else -- otherwise
			triggerClientEvent(playerElements, "onClientReceiveData", getRandomPlayer() or playerElements, false, pElement, pKey, pValue, pOnServerEvent) -- send simply
		end
	end

	return pElement, pKey, pValue -- perhaps, you would need those values afterwards, so let's return them.
end

--[[
/***************************************************

***************************************************\
]]

function onServerPlayerReady()
	if client then -- let's check if it's valid player - remember, do not use 'source'!
		setElementParent(client, playerElements) -- add player to our special group of "ready players"
		triggerClientEvent(client, "onClientDataSync", client, syncedData) -- we need to send copy of server-side data to client, otherwise client wouldn't have it!

		local element, key, value = setCustomData(client, "Key", "Value", false, nil, true, 1000)

		--outputChatBox("Element: "..tostring(element)..", Key: "..key..", Value: "..tostring(value)) -- Would print something like: "Element: userdata: 00000009, Key: Key, Value: Value" - just in case if you need this data
	end
end
addEvent("onServerPlayerReady", true)
addEventHandler("onServerPlayerReady", root, onServerPlayerReady)

--[[
/***************************************************

***************************************************\
]]

function onPlayerQuit()
	localData[source] = nil -- clear any local data stored under player index
	syncedData[source] = nil -- clear any synced data stored under player index
end
addEventHandler("onPlayerQuit", playerElements, onPlayerQuit) -- let's bind handler just for players which are stored in our 'playerElements' parent

--[[
/***************************************************

***************************************************\
]]

function onElementDestroy()
	localData[source] = nil -- clear any local data stored under element index
	syncedData[source] = nil -- clear any synced data stored under element index
end
addEventHandler("onElementDestroy", otherElements, onElementDestroy) -- let's bind handler just for elements which are stored in our 'otherElements' parent

In order to data being cleaned, you need either set your element parent to otherElements or set bound element to root (all elements) or resourceRoot (just this resource elements). For players it would work by default (unless you change player parent).

After reading this code you might think, why there's no force sync from client to server-side, the answer is simple (if you read previous part of this tutorial), we shouldn't believe that client will send valid data. So, we should use server for that.

Before going forward i want to include extra code by @IIYAMA which would help you solve how big timeout (in ms) you need for your buffer.

local sendingDelay = 100 -- ms
local fps = 60

local timeSlice = 1000/fps
local dataReduction = sendingDelay/timeSlice
print("~"..(dataReduction - 1 ).." x times LESS per "..sendingDelay.."ms")

Data handlers - awesome addition

Buffer isn't one and only solution which makes this data system powerful, there's something more - meet data handlers. They are desired to improve functionality of whole system by adding opportunity to execute functions at the moment when certain key(s) was set or changed. Besides, they can be registered for certain element type or event type. Think of it as equivalent of onClientElementDataChange. Each data handler provides useful parameters (element for which data changed/was set, key, old and new value, and server event which called that - if there was any). Yet, it decreases performance (i will try to optimise it as possible, since it's pretty new feature which i've created recently). If you don't want to use it simply remove/comment handleDataChange function in setCustomData.

Examples of using this data system

setCustomData(someElement, someKey, someValue, false, nil, "Buffer 1", 100)

-- Result: Synced data (with buffer), timeout 100 ms - no server event passed

setCustomData(someElement, someKey, someValue, false, "onServerEvent", false)

-- Result: Synced data, without buffer - defined a server event which might trigger data handler (if such exists)

setCustomData(someElement, someKey, someValue, true, "onServerEvent")

-- Result: Local data (non-synced) - defined a server event which might trigger data handler (if such exists)

getCustomData(someElement, someKey, false)

-- Result: Value from synced data - if exists, otherwise nil

getCustomData(someElement, someKey, true)

-- Result: Value from local data - if exists, otherwise nil

getCustomData(someElement, nil, false)

-- Result: All synced data for element someElement, if exists, otherwise nil

getCustomData(someElement, nil, true)

-- Result: All local data for element someElement, if exists, otherwise nil

addDataHandler("player", "Key", onClientDataChange, nil)

-- Result: Handler function which will be triggered for any player element, if key will be equal to "Key", and server event equals to nil.

addDataHandler({"player", "ped"}, {"Key", "Key 2"}, onClientDataChange, "onServerEvent")

-- Result: Handler function which will be triggered for any player/ped element, if key will be equal to "Key" or "Key 2", and event == "onServerEvent"

Efficiency test

--[[Efficiency test - custom data & element data in 100000 iterations (with sync off) - clientside]]

--[[Set local data (data handlers disabled)]]

-- Element data: 33 ms | Custom data: 17 ms

--[[Set local data (data handlers enabled)]]--

-- Element data: 31 ms | Custom data: 49 ms

--[[Get local data by key]]--

-- Element data: 19 ms | Custom data: 9 ms

--[[Get all local data]]--

-- Element data: no available function | Custom data: 10 ms

And here we are, at the end

For now, that's it. Stay tuned for next guide - which will be focused on squeezing performance out of Lua. Feel free to ask questions, if you have some. If you found any mistake in topic, please let me know, so i'll fix it.

Edited by majqq
Fix text.
  • Like 12
  • Thanks 2
Link to post

Updated tutorial.

- Added server-side data validation example.

- Added data handlers.

- Added efficiency test (compared to element data).

- Replaced type(pKey) == "nil" -> pKey == nil in get function.

  • Like 1
Link to post

I still don't know English, I read it several times but I didn't understand one thing, If I use setCustomData (someElement, someKey, someValue, false) on the client side, will it sync with all clients?

Link to post
2 minutes ago, Gaimo said:

I still don't know English, I read it several times but I didn't understand one thing, If I use setCustomData (someElement, someKey, someValue, false) on the client side, will it sync with all clients?

It won't sync due of security reasons. If you want force sync you'd need to write additional code, but doing that is unsafe unless you gonna take data from server-side. What are you trying to do?

Link to post
11 minutes ago, majqq said:

It won't sync due of security reasons. If you want force sync you'd need to write additional code, but doing that is unsafe unless you gonna take data from server-side. What are you trying to do?

I'm planning a group work, I want to create a visible marker for all members of the group and if one of them touches the marker I want it to disappear for everyone, I don't know exactly how I'm going to do this but I don't want to use setElementData, until then I was thinking about creating something like:

Create a table for all groups, groups = {}

Add players to the table -> table.insert (groups, {player1, player2 ...}

When creating a marker I would search the groups table with a for which was the group index and add the marker

groups.marker = createMarker(...)

And in checking if the player touched the marker he was going to use a forearm again, he was going to go through the entire group table checking if the marker belonged to any group and then he was going to delete it.

But I don't think this is going to be very smart, so I'm thinking about using your date system.

create a customData on the marker and when the player touches the marker check the customData.

Link to post
On 13/10/2020 at 21:19, Gaimo said:

Before I destroy an element do I need to erase its date?

Data gets cleared automatically if parent is valid.

On 13/10/2020 at 20:50, Gaimo said:

'm planning a group work, I want to create a visible marker for all members of the group and if one of them touches the marker I want it to disappear for everyone, I don't know exactly how I'm going to do this but I don't want to use setElementData, until then I was thinking about creating something like:

You can assign group to marker, on marker hit you will check if group of player is equal. For visibility use:

https://wiki.multitheftauto.com/wiki/SetElementVisibleTo

  • Thanks 1
Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...