Jump to content

[TuT] Lua tables as a efficient data system (Custom element data)


srslyyyy

Recommended Posts

  • Scripting Moderators

Hey. After two years of scripting i've decided to shared a bit of my knowledge with community, which i've learnt - special thanks goes to @IIYAMA for teaching me trigger and tables related stuff. Therefore i present you:


Custom element data

It is data system based on Lua tables and trigger(Server)(Client)Event, which was created and released back in 2020, over few months received a significant updates which introduced useful client-side features and not-so complex logic to fully control data flow through server-side. I do plan to actively develop it, mainly in improving performance which is already great, because i've scripted everything what i had in my mind.


But before going forward...

Let's do some comparision to element data. Color of divbox explains whether is a pro, con, or neutral difference.

Tables are resource dependent, that means you cannot get table and it's contents in other resource without special steps (using exports), while

Element data is resource independent, it's possible to get data from any resource.

Restarting resource causes table to lost stored data, while

Element data keeps it, until element gets destroyed.

Tables are considerably faster, while

Element data is slower.

After acknowledging differences you might say that advantage is taken by element data system because of resource independency, but in later part of tutorial i will point out and describe two scripted features in this data system (they aren't implemented in element data) which are important during sending anything to players, afromentioned features are heavily affecting both client and server performance.


Source code

Latest version is available on GitHub.


Syntax

Client:

mixed getCustomData(mixed pElement, string pKey, string pType)

mixed getElementsByKey(string pKey, mixed pValue, string pType, bool pMultipleResults)

bool setCustomData(mixed pElement, string pKey, mixed pValue, string pType, mixed pEvent, mixed pSyncer)

bool addDataHandler(mixed pElementTypes, mixed pTypes, mixed pKeys, function pFunction, mixed pEvent)


Server:

mixed getCustomData(mixed pElement, string pKey, string pType, mixed pRequester)

mixed getElementsByKey(string pKey, mixed pValue, string pType, mixed pRequester, bool pMultipleResults)

bool setCustomData(mixed pElement, string pKey, mixed pValue, string pType, mixed pReceivers, mixed pSyncer, mixed pEvent, mixed pBuffer, int pTimeout)

bool forceBatchDataSync(mixed pQueue, mixed pType)


Function acceptable types of data, returns and behaviours:

Client:

mixed getCustomData(mixed pElement, mixed pKey, string pType)


pElement: element or a string which holds data pack
pKey: string which holds data under certain name or nil
pType: string which defines data type, can be "local", "synced" or "private"


nil: on failure, or when certain key it's not existing
table: with all data, on success, when pKey is equal to nil
anything else: on success, this can be a string, number, table, bool


This function is used for retrieving data stored under element, you can retrieve certain key or whole data.

mixed getElementsByKey(string pKey, mixed pValue, string pType, bool pMultipleResults)


pKey: string, which defines a key used by element
pValue: string, bool, number, userdata in case if you want filter by certain value, or false/nil if you don't need value to be matching
pType: string, which defines data types, can be "local", "synced" or "private"
pMultipleResults: bool, which decides if you want get more than one element using certain data, true for multiple elements, false for single element


false: on failure
table: empty on failure, and containing element(s) on success - only if pMultipleResults is enabled
element: on success


This function allows you to get a element(s) which contains certain data key, if required you can enable matching key to specified value.

bool setCustomData(mixed pElement, string pKey, mixed pValue, string pType, mixed pEvent, mixed pSyncer)


pElement: element or string, which you wish to set data
pKey: string, which defines a key used by element
pValue: string, boolean, number, userdata or table which will be set as key value
pType: string, which defines type of data, can be "local", "synced" or "private"
pEvent: string or false/nil, defining which server event caused data to change
pSyncer: element or false/nil, responsible for data syncing


false: on failure
true: on success


This function allows you to set a data for element, with certain key, value and data type on client-side with some extras. It doesn't force sync due of security reasons.

Server:

mixed getCustomData(mixed pElement, string pKey, string pType, element pRequester)


pElement: element or a string which holds data pack
pKey: string which holds data under certain name or nil
pType: string which defines data type, can be "local", "synced" or "private"
pRequester: player or nil/false, which requests data - ignored when pType isn't equal to "private"


nil: on failure, or when certain key it's not existing
table: with all data, on success, when pKey is equal to nil
anything else: on success, this can be a string, number, table, bool


This function is used for retrieving data stored under element, you can retrieve certain key or whole data.

mixed getElementsByKey(string pKey, mixed pValue, string pType, mixed pRequester, bool pMultipleResults)


pKey: string, which defines a key used by element
pValue: string, bool, number, userdata in case if you want filter by certain value, or false/nil if you don't need value to be matching
pType: string, which defines data types, can be "local", "synced" or "private"
pRequester: player or nil/false, which requests data - ignored when pType isn't equal to "private"
pMultipleResults: bool, which decides if you want get more than one element using certain data, true for multiple elements, false for single element


false: on failure
table: on success/failure, when pMultipleResults is enabled
element: on success, when pMultipleResults is disabled


This function allows you to get a element(s) which contains certain data key, if required you can enable matching key to specified value.

bool setCustomData(mixed pElement, string pKey, mixed pValue, string pType, mixed pReceivers, mixed pSyncer, mixed pEvent, mixed pBuffer, int pTimeout)


pElement: element or string, which you wish to set data
pKey: string, which defines a key used by element
pValue: string, boolean, number, userdata or table which will be set as key value
pType: string, which defines type of data, can be "local", "synced" or "private"
pReceivers: player or table with players or nil/false, specifies which players will receive data, ignored if pType isn't equal to "private"
pSyncer: element or false/nil, responsible for data syncing
pEvent: string or false/nil, defining which server event caused data to change
pBuffer: string or false/nil, if string passed it will enable batch or buffer functionality (see below for explanation)
pTimeout: integer or false/nil, if it's == -1 then server will use batch, if it's >= 0 server will use buffer, ignored if pBuffer isn't enabled


true: on success
false: on failure


This function allows you to set a data for element, with certain key, value and data type with some extras on server-side.

bool forceBatchDataSync(mixed pQueue, mixed pType)


pQueue: string/false/nil, when string is passed server will search for specified queue to sync, otherwise it will select all queues, not dependent on it's name
pType: string/false/nil, when string is passed server will search for certain type of queue, can be "synced" or "private", otherwise it will select all queues, not dependent on type


true: on success
false: on failure


This function allows you to optimise data flow by collecting pack of data and sending it with merged trigger, further explanations available in later part of tutorial.

After reading detailed syntax, it's time to move on next topic and understand server-side data validation. Often skipped by majority of scripters, even though it's a very important thing.


Server-side data validation

It's a process when server validates data sent from client, if you aren't familiar with it, it might sound difficult, but in reality it isn't. Validation is made in server event function handler, via simple if conditions, it ensures that client sent valid data which later will be used by server. While MTA:SA anticheat is one of strongest ones and you can rely on it in most cases, it is scripter task to make sure that server-side logic is also scripted well, to completely eliminate possibility of harming your server by client sending fake (malicious variant), or incorrect data (default variant). Yes, even default client could be harmful, even unintentionally - let's take for a example ping as a main factor, due of delay between certain player and server, data for player might have different value than for player B. What to do in that case? Answer is simple, get data value from server.

Instead of:
Client -> Request data change with player value -> Server -> Process it

Do:
Client -> Request data change -> Server -> Check for client variable -> Get value from server -> Process it

But, what if (MTA) function isn't available server-side? Then you need use client for returning it and sending to server, e.g collision checks. While this might feels insecure you can (and should) always check whether type of data passed is valid. Special Lua function will be irreplaceable here.
 

local clientData = {}
local typeOfData = type(clientData)

print(typeOfData) -- "table"

You might wonder what is client variable? Shortly, it's a special global variable which is used in server events. This wiki article explains that when we want to receive a player who called the event, we should use client instead of source. Why? Because:

Any player with access to runcode/Lua injector can pass another player which might have admin permissions. This will result in executing server code even though it shouldn't happen by default.
Afromentioned source might be not an actual player, because you can pass pretty much every element e.g root, resourceRoot.

Perhaps you already know that nothing is 100% safe? That is accurate sentence, but when it comes to scripted server-side data validation - you can't bypass it, if it's correctly written. Well, probably you could do that, but you'd need to access server itself, however if it'd happen then it's game over. Example of validation is available here, since i do not pass any data from player side, if client then condition with parent check is sufficient.

Is there anything else you can do to protect your server more?

Actually yes. Since trigger(Client)ServerEvent is a function which calls a counter-side functions (and pass data to it) it's affecting traffic and CPU. It should be well maintained (element data isn't exception). If you do not understand events then feel free to check IIYAMA's events tutorial, while he explains basics i will be expanding this by showing how you can reduce calls and optimise passing data between sides, because triggers are main component of my data system. We'll be moving step by step from easier to a bit harder topics, here comes our first step.


Client-side delays

As i mentioned before that triggers should be well maintained, we should add some delay for client-side, which will be a obstacle for someone, who'd try to spam server with triggerServerEvent calls. It's easily achievable with getTickCount function.
 

--[[
/***************************************************
	Client
***************************************************\
]]

local dataToPass = "string" -- data we want pass to server
local lastTriggerCall = getTickCount() -- store last trigger call
local delayValue = 500 -- ms

function triggerEventByCommand()
	if getTickCount() - lastTriggerCall >= delayValue then -- if delay has passed
		triggerServerEvent("onServerEvent", localPlayer, dataToPass) -- trigger server event
		lastTriggerCall = getTickCount() -- update last trigger call
	end
end
addCommandHandler("trigger", triggerEventByCommand)

--[[
/***************************************************
	Server
***************************************************\
]]

function onServerEvent(pData)
	if client then -- check whether client exists (just for sake)
		local isString = type(pData) == "string" -- make sure that is string

		if isString then -- if so
			local validLength = #pData <= 4 -- check whether is less or equal to 4 chars

			if validLength then -- if so
				print("The checks have passed... Processing code.")
			end
		end
	end
end
addEvent("onServerEvent", true)
addEventHandler("onServerEvent", root, onServerEvent)

You can also implement same delay checks on server-side, but i believe that client-side should be sufficient.


Selecting lowest possible source element for trigger

Probably many of you (including myself), have passed root or resourceRoot as a 2-nd/3-rd argument in function:

triggerServerEvent ( string event, element theElement, [arguments...] )

triggerClientEvent ( [ table/element sendTo = getRootElement(), ] string name, element sourceElement [, arguments... ] )

That shouldn't be done, unless you know what you are doing. Because of CPU impact which wiki states on client and server variation. Most of people (once again, including myself) misunderstood this note. Which is not about attaching custom events to root, but about passing it to trigger itself. While resourceRoot is lower than root in element tree, you should use localPlayer instead - on client-side. When it comes to server-side you can also use same player which triggered certain event, as for example this is handled similarly in server-side part of custom data. But i do also check if sourceElement (pSyncer) is actual element to avoid warnings. We could say that attachedTo is some kind of filter which checks if sourceElement which triggered certain event is allowed (depending on it's position in element tree) to process the function. But you've should already know that if you read events tutorial.

addEventHandler ( string eventName, element attachedTo, function handlerFunction [, bool propagate = true, string priority = "normal" ] )


Optimising data flow

First and foremost my favourite way - batching. What is it, why and when you should use it? Batching is a process when you place data in certain queue, this data however isn't synced instantly, it's stored at server-side in additional table, till force sync will be called (forceBatchDataSync). This is different scenario when we compare it to element data, while custom data with batching enabled sends pack of data in trigger (1 queue = 1 trigger), element data sends each (!) data separately. In other words, you can do x100 setCustomData and they will be sent with 1 trigger, while x100 setElementData waste a lot of traffic and CPU by doing that separately. Generally you should use it always when you set more than one data.

Practical example, setting data upon spawning vehicle:
 

local dataTable = {{"Data 1", "Value 1"}, {"Data 2", "Value 2"}}

function spawnInfernus(pPlayer)
	local playerType = getElementType(pPlayer) == "player" -- check if it's player

	if playerType then -- if so
		local dataPack = false -- store it for later reuse
		local dataName = false -- store it for later reuse
		local dataValue = false -- store it for later reuse
		local playerX, playerY, playerZ = getElementPosition(pPlayer) -- get player position
		local infernusElement = createVehicle(411, playerX + 5, playerY, playerZ) -- create vehicle

		for dataID = 1, #dataTable do -- loop through data table
			dataPack = dataTable[dataID] -- get data at certain index
			dataName = dataPack[1] -- get data name
			dataValue = dataPack[2] -- get data value

			setCustomData(infernusElement, dataName, dataValue, "synced", false, pPlayer, "spawn_infernus", "queue_spawn_infernus", -1)
			-- set data for vehicle, with specified name and value, with type "synced", receivers false, pPlayer as syncer, "spawn_infernus" as event name and queue "queue_spawn_infernus" with timeout -1 (batch)
		end

		forceBatchDataSync("queue_spawn_infernus", "synced") -- force "queue_spawn_infernus" to sync from "synced" type
	end
end
addCommandHandler("spawninfernus", spawnInfernus)

Second way is using buffer, the difference between previous technique is that instead awaiting synchronization we create a timer which collects pack of data with given delay. Using it also results in data reduction, because calls are reduced. You can apply it to data which is being changed by multiple players, or in different scenario where you'd want to collect various data and sync them in same queue, i'll also include handy code by @IIYAMA which would help you adjust your delay.
 

local sendingDelay = 100 -- ms
local fps = 60

local timeSlice = 1000/fps
local dataReduction = sendingDelay/timeSlice

print("~"..(dataReduction - 1).." x times LESS per "..sendingDelay.." ms.")

An simple buffer example:
 

function bufferExample(pPlayer)
	local playerType = getElementType(pPlayer) == "player" -- check if it's player

	if playerType then -- if so
		local bufferChanges = getCustomData(pPlayer, "Buffer changes", "private", pPlayer) or 0 -- get changes for this data

		setCustomData(pPlayer, "Buffer changes", bufferChanges + 1, "private", pPlayer, pPlayer, "buffer_change", "queue_buffer_change", 3000)

		bufferChanges = getCustomData(pPlayer, "Buffer changes", "private", pPlayer) -- get once again changes for this data

		outputChatBox("Data has changed for player: "..getPlayerName(pPlayer).." -> Buffer changes: "..bufferChanges)
	end
end
addCommandHandler("buffertest", bufferExample)

Data handlers - awesome addition

Meet data handlers, equivalent of onClientElementDataChange, those are your own custom functions which could be attached to certain element, data type, key, and event.
Once the function is triggered, it provides set of useful parameters which could be used inside function scope, this is another feature which makes this data system powerful.
For examples, we'll refer to previously set data with batch and buffer.
 

function onClientInternusDataChange(pElement, pKey, pType, pOldValue, pNewValue, pEvent, pSyncer)
	if pSyncer == localPlayer then
		print("I am syncer of this data! At event: "..tostring(pEvent))
	end

	print("Key: "..pKey.." ("..pType..") has changed: "..tostring(pOldValue).." -> "..tostring(pNewValue))
end
addDataHandler("vehicle", {}, {}, onClientInternusDataChange, "spawn_infernus") -- requirements to trigger function: element needs to be vehicle, any data type and key name can trigger this, event needs to be equal to "spawn_infernus"

--[[
/***************************************************

***************************************************\
]]

function onClientBufferChange(pElement, pKey, pType, pOldValue, pNewValue, pEvent, pSyncer)
	if pSyncer == localPlayer then
		print("I am syncer of this data! At event: "..tostring(pEvent))
	end

	print("Key: "..pKey.." ("..pType..") has changed: "..tostring(pOldValue).." -> "..tostring(pNewValue))
end
addDataHandler("player", "private", "Buffer changes", onClientBufferChange, "buffer_change") -- requirements to trigger function: element needs to be player, data type == "private" and key name == "Buffer changes" can trigger this, event needs to be equal to "buffer_change"

How to use it?

Download latest version, and add it directly to your gamemode, to use functions directly or leave it as a separate resource to use functions with exports. First option gives best performance. Make sure that player parent isn't changed anywhere (excluding data system), so data will be cleaned upon player leaving. Add any element (besides player), to special parent.
 

local othersParent = getElementByID("otherElements") -- get parent element
local objectElement = createObject(1337, 0, 0, 3) -- create object

setElementParent(objectElement, othersParent) -- set object's parent

Efficiency test

Element data & Custom data in 100000 iterations (client-side, sync off)

Set local data:

ED: 31 ms < CD: 17 ms

Get local data by key:

ED: 19 ms < CD: 9 ms

Get all local data:

ED: n/a < CD: 10 ms

Edited by majqq
Various fixes
  • Like 14
  • Thanks 2
Link to post
  • Replies 52
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Hey. After two years of scripting i've decided to shared a bit of my knowledge with community, which i've learnt - special thanks goes to @IIYAMA for teaching me trigger and tables related stuff. Ther

Updated tutorial. - Added server-side data validation example. - Added data handlers. - Added efficiency test (compared to element data). - Replaced type(pKey) == "nil" -> pKey

Good Job bro, congrats!!  (TIO STEIN WHAT'S UP, FARPAS)

  • Scripting Moderators

Updated tutorial.

- Added server-side data validation example.

- Added data handlers.

- Added efficiency test (compared to element data).

- Replaced type(pKey) == "nil" -> pKey == nil in get function.

  • Like 2
Link to post
  • Scripting Moderators
2 minutes ago, Gaimo said:

I still don't know English, I read it several times but I didn't understand one thing, If I use setCustomData (someElement, someKey, someValue, false) on the client side, will it sync with all clients?

It won't sync due of security reasons. If you want force sync you'd need to write additional code, but doing that is unsafe unless you gonna take data from server-side. What are you trying to do?

Link to post
11 minutes ago, majqq said:

It won't sync due of security reasons. If you want force sync you'd need to write additional code, but doing that is unsafe unless you gonna take data from server-side. What are you trying to do?

I'm planning a group work, I want to create a visible marker for all members of the group and if one of them touches the marker I want it to disappear for everyone, I don't know exactly how I'm going to do this but I don't want to use setElementData, until then I was thinking about creating something like:

Create a table for all groups, groups = {}

Add players to the table -> table.insert (groups, {player1, player2 ...}

When creating a marker I would search the groups table with a for which was the group index and add the marker

groups.marker = createMarker(...)

And in checking if the player touched the marker he was going to use a forearm again, he was going to go through the entire group table checking if the marker belonged to any group and then he was going to delete it.

But I don't think this is going to be very smart, so I'm thinking about using your date system.

create a customData on the marker and when the player touches the marker check the customData.

Link to post
  • Scripting Moderators
On 13/10/2020 at 21:19, Gaimo said:

Before I destroy an element do I need to erase its date?

Data gets cleared automatically if parent is valid.

On 13/10/2020 at 20:50, Gaimo said:

'm planning a group work, I want to create a visible marker for all members of the group and if one of them touches the marker I want it to disappear for everyone, I don't know exactly how I'm going to do this but I don't want to use setElementData, until then I was thinking about creating something like:

You can assign group to marker, on marker hit you will check if group of player is equal. For visibility use:

https://wiki.multitheftauto.com/wiki/SetElementVisibleTo

  • Thanks 1
Link to post
  • Scripting Moderators

Code was updated, and now it's available only on GitHub.

- Localized buffer function.

- Fixed a case with reference to table, which was leading to saving data in incorrect table (local data used to be saved in synced and probably vice versa).
- Added condition which checks if data was changed last time - affects data handlers (!)

* Apparently data handlers doesn't require any further optimisations, the time required to process logic after implementing check it's way better than in case of element data.

Previous version of code didn't had that, and that's why it resulted in worse result.

Next update for this data system is coming soon, stay tuned :)

Also i want to wish you all merry xmas and happy (hope so) new year!

 

Edited by majqq
Link to post
  • srslyyyy changed the title to [TuT] Lua tables as a efficient data system (Custom element data)
  • Scripting Moderators

Code and tutorial updated.

- Fix issue with adding multiple types for data handler.  It seems that it doesn't work correctly, i will fix it today/tomorrow.
- Moved buffer function to main scope.
- Added batch feature (see first post).
- Added possibility to send data to certain player (this isn't fully complete - because latejoiner will receive this data anyways, excuse me for that, will fix it as soon as possible)
- Added possibility to select player responsible for data change.
- Added getElementsByKey shared function.

Edited by majqq
Link to post
  • Scripting Moderators
43 minutes ago, cocowen said:

what if i want to sync whole inventory to client , set table to value?

Best solution will be to use separate key per item, because in case of table of items - each update means sending this table across client/server-side. Also, it will be harder to maintain it.

Edited by majqq
Link to post
  • Moderators
4 hours ago, majqq said:

Best solution will be to use separate key per item, because in case of table of items - each update means sending this table across client/server-side. Also, it will be harder to maintain it.

You could add a batch process functionality. But to be honest I doubt many people will make use of it since it requires table knowledge.

 

Instead of:

item(server) > sync > item(client)

item(server) > sync > item(client)

item(server) > sync > item(client)

 

Doing this:

items > merge > sync > unpack > items

item(server)  \                 /  item(client)

item(server)    > sync >     item(client)

item(server)  /                 \  item(client)

batch = {

 {key="", value=""},

 {key="", value=""},

 {key="", value=""}

}

 

 

 

 

Edited by IIYAMA
  • Like 1
Link to post
  • Scripting Moderators
13 minutes ago, IIYAMA said:

You could add a batch process functionality. But to be honest I doubt many people will make use of it since it requires table knowledge.

It's already added, pBuffer should be enabled, and pTimeout needs to be equal to -1. After gathering pack of data, use forceBatchDataSync with queue name to sync certain queue or without passing anything to force all queues to sync.

  • Like 1
Link to post
  • Scripting Moderators

Code updated, make sure you use latest version. Issue with adding data handlers should be gone, would be nice if someone could approve that latest changes fixed it.

Link to post
On 02/02/2021 at 21:14, majqq said:

Best solution will be to use separate key per item, because in case of table of items - each update means sending this table across client/server-side. Also, it will be harder to maintain it.

if i have 20 items to sync , send whole items table  or send some or it such as 15 items  for 15 times ,which is better of performance 

Link to post
  • Scripting Moderators
1 hour ago, cocowen said:

if i have 20 items to sync , send whole items table  or send some or it such as 15 items  for 15 times ,which is better of performance 

If you want to sync more than one item, just use batch feature, it's easy and also efficient, because data is packed into table and sent with one big trigger, later is unpacked on client.

-- Server

setCustomData(client, "Item 1", 1, false, "all", client, "server_event", "item_queue", -1)
setCustomData(client, "Item 2", 2, false, "all", client, "server_event", "item_queue", -1)
setCustomData(client, "Item 3", 3, false, "all", client, "server_event", "item_queue", -1)

-- 1. client -> player who will get item
-- 2. "Item 1", "Item 2", "Item 3" name of item
-- 3. 1, 2, 3 -> value
-- 4. false (not local data) -> that is, we want to sync it with client-side
-- 5. "all" -> which players will receive data about client item count (for now it's one and only option)
-- 6. client -> element responsible for syncing, can be set to false
-- 7. "server_event" -> event used on client side - see below.
-- 8. "item_queue" -> name for data queue, you can name it however you want
-- 9. -1 -> timeout, it's set for -1 that means, it will await for your synchronization via function

-- Datas above are already set on server, now we will sync them via function because timeout = -1

forceBatchDataSync("item_queue")

-- Force queue "item_queue" to sync, 
-- we saved bandwith because all of 3 items were synced in merged trigger, and not separately

-- Client

function onClientItemUpdate(pElement, pKey, pOldValue, pNewValue, pOnServerEvent, pResponsibleElement)
	print("Element "..inspect(pElement).. " has received data under key "..tostring(pKey).." with value "..tostring(pNewValue))

	if pResponsibleElement == localPlayer then -- via responsible element you can for example refresh inventory
		-- do some stuff
	end
end
addDataHandler("player", {"Item 1", "Item 2", "Item 3"}, onClientItemUpdate, "server_event")

-- Register data handler for player type, keys which are equal to "Item 1" or "Item 2" or "Item 3"
-- For function onClientItemUpdate
-- And server event which is equal to "server_event"
-- Data handlers are very helpful, because you can process certain logic when function was triggered on data change.

 

Edited by majqq
  • Like 1
Link to post
  • Tut pinned this topic
  • Recently Browsing   0 members

    No registered users viewing this page.


×
×
  • Create New...