The SCM Interpreter
Filled with excitement I am here to announce a project that I have been working on together with @GTX named Sphene. In this post I will be talking about SCM, what Sphene has to do with it, our current development and what we plan for the future. A revelation video will accompany this post a few weeks from now.
Before I tell you what Sphene is I first need to give you some background on what SCM is as this plays an important factor in this project. I will not keep you waiting so let's get starting! What is SCM?
Many of you may have heard, seen or even tampered with a well known file "main.scm" in your GTA: San Andreas installation. This file (notice the '.scm' extension) contains all the mission scripts that are available in singleplayer. Essentially the whole story-line and its side-missions are stored in here. The reality for side-missions is a bit more complicated but for the sake of this explanation we'll keep that aside as to keep it simple.
SCM is the name of the language this file has been written in by Rockstar Games. Since it was made by them and the file is compiled (converted into a format that can be easily read by the game but not easily by a human) we unfortunately do not know what the original code looked like. Modders all across the world use a tool named SannyBuilder to write SCM themselves but this is very unlikely to look anywhere near the original format created by Rockstar Games. But, this does not matter much to us as it still compiles into the same format readable by the game.
Essentially a compiled SCM file contains a big variety of "instructions" which tell the game what to do. For example there is an instruction that makes the game spawn a car on specific coordinates, tell a ped to drive or that tells the game to jump to a different location in the script and execute the instructions on this new position.
For those wanting more in-depth information I highly recommend to read the SCM section in the "GTA SA modding book" by fastman92 which can be found here.
So, what does Sphene have to do with this?
Sphene is a SCM Interpreter, as in, it can run files created in the SCM language and which are compiled, including the "main.scm" file from the singleplayer game. This means that we can recreate all of singleplayer within Multi Theft Auto and further extend it with a big variety of features and improvements. Sounds great, doesn't it? It sounds easier than it is in reality however.
The reality is that, as explained in the SCM section, these scripts tell the game what steps to perform, but the game still has to perform them. What does this mean? Well, it means we have to implement each instruction ourselves and make sure we stick as close to the actual game while doing so. Some instructions are fairly easy and quick to implement while others are a lot more complicated. Why? Because on many instructions the game doesn't perform just a single action.
Let us take the instruction (or as we commonly refer to an instruction: opcode) to make a car drive as an example:
0704: car $car_pointer drive_to 1250 -75.5 13.25
Taken from 'SASCM.ini' included in SannyBuilder 3.
The identifier for this instruction is '0704', through this Sphene knows this is the car_drive_to instruction and how many parameters (information given together with this instruction, in this case what car we want to get driving and to which location) to expect. Seems easy enough, except that there is no simple MTA function to get a car driving to a specific location. No, instead we have to write our own logic to make this possible. This can become very complex very quickly, especially as the exact functionality of many instructions isn't even known.
Because of this complexity and the need to make Sphene work as close to the actual game as possible we have started to reverse engineer GTA SA. Reverse engineering means that we try to make sense of the compiled code of the actual game and try to turn it into human readable code. This is easier said than done though, it's a lot more complex than reverse engineering a SCM file is.
Luckily the big modding community (including MTA devs/contributors) have managed to reverse engineer big chunks of the game already, we just have to fill in the gaps that hasn't been reverse engineered yet but that contains chunks of code we require to make Sphene as accurate as possible.
Contributing to Multi Theft Auto
Using this knowledge and to make development easier for us (and simultaneously contributing to the MTA community as a whole) we have started contributing to the Multi Theft Auto codebase. There already is a work in progress pull request (a request for code to be added to MTA) to make it possible for players to drive client-side vehicles, damage them and other improvements. This is not only useful for us but many other servers as well.
Okay, what is the purpose then? Why don't we just play singleplayer?
Good question. Sphene will introduce many new options to make the game behave differently. This can be a setting to have much smarter ped AI's (making the game more difficult) to other settings to enhance the gameplay or raise its difficulty. We're not just interpreting the SCM but can actively improve it.
There surely must be more to it?
Oh, you bet. Did you ever want to play the storyline together with a friend (or multiple friends)? We are introducing Co-Op which allows exactly this. Naturally this version of the game will contain small changes to accommodate for the existence of multiple players and will be a lot harder. Although extra settings can raise that difficulty even more (1 HP limit anyone?).
(Click to enlarge) This is a concept design of the Co-Op lobby designed by AnarchY.
Did I forget to mention that we are also planning support for GTA III and GTA Vice City (data files will have to be provided by yourself in order to load these in, to make sure you do own these games legitimately) into Sphene? I did? Well, I am happy to announce that we have already start adding basic support for these games and are hoping to make them available not long after we complete the support for GTA: San Andreas. The difficulty of implementing support for these games is of course greater as we have to import their full maps, recreate their controls, etc...
That's nice, but how is the current development going?
I am glad that you are asking. Sphene started out as a personal experiment but has quickly grown into a big and stable project. We started with implementing support for basic instructions and basic game logic that allowed us to fully get a tiny, custom, SCM file with a small mission working.
This mission consisted of the following steps:
Step in the nearly exploding car marked by the arrow.
Drive the car to a checkpoint without further damaging it.
Get out of the car and kill the NPC with the arrow above its head.
Very simple, but great for initially testing the interpreter.
Screenshots were taken from an internal video at the time of said development. A small debug panel (improved in later stages) is seen on the right showing the instructions being executed by the interpreter.
(Click to enlarge)
This worked great, so now it was time to start implementing the instructions for the actual game. This proved to be challenging very quickly due to the high amount of instructions the game calls before even visually showing anything to you. But, eventually we did implement the instructions and proper text drawing support for the well known start of the game.
That was a great start. Although it didn't go as well as planned as this text kept disappearing and re-appearing in a loop. Great. Now I had to figure out why this was the case. In order words, I had to start reverse engineering all the instructions Sphene was going through and manually going through the compiled SCM file instruction by instruction to make sure Sphene was interpreting everything correctly.
Eventually I managed to find the issue and resolved it. A larger version of the debug panel was then being developed (for more in-depth information) and later on improved multiple times. It didn't take too long before everything was implemented to allow us to get to the famous "Grove Street -Home." sequence. Complete with audio!
(Click to enlarge)
We then proceeded to improve performance further and mostly do bug fixing. Currently the interpreter can handle instructions up to the sequence in the first mission where you need to get on a bike after a Ballas drive-by occurs. Although due to the amount and type of instructions implemented we did already make it possible for Sphene to run the Kickstart and Bloodring (partially) minigames as well. This truly shows that when we implement more and more instructions a lot more of the game will automatically start becoming available.
(Click to enlarge)
We are of course still implementing a lot of instructions, improving our overall code (fixing bugs and improving performance), adding more game logic, etc. Not only that, we actually have attempted (and will continue working on it in the future) to implement cutscenes. This did not go well at first as it caused a lot of crashes, misaligned objects, etc. But we got it reasonably working, aside from the NPC's that are not animated whatsoever and float weirdly in the air. We hope to get cutscenes up and running soon.
The lobby (for Co-Op) will also be implemented soon and similarly we will start develop on the Co-Op portion of Sphene. After we launch a first version (with GTA: San Andreas support) we will continue development on the GTA: Vice City and GTA III portions.
We also plan (and slowly started) to develop a decompiler and compiler for SCM straight into Sphene and build our own language around it that compiles to SCM. This will allow for user created storyline's, missions, etc that also work in singleplayer if you so desire. The reason we'll be building our own language rather than using the SannyBuilder syntax is simply the fact that there is no standard SCM coding syntax out there and the SannyBuilder one is often too complex and not intuitive for most people. We want our implementation of it to be closer to what people expect from modern programming languages.
(Click to enlarge)
That's it for this post. Please leave any questions and/or remarks in the comments!