Sign in to follow this  
Tut

Working with 3D maps

Recommended Posts

Introduction

This guide intends to teach 3ds Max users how maps are converted and imported/exported through 3D applications, to then be used in professional renders, visualizations, machinma's or to utilise 3ds Max as a precise map editor. The below video demonstrates some of the possibilities that are being taught through this guide. There is a download link in the video description to IPL's that I have decompiled and filtered.

Table of contents

  • Prerequisites
  • Access to GTA SA assets (read https://forum.mtasa.com/topic/119240-mta-modding-in-3d/ on how to extract, etcetera)
    • Converting/decompiling GTA SA assets for compatibility
  • Caution for performance and processing
  • Importing IPL and IDE to 3D application
  • Model issues and solutions
    • Solution: Alpha is severely broken or has the wrong material applied on faces
    • Solution: Random polygons appear black with flipped normals (.001 weld + reset/adjust smoothing groups)
  • Common uses
    • Converting to FBX
    • Mental Ray adaption
    • Designing and rendering a scene
    • Mapping using Max

 

Prerequisites

  • Must have Autodesk 3ds Max of any desired version.
  • Must have Kam's Maxscripts.
  • Must have The Hero's RW importer.
  • Must have GIMS EVO (Max 2016 or below). The beauty of this tool is it can convert various types of materials containing diffuse texture, into standard or GTA 5 material. It supports Kam's and RW material to be converted to standard. It is recommended to add the tool to a secondary copy of 3ds Max due to the heavy size of it. It is possible to have multiple copes of 3ds Max under 1 Student License.

 

Access to GTA SA assets

This guide assumes you have already extracted the entirety of the .img archives found in \models directory of your GTA SA installation. Extracting the interior and cutscene archives is ideal as well. I'm not going to cover the process in this guide, instead please read my MTA Modding in 3D guide.
The following files are needed and has a description for where they're stored.

- DFF (gta3.img, cutscene.img, gta_int.img @ Grand Theft Auto San Andreas\models)
- TXD (gta3.img, cutscene.img, gta_int.img @ Grand Theft Auto San Andreas\models)
- IPL (gta3.img, gta_int.img @ Grand Theft Auto San Andreas\models)
- IPL [binary] (Grand Theft Auto San Andreas\data\maps)
- IDE (Grand Theft Auto San Andreas\data\maps - these are optional as they store 2dfx etc.)
- IFP (Grand Theft Auto San Andreas\anim - these .img archives contain IFP archives which stores a good bunch of animations each. It's possible that gta3.img and cutscene.img contains IFP files too.)

 

Converting/decompiling GTA SA assets for compatibility

Before certain data can be processed by 3ds Max, it needs to be decompiled and cleaned up. Fortunately there are modding tools available which can partially get it done, leaving only small bits of work to be done by hand. GTA SA has a set of binary and non-binary IPL files. Every IPL found in gta3.img and gta_int.img are binary, meaning they're in a non-readable format and cannot be read by 3ds Max, as opposed to IPL files found in \maps\ which can be read.
IPL files found in .img archives do not contain LOD models, whereas ones in \maps\ does. To help identify a LOD, you can look for lines that ends with -1, although not all of these are LOD's. If you want to load in LOD's separate from high detail's, you'll have to filter out LOD's by hand. 

In order to decompile binary IPL's you need the Binary decompiler tool from gtainside and the IPL name restorer from GTAforums. By simply decompiling the IPL files, they'll rename all of the model names to "unknown" but retain the ID's. The IPL name restorer tool will then restore the ID's by cross searching IDE files. There are cases where the tool fails to perform the rename, this namely happens for models which are animated such as burger shot, mills, signs etcetera. To correct this, simply use Prineside, look up the ID and get the model name associated with the ID.

For those who care to keep LOD's separate from high detail's, simply use a coding editor e.g Notepad++ and use the search function and disable "case dependant" and search "lod". You'll want to have 2 versions of every non-binary IPL, one for LODs and one for high details. For high detail IPLs, delete anything with "lod" in their lines, and opposite for LOD IPLs.
The first video in the thread contains decompiled IPL's, sorted with LOD's and without LOD's, and has interior world decompiled as well.
Several IPL's can be merged together if so desired, although it may be wise to keep them separate to avoid overloading your scene with thousands of objects. 

 

Caution for performance and processing

For anyone who doesn't have experience with game model conversion and import of game maps, it's crucial to know that a single part of the map, where that part of the map is not even half of a city, can have fatal consequences for your 3D application and cause shutdowns or hour long freezes if not optimised and dealt with. Your computer will not be very welcoming to thousands of objects and especially materials/textures. It can take 1 IPL district for your viewport to reduce to 5 FPS even without materials visible in viewport. Ways to go about this is using instances and xref/proxies. It seems that IPL imports instances, but at times they lose their instance status and become unique from eachother. If this issue is encountered, you'll want one model to be the parent of a lot of other instances. To do this, either check this Autodesk forum post or open Maxscript listener and add the following code. 

for obj in selection do instancereplace obj $foo

Replace "foo" with the name of the model that you want to use as instance parent. For example you have 11 palm trees, 10 of which are named palm_tree and 1 is named palm_tree_parent. Select those 10 and then replace the word "foo" with "palm_tree_parent". Whenever you make geometry changes to "palm_tree_parent" the 10 others will be affected.

Instanced models is a good start on scene optimisation, but is far from great. Deleting objects not seen by the camera or shadows, reflections, etc. can do a great deal of change for your viewport and render speed. Do yourself a favor and google scene/render optimization. I learned a lot for my demoreel through the following articles:
https://evermotion.org/tutorials/show/10105/optimizing-3d-scenes-for-faster-rendering
https://cgtricks.com/save-memory-when-working-with-big-scenes-3dsmax/

That being said, it's important to be aware of how big an impact these 16 year old maps can have on your $2000 workstation. Your computer won't take the maps lightly. If needed, consider splitting map parts into several max project scenes to avoid overload and rendering/work issues. Be extremely cautious with detail of reflection, refraction, shadows, lighting as these REALLY eat up rendering time from a solid 2 minute to 10 minutes+ if done by somebody without sufficient knowledge. Sometimes alpha materials may also slow down the renderer. There are also ways to enhance rendering speed (if using Mental Ray) by choosing IBL (Image based lighting), this can cut down rendering time by 2-6 times rather than using Final gather, however the quality can degrade so it's really a question about whether a shorter render time is desired at the expense of render quality.

 

Importing IPL and IDE to 3D application

Certain GTA scripts such as Kam's vanila and Kam's Goldfish editions comes with IPL import functions titled as "Map IO". These functions can handle not only IPL and IDE import with several settings, but also export. Exporting as .IPL can prove useful for mappers who then convert their IPL to MTA map editor format. I have just found that Goldfish's Map IO allows to to import not only standard materials (negating the need to convert gta material to standard scanline) but also allows to import binary IPL by specifying a local .dat file. This guide however focuses on the traditional way of importing and handling maps using vanilla scripts, not Goldfish's, so that those who want to know more in depth about the formats gets the option to do so. Nevertheless, below spoiler contains a quick overview of doing it using Goldfish's latest script.

Spoiler

GOLDFISH EDITION OF KAM'S SCRIPTS METHOD

What you're able to do in a split second is: 
- Importing GTA map with standard material, which can then be converted to Mental Ray renderer materials with ease, avoiding having to convert materials using GIMS EVO.
- Importing binary data (non-readable compiled text) without having to decompile anything... I spent 2 days filtering and decompiling IPL files for this guide and just saw this. :o](*,)
 

This importer does the same as vanilla kam's though. Material ID's are :Oed on import.

kams-rw-comparison.png

Kam's and RW imported models above.

Simply select your DFF path (the place where you extracted all of gta3.img's contents), select your image format of choice, select your material of choice, then select your IPL of choice. If you select any from gta3.img or gta_int.img, you'll have to afterwards select a file in your GTA SA installation\data\.dat for the IPL loader to cross search IDE and IPL files to load the binary data.

I guess you'll be able to export the imported binary IPL's to decompiled IPL's through same script. I haven't tried but just something that crossed my mind. This might be an easy workaround rather than having to use the decompiler and model name restorer method.

The below image shows how data files can be imported into 3ds Max along with models, by using Kam's orginal script "Map IO".

unknown.png

  1. Path to your DFF files. Ideally, you've extracted all of the IMG archive's contents into 1 folder.
  2. Version of the game models. Some DFF models are different depending on game to game (III, VC, SA)
  3. Select desired image format. Choose the one that your TXD files were extracted as, in my case it's TGA.
  4. Import IPL file.
  5. Import IDE file.
  6. Import ZON file.
  7. Import 2dfx data (after IDE is imported).

For the vast majority of binary IPL's, only props, buildings and roads are included. The regular IPL's found in GTA SA installation\maps\ usually contain the land masses and such. The binary IPL's contain stream numbers, so they're split into several different IPL's presumably to put less strain on the GTA streamer. These can be merged into one if desired, but not ideal.

 

Model issues and solutions

There aren't any perfect GTA scripts and tools available to public to flawlessly import maps. For all of them there are either limitations or incorrections. Same applies to Blender. This guide contains some of the issues and possible solutions, but they're far from perfect. A minor issue which doesn't deserve any header is where IPL doesn't import anything, or an error message appears. This is usually the case when a different GTA script has been opened in the current 3ds Max instance. Simply close 3ds Max and run again. If it still happens, repeat the step but try on a new project scene, then save as new project file and merge into the main project scene.

 

Solution: Alpha is severely broken or has the wrong material applied on faces

All editions of Kam's scripts has issues with importing face or material index correctly. What this means for imported models is that they have randomly flipped faces (identified by black faces) or have the wrong material applied on the wrong place, e.g wall bricks on a house roof or tree bark on tree leaves.

To fix these issues, use either CJ2000's RW importer or The Hero's RW importer. These two import materials and normals nearly correctly, although by looking at the models in viewport there's still black (flipped) faces. This is not the case when rendering the model, however. It's important to note that RW does not always import the rockstar models perfectly. There are cases where faces must be manually flipped, or enable doublesided for particular models. Disabling backface culling (on render) solves it most of the times.

It's possible that neither of the above scripts imports 2dfx data unlike Kam's. In which case, it may be necessary to import map with Kam's, preserve the 2dfx positions and then replace corrupted models with RW imported ones.

If several models are corrupted, it may be a good idea to look into using some object replacement scripts or functions if they come with 3ds Max. This can speed up with auto-aligning correct models to the corrupted ones. These articles may help:
https://jamiesjewels.typepad.com/jamies_jewels/2011/10/85-3ds-max-quicktip-substituting-objects.html
https://knowledge.autodesk.com/support/3ds-max/learn-explore/caas/CloudHelp/cloudhelp/2017/ENU/3DSMax/files/GUID-2CABA3D7-ECFA-4D0D-A8C2-E86600BEFBE4-htm.html

 

Solution: Random polygons appear black with flipped normals (.001 weld + reset/adjust smoothing groups)

The "Alpha is severely broken or has the wrong material applied on faces" section explains part of why this occurs.

A way to fix the viewport issue is welding all vertices at 0.001 threshold. This removes any overlapping vertex. The model's smoothing groups can then be reset or adjusted, provided the model is imported with 1 total group which makes for a very smooth and unrealistic appearence.

To correct the black polygons, use the "Flip normal" function. Simply flip any black polygons which are supposed to be white. This is not relevant to fix if your end goal is to render the map. Simply follow the method for "RW" scripts from the above section. It's only ideal to flip the faces if it annoys you to view the models corrupted in viewport.

 

Common uses

A lot of things are possible by importing GTA map into 3ds Max. For instance, film makers are able to integrate their own models with ease, customized animations and even physics, FX and much more. MTA mappers can also use the program to create mappings in 3D with a lot of powerful tools to ease the workflow. This can prove exceptionally useful for race maps due to Max's array functions to instance an object a thousand times along a curve to generate roads, loops and what not. 

 

Converting to FBX

Instead of being restricted to Renderware and DFF formats, modelers can now import GTA map and then convert materials and the model components to be suitable with other programs and engines such as Cinema 4D, Unreal Engine and Unity. The conversion to FBX is in fact very simple.

In order for DFF models to be converted, their material and lights needs to be converted too. GTA/RW material needs to be converted to standard scanline material. Normally it's only a matter of changing the material to standard and copying over the diffuse texture, and in rare cases, a specular and environment image as well. Lights may need to be adjusted to show the color that they're given in their IDE (if they import as dummies). Once that's done, the models may be exported as FBX (or any other model format). To correctly save as FBX format, follow the below steps.

Go to File -> Export -> Export selected. This ensures that only the selected assets are being added to the FBX file, and that none other e.g non-converted models interferes and breaks the file.

unknown.png

 

Next, find a location for the FBX file and name it to your liking. Below are the ideal settings for a GTA map to be exported as FBX.
☑️ Smoothing Groups - enable if your lighting won't rely on vertex colors. You'll instead want to use smoothing groups to define model's shading.
☑️ Preserve instances - enable if your scene utilises instances, disable if your scene only contains unique models.
☑️ Preserve edge orientation
☑️ Embed Media - Max will reference the textures used in the models, search for them in your texture folder, then paste the ones used in the scene into the FBX file. Very useful to cut down on texture folder size as it'll only use the ones that are used by your map. Can save some computer disk space. This can be useful if your client or member doesn't have access to your global texture folder.
☑️ Units - normally you'll want metric/meters as that's the system used by SA.
☑️ Axis Conversion - SA is Z-up whereas VC is Y-up (wrong?), so keep as Z-up.
It's optional whether you should export lights or not.

unknown.png

 

Look through the settings just to ensure that everything adapts to your preferences. If it does, hit OK and it will export. If you get an error message, make sure you've only selected the models with standard material.

 

Mental Ray adaption

For those who own a 3ds Max copy of version 2017 or higher, has the ability to change standard scanline materials to Physical among other material types within a few clicks using the Scene Converter (through the Rendering menu). For simple materials with diffuse maps, there are also other tools and 3rd party plugins that can convert between Vray/Corona/Arnold/Mental Ray and so forth.

Having Standard Scanline materials in a Mental Ray scene can add to performance and render speed; it's always a good idea to keep materials corresponding to the active renderer. Arch & Design material also has a lot more functionality than Scanline does. In order to convert Scanline to Arch & Design, install the Material Converter from the following site. https://www.motivacg.com/downloads/scripts-3dsmax/. With this tool, simply choose the materials type according to their renderer. On conversion, all standard materials including multimaterials are converted into Arch & Design for use with the Mental Ray renderer.

Designing and rendering a scene

render5ao.png

3D rendering can be a beneficial way of visualizing a project or simply create art out of San Andreas. The above was shot on 3ds Max using Mental Ray as renderer in a scene lit by daylight system and parti volume shader as means of environment fog. A separate render pass was exported on the graphics card for Ambient Occlusion. The beauty and AO map were composited together and then color corrected. I tried to imitate "Welcome to the 80's" style of his GTA SA remastered CG shots. Down below I'll have a video and few lines of text on what I did to achieve part of what you see. It will by no means be the perfect guide to produce this exact image, but it hopefully will help people get started and troubleshoot issues through the process of setting up a scene.

In the below video's description there are additional links and information. It's important to pause the video once a new subtitle appears, as they do not show for a very long duration.

 

For those looking to add character animations to their project:

  • Own GTA Anim Manager. Minor program that allows you to dig into IFP archives and change/modify their contents. Useful for adding/removing IFP animations in IFP archives.
  • Own Kam's IFP scripts (comes with the DFF IO etc). Allows you to import animations onto skin rigs.

It is a good idea to make necessary changes to rigs before applying animations. Necessary changes could be subdivision/smooth, material setup etcetera. 

After an animation is applied it's recommended to parent the Root bone to a dummy, so the new dummy can rotate the rig without the rig getting deformed while posing it.

 

Mapping using Max

Over the years a few community contributors have designed tools which export map files from Max, as well as internet converters which convert IPL to MTA. Which one to use really boils down to user preference. The most common however would be Kam's Map IO. This has the functions to export position/rotation data through IPL and IDE file types. These can be parsed/converted through 3rd party tools not necessarily related to Max. Another Max method of generating MTA maps is exporting using 50p's Map exporter.

The beauty of creating maps in Max versus MTA map editor, is the ability to work in 3D and really isolate the mapping from everything else with a few clicks. It also offers great scene management, granting users options to import SA map for position reference, hide the SA map while creating their custom map etcetera. It also allows for visualizations of mappings using modern 3D graphics engines. 3ds Max has a lot of tools to create arrays or scatter objects around the surface of another object, those objects' pos and rot data can then be converted to MTA map file format. Mapping in Max can prove exceptionally handy for race mappers, or for those looking to group objects and move them all together, map with attention to precision or instance thousands and thousands of objects.

 

  • Like 3

Share this post


Link to post
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.