Custom Meshes & Animations - Where To Start?

[Spring E. Thing]

In the same spirit as my earlier thread about Lua movesets (Custom Lua Moveset - Where To Start?), I'm looking for guidance on where to start with custom meshes and animations loaded through Lua.

The difference between this thread and the last one is that, to my knowledge, the tools to write custom mesh files and animation data is through Blender, and unlike Lua I'm actually extremely experienced in Blender. As such, here's all I need to know:
- Where to get this Blender add-on (if all this is indeed done through Blender)
- The basic requirements for an export from the add-on to be considered valid mesh and/or animation data in Co-Op DX and maybe some usage information for the add-on
- How to interface with the mesh and/or animation data through Lua once exported

Much like the thread linked at the start of this post, I will top-up this thread with any strife I experience in accomplishing the things I am yet to know.
I've also done a bit of digging in existing Co-Op DX mods for information pertaining to the questions of this thread already, however it's pretty hard to make anything of them other than answering a bit of my question about Lua interfacing.

Thanks for reading!

 

[yes]

If you are good with blender here's the fast 64 plugin If you want to make custom animations refer to this tutorial here

 

[Spring E. Thing]

If you are good with blender here's the fast 64 plugin If you want to make custom animations refer to this tutorial here

Thanks for the resources! The add-on you linked seems to be the right one, however the animation tutorial you linked seems to export animations in a way that is alien to how I've seen animations interfaced with through Lua so far in my research. As such, I'll take the add-on you linked as-is for now and just post any more specific questions I have as I go along.

 

[yes]

I don't really know how to code

 

[Spring E. Thing]

TL;DR
I'm trying to 'Export Object GeoLayout' but I keep getting an error that reads 'Mesh' has no attribute calc_normals_split. Additionally, my Fast3D material renders as though the shader powering it is broken.

Alrighty I've gone and hit my first major roadblock.
So using a mix of messing around and watching this video (I'm aware that this video is for Co-Op EX and not Co-Op DX but the core systems of both are pretty much identical and the knowledge most certainly transfers for something as core as Fast64 interfacing with one or the other):

I have determined the very basics to get a model to export through Fast64 as a .bin file, which seems to be the format expected by the libraries Co-Op DX provides to Lua mods. I have configured the right-hand panels of the add-on in the 3D viewport to match as closely as I can with the video linked above with the directory to the SM64 Decomp local repo provided (although my local copy is just a .zip download and not an actual repo but that definitely shouldn't matter if my knowledge of git is correct).
The mesh I have created is as basic as it gets, just an active UV map named UVMap to meet the requirements of the 'Export Object GeoLayout' button, a Fast3D material on the default settings of the 'SM64 Shaded Solid' preset, and a primitive 'Brilliant Diamond' mesh data for an object named dobj (the mesh itself named dmesh):

However, I'm facing two issues, one minor and one major.
Firstly: The major issue. Every time I try and 'Export Object GeoLayout' with this primitive 'Brilliant Diamond' object data as my active selection, I get an error that reads the following:

In an effort to remedy this, I tried clicking the button pictured below, something I've never had to do before but figured it to be a quirk of the exporter. However, this had no effect, the error screenshotted above can be found regardless of if Custom Split Normals Data is added to the mesh data or not. I'd also like to note that no object in the scene has the name Mesh as I already pointed out, which is very odd.

Secondly: The minor issue. Why is my mesh the colour that usually appears when something is wrong with a shader in most 3D software? Is this indicative of a greater issue with my installation of the add-on which could be the reason behind the more major issue I'm facing? I installed the add-on like any other .zip add-on I've ever installed to Blender, so it shouldn't be a problem especially if I got this far. My Blender version is 4.1 if that makes a difference, a very recent release.

Any ideas guys?

 

[yes]

What model are you trying to change it too?

 

[Spring E. Thing]

What model are you trying to change it too?

I appreciate the engagement but overwriting an existing model isn't the goal of the thread. Basically, Lua files loaded as part of mods for Co-Op DX can load models exported by Fast64 (presumably exported by Fast64, anyway) so long as they are formatted as raw binary data with the extension of .bin. From surveying other Lua mods out there, I have gotten a better grasp on how through Lua and specifically the lua_definitions library provided by the Co-Op DX team that I can load these .bin files during the execution of my .lua files in-engine, right now my roadblock is actually writing these .bin files since Blender is giving me "backchat" to put it coloquially.

 

[Spring E. Thing]

TL;DR
I'm trying to 'Export Object GeoLayout' but I keep getting an error that reads 'Mesh' has no attribute calc_normals_split. Additionally, my Fast3D material renders as though the shader powering it is broken.

Alrighty I've gone and hit my first major roadblock.
So using a mix of messing around and watching this video (I'm aware that this video is for Co-Op EX and not Co-Op DX but the core systems of both are pretty much identical and the knowledge most certainly transfers for something as core as Fast64 interfacing with one or the other):

I have determined the very basics to get a model to export through Fast64 as a .bin file, which seems to be the format expected by the libraries Co-Op DX provides to Lua mods. I have configured the right-hand panels of the add-on in the 3D viewport to match as closely as I can with the video linked above with the directory to the SM64 Decomp local repo provided (although my local copy is just a .zip download and not an actual repo but that definitely shouldn't matter if my knowledge of git is correct).
The mesh I have created is as basic as it gets, just an active UV map named UVMap to meet the requirements of the 'Export Object GeoLayout' button, a Fast3D material on the default settings of the 'SM64 Shaded Solid' preset, and a primitive 'Brilliant Diamond' mesh data for an object named dobj (the mesh itself named dmesh):

However, I'm facing two issues, one minor and one major.
Firstly: The major issue. Every time I try and 'Export Object GeoLayout' with this primitive 'Brilliant Diamond' object data as my active selection, I get an error that reads the following:

In an effort to remedy this, I tried clicking the button pictured below, something I've never had to do before but figured it to be a quirk of the exporter. However, this had no effect, the error screenshotted above can be found regardless of if Custom Split Normals Data is added to the mesh data or not. I'd also like to note that no object in the scene has the name Mesh as I already pointed out, which is very odd.

Secondly: The minor issue. Why is my mesh the colour that usually appears when something is wrong with a shader in most 3D software? Is this indicative of a greater issue with my installation of the add-on which could be the reason behind the more major issue I'm facing? I installed the add-on like any other .zip add-on I've ever installed to Blender, so it shouldn't be a problem especially if I got this far. My Blender version is 4.1 if that makes a difference, a very recent release.

Any ideas guys?

MAJOR UPDATE
TL;DR
I think Blender 4.1 is too new for the latest release of Fast64 given that it has only been out a few weeks at the time of writing this.

In search to resolve some issues of this thread on my own, I found this tutorial ( GitHub - TrashcamSaysFrick/sm64ex-coop-modding-resources: resources and that may help with the modding of sm64ex-coop ). Whilst incomplete (and likely abandoned), it DOES give enough information for one to first download a premade mario.blend guaranteed to be exportable as-is as geometry data for Mario. However, upon trying to 'Export Armature Geolayout' for mario.blend, I get that same calc_normals_split error as I did in my own .blend file:

Considering this error is happening on a .blend that has pretty confident evidence to work as an export with the Fast64 Blender add-on out-of-the-box, I'm going to roll back to an earlier version of Blender and see if my luck changes.

 

[Spring E. Thing]

ANOTHER MAJOR UPDATE

Well what do you know, only 2 hours ago during the very aggregation of this thread, a fix for this exact issue was merged into the main branch of the Fast64 Blender add-on repo:

I must say, I don't think I've ever seen timing quite like this before. I'll simply build the very latest source for the add-on and use that instead. Will post with my latest issues should I have any more!

EDIT
Whilst the fix above did indeed work, it was not without some strife. I didn't know up until now that the one Scene of my .blend needed some kind of World set other than none at all for Fast64 to function properly. Setting a default World for the Scene, not only did the fix above start working, but the fluorescent pink of my objects in the Scene went away too!
I'm sure there's more this thread can be used for yet with future roadblocks I'm almost certain to hit, but this one big issue is knocked out now at least.

 

[Spring E. Thing]

TL;DR
I clicked the 'Export Armature Geolayout' button with the export type set to 'Insertable Binary' and the game crashes upon trying to host a server with a Lua file that reads the exported .bin file.

Hey guys, back again, just trying to get to grips with Fast64, got to a point where I can consistently export thing without error however it'd seem that my exports are so messed up that they crash the game upon trying to be read by a .lua file.
Below you will find a link to the .blend file I'm trying to export (heads up though, I saved this in Blender 4.1, I doubt it'll read right in earlier versions):
https://files.catbox.moe/unfqx6.blend

What I do is I make the armature present in the .blend (car_geo) into my active selection then click 'Export Armature Geolayout' with my export type set to 'Insertable Binary'. The .bin file that is output by this export goes into a subdirectory of my mod folder called actors with the name car_geo.bin. Then, in Lua, I try and read the .bin with the following line:

local E_MODEL_MOTOMARIO_CAR = smlua_model_util_get_id('car_geo')

After reading the .bin into memory, as a test, I try and spawn_sync_object() an id_bhvBreakableBox with the model swapped to that of the exported .bin, not because that's how I want the .bin to function when paired to a behaviour in the end but just to make sure I've got the Fast64 workflow down. Here's the exact spawn_sync_object() line I'm using for reference:

spawn_sync_object(id_bhvBreakableBox,E_MODEL_MOTOMARIO_CAR,m.pos.x, m.pos.y, m.pos.z,function(o)end)

Well, when all of this is said and done, every time I try and host a server with the Lua that read this .bin enabled, I get this crash dump screen that I personally can't interpret very well:

I'm hoping someone here can understand what this is trying to tell me? Many thanks!

 

[EmeraldLockdown]

By some miracle I can help you with this. Don't use insertable binary, use C. Put that in the actors folder, the game will generate a .bin for you.

 

[Spring E. Thing]

By some miracle I can help you with this. Don't use insertable binary, use C. Put that in the actors folder, the game will generate a .bin for you.

Ah gotcha, didn't expect that kind of functionality from Co-Op DX just reading my .lua, awesome all the same! It's pretty late here now so I won't try this right away but you can expect a test of this first thing tomorrow and anything that may come about from it.

 

[Spring E. Thing]

TL;DR
Exporting with an export type of 'C' and my decomp repo set to a local pull of the Co-Op DX repo produces an 'Error' mesh in-game. Is this the fault of my export or the fault of my Lua?

Alrighty, it's tomorrow now, and I did as EmeraldLockdown suggested. Switching my export type to 'C', the server now starts without crashing BUT the model itself is the classic 'Error' mesh, clearly indicating I did something wrong:

On top of that, I assumed by exporting to C code files through EmeraldLockdown's description that Co-Op DX trying to read those C code files would then automatically write a .bin equivalent (since AFAIK the mod is to be distributed with the models in .bin format based on evidence I've seen from several other mods). As far as I can tell, this is not the case, as all I have are still the C export files and not the .bin file(s) I've seen in other mods attempting to load custom mesh data. Also, just in a practical sense, I know that C is precompiled whereas Lua is not, so I'm not sure why I'd want a C export unless Co-Op DX is manually reading those C files line-by-line and only expecting very specific syntax out of 'em, but I'm more than willing to trust those that have told me to just export as C files regardless.

So, any ideas guys?
I will link the .blend file again below as it is unchanged from when I sent it earlier other than switching the export type from 'Insertable Binary' to 'C' and also pointing the Fast64 add-on to my source repo of C:\SM64COOPDX-REPO. My source repo is of the Co-Op DX source and not the SM64 Decomp source because the Co-Op DX source is just a several forks deep forking of that SM64 Decomp to a degree I imagine would be compatible enough with Fast64, but maybe that's my issue here? IDK, take a look (Blender 4.1 and up only):
https://files.catbox.moe/unfqx6.blend

 

[EmeraldLockdown]

What does the lua model look like? It should be modelname_geo

 

[Spring E. Thing]

What does the lua model look like? It should be modelname_geo

Good question.
In Blender, my both my Armature and its data are named car_geo with my Object and its mesh data named car. In the exporter panel, it is much the same:

In Lua, this is how I'm loading the model, with the script this is in being in the same directory as the actors subdirectory containing my car_geo.bin file (specifically in the main.lua file right at the top outside of any functions):

local E_MODEL_MOTOMARIO_CAR = smlua_model_util_get_id('car_geo')

 

[EmeraldLockdown]

Your exporting it wrong, you should export a folder callled car in the actors folder, not the actor in the actors folder

 

[Spring E. Thing]

Your exporting it wrong, you should export a folder callled car in the actors folder, not the actor in the actors folder

TL;DR
The model's now loading in, but about a quarter of the polys are culled. How come?

Ah, that'd do it I guess. I'll fix that now and report back shortly, probably in an update to this post.

UPDATE
Success! The model is now loading in, thanks a ton! A little bigger than I'd like but that's easily fixed.

Now, as you can see in the image, there's an issue of about quarter of the polys being culled for some reason. I know it can't be the face normals or orientation since I 'Recalculate Normals' regularly in Blender as I model AND a Face 'Orientation' overlay shows all polys to be blue as they ought to be:

I'm going to investigate the mesh to see if there's any commonality between the polys that were chosen to be culled when rendered in-game (perhaps I have to manually triangulate before an export?), but if I don't find anything in common across those polys then please let me know if you know why about quarter of my polys are culled in-engine. Here's a look at the wireframe for the mesh so you can see it's all either quads or tris as it stands right now:

UPDATE
Some better views of the exact polys that are getting culled:

 

[Spring E. Thing]

TL;DR
Culling was caused by non-binary weighting for the mesh's weight painting, see below for more details.

So I randomly remembered a tidbit of info from years ago that SM64's engine cannot handle weighting on meshes that are part of an armature that is non-binary. That is to say that vertex weights on SM64 are expected to either be fully weighted to one bone or to no bones at all (zero OR one). I did NOT create the mesh topo nor the Blender Armature itself with this in mind and so several vertices around the area of the culled polys were weighted partially to one bone and partially to another (FROM zero TO one).
Whilst I have not altered the weighting yet to test this theory, I did decide something close enough and less destructive to support my theory was to export as an object rather than an armature which would disregard the concept of weight painting altogether. Sure enough, the polys stopped being culled when I spawned the mesh in with an object export rather than an armature export (don't know why as an object rather than an armature the spawn has a rotation change but that's besides the point):

As such, I am heavily suspecting that once I re-weight the mesh to only use binary weightings and redesign the armature and the topology a bit to accomodate this, my culled polys SHOULD stop being a problem. Will keep you all posted!

UPDATE
Yep, it was the non-binary weighting:

If the rest of this thread is anything to go by, I do not expect the rest of my journey learning mesh and animation importing to be 100% smooth sailing, so I'll be keeping the thread going with any further issues!

 

[EmeraldLockdown]

Not at my computer, however enabling advanced in material settings and heading to a certain tab should let you turn off culling