Static model import/export for Blender: development

[6667a9bf3cd4anemyax]

The standard locations are the .\txt and .\txt16 folders relative to the .bin file.

[6667a9bf3d059Olfred]

Somehow that isn't working for me.

On another note, I checked the BSP sorting on a more complex model. On them it seems to break more than it fixes. But happy to see that you decided to work on it. In cases where it was needed I still had to rely on bsp.exe which is a way more complex procedure for me. And I can't make use of some benefits when I use your exporter

Here's my testing example.
BSP - without sorting - with sorting enabled
[hats exporters.jpg expired]

[6667a9bf3d1acnemyax]

BSP sorting is a bitch. You have to get two things just right: firstly, the criterion for finding the best faces to put "left" and "right" at each node and secondly, the tree walk algorithm. I've tried lots of things but haven't been able to get the perfect match. However, I'm still hoping to.
I'm also going to add an alternative sorting technique where you tweak the order manually by assigning vertex groups. It isn't a universal solution, but at least it doesn't mangle meshes.

[6667a9bf3d2c7Olfred]

I don't know much about BSP sorting and probably couldn't code anything like that.
Would any BSP sorting algorithm suffice or would you need to exactly replicate the Dark engine BSP sorting?

[6667a9bf3d55bnemyax]

Would any BSP sorting algorithm suffice or would you need to exactly replicate the Dark engine BSP sorting?

There are several approaches, and I've been trying to adapt the oldest and most naive one (use existing faces as splitting planes). The engine toolkit's algorithm doesn't have to be replicated exactly. Some other good implementation can differ from it and still produce comparable results.

[6667a9bf3d658Olfred]

So what about using the one built into the blender game engine?
If you can't connect with it directly, the source should still be free.

[6667a9bf3d74cnemyax]

Yeah, a read through Blender's implementation might help. Thanks for suggesting it.
Mind you, it's part of Blender boolean operations, not of BGE. That engine has no use for BSP.

[6667a9bf3d82cOlfred]

Oh, I thought every engine needs some kind of sorting algorithm like that.
Well, maybe not the raycast ones.

[6667a9bf3d90bnemyax]

Other engines know how to do scene-wide depth sorts at each frame or use order-independent transparency.

[6667a9bf3d9e2Olfred]

Yeah, I thought you could just put the model through that and use the outcome after that.

[6667a9bf3db89nemyax]

Update 0.3.20160811

• The Export only from visible layers option is now available, enabled by default.
• Another hierarchy bug fixed; where subobjects had three or more children, only one of them was drawn in the exported model.

Download: https://sourceforge.net/projects/blenderbitsbobs/files/

[6667a9bf3df4fnemyax]

Update 0.4.20160826
In addition to half-baked BSP, there’s now a new way to have polygons sorted on export; this time you are in control. It can be done through the use of named vertex groups.
Image: http://pix.academ.info/images/img/2016/08/27/9776d9766cf5510241cf3b98f71ded39.jpg

This technique isn’t a one-size-fits-all solution, but it leaves the poly count unchanged. Considering the kind of engine we're dealing with, you’d be hard pressed to find a situation where the method would be useless.
Vertex group naming is the one thing you need to be careful with; it may involve some trial and error.
Here’s an example to put things in perspective. Suppose you have a wine glass in your mission. For the glass to look properly transparent, its polygons must be rendered from back to front. You use vertex groups to batch your polygons for rendering, in alphabetical order.
Here’s how you can arrange the polygon batches in the glass:
Image: http://pix.academ.info/images/img/2016/08/27/cc83b0f27c83a13e353ad6482b950ad9.jpg

The numbers mark the drawing order. Now, for each of the batches:

• Create a vertex group.
• Deselect everything and select the polygons you need.
• Assign your selection to the vertex group.

Example:
Image: http://pix.academ.info/images/img/2016/08/27/99a41f64e33bb95723c508fd478d0096.jpg

If you can remember ZX Spectrum Basic, the sparse numbering technique from that era may come in handy here.
Here’s what you get with that kind of grouping:
Image: http://pix.academ.info/images/img/2016/08/27/0beed400a20135787c0dffab9ca58204.jpg

Admittedly, the glass will still have some problems if you look from below, but you’re unlikely to do that kind of crawling in-game.

Notes:

• Strictly speaking, it’s vertices not polygons that are added to groups. The exporter considers a polygon “in” a group if all of its vertices are in it. If a polygon finds its way into multiple groups, only the first one alphabetically is used.
• Polygons that aren’t associated with any groups are drawn last.
• In objects that are lumped together during export, vertex group membership is merged, including identically-named vertex groups.

Download: https://sourceforge.net/projects/blenderbitsbobs/files/

[6667a9bf3e050Olfred]

That's a really nice addition.
I used to split my model into different objects and sorted them in the outliner.
With this the model will be less of a mess, which is really nice.

[6667a9bf3e1b0voodoo47]

I have no idea what you guys are talking about, but if it means I'll be getting prettier ripped off heads, then I'm in for all the fun.

[6667a9bf3e2bdOlfred]

Do you know these problems where something gets rendered in the forground which is supposed to be in the background and vice versa?
This basically allows you to sort the order manually.

[6667a9bf3e3a9voodoo47]

thumbs up for properly sorted heads.

[6667a9bf3e632nemyax]

if it means I'll be getting prettier ripped off heads

Well, ripping heads off prettily is pure skill, but sorting polygons was a numbing, thankless chore. Not any more.

[6667a9bf3e731Olfred]

Besides, there isn't really much to missort on a ripped of head.
Except you fuck up the vertex order on the face and it faces in the wrong direction.

[6667a9bf3e805voodoo47]

*acknowledges both post trying to not look dumb*

[6667a9bf3e9a1nemyax]

Update 0.4.20161124
Some bugs fixed:

• A regression that caused the script to crash on vhot export
• Incorrect parenting in exported models

https://sourceforge.net/projects/blenderbitsbobs/files/

[6667a9bf3eb71nemyax]

This isn't an official update yet, but I hope it's getting there. I've split the script in two, for import and for export. The results are in the attachment.
In the importer, I've added support for hard edges, and that's about it.
The exporter, however, has undergone some major changes. If the BSP option is selected, it now packs an actual BSP tree into the file instead of trying to do a one-size-fits-all polygon sort. A lot of the code has to accommodate for that. I'm not happy with how the resulting BSP trees are rendered, but it's arguably still an improvement over the previous method. The vertex-group-sort and no-sort options should work as before. Oh, and I changed the subobject parenting logic (again).
Olfred
Would you kindly do some real-world testing on these versions? Especially with multi-object models where joint numbering matters.
[dark-bin-import-export-20171220.zip expired]

[6667a9bf3ec81Olfred]

Early christmas present!
I'll try to get to testing as soon as possible. But with christmas just around the corner and other things taking up my time I might only get to it in the next year.

[6667a9bf3ed66nemyax]

No rush, of course! I'd rather make sure it works before I upload it.

[6667a9bf3f144Olfred]

I've finally gotten around to test a bit.
About the importer. I wonder how you manage to identify hard edges. On the models where I know where I set the hard edges, on import it looks correct. On models directly pulled from the game, I get hard edges, but they don't always make sense. But on the other hand, the models aren't the cleanest so it probably is correct.
One little function could be added to it, if you get a hard edge, it would be nice to automatically get the EdgeSplit Modifier added with "Edge Angle" ticked off and "Shard Edges" ticked on.

Now let's get to the exporter. I couldn't really test it all to my satisfaction because I currently have no "unclean" model at my disposal. I ought to create one in the future. But nonetheless I found some problems.
First of all I tried fumbling about with the christmas hat, as I knew it had some ugly issues if you use transparency on it. But as mentioned before. I cleaned the model up in several ways and now I can't revert it back to it's original trouble bringing state. Originally I split it open into several objects so it will sort somewhat nice when exported. But, now when I merge it all together, the initial sorting which was done stays intact, so I didn't have much luck in testing the desired results. But, I ran into another problem. When I try to export the xs_hat_vg.blend (attached) with vertex group sorting, I get an error message (vertex group export.jpg).

After that I used the botlevo.bin which was previously fixed by me to run some other tests. The models in the screenshots are from left to right, vertex group sorted, created with bsp.exe and using the new bsp tree exporter.
First of, I just looked at the models without adding "Transparency (alpha)" under the Render options in ShockEd. As seen in the screenshot (vodka_no_trans.jpg) it creates some fragmentation when rendered.
But as it can be seen, objects in front of other transparent objects won't render anything behind it. To do this correctly the Object needs "Transparency (alpha)" added in the game. I believe voodoo47 and ZylonBane added this to all object with transparency on SCP already. IIRC it's something specific to NewDark?
Well, if I add it to the Object the fragmentation is gone, but now the transparent part gets rendered in front of the non-transparent part of the bottleneck as seen on vodka_trans.jpg.
Comparing all three, the bsp.exe one still creates the best result. Even on the manually vertex sorted one you get some errors when viewed from very specific angles.

Lastly I just imported the tu_s.bin and then directly exported it again, using the old importer/exporter and the new one. Both resulted in a not correctly working model. But at some point this worked flawlessly, I just don't know which version I was using at that time. I might be able to reconstruct it if necessary.
I haven't spend much more time on this. If wanted I can do it.
[vertex group export.jpg expired]
[xs_hat_vg.blend expired]
[vodka_no_trans.jpg expired]
[vodka_trans.jpg expired]