Ok so using GoAsm assembler and GoLink linker, along with the Easy Code GUI for the GoAsm suite, I wrote my own DLL file, complete with some exported functions, entirely in assembly language. Problem is, Visual Basic 6 kept giving me the Bad DLL Calling Convention error.
After spending an hour of troubleshooting and trial and error trying to figure out what part of my function VB6 didn't like, I finally figured that I needed to PUSH the EDI register onto the stack at the beginning of my DLL function (part of its operation modified the EDI register), and then at the end of my function I needed to POP the value out of the stack back into the EDI register. After that, I no longer got any errors at all from VB6 about Bad DLL Calling Convention.
Why did VB6 not like the fact that the EDI register got modified? What is so special about that register? In the course of my function operating, other registers also got changed (EBX, ECX, EDX, and ESI). Yet the only one that caused VB6 to throw an error was the EDI register.
After spending an hour of troubleshooting and trial and error trying to figure out what part of my function VB6 didn't like, I finally figured that I needed to PUSH the EDI register onto the stack at the beginning of my DLL function (part of its operation modified the EDI register), and then at the end of my function I needed to POP the value out of the stack back into the EDI register. After that, I no longer got any errors at all from VB6 about Bad DLL Calling Convention.
Why did VB6 not like the fact that the EDI register got modified? What is so special about that register? In the course of my function operating, other registers also got changed (EBX, ECX, EDX, and ESI). Yet the only one that caused VB6 to throw an error was the EDI register.