Note, that this is really just a temporary workaround. beaTunes will most likely automatically overwrite the BEATUNES_MEMORY value each time you start it. I'm working on changing this for the 32 bit version. Version 5.0.4 will contain this change.
Version 4 could handle the work okay?
The issue here is that beaTunes 5 automatically adjusts certain settings when it detects that certain collection sizes. Unfortunately, 32 bit systems don't allow to reserve large parts of RAM, which is the thing beaTunes bumps up against. beaTunes 4 didn't have this automatic mechanism built in, so people didn't run into the issue you are seeing, but instead into other issues.
PS: Just for your information: Most likely I will drop 32 bit support for beaTunes 6. This won't happen any time soon—I haven't even worked on creating a todo list for that, much less started working on implementing feature. But it's likely that other software vendors will go the same route. So the next time you update your system, you might want to consider using the 64 bit variant.