Need some heavy-duty debugging suggestions

[Warp]

I have finally found myself in the worst possible nightmare for a C/C++/ObjC programmer: I am experiencing random crashes, have absolutely no idea why, and no debugging tool in existence is being of any help whatsoever. I have tried everything that I could think of, and even more, to no avail.

The program always crashes in the main runloop, which isn't helpful at all. The stack trace is:

#0 0x9161109a in mach_msg_trap ()
#1 0x91611807 in mach_msg ()
#2 0x013ef4a6 in __CFRunLoopServiceMachPort ()
#3 0x0134c874 in __CFRunLoopRun ()
#4 0x0134c240 in CFRunLoopRunSpecific ()
#5 0x0134c161 in CFRunLoopRunInMode ()
#6 0x02ed1268 in GSEventRunModal ()
#7 0x02ed132d in GSEventRun ()
#8 0x004e342e in UIApplicationMain ()
#9 0x00007ee8 in main (argc=1, argv=0xbfffedc8) at [...]main.m

This would be mostly indicative of a memory error somewhere. However, I can't find the error with any tool. I have set these gdb environment variables for some heavy-duty debugging:

MallocHelp=YES
NSZombieEnabled=YES
NSDeallocateZombies=NO
MallocCheckHeapEach=100000
MallocCheckHeapStart=100000
MallocScribble=YES
MallocGuardEdges=YES
MallocCheckHeapAbort=1

I have confirmed that gdb is indeed reading these settings (it echoes them on the console) and that they do indeed have effect (if I eg. deliberately dealloc an object and then access it, I'm getting an error).

No cigar. The program is still randomly crashing in the exact same way, with no indication of what the problem could be.

Since I'm programming in Objective-C++ I have also set the gcc option _GLIBCXX_DEBUG to catch possible errors in accessing STL data containers (and I have confirmed that it works by making a deliberate out-of-bounds access). No cigar.

I have tried to get valgrind to run on the simulator, but I can't. (It runs on normal MacOS X apps, but it won't work with iPhone simulator apps. It just gives an error about not being able to execute the binary.)

I'm running out of options here. The program is fairly large, and going over everything with a fine tooth comb could take weeks, if not longer, and there would be no guarantee that I could find the bug.

I have noticed that the program always crashes at the same places, namely after launching a Game Center request and before the completion handler block is executed (these are the only code blocks used in the program). However, there's nothing I can see as being wrong with them, as the crashes happen even if the blocks are empty. Also note that the crashes happen randomly, not every time. It still looks like there's a memory error somewhere else, and it's showing symptoms on these memory blocks just by chance.

[Duncan C]

Originally Posted by Warp

I have finally found myself in the worst possible nightmare for a C/C++/ObjC programmer: I am experiencing random crashes, have absolutely no idea why, and no debugging tool in existence is being of any help whatsoever. I have tried everything that I could think of, and even more, to no avail.

The program always crashes in the main runloop, which isn't helpful at all. The stack trace is:

#0 0x9161109a in mach_msg_trap ()
#1 0x91611807 in mach_msg ()
#2 0x013ef4a6 in __CFRunLoopServiceMachPort ()
#3 0x0134c874 in __CFRunLoopRun ()
#4 0x0134c240 in CFRunLoopRunSpecific ()
#5 0x0134c161 in CFRunLoopRunInMode ()
#6 0x02ed1268 in GSEventRunModal ()
#7 0x02ed132d in GSEventRun ()
#8 0x004e342e in UIApplicationMain ()
#9 0x00007ee8 in main (argc=1, argv=0xbfffedc8) at [...]main.m

This would be mostly indicative of a memory error somewhere. However, I can't find the error with any tool. I have set these gdb environment variables for some heavy-duty debugging:

MallocHelp=YES
NSZombieEnabled=YES
NSDeallocateZombies=NO
MallocCheckHeapEach=100000
MallocCheckHeapStart=100000
MallocScribble=YES
MallocGuardEdges=YES
MallocCheckHeapAbort=1

I have confirmed that gdb is indeed reading these settings (it echoes them on the console) and that they do indeed have effect (if I eg. deliberately dealloc an object and then access it, I'm getting an error).

No cigar. The program is still randomly crashing in the exact same way, with no indication of what the problem could be.

Since I'm programming in Objective-C++ I have also set the gcc option _GLIBCXX_DEBUG to catch possible errors in accessing STL data containers (and I have confirmed that it works by making a deliberate out-of-bounds access). No cigar.

I have tried to get valgrind to run on the simulator, but I can't. (It runs on normal MacOS X apps, but it won't work with iPhone simulator apps. It just gives an error about not being able to execute the binary.)

I'm running out of options here. The program is fairly large, and going over everything with a fine tooth comb could take weeks, if not longer, and there would be no guarantee that I could find the bug.

I have noticed that the program always crashes at the same places, namely after launching a Game Center request and before the completion handler block is executed (these are the only code blocks used in the program). However, there's nothing I can see as being wrong with them, as the crashes happen even if the blocks are empty. Also note that the crashes happen randomly, not every time. It still looks like there's a memory error somewhere else, and it's showing symptoms on these memory blocks just by chance.

Intermittent crashes in a large program are every programmer's nightmare.

Have you done a clean, followed by build and analyze, and gone over the results of the analysis very, very carefully, eliminating every warning?

Have you run with the leaks instrument, and with the zombies instrument? I haven't been able to get the NSZombies environment variable to work with recent versions of XCode, but have found a couple of obscure crashes using the Zombies instrument. Your problem sounds like an over-release bug to me, or an uninitialized stack-based pointer variable.

The fact that the crashes occur in the same places is good. That tells you were to concentrate your debugging efforts. Sometimes these bugs can take weeks to hunt down however. It's grubby, nasty, painful work to be sure, but it's gotta be done.

[Warp]

Ok, this is getting really weird. Basically, I have been in a wild goose chase trying to find a problem in my code (all the tens of thousands of lines of it), when in fact it's not my code that's faulty. I painstakingly went through all of my code, commenting out parts of it and seeing if it still crashes (a task made harder and more annoying because the crashes happen relatively randomly, and never immediately; you always have to wait at least a minute or two before the crash might happen), until there was nothing else left than a completely bare-bones minimal program with one button which sends a gamecenter score, and nothing else. And it still crashes. After a long and painful facepalm I tried the obvious next step: Create a new project from scratch which has nothing else than a button which sends the gamecenter score. Yep, it crashes too. (But only after waiting a minute or two before sending it, and not always even then.)

The crashes happen on the simulator (regardless of iOS version) and on an iPod Touch running iOS 4.1. For some reason I'm unable to get it to crash on an iPod Touch running iOS 4.2.

I'm left with three possibilities: There's something wrong with the way I make the gamekit calls (I don't see how this is possible because it's as bare-bones as it can be), there's something odd with the build settings (there shouldn't be, because it crashes even with default settings on a newly-created project), or there's a bug in either the compiler or iOS which nobody else has noticed. All of these seem extremely unlikely.

As already said, no tool is reporting any error in the program (such as zombie objects or leaks).

Is there something wrong in how I call the gamekit functions? In the app delegate (inside the didFinishLaunchingWithOptions method) I have this:

        [[GKLocalPlayer localPlayer]
            authenticateWithCompletionHandler: ^(NSError *error)
            {
            }];

And in my view controller file I have this:

    void submitGCScore(NSString* leaderboardID, unsigned score)
    {
        GKScore *scoreReporter =
            [[[GKScore alloc] initWithCategory:leaderboardID] autorelease];
        scoreReporter.value = score;
        printf("sumitGCScore(\"%s\", %u);\n", [leaderboardID UTF8String], score);
        [scoreReporter reportScoreWithCompletionHandler:^(NSError *error)
         {
             printf(" - scoreReporter reportScoreWithCompletionHandler: %p\n", error);
         }];
    }

I then call it from the button callback like:

    submitGCScore(@"someid", 5);

The crash always happens after the first printf() and before the second. The stack trace always points to the main runloop (and thus isn't helpful). (No, removing the printfs doesn't remove the crashing.)

[Warp]

It seems that I'm not going crazy after all, because somebody else is having the exact same problem as me: iphone - Crash on reporting score to GameCenter? - Stack Overflow

Not surprisingly, there's no answer to the problem there.

[Duncan C]

Originally Posted by Warp

It seems that I'm not going crazy after all, because somebody else is having the exact same problem as me: iphone - Crash on reporting score to GameCenter? - Stack Overflow

Not surprisingly, there's no answer to the problem there.

Since you've created an all but empty project that exhibits the problem, it sounds like it's time to go to Apple's developer site and submit a radar bug.

I just submitted a bug a couple of days ago regarding NSSavePanel on Lion. Apple does indeed screw up sometimes. (Especially with Lion. It's full of problems. Sigh...)

Do a google search on "Apple Radar bug" to find their bug submission system.