Bad Data Will Ruin Your Good Service

Mapping and GIS information is one of the classic “big data” problems. An awful lot of work goes into the formatting and display of mapping information in Google maps/Google earth/Apple maps etc. Satellite and aerial Photography calibrated against position information, layouts of roads and streets, boundaries, etc. It’s a big problem, it’s always been a big problem, and it’s a good example of how reliance on that data can ruin an otherwise good service.

I live in a decidedly rural area, and for many years Google’s maps products had my road labelled with the wrong name. That was annoying, but wasn’t a big deal because not much depended on that information being correct. It actually did get fixed just a few months before Apple released their new replacement for Maps on iOS which uses Apple’s data instead of Google’s data.

In September 2012, Apple replaced the Google-data driven maps with it’s own Apple-data driven maps application. Mapping is a big problem, and I don’t think Apple treated it lightly. They acquired companies and brought in data from different sources to start from scratch to try to catch up to the level of detail Google has in their datasets.

Problem: my road which had just been fixed in the Google datasets, was now wrong in Apple datasets. It was really wrong, too. In the Google sets the road was named wrong, but the house numbers were more or less correct. In the new Apple datasets, the road I live on disappeared. The road overlay itself does show on the map, but now has no name and doesn’t correspond to any civic address, just a rural area name and a postal code which covers a broad area. They actually have the first hundred meters of the road labelled correctly on the map, which then mysteriously ends (even though the real and overlayed roads continue). Consequently all the civic address numbers for the entire road are bunched up in this little 100 meter-or-so stretch. So not only does the dataset not know where I live, it thinks my address is in a place about six kilometers away.

This, naturally, would play havok with navigation directions. Thankfully, I already know where I live and don’t need GPS to find my way back home. If I have to tell anyone else navigating to my place, I have to say “If you’re using Apple’s maps it will lead you to a place about six kilometers away, just keep going down the road

This is where things start going bad. Several services in iOS are now based around this “geofencing” concept. Applications and system operations get triggers based on proximity to a fixed geographic reference. The easiest example of this is the iOS “Reminders” application, where you can put in a reminder to do an alert when you enter/leave a geofenced area. i.e. – “Remind me when I get home to take out the garbage”

reminder

This geofencing idea is fantastic, since I never remember to actually look at the reminders app, and doing reminders based on times can be inconvenient. “Remember to take out the garbage at 7pm” “Oh wait, it’s 7pm and I’m still at work, guess no garbage removal then”

Geofencing completely falls apart when the system can’t determine what “home” is. “When I get home” sounds pretty simple, but if the map data returns that “home” is a place six kilometers from the actual home, geofencing will never be set off. You’ll never get a reminder to take out the garbage, and soon you will be wallowing in filth.

Lots of applications use the Apple map data as a backend for their geofencing, and they will all break because of the bad data. Garbage in, Garbage out as they say, or in my case – Garbage never taken out.

Here’s where things get silly: location services in iOS uses real GPS (and A-GPS, depending on your point of view). GPS returns coordinate data for position as latitude/longitude: you know those crazy numbers like “48.945462, -69.676252” which give a position here on planet Earf. At some point in this chain iOS converts latitude/longitude GPS coordinates from the GPS device into civic address map data. The problem is that despite the fact that location services’ position derived from GPS coordinates, you cannot use GPS coordinates to specify a location in the iOS address book (which is about the only way to specify places for use by other applications).

This would be the easy solution to poor civic address data. Just be able to enter the latitude/longitude GPS data into the address book of a location, and let location services do the rest. After all, GPS data is the source of all this information. Dummy up a new field in the iOS address book, call it something like “geo” and enter the GPS coordinates, you would be able to just use that data as a location. “Remind me when I get to geo to take out the garbage.”

Sorry, it doesn’t work like that. You can enter that information into address book, but iOS will tack the country onto the end of it (“Canada” in my case), ignore the latitude/longitude and try to pick some position that indicates the country, like this.

welcome to canada

Which makes it even more useless than being six kilometers away, now it’s about a thousand kilometers away. Even better: Apple’s Maps actually understand the coordinates, because if you go into the field at the top of the app and backspace over “Canada” and re-submit it, it works correctly. You can’t remove the Canada from address book, though, and address book is where all the queries for “home”, “work”, “geo”, “dog pound” go through.

When iOS 6 initially was released, and it had this bad map data, I was disappointed. After all, it had been wrong for years in Googles map dataset, and only just got fixed. What the new Apple-derived maps application had that the old Google one did not, however, was a way to fix it. Right in the application, “Report a Problem.”

report a problem 1 report a problem 2

I used this to report the problem. In fact, I’ve used it many times. It’s going on eight months since the release of iOS6 and the data has never been fixed. I’ve also tried using the “send feedback to Apple” section on their website to no avail. There’s actually no guarantee that this will ever get fixed, not in iOS7, or anything later because it’s not iOS that’s the problem: it’s the data it’s sourcing.

I don’t even know where this bad data comes from. According to the maps app, data comes from (but I’m sure isn’t limited to) Tomtom, Acxiom, AND, CoreLogic Inc, DigitalGlobe, DMTI, Factual, Getchee, INCREMENT P CORP, Intermap, LeadDog, Localeze, MapData Services Pty Ltd, MDA Information Systems Inc., Urban Mapping, Waze, Yelp, CanVec, CIGAR, Flickr, GeoNames, GlobCover, NASA, OSDM, OSM, StatCan, Tiger/Line, and VMAP0. Someone in that group has my road entered wrong, and there’s no way to find out who, or get them to fix it. My only interface into this is the Apple “Report a Problem.”

The point of all this is that because of this Bad Data, all these services that hinge on the geolocation data are rendered useless. I can’t take advantage of them, it’s as if they do not exist. Apple and other companies invest lots of money in time developing good software and hardware, and it’s rendered completely useless by a few bad entries in a database table somewhere, which is the real shame.

What can you do about this? The easiest solution really is, “just fix problems when they’re reported.” But that takes manpower, and as I wrote earlier, who knows even where this data is wrong. For all I know Apple gets ten thousand “problems” submitted every day which are kids who think it’s funny to ask that the street they live on be renamed “BUTT.”

There are solutions like the OpenStreetMap project (and similarly, Waze) which uses “crowdsourced” information. That’s great as long as you’ve got a crowd to source. When Openstreetmap was younger than it is now it had zero information at all about the area I live in. No streets, no roads, no names, no nothing. I invested a decent amount of my time to use my GPS logger to map out roads and enter the information and names for my region into their database, only to have another member of “the crowd” come along and delete it all. Openstreetmap is a little better populated these days, but it doesn’t mean anything with respect to the Apple maps problem since Openstreetmap’s data is not reflected in Apple’s data.

While writing this piece I went to Openstreetmap and checked. They do have the road name correct, but the civic house numbers are about one kilometer off. I’d consider editing it to fix it, but why bother if someone else will just revert it?

This is the problem of Bad Data, and isn’t just Apple’s problem, it’s just that Apple’s map problem is the best illustration I can give with personal experience.

So, now you want to move to OSX Mountain Lion.

I’m not kidding, I’m word-for-word copying this old post because little nothing has changed, and you feel a need to read this.

Want to move to OSX Mountain Lion? Concerned? Be Safe:

  1. Make sure you have a Time Machine backup. If you don’t have a time machine backup, you’re a fool.
  2. Make sure your machine can actually run OSX Mountain Lion, “iMac (Mid 2007 or newer), MacBook (Late 2008 Aluminum, or Early 2009 or newer), MacBook Pro (Mid/Late 2007 or newer), MacBook Air (Late 2008 or newer), Mac mini (Early 2009 or newer), Mac Pro (Early 2008 or newer), Xserve (Early 2009).” No, your G5 will still not run OSX Mountain Lion
  3. Use SuperDuper or CarbonCopyCloner to make a clone of your existing system out to another hard drive. Buy an external drive, or buy a drive dock and buy bare drives to plug into it. Docks are great for Time Machine backups too. Do not whine that hard drives are expensive. How much is your time worth to you?
  4. Buy and download the OSX Mountain Lion Installer. Don’t pirate it, for fuck’s sake. It’s only $30 $20. Save a copy of the Installer somewhere safe, make an installable DVD or USB Key while you can.
  5. If you’re using Filevault 1, turn it off (and you should be using Filevault, if you’re using a portable machine). It’ll make transitioning easier, and you’ll want to enable the “whole disk filevault” after Mountain Lion is installed. If you’re using Filevault 2 (the Filevault introduced in Lion 10.7), you can leave it enabled – OSX Mountain Lion can install to an existing whole-disk-encrypted volume without any problems.
  6. Install OSX Mountain Lion to the SuperDuper’d Drive and not your internal drive. Don’t do anything to your internal drive.
  7. (alternately: install OSX Mountain Lion to the internal drive, and disconnect your SuperDuper’d drive for safety. This is more work to get back to normal though for when your OSX Mountain Lion goes wrong.)
  8. Boot from the drive you put OSX Mountain Lion on (the SuperDuper’d drive)
  9. Find out exactly how many Rosetta apps you didn’t know you have. Start looking for replacements, since Rosetta is dead. PowerPC has been dead for years. Move with the times. I’m also looking at you, stupid developers still shipping PowerPC code
  10. Use it like this for a few days. Try out everything you normally use to find out it doesn’t work anymore. Keep track. Nicely ask developers to get OSX Mountain Lion versions out.
  11. If it doesn’t work out, don’t worry. Just boot from your untouched Snow Leopard Lion drive, carry on as if nothing happened. If you fuck things up, use your Time Machine backup to save your ass.

Did that all turn out nicely? Fine. Your choice at this point:

  • Install OSX Mountain Lion to your regular drive
  • SuperDuper your external clone drive with OSX Mountain Lion back to your internal drive. Disconnect external drive and keep it safe.

Make sure to keep your Time Machine backup for when you need to roll back.

Never work without a net.

Invalid Opcode at 0027 with Spinrite

I’m only making this post because if you’re searching the intarweb for the answer to this like I did, this is probably it.

I was trying to use my copy of Spinrite 6 on someone’s computer to resolve their bad hard drive issues. This was a Dell XPS430. When you would try to start Spinrite from the cdrom drive you’d get this strange error:

Invalid Opcode at 0027 0004 0293 0073 0000 0000 F800 01E1 6800 728E FFF0 76F8 51E2

Which is not helpful. This turns out to be associated with the RAID controller on the XPS430. Spinrite can’t work with the drive(s) with the RAID controller active. Go into the BIOS setup (F2 at startup), and turn off the RAID controller. Then Spinrite will start normally and let you work on the drives. Remember when you’re all done to turn the RAID controller back on before you reboot back into Windows.

That’s it. Hope this helped you.

Don’t lose your shit when you lose your shit.

Ever had your laptop stolen? That would suck, wouldn’t it. It’s pretty inconvenient to be without your facebooks and your twoofters, sure, but what’s really inconvenient is all your information getting into the hands of some thieving bastard.

Sure, in most cases people who steal your laptop are going to just sell it to some pawn shoppe for crack or linden dollars. In lots of cases they’re just bozos and take it home and start using it themselves. The problem is that if you’re like most people, your personal info is still on that machine for them to access.

I’m not thinking of stupid stuff like those ‘pictures’ of you and that guy’s wife, or all your l33t torr3ntz of Master Chef. I’m talking about stuff like your banking passwords, your email passwords. Use Firefox/Chrome/Internet Asploder? Do you have it ‘save passwords’? You’re screwed. Have your email program remember your password? You’re screwed. Lose your laptop and in a short time, a determined thief can clean you out.

What about your iPhone? Your Android phone? Same thing. There’s the inconvenience of having your shit stolen (or just lost), but the terror knowing that all those camera phone pictures of you and that guy’s dog wife are loose and in the wild.

So take some steps to keep yourself from freaking out.

You have to do these steps before you lose your stuff. Don’t do it after because that would be stupid.

Backups

I’ve been harping with you miserable fucks for years now to make backups, and you still don’t. This will also have the side benefit of saving your ass if your hard drive dies or you just accidentally drop your laptop off a bridge.

You have no excuse. Backup and Restore is built into Windows 7 now, and those insufferable mac people have had this for a while as well. You don’t have to do anything, you just need to set that shit up. Set that shit up already.

Disk Encryption

Yes, encrypt your damned hard drive. If you’ve got a portable machine that people can walk off with, you want to do this.

Using Windows? Then use Truecrypt or Bitlocker and enable the “whole disk encryption”. Using a Mac? Turn on Filevault for fuck’s sake. The Filevault in OSX Lion will encrypt the entire drive, and Filevault in 10.6 and lower will encrypt your home directory and make it safe. These are Free.

Yes, you can even do disk encryption in Linux. Don’t worry, nobody wants to steal your linux laptop.

The nice thing about using disk encryption like this is that it takes almost no effort on your part, you spend a few minutes clicking the mouse (I know that’s hard for you), and it’s done. Then all you have to do is remember your damned password.

Passwords

This one is kind of simple. If you have a computer, set it to require a password to log in. Sure, if you haven’t encrypted the disk it’s fairly trivial to bypass that, but look at who most thieves are: morons. They aren’t going to try to beat the password, they’ll just move on. It also keeps casual people from poking at your shit.

For your other passwords, use something like 1password, Lastpass, Keepass, or Supergenpass or even the built-in sync in Firefox and Chrome to manage your passwords. That way you have them synchronized off that laptop, and you can systematically go and change all your passwords – which you should do if your laptop gets stolen.

Scorched Earth

Here’s where you break out the heavy artillery. Remember you have to set these up before your shit gets stolen. If someone steals your iPhone/Android/Mac/PC, you want the ability to either

  • Get it back (unlikely)
  • Prevent them from using it

If you’ve encrypted the drive, don’t have an easily guessable password, and haven’t written it on a sticky note inside the case lid, that will probably do the job right there. The thieves’ only recourse will be to format the hard drive (destroying your data, keeping it out of their hands).

There are several solutions to tracking down your lost stuff, which is especially important for iPhones/Android phones, which can be pretty trivial to break into. For the iPhone it comes free with the phone: Find my iPhone is a service you had to pay for MobileMe to get, but is now free. Set this up right now. It lets you track the phone’s location (also good if you’ve just lost it behind the couch), put messages on the screen (“give me back my phone you knob”), or – most useful – wipe the phone from remote so nobody can get at the stuff on it. Then your phone just becomes another black market phone for drug dealers and terrorismists.

I don’t have an Android phone, but there are solutions for Android including Prey for Android. I’m sure you know a hundred better ones, and I don’t care.

For the laptop, there is the Prey Project, which is free for three machines – you can pay for more if you’re filthy rich and own so many computers. Prey will let you shut the machine down, get pictures from the webcam of the thief, location, etc. It’s free – so why the hell not? There are other solutions like Lojack, but I have no experience with them. I’m sure you’ve used a hundred better ones, and I don’t care.

Let’s face it: if you go to the cops with the exact location of the criminal, his picture in a mug shot, and him holding up a big sign saying “YO DAWG I STOLE THIS GUY’S LAPTOP”, that’s still not going to be enough to get cops to get off their donut-eating behinds to come out and do something about it. I have first-hand experience with this aspect of law enforcement. So the next best thing is going to be just being able to shut down and wipe the machine from remote to keep their grubby mitts out of your stuff.

The point is, setting up this stuff is easy, and is well worth it for the peace of mind you get when your shit gets ripped off.

So, you want to move to OSX Lion.

Want to move to OSX Lion? Concerned? Be Safe:

  1. Make sure you have a Time Machine backup. If you don’t have a time machine backup, you’re a fool.
  2. Make sure your machine can actually run OSX Lion, “Mac computer with an Intel Core 2 Duo, Core i3, Core i5, Core i7, or Xeon processor”. No, your G5 will not run OSX Lion
  3. Use SuperDuper or CarbonCopyCloner to make a clone of your existing system out to another hard drive. Buy an external drive, or buy a drive dock and buy bare drives to plug into it. Docks are great for Time Machine backups too. Do not whine that hard drives are expensive. How much is your time worth to you?
  4. Buy and download the OSX Lion Installer. Don’t pirate it, for fuck’s sake. It’s only $30. Save a copy of the Installer somewhere safe, make an installable USB Key or DVD while you can.
  5. If you’re using Filevault, turn it off (and you should be, if you’re using a portable machine). It’ll make transitioning easier, and you’ll want to enable the “whole disk filevault” after Lion is installed.
  6. Install OSX Lion to the SuperDuper’d Drive and not your internal drive. Don’t do anything to your internal drive.
  7. (alternately: install OSX Lion to the internal drive, and disconnect your SuperDuper’d drive for safety. This is more work to get back to normal though for when your OSX Lion goes wrong.)
  8. Boot from the drive you put OSX Lion on (the SuperDuper’d drive)
  9. Find out exactly how many Rosetta apps you didn’t know you have. Start looking for replacements, since Rosetta is dead. PowerPC has been dead for years. Move with the times. I’m also looking at you, stupid developers still shipping PowerPC code
  10. Use it like this for a few days. Try out everything you normally use to find out it doesn’t work anymore. Keep track. Nicely ask developers to get OSX Lion versions out.
  11. If it doesn’t work out, don’t worry. Just boot from your untouched Snow Leopard drive, carry on as if nothing happened. If you fuck things up, use your Time Machine backup to save your ass.

Did that all turn out nicely? Fine. Your choice at this point:

  • Install OSX Lion to your regular drive
  • SuperDuper your external clone drive with OSX Lion back to your internal drive. Disconnect external drive and keep it safe.

Make sure to keep your Time Machine backup for when you need to roll back.

Never work without a net.

You don’t give a shit what I think of Duke Nukem Forever, and neither do I.

This post spoils every game ever made, if you don’t like that, too bad. There’s also no music associated with this post. You would have thought I’d have done the obvious thing, wouldn’t you?

I recently purchased and played Duke Nukem Forever, I played it, I liked it – with caveats, and since it’s such a popular topic for people to bash on it I figured I’d get my two cents in. Although this rambling bullshit reads more like thirty-five cents.

I’ll start off by mentioning the fact that you can get the “original” Duke Nukem 3D from GOG.com, which as of this writing is only US$5.99. It’ll run on just about any hardware these days, so you haven’t got much of an excuse. You can match that up with the EDUKE32 project, and run it with nice modern resolutions on everything including your toaster.

We’re in an age these days where getting a vidjagame to market takes lots of people, lots of time, and lots of money. The story of getting this game to market is a tortuous path and I’m not going to bother going over any of it here, there are tons of people who’ve already beat that dead horse.

These days video games are really more interactive cinema than games. My first slap in the face for this was the first Mass Effect game. It’s a computer program that pretends to be an RPG, but is really just “run down a hallway, mash buttons for a minute, then watch a long cutscene”. Mass Effect 2 was the same thing, but more. Run down a hallway, mash buttons, watch cutscene with intricately detailed breasts, then run down the next hallway.

Mass Effect 1/2 “worked” for me, not because they were really that great a game, but because they had a pretty decent story behind them. i.e. – if you just strung out the game’s cutscenes one after the other that would probably have been just as an enjoyable an experience as ‘playing’ the game.

Let me go back a little bit in time. The original Duke Nukem 3D (and Doom, and Marathon, and countless other games of that era) were essentially “run around and kill everything, and eventually find the button that ends the level” games. Usually you had to do some exploring, find the red/blue/ultraviolet key, and then fight a hundred hojillion monsters.

That’s not to say that Duke Nukem 3D/Doom weren’t fun. They were great fun. I was in graduate skool at the time those came out and spent lots of time playing them and enjoying it. I wasn’t a big fan of the violence, but I was huge fan of the exploring concept. I just wanted to run around and see all the cool stuff the game designers had put into it.

Marathon was one of those games that rewarded you for your “running around looking at stuff” mode. Marathon had, I thought, I pretty nice storyline attached to it. Even though the story went completely insane for Marathon Infinity. The drawback of Marathon was that you only “got” the story by reading the terminals, and by “reading” I mean reading. You activated a terminal and usually had to read a wall of text. No voice would read it, no cutscene. If you hadn’t killed all the Pfhor one would usually come up and beat you to death as you were reading. A player could get through all the Marathon games without paying attention to a single thing, and just run around killing things and pushing buttons.

(These days you can download Aleph One, with all the original Marathon levels for free, and I recommend the Marathon Rubicon scenario).

From there, probably my favourite game of all time was the original Deus Ex. Compelling story, voice acting, cutscenes, and importantly: large game levels with tons of ways to complete the objective. A lot of people put this down as choosing either “guns blazing” or “stealth”, but I felt it was a lot more than those options. Accidentally trip an alarm? That’s okay, you haven’t screwed the level. Run past some guys rather than kill them? You can get away with that. I loved the fact that there were tons of plot elements that you didn’t have to do in order to advance the game, so these “secrets” were truly secret. That made the replayability great, since you could go back and and explore areas you hadn’t even noticed the first time. The multiple endings thing was touted at the time, but it didn’t make a lot of difference to me since you could just save and complete all three within about five minutes of that save point.

Deus Ex 2? No. Didn’t like it. Universal ammo and crummy graphics? No. Sniper rifle that made characters do a 360 cartwheel? No.

Deus Ex 3… err Project Snowblind? So disappointing I didn’t even finish it.

Deus Ex: Human Revolution? Not out yet, we shall see. I want to believe.

Games have become bigger and bigger investments for companies. With this inflation, people expect more and more from games. If you bring out a “simple platformer” game these days you’re an indie developer, are expected to charge five bucks, and not be very complicated.

I’m not going to defend Duke Nukem Forever, as overall I thought it was a fairly average game. I don’t think it’s the worst game ever made though, unlike lots of folks. No, the worst game ever made is X3: Terran Conflict – which consists of spending an hour of real time flying somewhere in space while doing nothing, getting your ass trashed in five seconds, and then having to start over.

What’s my favourite game ever? No question, Nethack. No, fuck you. I had more fun pushing ascii characters around a randomly generated map than I ever had being called names by 14-year olds.

So, why don’t I start off with some criticism: people complain about Duke Nukem Forever’s linear gameplay. I agree. The entire game is Duke-on-a-rail. Run down the hallway, get stuck shooting enemies until you kill them all, then the door opens to the next section. In lots of cases you can’t even backtrack. That’s not new in games though. Mass Effect 1/2, F.E.A.R., are all games essentially on rails. Mass Effect 2 had the story to back it up. Here, I’ll put that graphic here that everyone’s been using these days, just so I’ll look cool.

(I wish I knew who made that graphic originally, so I could properly give them credit).

The “on rails” part is really apparent in that disappointing underwater sequence. In Duke Nukem 3D you could “find” scuba gear and swim around underwater levels at length until the air ran out. In Duke Nukem Forever you have exactly enough air to swim to the next conveniently placed set of bubbles coming from a pipe. Too slow and you’re dead. That’s the worst kind of rails because you feel the underwater area is open and explorable, but you can’t go anywhere because you can’t make it ten feet if there aren’t convenient pipes bubbling in that direction. In fact, you just end up following the pipes (rails) to the conclusion.

The antithesis to this, for me, at least, was Elder Scrolls IV: Oblivion. When Oblivion came out I was so enamored with the visuals and the big open playfield that I was actually completely disinterested in doing any actual part of the game. I was far more interested in going to the top of hills and looking at vistas than going through the linear dungeons and – worst of all – trying to find 20 Nirnroot for some dumb fuck NPC who then turns around and demands I go get even more. Fuck Nirnroot.

The good inbetween “wanting to look around and see stuff” and “actually wanting to play the game” were Fallout 3 and Fallout: New Vegas. I think Fallout 3 was the first game since Deus Ex where once I’d finished it, I went back and started over again from scratch just to see more things the second time around.

And to the other side of the scale were the Grand Theft Auto games (GTA3, GTA San Andreas), where the game is big, open, and – to me – completely uninteresting. The “quests” in the GTA games were so tedious for me to try to complete, since you had to complete key parts to advance the game along. I was not interested in chasing a train on a motorcycle, or driving around trying to find 15 dropped porn mags in 1 minute. The highest point of fun for those was that in GTA San Andreas seeing how stupid a jump I could make in a motorcycle. I never finished any of the GTA games because they were so aimless they were boring. I bought GTA4, but it’s unplayable on my computer, so I never bothered even trying to get into it.

Starting with games like Quake 3 Arena and Tribes 2, games started to shift away from even having single-player modes. I don’t like to play online games, principally because I don’t like being called names by 14-year olds. Single-player games have suffered in complexity as a result, because I think game developers put more effort into the low-hanging fruit of decent multiplayer support than making it enjoyable for us rejects.

So, Duke Nukem Forever’s story: really, there isn’t any story here. The game is largely “I thought of this cool scenario for Duke Nukem, let’s get him from A to B to make this happen” I think of the scenario on the top of Hoover dam against the Battlelord where you run down a hallway (okay, the top of the dam), to trigger a scene where a big dude comes out and starts gunning at you. Then you realize why there are all these ammo-replenishing crates that were scattered down the length of the dam for you. The ends of the boss battles are all designers saying “Yeah, then Duke climbs up his back and rips off his tentacle, how are we going to implement that? Make players bash the spacebar”. Attempting to make a game out of a cinematic sequence.

Enemy models: The original Duke Nukem 3D, despite being a game where you essentially ran around and killed everything, had some pretty cartoony models. Cats with jetpacks, pigs with LAPD uniforms. Low-resolution sprites are easy to manage like that, and you have to make them cartoony since you can’t really make them realistic. You might as well run with that.

Duke Nukem Forever tries to show off a modern graphics engine with realistic gritty models. The large boss characters (that Battlelord, the Energy Leech, that dude at the end), who cares – those are big complicated guys. I was really disappointed with the current pigs/cats designs. The pigs/cats had no charm. The pigs are now just muscular humans with pig heads and pants and no shirts, and the cats with jetpacks – well, you can’t even really tell they’re cats. The colour palette of Duke Nukem Forever is in the vein of “if we make everything gray-to-black, people will think it’s gritty and real”. In fact, I had played quite a bit into the game before it clicked in my head, “oh yeah, those are the pigs/cats from the first game”. I’ll come back to this in a little bit.

These movies-become-games things are pretty hard to design and have the cinema flow properly from gameplay. Look at the Valve stuff – Half Life 2, Portal, and especially Portal 2. Portal 2 is a heavily scripted game and pretty dependent on playing along a rail. Do such a task, trigger an event at the end of it, which leads to another task, etc. Portal 2 does a really good job of obscuring that fact from the player, though. As you play through the game you’re not constantly being banged over the head with “once you cross that line on the floor, something will happen”, you get the perception that events are naturally flowing from your player’s action. These are hard to get right, and I figure are undoubtedly the reason why the Half-Life 2 Episodes take so damned long to come out.

Doom 3 was a big loser in this aspect. Most of Doom 3 revolved around “walk over a trigger point, lights go out, demon appears behind you, get your ass kicked”. After a while that got tedious. F.E.A.R. was similar. Duke Nukem Forever also suffers from this. In order to preserve the cinematic experience, you have to restrict what the player can do. No, you can’t go through that door until the next scene plays out. No, you can’t jump up on that surface because that’s not part of the script. Yes, you can destroy that because it’s important to trigger an event, but you can’t destroy that other thing because we haven’t planned for it. Key example of this are those little bouncy alien-trampoline things. You jump on one and it only bounces you exactly where it’s designed to throw you for the next part of the sequence. In Half Life, which also had bouncy things, you had to figure out how to use the bouncy things to get you where you wanted to go. Duke Nukem Forever just takes you there by the hand.

That was one thing I really liked about Deus Ex, was that most of the ‘objects’ in the world you could pick up and move around. Sure they weren’t useful, like plants and bowling trophies, but you felt like you had some measure of control because you could knock shit over. Contrast that with another game I played from that period, No One Lives Forever, which was pretty humourous and fun to play, but always frustrated me because you couldn’t push the chairs around on the floor. Everything might as well have been a crate. (Coincidentally, the second game in that NOLF series was also pretty heavy on the triggered-event-rail-story problem).

Essentially, designing a game that ‘plays’ well, and a game that looks like a good movie are two different things and hard to get right unless you put a lot of effort into it, and I don’t think the Duke Nukem Forever folks had a lot of time to do that. This game has been ‘in development’ for twelve years, but really I think it’s only been a year they actually worked on this version. If you look at the previously released ‘footage’ from the game you see most of them were mockups, not playable, or in most cases contained story elements that never made it to the final Duke Nukem Forever. When the final company in the chain got it’s hands on the task of making a game, they made a game quickly and put it out.

Watch The 1998 E3 Trailer, The 2001 E3 Trailer, none of that is in this game.

I do like that in the ‘game extras’ they include a timeline detailing out a lot of the horrid history of this game, including all those movies. At least they’re not trying to hide it.

The “no secrets” problem: Duke Nukem 3D was big on having hidden things to find, usually giving you something like atomic health or a gun, or something. Same thing for finding hidden rooms in Wolfenstein 3D: it wasn’t that what was in there was necessarily cool, it was that you explored around and found something ‘secret’ in the game. Duke Nukem Forever has, near as I can tell, zero secret stuff. The closest thing is “find all the items that give your ego bar a boost” (also an achievement, I hate achievements). No secrets means no incentive to want to explore the game areas, on rails or not, you don’t want to hang around to look for anything.

Technical issues: I’m an old guy and not actually a gamer. My PC is a little old, but played Duke Nukem Forever just fine. The loading times were absymal. Not seconds, minutes. I never timed it, because usually I’d just leave the room and come back when it was ready to play. Couple that with the checkpoint-save method (you never save, if you get greased you have to start over from the previous checkpoint), meant a lot of areas were damned frustrating until you figured out the only-scripted-way to get out without dying. The loading times were a real killjoy.

Okay, thats a lot of complaining on my part. Duke Nukem Forever has a lot of shortcomings as a game, and most of them come down to unrealized expectations. “This game could have been so cool if… “, “It would have been so much better if…”, “I wish this were like the original game where…”.

The thing is, I played Duke Nukem Forever to the end, I finished it, and overall I wasn’t actually disappointed. The game has a lot of faults, and it isn’t Duke Nukem 3D with modern graphics. I thought a lot of the cinematic stuff was well done, even if the story was pretty dull. Duke Nukem Forever could never live up to the hype and anticipation of a game so legendarily long in development. So at that point it’s best to turn that off and just live in the moment.

Having said that, let’s talk about the parts of Duke Nukem Forever that I really did like. Lots of reviewers got their Hatorade out for this game, but some bits they hated were what saved it for me.

The nostalgic throwbacks and references: Early in the game Duke has to force open a door (by stupidly bashing the spacebar, but forget that) for the first time. The computer voice says you can’t go through because you need the red key. Duke says essentially, “I don’t need no red key” and forces the door open. Given that most of Duke Nukem 3D was “where the fuck is the red key?”, this really was great. The comment about the power armour (i.e. Halo) being for pussies.

I enjoyed the shrunken-Duke-in-RC-car bit. I enjoyed running everyone over in the Monster Truck and making those jet-assisted jumps. I even enjoyed the pre-game intro sequence with Duke throwing a pair of dice with all the company logos who’ve had to work on this thing, leading into a recap of all of Duke Nukem 3D in stylistic form.

The whole ego bar thing: lots of reviewers hated this, but I thought it was well suited. The Duke “character” in all the games revolves around him being an over-the-top caricature. It sure beat wandering around with 3% health knowing you’ve used up all the health packs on the level, and begrudginly having to reload an older save game. People said the regenerating health promoted you to hide more and have less action, but I didn’t find that. Sure you ducked behind a pillar for a moment, but then your ego bar/health comes back up and you run back out into the action. This leads me to my next bit.

Action: Duke Nukem 3D was game about blowing shit up. You ran around and killed everything. Games these days have leaned towards ‘tactical’ stuff. Your Modern Warfare, your Battlefield, Mass Effect 2. Stay safe, hide behind things, use a sniper rifle when possible. Duke Nukem 3D and Duke Nukem Forever are games designed around getting maximal fun by throwing yourself into a hail of gunfire with your own guns emptying out until everyone is dead. I got a lot more enjoyment by charging at dudes with the shotgun than hiding behind a slot machine. At one point you man a cannon and blow up and entire mothership, what’s not to like about that? The end-boss battle with the Cycloid Emperor is where Duke Nukem Forever feels the most like a Duke Nukem 3D successor.

Carrying only two weapons: I didn’t care, you always find enough weapons lying around to trade when you need something bigger.

I liked a lot of the ‘destructability’ of the environments, in particular pillars – which when you gunned them turned into masses of rebar and concrete for a nice visual touch. I wish the game had more of that. Blow up slot machines, knock shit over. That takes more design and thinking and time though, and this game didn’t get that. One of the best things about Fallout 3 I liked to do was blow up abandoned cars, something you don’t get to do a lot of in Fallout New Vegas because there aren’t many abandoned cars.

As I’ve said, I’m old and lame, and one of the parts I got the biggest kick out of were these:

Yeah that’s right, I’m an engineer, I appreciated the realistic looking wiring and fire controls panels.

It falls into the pop culture references category, but here’s a simple bit that I think impressed me the most.

During a part which turns out to be a dream sequence, there’s that little spinning “totem” from the movie Inception on a table, happily spinning away, back and forth. That bit there sold me on the game. Seriously.

A lot of reviews of this game complained about the treatment of women, and aspects of the game they considered misogynistic. Seriously, did you not play Duke Nukem 3D? I’m not talking about the pixellated strippers – if you shot the strippers in Duke Nukem 3D you usually got penalized by having more enemies show up. I’m talking about the final battle with the Alien Queen, where in the game’s final cutscene Duke shoves a bomb up some orifice (take your pick which one you think he uses), makes a glib statement about abortion, and blows her completely up. Both games are excessively over-the-top on Machismo – just like most action movies.

Finally, the bit that everyone seems to mention: fishing out the turd from the toilet at the beginning of the game (which you can also do later on). I think that was well executed. After all, how many games do you try and go around and ‘action’ everything to see what you can do with it? Given that lots of Duke Nukem Forever you actually can’t interact with, I thought it was neat you could pick up that turd, and then get the voiceover along the lines of “who the hell picks up a turd, now what am I supposed to do with it?”, then toss it against the wall. I thought that was comedic genius. It’s also backed up by one of the million different ‘play tips’ that you get to read while waiting a half hour during the loading screens, which says something to the effect of “we wanted to to make picking up the turd make your ego go down, but didn’t”.

Duke Nukem Forever is a flawed game. Flawed a lot more than lots of games. Worst game ever? Hardly. Enjoyable? For me, it was. Worth full price? Hell no, wait for this to go on sale for ten bucks.

Now, making the game better. This was the whole reason I decided to write all this bullshit in the first place, because I saw this comment on reddit and didn’t want it to go by to be forgotten. This game was so long in development, imagine how this could have turned out:

Sam: Dude, I had this epic vision of Duke Nukem Forever
Kenneth: so did 3drealms
Sam: like why throw out each previous engine when you have to catch up with the times?
Sam: Here’s what should have happened.
Sam: Duke powers through two gorgeously rendered Crysis-like levels before the aliens hit him with a Retrotime Gun
Sam: then he ends up in the pixelated quas-3d of the original Duke Nukem
Kenneth: epic
Sam: and is like “what the helll”
Sam: and he has to fight his way back through 10 years of FPS gaming engines
Sam: see this is what I was desperately hoping was going on through all those ten years
Sam: they weren’t WASTED years
Sam: they were carefully stored
Sam: DNF was to be the epic panopolistic archaeological journey of the Perfect FPS
Sam: oh wait except it wasn’t, they were a bunch of incompetents bumbling away a decade
Sam: O WELL

Seriously, if the game had gone this route, and actually switched game engines and visual design during the game, I think that alone would have made it the game of the year. That’s the Duke Nukem Forever I want to play.

I don’t care what you think, AWTFY.

Recovering Installed Firefox Extensions (i.e. – Socialite)

Let’s ignore the fact that today that my car got repossessed – not because of me, but because the previous owner that I bought it from a year ago declared bankruptcy and the credit union still had a lien on the vehicle, all news to me. As usual, it gives evidence to my conjecture that people are stupid and everyone will let you down. But enough of that.

I like Reddit. I also like the firefox extension for Reddit called Socialite. Socialite gives you an extension-level toolbar at the top of the screen when you click on a link to an article in Reddit. Some people like that for the up/downvote and submitting links. I don’t use it for any of that garbage. I use it to get me back to the comments page for the link, since I tend to open a half-dozen tabs at a time and then can never find the original comments thread for it again.

Problem #1: The current Firefox is 3.6 (4.0 is in Beta, I don’t use 4.0). The current version of Socialite does not install under 3.6, despite saying it has compatibility. The author at one point pushed out an update (1.3.3.8) that did install and work under firefox 3.6 – but it was labelled as experimental.

Problem #2: This 1.3.3.8 extension has disappeared completely from the mozilla add-ons site. Or at least, I can’t find it anymore. 1.3.3.1 doesn’t install, and 1.3.3.8 is gone. I don’t think the author is working on it anymore since it’s been 2009 since an update. I can’t blame them, I wouldn’t want to deal with updates. There are also reports that voting is broken on Reddit with 1.3.3.8 (perhaps due to an API change in reddit itself).

Problem #3: Alternative? The reddit “toolbar” you can turn on in your Reddit preferences really just sucks. Frequently you don’t get the page content because it’s been re-framed. Things like Reddit Enhancement Suite really don’t do this function.

So, what? I had the problem I was installing Firefox on a different machine and couldn’t install Socialite 1.3.3.8 from the Mozilla Add-Ons (and hence how this whole thing started). The saving grace: I already had Socialite installed on another computer. Except…

Problem #4: It’s not that obvious how to move an extension from one Firefox installation to another. I could move the whole profile, but I didn’t want to do that.

So, I eventually fixed this by doing some google searching and found a Firefox extension called FEBE or, the “Firefox Environment Backup Extension”. FEBE allows you to back up stuff from your Firefox installation. The nice part? It collapses the extensions back into .xpi files.

Long story short: I installed FEBE onto a machine with working Socialite. Did the backup, pulled the Socialite{1.3.3.8}.xpi from the backup, copied it to another machine, and installed it into Firefox. With .xpi files all you need is to open it from “File -> Open File …” and pick the Socialite{1.3.3.8}.xpi file. You’ll get the standard warnings about installing things you don’t trust (which, if I look, seems to be just about every got-damned extension on the Mozilla Addon’s site, since none of them have verified authors). Worked like a charm.

Also probably actually useful for backing up your Firefox extensions in general.

Addendum: You can get Socialite 1.3.3.8 as source from Github, but you’ll have to build it yourself.

Addendum: May 12, 2011: Socialite 1.4 is finally released and works with Firefox 4. I can finally upgrade

Get yer mail from Thunderbird into GMail in Zero Steps

(There is no music associated with this post)

I had been a long and loyal Thunderbird user for many years now. My problem with Thunderbird became:

  • POP mail scattered about many machines, so I had to make sure “leave mail on server” was always enabled, so I could get my mail from whatever machine I was at.
  • IMAP not an option, since the workplace had at 25MB mail quota. Sigh.
  • Thunderbird 3.x disabled two of my favourite add-on’s for Thunderbird 2.x (“Attachment Sizes” and “Tb Progress History Extension”), that 3.x still can’t do itself – this in itself is lame, but not the focus.
  • Thunderbird 3.x’s search blows total goat.

I actually have a lot of email accounts for work and crap. Since my little trip to the hospital last year, I started consolidating everything into Google’s GMail so that I could get all my mail in one spot, and be able to actually find things via search. Some accounts I’m able to forward from, and some accounts I still have to have GMail get via POP3, but it works.

My holdout is Thunderbird on my desktop at work, which still brings in mail via POP3 because of my years of mail built up in mbox’s on the hard drive.

Up until last week I’d been using Google Desktop Search to try and find things in Thunderbird’s mail, since 3.x’s new search blows goat and the 2.x style search is slow as hell. Much to my disdain I found out that somewhere along the way Thunderbird 3.x broke Google Desktop Search and I haven’t had any mail indexed since August. Uninstal/reinstall/reindex GDS to no avail. Turns out it’s just a lame-ass bug in GDS, which probably won’t get fixed anytime soon.

Windows Desktop Search 4.0 for XP doesn’t even bother to try to index Thunderbird. I understand that Search on Windows 7 will, but my computer at work is eight years old. We’re stuck on XP until the building burns down.

So I took the bold step of migrating back to Thunderbird 2.x, which the indexing reputedly still works for. It brought up again the spectre of “I wish I just had all that old mail in GMail”. I figured I would try again.

Previously I’d tried GMail Loader, which does work but was about the worst solution: it emails all your old mail to your GMail account. Hence, inbox full of ten-year-old mail in about five minutes. Fuck that noise.

I started trying to use Google’s Outlook Migration Assistant, which sounds nice but only works if you’re a Google Apps customer (not the regular GMail, but if you have a Google Apps for Domains account that you pay for). That also would have necessitated moving all my mail from Thunderbird into Outlook for the sole purpose of moving it back out again. Feh.

So, out of the blue I hit on a solution that’s easy, actually works, and doesn’t require any other software. It’s so easy I kicked myself and wondered why I hadn’t thought of it before, stupid me:

  • In GMail, go to your account settings and enable IMAP. That means you can get your GMail via an IMAP client
  • In Thunderbird 2.x/3.x, add your GMail account (via IMAP, not via POP3) to your list of accounts. Once you’re setup, you’ll see all your GMail tags showing up as folders.
  • The easy part: drag your Thunderbird email folders into your GMail IMAP account. That’s it. Thunderbird moves it all in. It’ll show up with a GMail tag the same as the name of the folder in Thunderbird.
  • No, that’s it. That’s the last step. You’re done. All your old mail from Thunderbird in GMail, no fuss. Just a lot of waiting as it copies across the intarweb.

Since it uses only Thunderbird itself, this should work on any system Thunderbird works on (Windows, OSX, Linux, Plan 9).

Now the Thunderbird on the desktop is only around for historical legacy purposes for when I get fired and they want all my old mail.

Solution for iOS4 on iPhone 3G

After much experimentation and research, I’ve determined how to fix iOS4 running on the iPhone 3G. No warranty is expressed or implied. YMMV.

Preparation:

  • First off, make sure you’re updated to iOS 4.0.2
  • Make sure your phone is backed up, and do two hard resets (holding down the sleep button and home button until it resets).
  • Then do a normal power off (hold the sleep button, slide to power off). Wait 30 seconds and power back on
  • Do a DFU restore of iOS 4.0.2
  • After you’ve done the restore, don’t restore from the backup. Reinstall all your applications and re-enter all your data. Don’t sync any music yet.
  • Backup the iPhone again

The Important Part:

  • Put every application into Folders. Not just willy-nilly. It’s important that you put all the applications of the same category into the same folder (i.e. – All the “Social” applications in one folder, all the “Navigation” into a different folder). I’m not sure if you should have them sorted alphabetically in the folder, but it can’t hurt.
  • Go to Settings -> General -> Home Button -> Spotlight Search and turn off Spotlight for everything except Notes.
  • Turn off Push Notifications. Wait at least 15 minutes for the servers to catch up. Then turn Push Notifications back on
  • Sync your music. Use iTunes to sync the “only checked songs” and then incrementally sync the iPhone by genre (i.e. – check all the 50’s music, and sync. Then check all the 60’s music, and sync. Check all the Pop music, and sync.). After each Sync do a power off-power on sequence.
  • Convert all your videos to h.264, but make sure you’re using Main profile with 8 b-frames, CABAC, and CQ mode of 19. Use FFMPEG. It’s actually pretty easy to write a bash shell script to do all this, so I won’t bother including it here. Make sure you get all those little options right
  • If you have ringtones, make sure none are longer than 29.97 (30p) seconds, and none shorter than 23.976 (24p). Use Garageband or Audacity for this
  • Give up on iOS4 on the 3G and get an iPhone 4 instead, nothing’s going to fix that bullshit.