Beware of succumbing to the the allure of coding with Generative AI
Not a Medium member? Read for free here.
Have you ever wondered if the “4% rule” for retirement planning is a dependable heuristic? I did. I’ve also wondered what the long term impact of retiring early would be. Should I start collecting Social Security immediately or live off of savings for a while to maximize my benefit?
Great questions, among many others, and tough to answer with a spreadsheet alone.
So, ever the programmer, I decided to build a “Retirement Simulator” to add to Gizmo Life Planner, my free personal Life Planner.
Naturally, any self-respecting retirement planner needs an actuarial table to predict lifespan (and concomitantly, force one to ponder mortality whilst coding). It happens that the Social Security Administration’s (SSA) website provides exactly the information I needed in a ponderously large HTML table. I wondered if ChatGPT could help get this thing into Typescript, so I asked (and held my breath):
“Please convert the following table of actuarial data into Typescript: <copy pasted data from the SSA website here>”
Boom! I got the following, which compiles beautifully without error. Fantastic! ChatGPT is amazing! Since this seemed too good to be true I spot checked a handful of rows. All good. Rock and roll, on to the next task.
export const actuarialData: IActuarialEntry[] = [
{ age: 0, Male: { deathProbability: 0.005860, numberOfLives: 100000, lifeExpectancy: 73.54 }, Female: { deathProbability: 0.005063, numberOfLives: 100000, lifeExpectancy: 79.30 } },
{ age: 1, Male: { deathProbability: 0.000420, numberOfLives: 99414, lifeExpectancy: 72.97 }, Female: { deathProbability: 0.000393, numberOfLives: 99494, lifeExpectancy: 78.70 } },
{ age: 2, Male: { deathProbability: 0.000272, numberOfLives: 99372, lifeExpectancy: 72.00 }, Female: { deathProbability: 0.000223, numberOfLives: 99455, lifeExpectancy: 77.74 } },
{ age: 3, Male: { deathProbability: 0.000225, numberOfLives: 99345, lifeExpectancy: 71.02 }, Female: { deathProbability: 0.000177, numberOfLives: 99432, lifeExpectancy: 76.75 } },
{ age: 4, Male: { deathProbability: 0.000184, numberOfLives: 99323, lifeExpectancy: 70.04 }, Female: { deathProbability: 0.000144, numberOfLives: 99415, lifeExpectancy: 75.77 } },
{ age: 5, Male: { deathProbability: 0.000157, numberOfLives: 99304, lifeExpectancy: 69.05 }, Female: { deathProbability: 0.000122, numberOfLives: 99400, lifeExpectancy: 74.78 } },
{ age: 6, Male: { deathProbability: 0.000140, numberOfLives: 99289, lifeExpectancy: 68.06 }, Female: { deathProbability: 0.000109, numberOfLives: 99388, lifeExpectancy: 73.79 } },
{ age: 7, Male: { deathProbability: 0.000128, numberOfLives: 99275, lifeExpectancy: 67.07 }, Female: { deathProbability: 0.000102, numberOfLives: 99378, lifeExpectancy: 72.79 } },
{ age: 8, Male: { deathProbability: 0.000122, numberOfLives: 99262, lifeExpectancy: 66.08 }, Female: { deathProbability: 0.000098, numberOfLives: 99367, lifeExpectancy: 71.80 } },
{ age: 9, Male: { deathProbability: 0.000123, numberOfLives: 99250, lifeExpectancy: 65.09 }, Female: { deathProbability: 0.000097, numberOfLives: 99358, lifeExpectancy: 70.81 } },
{ age: 10, Male: { deathProbability: 0.000129, numberOfLives: 99238, lifeExpectancy: 64.10 }, Female: { deathProbability: 0.000103, numberOfLives: 99348, lifeExpectancy: 69.82 } },
{ age: 11, Male: { deathProbability: 0.000138, numberOfLives: 99225, lifeExpectancy: 63.10 }, Female: { deathProbability: 0.000113, numberOfLives: 99338, lifeExpectancy: 68.82 } },
{ age: 12, Male: { deathProbability: 0.000164, numberOfLives: 99211, lifeExpectancy: 62.11 }, Female: { deathProbability: 0.000131, numberOfLives: 99327, lifeExpectancy: 67.83 } },
{ age: 13, Male: { deathProbability: 0.000220, numberOfLives: 99195, lifeExpectancy: 61.12 }, Female: { deathProbability: 0.000157, numberOfLives: 99314, lifeExpectancy: 66.84 } },
{ age: 14, Male: { deathProbability: 0.000310, numberOfLives: 99173, lifeExpectancy: 60.14 }, Female: { deathProbability: 0.000190, numberOfLives: 99298, lifeExpectancy: 65.85 } },
{ age: 15, Male: { deathProbability: 0.000446, numberOfLives: 99143, lifeExpectancy: 59.16 }, Female: { deathProbability: 0.000233, numberOfLives: 99279, lifeExpectancy: 64.86 } },
{ age: 16, Male: { deathProbability: 0.000637, numberOfLives: 99098, lifeExpectancy: 58.18 }, Female: { deathProbability: 0.000291, numberOfLives: 99256, lifeExpectancy: 63.88 } },
{ age: 17, Male: { deathProbability: 0.000868, numberOfLives: 99035, lifeExpectancy: 57.22 }, Female: { deathProbability: 0.000355, numberOfLives: 99227, lifeExpectancy: 62.90 } },
{ age: 18, Male: { deathProbability: 0.001100, numberOfLives: 98949, lifeExpectancy: 56.27 }, Female: { deathProbability: 0.000418, numberOfLives: 99192, lifeExpectancy: 61.92 } },
...
];
Weeks later, I was stymied by some very odd behavior with certain simulation scenarios, especially around inheritance planning. I debugged for hours. Finally, I discovered the worst kind of ChatGPT error — one that seems correct, but isn’t. You can see from the table below that errors in ChatGPT’s conversion accumulated gradually. They’re not evident in the first 20 or so rows (where I checked), but then errors begin to accumulate towards the end of the table where they are wildly inaccurate — but not in a way that causes the program to crash. Just enough to give suspicious results.
{ age: 116, Male: { deathProbability: 0.156784, numberOfLives: 23953, lifeExpectancy: 0.56 }, Female: { deathProbability: 0.026251, numberOfLives: 22210, lifeExpectancy: 0.00 } },
{ age: 117, Male: { deathProbability: 0.163610, numberOfLives: 23619, lifeExpectancy: 0.51 }, Female: { deathProbability: 0.026590, numberOfLives: 21027, lifeExpectancy: 0.00 } },
{ age: 118, Male: { deathProbability: 0.171030, numberOfLives: 23251, lifeExpectancy: 0.47 }, Female: { deathProbability: 0.026914, numberOfLives: 19844, lifeExpectancy: 0.00 } },
{ age: 119, Male: { deathProbability: 0.179048, numberOfLives: 22853, lifeExpectancy: 0.43 }, Female: { deathProbability: 0.027225, numberOfLives: 18663, lifeExpectancy: 0.00 } },
{ age: 120, Male: { deathProbability: 0.187664, numberOfLives: 22425, lifeExpectancy: 0.40 }, Female: { deathProbability: 0.027525, numberOfLives: 17483, lifeExpectancy: 0.00 }
In the end, after wasting quite a few hours debugging, I went back to the old school method of copy/paste plus a script to do some transformations to get the correct code into Typescript. The old school approach took about 15 minutes.
There’s a lesson here. In my enthusiasm to save 15 minutes or so, I trusted ChatGPT’s answer, where I should have known better. I know Generative AI hallucinates, and I know that generated characters depended on what has come before, which can lead to insidious, cumulative errors.
But, in my greedy efforts to save a few minutes, none of this occurred to me. Next time, I’ll know better.