It's a play on words: while the title conjures "New York State," the singer is clearly intent on the Big Apple experience; the things mentioned in the lyrics are found in and near New York City ("a New York state of mind").
The waymark is looking down on 8th Avenue at 48th Street, just 3 blocks from Times Square.
Billy Joel wrote the song and performed it on Turnstiles, but he was a little too fresh-faced at the time to do it justice; that didn't happen until Streisand's version on Superman.