No, you're not alone in having this fear. Lots of us have this fear, and personally, I think it's a good thing in a way, to keep us alert and on our toes. One of the biggest fears I have is that your mind will go before you know it. Scary, but it does happen.
To counter it, keep your doctor appts. If you have an 'episode' of forgetfulness, jolt it down on a calendar and keep track of these events too, perhaps some sort of pattern will help somehow. Perhaps some outside environmental factors, such as hot and muggy weather outside could factor in such events.
Develop and maintain daily routines that keep your physical and mental facilities active and alert. If you don't have a regular one-on-one communications w/ anyone in the outside world, develop and engage in some activities that will help keep you active and mentally active by doing things with others, on a regular basis. Keep in contact w/ others your same age, most of us do have friends still around, that our age, give or take a few years. Keeping in check w/ each other helps all of us.
I am not a specialist by any means, these are just things I do, to keep myself mental and active. I'm not a spring chicken by any means, but I am still here, I'm still alive, and want to experience a full life as much as I can, at its finest and fullest. We all do.