Uhm. What you had to do instead of ray casting in real time:

- build a graph of connectivity of all areas of the map OFFLINE

- make sure that graph also has information about disconnected components and never apply A* to points which are disconnected from each other

Do A* on that graph.