You don't need to worry about the grammar police. You need to worry about the person with whom you are trying to make an appointment. Saying "let me know when works for you" is about like saying "let me know where you would like to meet at." The former is at best an example of bad style. There is no good reason to intentionally risk making a bad first impression. No one is going to say "I really like this guy's stance against the grammar police. Let's meet with him as soon as possible."