The face authentication is a challenging task to validate the user with uncontrolled environment like variations on expression, pose, illumination and occlusion. In order to address these issues, the proposed work provides solution by considering all these factors in inter and intra personal face authentication. During enrollment process, the facial region of still image for the authorized user is detected and features are extracted using local tetra pattern (LTrP) technique. The features are given as input to the neural network namely fuzzy adaptive learning control network (FALCON) for training and classification of features. During authentication process, an image that can vary with expression, pose, illumination and occlusion factors is taken as test image and the test image is applied with LTrP and FALCON to train the features of test image. Then, these trained features are compared with existing feature set by using new proposed multi factor face authentication algorithm to authenticate a person. This work is evaluated among 1150 face images which are collected from JAFFE, Yale, ORL and AR datasets. The overall performance of the work is evaluated by authenticating 1106 images from 1150 constrained images. The second phase of the research work finally produces highest recognition rate of 96% among conventional methods.