An online learning differential evolution algorithm (OLDE) integrated with deep reinforcement learning is proposed to solve complex optimization problems. First, a neural network model maintained by a double deep Q network algorithm is introduced to select the proper parameter adaptation method and control the mutation and crossover of the population. The history information generated by the search process is collected as the training data of the model. The adaptive ability of OLDE is enhanced due to the online learning method. Second, a long-term strategy is proposed to reduce computational complexity and boost learning efficiency. Finally, an adaptive optimization operator is designed to select a suitable mutation strategy for the different search processes. The experimental results reveal that the proposed algorithm is superior to comparison algorithms on CEC 2017 real-parameter numerical optimization.