In high voltage direct current (HVDC) systems, detecting faults has gained a lot of importance. Due to bulk power transmitted through the system where any faults not detected or properly treated will cause to loss of a huge amount of energy and maybe lead to a fall in the transmission system. Fault detection in HVDC transmission systems is crucial to prevent system failure. Voltage source converter (VSC-HVDC) transmission technology is expected to be heavily used in future power systems, but it is more prone to faults than AC transmission. This paper aimed to investigate algorithms to enhance fault detection accuracy in VSC-HVDC transmission lines. The authors used grey wolf optimization (GWO), particle swarm optimization (PSO), and cat swarm optimization (CSO) optimization algorithms to select the best features of fault signals to train artificial neural networks (ANNs) for detection and classification. The models were then combined to maximize system reliability. Implementing these models using MATLAB yielded positive results, with grey wolf optimization providing the highest accuracy of over 99%. The combination of these algorithms resulted in a highly accurate and reliable fault detection and classification system.