Abstract: Objectives To address the issues of missed detections and false positives in ship detection within coastal environments characterized by dense small targets and severe background interference. Methods A ship target detection method based on the WTDS-YOLO model is proposed. Using YOLOv11n as the base model, it combines lightweight wavelet transform convolutions (WTConv) with separable convolutions to enhance the model's noise resistance. Second, a Dynamic Multi-Scale Adaptive Feature Modulation (DyMSAFM) module is embedded to enhance small target detection capability while maintaining processing speed. Finally, ShapeIoU Loss is employed to further accelerate model inference. Results The model achieves an average precision (mAP50) of 87.6%, recall of 77.8%, and F1 score of 83.0%, representing improvements of 5.4%, 5.6%, and 6.0% respectively over the original YOLOv11n model. While significantly enhancing accuracy, it achieves an inference speed of 101.56 FPS. Conclusions The proposed algorithm delivers high accuracy and fast processing speed, making it suitable for real-time maritime monitoring and port management scenarios.