To address the challenges of sample imbalance, insufficient learning of hard-to-classified samples, and difficulties in small target segmentation in rock fracture detection and segmentation tasks, this paper proposes YOLO11n-seg-RF, an improved lightweight algorithm based on YOLO11n-seg. The proposed method incorporates three key components: 1) a Multi-Receptive Field Joint Enhanced Convolutional Block Attention Module (JECBAM) to enhance feature representation, 2) a Grouped Channel Attention-based Feature Fusion Module (GCAConcat) for effective multi-scale feature integration, and 3) a Simplified Spatial Pyramid Pooling Fast module (SimSPPF) to optimize spatial information aggregation. Additionally, the Focaler-IoU loss function is adopted to improve segmentation accuracy for fine-grained and multi-branch fractures. Experimental results on a custom rock fracture dataset demonstrate superior performance. Detection metrics achieve 88.7% Precision (Box), 77.5% Recall (Box), 84.2% mAP0.5 (Box), and 67.3% mAP0.5:0.95 (Box). Segmentation metrics reach 78.5% Precision (Mask), 68.0% Recall (Mask), 68.0% mAP0.5 (Mask), and 27.0% mAP0.5:0.95 (Mask). The model achieves real-time inference at 144 FPS with only 2.47M parameters, outperforming baseline YOLO11n-seg and other mainstream instance segmentation models. Ablation studies confirm the effectiveness of each proposed module, showing significant improvements in detection/segmentation accuracy while reducing model complexity. Generalization experiments on public datasets (crack-seg and carparts-seg) demonstrate superior cross-domain performance, with mAP0.5 (Box) and mAP0.5 (Mask) exceeding comparative models. Practical validation in mining engineering applications reveals that the algorithm successfully identifies core fractures in borehole samples, enabling rapid estimation of uniaxial compressive strength through established porosity-compressive strength equations derived from fracture ratio analysis and uniaxial compression tests, thereby verifying the practical engineering value. To address the challenges of sample imbalance, insufficient learning of hard-to-classified samples, and difficulties in small target segmentation in rock fracture detection and segmentation tasks, this paper proposes YOLO11n-seg-RF, an improved lightweight algorithm based on YOLO11n-seg. The proposed method incorporates three key components: 1) a Multi-Receptive Field Joint Enhanced Convolutional Block Attention Module (JECBAM) to enhance feature representation, 2) a Grouped Channel Attention-based Feature Fusion Module (GCAConcat) for effective multi-scale feature integration, and 3) a Simplified Spatial Pyramid Pooling Fast module (SimSPPF) to optimize spatial information aggregation. Additionally, the Focaler-IoU loss function is adopted to improve segmentation accuracy for fine-grained and multi-branch fractures. Experimental results on a custom rock fracture dataset demonstrate superior performance. Detection metrics achieve 88.7% Precision (Box), 77.5% Recall (Box), 84.2% mAP0.5 (Box), and 67.3% mAP0.5:0.95 (Box). Segmentation metrics reach 78.5% Precision (Mask), 68.0% Recall (Mask), 68.0% mAP0.5 (Mask), and 27.0% mAP0.5:0.95 (Mask). The model achieves real-time inference at 144 FPS with only 2.47M parameters, outperforming baseline YOLO11n-seg and other mainstream instance segmentation models. Ablation studies confirm the effectiveness of each proposed module, showing significant improvements in detection/segmentation accuracy while reducing model complexity. Generalization experiments on public datasets (crack-seg and carparts-seg) demonstrate superior cross-domain performance, with mAP0.5 (Box) and mAP0.5 (Mask) exceeding comparative models. Practical validation in mining engineering applications reveals that the algorithm successfully identifies core fractures in borehole samples, enabling rapid estimation of uniaxial compressive strength through established porosity-compressive strength equations derived from fracture ratio analysis and uniaxial compression tests, thereby verifying the practical engineering value.