This paper presents OmniPatch, a novel training framework designed to generate universal adversarial patches that can attack semantic segmentation models across different architectures including both Vision Transformers (ViT) and Convolutional Neural Networks (CNN). The approach addresses the critical challenge of black-box adversarial attacks in autonomous driving systems where target model parameters are unknown. By learning patches that generalize across images and architectures without requiring access to target models, this work provides a practical solution for evaluating robustness of deployed perception systems. The framework specifically targets semantic segmentation, which is essential for safe autonomous driving navigation.