Early diagnosis of tuberculosis (TB) is an essential and challenging task to prevent disease, decrease mortality risk, and stop transmission to other people. The chest X-ray (CXR) is the top choice for lung disease screening in clinics because it is cost-effective and easily accessible in most countries. However, manual screening of CXR images is a heavy burden for radiologists, resulting in a high rate of inter-observer variances. Hence, proposing a cost-effective and accurate computer aided diagnosis (CAD) system for TB diagnosis is challenging for researchers. In this research, we proposed an efficient and straightforward deep learning network called TBXNet, which can accurately classify a large number of TB CXR images. The network is based on five dual convolutions blocks with varying filter sizes of 32, 64, 128, 256 and 512, respectively. The dual convolution blocks are fused with a pre-trained layer in the fusion layer of the network. In addition, the pre-trained layer is utilized for transferring pre-trained knowledge into the fusion layer. The proposed TBXNet has achieved an accuracy of 98.98%, and 99.17% on Dataset A and Dataset B, respectively. Furthermore, the generalizability of the proposed work is validated against Dataset C, which is based on normal, tuberculous, pneumonia, and COVID-19 CXR images. The TBXNet has obtained the highest results in Precision (95.67%), Recall (95.10%), F1-score (95.38%), and Accuracy (95.10%), which is comparatively better than all other state-of-the-art methods.