Phellinus baumii extract (PBE) possesses considerable α-glucosidase-inhibited activity. This study investigated the hypoglycemic effect in vitro and in vivo using a glucose consumption assay in HepG2 cells, intragastric administration for ten weeks in STZ-induced mice, and intestinal flora fermentation in patients with type 2 diabetes to reveal the possible underlying mechanisms. PBE was prepared, including α-glucosidase-inhibited ethanol extract (EE) and aqueous extract (AE). In vitro, PBE promoted glucose consumption and enhanced glycogen content and hexokinase activity but lowered phosphoenolpyruvate carboxylase kinase activity in HepG2 cells. In vivo, PBE treatment significantly reduced the body weight (p < 0.05) and fasting blood glucose levels of diabetic mice (p < 0.01), with the lowest blood glucose level observed in the EE+AE group. Furthermore, the serum insulin levels and insulin resistance index (HOMA) of PBE-treated groups decreased significantly (p < 0.01). Moreover, gene expression levels of the IRS-1/PI3K/AKT pathway were significantly upregulated by PBE treatment (p < 0.01). In vitro fermentation demonstrated that EE significantly inhibited the production of H₂S and NH₃ in the intestinal flora fermentation model in diabetic patients (p < 0.05). In addition, the ratio of Firmicutes to Bacteroidetes was reduced, the growth of Lactobacillus and Prevotella 9 was promoted, and Pseudomonas aeruginosa was inhibited. This study provides new insights and clues for using PBE as a functional food and clinical drug for glycemic control.